# SocialSecretary — Operator Guide ## Version 1 — September 2026 *What to look at, how often, what it means, and what to do.* This guide assumes the installation in INSTALL.md: SocialSecretary under systemd as the `secretary` user, everything under `/opt/socialsecretary`. Commands use `YOUR_DOMAIN.COM` and `YOUR_HANDLE` as INSTALL.md does; the actor directory referred to throughout is ``` /opt/socialsecretary/actors/YOUR_DOMAIN.COM/YOUR_HANDLE/ ``` SocialSecretary is designed to need very little from you. A healthy installation runs for months without intervention. But "needs very little" is not "needs nothing," and the difference between the two is knowing what normal looks like so that abnormal is recognisable. That is most of what this guide is for. --- ## 1. The Weekly Check Five minutes, once a week. Four commands. Do them in this order. ### 1.1 Is it running? ```bash sudo systemctl status socialsecretary ``` Want: `Active: active (running)` and an uptime that matches your last restart. If it says `activating (auto-restart)`, it is crash-looping; go to §4.1. If `inactive (dead)`, it was stopped and not restarted; `sudo systemctl start socialsecretary` and then find out why in the journal. ### 1.2 What has it been doing? ```bash sudo journalctl -u socialsecretary --since "1 week ago" --no-pager | grep -v "received Delete\|dropped" | tail -50 ``` The `grep -v` removes the routine deletion traffic (§2.2) so that anything else stands out. Want: follows, deliveries, retries — the traffic of a working account — and no lines containing `WARNING`, `error`, `failed`, or `threw`. Each of those is explained in §3. ### 1.3 Is anything stuck? ```bash ACTOR=/opt/socialsecretary/actors/YOUR_DOMAIN.COM/YOUR_HANDLE sudo ls $ACTOR/processing/ $ACTOR/failed/ $ACTOR/outbox/ sudo cat $ACTOR/retry.json ``` Want: three empty directories and `[]`. Anything in `processing/` is §4.3. Anything in `failed/` is §4.4. Anything in `outbox/` older than a minute means the watcher is not picking envelopes up — §4.2. Entries in `retry.json` are §4.5. ### 1.4 Is the client keeping up? ```bash sudo find $ACTOR/mailboxes -maxdepth 1 -mindepth 1 -type d -not -name processed \ -exec sh -c 'echo "$(find "$1" -maxdepth 1 -name "*.json" | wc -l) $(basename "$1")"' _ {} \; ``` This counts unprocessed files in each live mailbox. Want: small numbers that go down when your publishing client runs. Numbers that only go up mean the client is not reading its mail — see the Mailbox Specification for whose job that is. That is the whole weekly check. Everything below is reference for when one of those four steps shows something. --- ## 2. What Normal Looks Like ### 2.1 The actor directory A healthy actor directory after five months in production, one actor, three followers, twenty-odd posts: ``` drwxrwsr-x secretary secretary . drwxrwsr-x secretary secretary 3.1M log/ (activity_log-*.jsonl, rolling) drwxrwsr-x secretary secretary 128K archive/ -rw-rw-r-- secretary secretary denylist.txt -rw-rw-r-- secretary secretary followers.json drwxrwsr-x secretary secretary 116K index/ -rw-rw-r-- secretary secretary last_broadcast.txt drwxrwsr-x secretary secretary 1.8M mailboxes/ drwxrwsr-x secretary secretary outbox/ (empty) drwxrwsr-x secretary secretary processing/ (empty) drwxrwsr-x secretary secretary failed/ (empty) -rw-rw-r-- secretary secretary retry.json ([]) ``` Everything owned by `secretary:secretary`. Anything owned by another user was written by something other than SocialSecretary — usually a hand edit done as root — and will cause a permissions error the next time SocialSecretary tries to write it. Fix with: ```bash sudo chown -R secretary:secretary /opt/socialsecretary/actors ``` Data files should not have execute bits. If `ls -l` shows `-rwxrwxr-x` on a file under `log/` or on `last_broadcast.txt`, they were created under an unusual umask; `sudo chmod 664 $ACTOR/*.txt $ACTOR/*.json $ACTOR/log/*.jsonl` is harmless and tidy. `log/` itself should be `2750` (the setgid bit keeps new files group-owned); `sudo chmod 2750 $ACTOR/log` if it is not. **Upgrading from before September 2026:** the first start after upgrading moves the old `activity_log.jsonl` into `log/`, named for its first line's timestamp, unread and unmodified. You do not need to do this yourself, and you will see it happen once in the journal: `[log] migrated … → …`. ### 2.2 The journal on a normal day ``` [inbound] received Delete from https://mastodon.social/users/doozy [inbound] dropped unverifiable Delete(actor) — account deleted: https://mastodon.social/users/doozy [inbound] received Delete from https://mastodon.social/ap/users/116112797909806658 [inbound] dropped unverifiable Delete(actor) — account deleted: https://mastodon.social/ap/users/116112797909806658 ``` This is the bulk of what you will see, and it is nothing. Large Mastodon instances tell every server that has ever seen an account when that account is deleted — spam purges, mostly. Once one of their users follows you, you are on the list. Expect tens to hundreds of these a day, more from mastodon.social than anywhere else. They are verified as far as they can be (the deleted account's key is gone, so the signature cannot be checked) and dropped before routing. As of 1.0.1 each one is still one line in the activity log (`activity.rejected`, reason `signature`) — the log's job is to be comprehensive, and dropping something is not the same as it never having happened. If this volume bothers you, `logUnverifiableSelfDeletes: false` in `socialsecretary_config.mjs` turns off logging for this one specific, routine case; it never affects a genuine signature failure. Default is on. Either way, nothing about the journal output above changes — this is a change to the log file, not to what you see with your own eyes. Post deletions, as opposed to account deletions, only come from servers that received the post. You will see far fewer, and a `Delete` for a post you actually hold (a reply in your mailbox) is filed to `deletions/`. Real traffic looks like: ``` [inbound] received Follow from https://mastodon.social/users/someone [inbound] added follower: https://mastodon.social/users/someone (sharedInbox: https://mastodon.social/inbox) [inbound] sent Accept to https://mastodon.social/inbox [outbound] 1 envelope(s) in outbox [outbound] processing create: my-post-20260907120000 [outbound] delivering Create to 3 inbox(es) [outbound] my-post-20260907120000: 3/3 delivered [outbound] watermark advanced to 2026-09-07T12:00:00Z [inbound] received Like from https://mastodon.social/users/someone [inbound] routed Like to mailboxes/likes/ (object: https://YOUR_DOMAIN.COM/posts/my-post) ``` A post with a mention that resolved, and one that did not, looks like: ``` [outbound] resolved @alice@mastodon.social → https://mastodon.social/users/alice [outbound] could not resolve @typo@mastodon.social: GET https://mastodon.social/.well-known/webfinger?resource=acct:typo@mastodon.social returned 404 ``` The post still goes out; the person behind the typo simply is not notified. Both lines are also in the activity log in full (`handle.resolved` / `handle.unresolved`), which is the only place that keeps a permanent record of which mention failed and why — the journal is not kept forever, the log is (§2.3). ### 2.3 What grows, and how fast | Path | Grows when | Rate (one actor, modest use) | Bound | |---|---|---|---| | `archive/` | every post, update, delete, like, boost, or undo you send | ~5 KB per object | none — this is your permanent record | | `log/` | every verified activity, both directions, filed or not — the third contract (§5.4) | a few hundred bytes per event; a quiet account is a few MB a year, a busy one more | self-managing — rolls to a new file at `logRollSizeMB` (default 50), never overwrites | | `mailboxes/*/` | replies, likes, boosts, mentions arrive | proportional to engagement | bounded only by the client consuming it | | `mailboxes/processed/` | the client moves consumed files here | same | clear at will (§5.5) | | `index/received/` | every mailbox write | one ~200-byte file per remote object, actor, or engagement | proportional to engagement; never needs clearing | | `index/by-url/` | every archive write for an object with a `url` | one small file per object you publish | same as archive | | `index/by-envelope-id/` | every archive write for an envelope with an `id` — as of 1.0.1, that is all of them | one small file per envelope | same as archive | | `followers.json` | follows and unfollows | ~250 bytes per follower | — | | `retry.json` | a delivery soft-fails | — | should return to `[]` | | `failed/` | an envelope's broadcast throws | — | should stay empty | Nothing here will fill a disk in any timeframe you need to plan for. On a 1 GB VPS, the first thing to run short is not disk. --- ## 3. Journal Lines and What They Mean Everything SocialSecretary says goes to the journal (`sudo journalctl -u socialsecretary`). Each line is prefixed with the module that wrote it. The ones that call for attention: | Line | Meaning | Action | |---|---|---| | `[outbound] WARNING: N stale envelope(s) in processing/` | A previous run did not finish. The files are named on the following lines. | §4.3 | | `[outbound] failed: … — moving to failed/` | An envelope could not be processed at all — any type, including a like, boost, or undo whose target post could not be fetched (new in 1.0.1; previously this was a skip with no `failed/` entry). | §4.4 | | `[outbound] cannot parse : … — left in outbox/` | A file in `outbox/` is not valid JSON. Left in place; seen again every run until removed. | Fix or remove the file | | `[outbound] soft failure from : … — queuing retry` | A follower's server timed out or returned 5xx. Normal in small numbers. | Nothing, unless it persists — §4.5 | | `[outbound] retry → failed N times — giving up (follower kept)` | That inbox stayed down through every retry. The delivery is abandoned; the follower stays. | Nothing | | `[outbound] retry → : no stored body (pre-September-2026 entry) — dropping` | A retry queued by pre-1.0.0 code, which cannot be replayed faithfully. Seen once, after upgrading. | Nothing | | `[outbound] hard failure from — pruning` | A follower's inbox returned 404/410. They are gone; removed from followers. | Nothing | | `[outbound] could not resolve @h@d: …` | A mention or recipient handle could not be resolved; the post goes out without that mention tag (or, for a `direct` post with no other recipient, the send fails — see `failed/` above). | Check the handle | | `[outbound] update for unknown object — skipped, not converted to a create` | An update named an `id` Secretary never sent. Not sent in any form — see the Handoff Envelope Specification, Update Envelopes, for why this is a strict failure rather than a fallback. | Check your client's records: a bad `id`, drifted local state, or an original create that failed earlier without your noticing | | `[outbound] delete for unknown object — nothing to unsend` | A delete named an `id` Secretary never sent, or already deleted. | Nothing | | `[outbound] undo : no record for envelope id ` or `… is a , not a like or boost` | An undo named something that is not a like or boost Secretary sent. Not sent. | Nothing, unless the reference was meant to match — check your client's records | | `[inbound] signature verification failed for ` | A request claiming to be from someone could not be verified. Occasional ones are noise; a stream of them from one domain is either a broken sender or a clock problem on your side. Also written to the activity log as `activity.rejected` (reason `signature`). | Check `timedatectl`; your clock must be within a few minutes of true | | `[inbound] could not fetch public key for verification: …` | The sender's server was unreachable when we tried to verify. Returned 200; the sender will retry. The routine case — an account deleting itself — has its own line and its own explanation in §2.2. | Nothing unless persistent for one domain | | `[inbound] blocked domain: ` | Denylist working as configured. Also written to the activity log as `activity.rejected` (reason `denylist`), corrected from earlier drafts of this guide, which said denylist hits were not logged. | Nothing | | `[inbound] follower deleted their account — pruned and filed` | A follower's account was deleted. Removed from followers, filed to `deletions/`. | Nothing | | `[inbound] failed to write mailbox file …` | Disk or permissions problem. The activity is in the log but not the mailbox. | Check disk space and ownership (§2.1) | | `[index] Port 3000 is already in use` | Something else is on 3000. Under systemd this should not happen. | `sudo ss -ltnp \| grep 3000` | | `Error: private key not found` (startup) | Key not at the derived path. | INSTALL.md Step 5 | | `[log] migrated … → … (contents unchanged)` | The one-time move of the old `activity_log.jsonl` into `log/` after upgrading to 1.0.1. | Nothing | Lines you will see and can ignore: `received from …` (every inbound request, logged before verification), the `dropped` family (§2.2), `routed to mailboxes/…` (a mailbox write), `N/N delivered`, `resolved @h@d → …` (a mention that resolved fine), and `no handler for activity type: ` (an activity of a kind SocialSecretary does not act on — `Update`, `Flag`, and so on — logged and nothing else). The console is a running narrative for whoever is watching it; the full, structured account of any one of these is in the activity log (§2.3, and the Activity Log Specification) — that is where you go to answer "what exactly happened to that one post," not the journal. --- ## 4. Procedures ### 4.1 It is crash-looping ```bash sudo journalctl -u socialsecretary -n 100 --no-pager ``` The last lines before each `Started socialsecretary.service` are the error. Almost always one of: a syntax error in `socialsecretary_config.mjs` after an edit (fix the edit; `node --check socialsecretary_config.mjs` finds it), a file under `/opt/socialsecretary` not owned by `secretary` (§2.1), or the private key missing after a restore. systemd does **not** give up. With `RestartSec=5` the attempts are spaced too far apart to trip its burst limit, so it retries every five seconds indefinitely — the restart counter in the journal will climb into the thousands. This is deliberate: the moment you fix the cause, the next attempt succeeds and the service is back with no further action. The cost is a noisy journal while it is broken, and the fact that nothing *stops* to get your attention. That is what the external check in §5.6 is for. Once the cause is fixed, you need do nothing; watch `systemctl status` turn green. A config that still contains `YOUR_DOMAIN.COM` — the stub installed over the real file — fails at startup with a message saying exactly that (from 1.0.1; 1.0.0 reports it as a missing private key at a placeholder path). It has happened. Restore the real config from backup (§5.1). ### 4.2 Envelopes sit in outbox/ and nothing happens The outbox watcher fires on filesystem events, which are unreliable on some filesystems (NFS, some container mounts). The trigger endpoint exists for this reason. Try it: ```bash curl -X POST http://localhost:3000/trigger ``` If that broadcasts, tell your publishing client to ping the trigger after every write. If it does not, the journal will say why. ### 4.3 Stale files in processing/ SocialSecretary moves each envelope from `outbox/` to `processing/` before broadcasting it and removes it afterwards. A file still there when a run starts was left by a run that did not finish — a crash, a power loss, a `systemctl stop` at the wrong moment. SocialSecretary refuses to guess whether it was delivered, because re-sending a delivered post duplicates it on every follower's timeline. It warns, by filename, and leaves the decision to you. This applies to any envelope type stuck in `processing/` — a delete, like, boost, or undo included, now that all of them really do pass through it (before 1.0.1, a delete never reached this far at all; see the Handoff Envelope Specification). To decide, look the object up in the archive: ```bash ACTOR=/opt/socialsecretary/actors/YOUR_DOMAIN.COM/YOUR_HANDLE sudo cat $ACTOR/processing/.json | grep '"url"' sudo grep -rl '' $ACTOR/archive/ | head -1 | xargs sudo cat | grep transportState ``` - `"transportState": "broadcast"` — it was delivered. Delete the file from `processing/`. - `"transportState": "queued"` or no archive record — it was not delivered. Move it back: `sudo mv $ACTOR/processing/.json $ACTOR/outbox/` and trigger. The watermark will not stop it: its `published` is newer than `last_broadcast.txt` if it was never sent. ### 4.4 Something in failed/ Each file in `failed/` has a sibling `.json.error.txt` with the exception. Read it. The envelope is exactly as your client wrote it. The usual causes: - A malformed envelope — bad JSON, missing `id`, `published` not ISO 8601. - A `direct` post whose only recipient handle could not be resolved. There is no fallback address for a direct post; Secretary refuses to send it publicly by mistake, so it fails instead. - A like, boost, or undo whose target post could not be fetched, or whose author's actor document has no inbox. New in 1.0.1 — before, this was a skip with a console warning and no trace in `failed/`; now it is visible the same way a bad content envelope always was. Fix the cause, then either move the envelope back to `outbox/` or delete it. SocialSecretary never retries `failed/` on its own. ### 4.5 retry.json is not empty Each entry is a delivery to one inbox that soft-failed (timeout, 5xx) and will be retried the next time a broadcast runs. Retries are attempted only during a broadcast — if you have not posted, they wait. Each entry stores the exact activity that failed, and a retry sends exactly that; the `attempts` field counts tries, and after `retryMaxAttempts` (default 12) SocialSecretary gives up on that delivery and logs it. The follower stays. `retryMaxAttempts` has been documented since 1.0.0 but was not actually read from your config until 1.0.1 — a value you set had no effect; the built-in default of 12 always applied regardless. If you had set it to something else, it takes effect for the first time on this upgrade. You will rarely need to touch this file. If you want to abandon a pending delivery sooner, clear it: ```bash sudo -u secretary sh -c 'echo "[]" > $ACTOR/retry.json' ``` (a publishing client may offer an equivalent command of its own). The follower stays in `followers.json`; if their server comes back, they receive your next post normally. ### 4.6 Restart ```bash sudo systemctl restart socialsecretary ``` Required after editing `socialsecretary_config.mjs` or deploying new code. Not required after editing `denylist.txt` (read on every request), `actor.json`, `webfinger`, or `avatar.png` (static files, never read by SocialSecretary). ### 4.7 Deploy a new release ```bash cd /tmp curl -O https://socialsecretary.pub/dist/socialsecretary-latest.tar.gz curl -O https://socialsecretary.pub/dist/socialsecretary-latest.tar.gz.sha256 curl -O https://socialsecretary.pub/dist/socialsecretary-latest.tar.gz.asc gpg --auto-key-locate clear,wkd --locate-keys rob@socialsecretary.pub # fingerprint below gpg --verify socialsecretary-latest.tar.gz.asc socialsecretary-latest.tar.gz # Good signature sha256sum -c socialsecretary-latest.tar.gz.sha256 # OK sudo tar -xzf socialsecretary-latest.tar.gz -C /opt/socialsecretary --strip-components=1 sudo chown -R secretary:secretary /opt/socialsecretary sudo systemctl restart socialsecretary sudo journalctl -u socialsecretary -n 20 ``` Before trusting the signature, confirm `--locate-keys` printed this exact fingerprint (INSTALL.md's Step 3 explains what each of the three lines above proves; this section just gives the commands): ``` 735C CBB0 C5C2 73DD E2C5 0FA1 A84B 1564 762B 3085 ``` `--locate-keys` fetches the key from `socialsecretary.pub` over WKD and adds it to *this machine's* gpg keyring — a one-time thing per machine, not per release, and harmless to re-run (importing an already-known key is a no-op). This line was missing from this section through 1.0.1's first real run of this whole sequence against a server: Rob's own machine already had the key from building the release, so the gap went unnoticed until it hit a server gpg had never touched, and `gpg --verify` failed with no explanation beyond "I don't have the key" (L-032). Without this line, `gpg --verify` cannot succeed on any machine that hasn't separately imported the key some other way — it silently depends on that having happened, rather than being part of the deploy procedure itself. A release never contains `socialsecretary_config.mjs`, `keys/`, or `actors/`, so extracting over the installation cannot touch your settings, key, or data. It does replace `stubs/`, which is correct — the stubs are part of the release. Read the release notes first; a release that changes config fields says so. The same line holds outside this exact procedure, however these files are being moved around — reviewing them, copying them by hand, anything other than this tarball. Every `.mjs` module and everything under `stubs/` is disposable: overwrite it, upgrade it, replace it wholesale, always. `socialsecretary_config.mjs`, `denylist.txt`, `keys/`, and `actors/` never are — they hold this installation's own settings and state, not the project's code, and nothing about a release, a stub, or a fresh copy of either should ever land on top of them. L-021 in the project ledger is what this line is for: a stub config, handled alongside real deployable files during a review, was copied over the live one and took the service down for a day. `config.mjs` now refuses to start on a config that still carries the stub's placeholder values (§4.1), which catches the mistake even when this paragraph doesn't — but the paragraph is the one that stops it from happening at all. ### 4.8 Block a domain Add it to `denylist.txt` in the actor directory, one domain per line. Takes effect on the next request. Blocked senders receive 403 and are logged to the activity log as `activity.rejected` (reason `denylist`) — corrected from an earlier version of this guide, which said the opposite. Existing followers from that domain are not removed automatically; delete their entries from `followers.json` (§4.9) if you want them gone. ### 4.9 Edit followers.json by hand Rarely necessary — follows and unfollows maintain it — but if you must: stop SocialSecretary first (it holds a lock on writes but not on your editor), edit as `secretary` or fix ownership afterwards, keep it valid JSON, start again. Back it up before you begin. It is the only record of who follows you; there is no way to rebuild it. --- ## 5. Maintenance ### 5.1 Back up Four things. Everything else is regenerable or replaceable. | What | Why | |---|---| | `keys/` | Your identity. Lost key = lost identity; every follower must re-follow | | `socialsecretary_config.mjs` | Your settings | | `actors/…/followers.json` | The only record of who follows you | | `actors/…/archive/` | Every object you have published; cannot be re-derived | `mailboxes/` is worth including if engagement matters to you. `log/` is an audit trail; back it up if you would want it after a disaster. `index/` regenerates from `archive/` and `mailboxes/`. A nightly `tar` of `/opt/socialsecretary` minus `index/` and `mailboxes/processed/` is under 50 MB for a small account and covers all of it. ### 5.2 Key rotation Not supported in v1. The public key is cached in every follower's instance; there is no standard way to tell them it changed. If you must rotate, followers re-follow. Announce it first. ### 5.3 Upgrade Node SocialSecretary uses only built-in modules and runs on any Node 18 or later. Upgrade Node with apt as normal and restart SocialSecretary. ### 5.4 The activity log `log/` is the third contract, alongside the Handoff Envelope and Mailbox Specifications: a complete, append-only account of what SocialSecretary sent and received, both directions, one JSON line per event. Unlike 1.0.0's `activity_log.jsonl`, **you do not rotate it — do not point logrotate at it.** SocialSecretary manages its own growth: it appends to the current file and starts a new one, named for the moment it started, whenever the current file would pass `logRollSizeMB` (default 50, set in `socialsecretary_config.mjs`). A quiet account keeps one file for years; a busy one accumulates several. Nothing is ever truncated or overwritten in place. Deleting an old file is always safe — this is a property of the format, not a convenience claim. Every line carries every id it needs; no line refers back to an earlier one; SocialSecretary never writes a "continued from" marker. If you want the disk space back: ```bash ls -1 $ACTOR/log/*.jsonl # oldest names sort first rm $ACTOR/log/activity_log-2026042*.jsonl # example: everything from April ``` Do not delete or truncate the *current* file (the one with the greatest name) while forming an opinion about what "current" means from a stale `ls` — if you are unsure, check again right before deleting; Secretary handles a deleted current file gracefully (it starts a new one on its next write) but there is no reason to test that path on purpose. Never rename a file in `log/`: filename order is the only ordering guarantee a reader has, and this specifically is not safe. See the Activity Log Specification for the full contract, including how a client (or you, with `jq` and `tail`) reads it efficiently without loading the whole directory. Nothing reads the log but you and whatever publishing client you point at it — SocialSecretary itself never reads its own log back. ### 5.5 Clear processed mailboxes ```bash sudo rm -rf $ACTOR/mailboxes/processed/* ``` Safe at any time. It removes only the ability to reprocess old mail and to rebuild the received index from scratch with `tools/triage_deletions.mjs --seed-index`. If you expect to need either, keep it. ### 5.6 Monitor from outside The single most useful external check is whether your actor document is being served: ```bash curl -sf https://YOUR_DOMAIN.COM/actor.json > /dev/null && echo OK || echo DOWN ``` That tests DNS, TLS, and Caddy. It does not test SocialSecretary itself; for that, `curl -sf https://YOUR_DOMAIN.COM/followers` exercises the process. Either line drops into any uptime monitor. --- ## 6. Tools ### `tools/triage_deletions.mjs` Written for a one-time cleanup and kept because the situation it handles — a `deletions/` mailbox that has grown beyond what the client can usefully process — can recur if the received index is ever lost. Read its header. Report by default; `--seed-index` and `--purge` are explicit. ```bash sudo -u secretary node /opt/socialsecretary/tools/triage_deletions.mjs --actor-dir $ACTOR ``` --- ## 7. Removing SocialSecretary ```bash sudo systemctl disable --now socialsecretary sudo rm /etc/systemd/system/socialsecretary.service sudo systemctl daemon-reload # keep or remove — this is your identity and history: sudo tar -czf ~/socialsecretary-final.tar.gz /opt/socialsecretary sudo rm -rf /opt/socialsecretary sudo userdel secretary ``` Your followers' instances will keep trying to fetch your actor document for a while and then give up. There is no "delete account" activity to send that every instance honours; the Fediverse forgets you by attrition. --- *SocialSecretary Operator Guide — Version 1 — September 2026*