SocialSecretary — Operator Guide

Version 1 — September 2026

What to look at, how often, what it means, and what to do.

This guide assumes the installation in INSTALL.md: SocialSecretary under systemd as the secretary user, everything under /opt/socialsecretary. Commands use YOUR_DOMAIN.COM and YOUR_HANDLE as INSTALL.md does; the actor directory referred to throughout is

/opt/socialsecretary/actors/YOUR_DOMAIN.COM/YOUR_HANDLE/

SocialSecretary is designed to need very little from you. A healthy installation runs for months without intervention. But "needs very little" is not "needs nothing," and the difference between the two is knowing what normal looks like so that abnormal is recognisable. That is most of what this guide is for.

1. The Weekly Check

Five minutes, once a week. Four commands. Do them in this order.

1.1 Is it running?

sudo systemctl status socialsecretary

Want: Active: active (running) and an uptime that matches your last restart. If it says activating (auto-restart), it is crash-looping; go to §4.1. If inactive (dead), it was stopped and not restarted; sudo systemctl start socialsecretary and then find out why in the journal.

1.2 What has it been doing?

sudo journalctl -u socialsecretary --since "1 week ago" --no-pager | grep -v "received Delete\|dropped" | tail -50

The grep -v removes the routine deletion traffic (§2.2) so that anything else stands out. Want: follows, deliveries, retries — the traffic of a working account — and no lines containing WARNING, error, failed, or threw. Each of those is explained in §3.

1.3 Is anything stuck?

ACTOR=/opt/socialsecretary/actors/YOUR_DOMAIN.COM/YOUR_HANDLE
sudo ls $ACTOR/processing/ $ACTOR/failed/ $ACTOR/outbox/
sudo cat $ACTOR/retry.json

Want: three empty directories and []. Anything in processing/ is §4.3. Anything in failed/ is §4.4. Anything in outbox/ older than a minute means the watcher is not picking envelopes up — §4.2. Entries in retry.json are §4.5.

1.4 Is the client keeping up?

sudo find $ACTOR/mailboxes -maxdepth 1 -mindepth 1 -type d -not -name processed \
  -exec sh -c 'echo "$(find "$1" -maxdepth 1 -name "*.json" | wc -l) $(basename "$1")"' _ {} \;

This counts unprocessed files in each live mailbox. Want: small numbers that go down when your publishing client runs. Numbers that only go up mean the client is not reading its mail — see the Mailbox Specification for whose job that is.

That is the whole weekly check. Everything below is reference for when one of those four steps shows something.

2. What Normal Looks Like

2.1 The actor directory

A healthy actor directory after five months in production, one actor, three followers, twenty-odd posts:

drwxrwsr-x  secretary secretary          .
drwxrwsr-x  secretary secretary  3.1M    log/             (activity_log-*.jsonl, rolling)
drwxrwsr-x  secretary secretary  128K    archive/
-rw-rw-r--  secretary secretary          denylist.txt
-rw-rw-r--  secretary secretary          followers.json
drwxrwsr-x  secretary secretary  116K    index/
-rw-rw-r--  secretary secretary          last_broadcast.txt
drwxrwsr-x  secretary secretary  1.8M    mailboxes/
drwxrwsr-x  secretary secretary          outbox/          (empty)
drwxrwsr-x  secretary secretary          processing/      (empty)
drwxrwsr-x  secretary secretary          failed/          (empty)
-rw-rw-r--  secretary secretary          retry.json       ([])

Everything owned by secretary:secretary. Anything owned by another user was written by something other than SocialSecretary — usually a hand edit done as root — and will cause a permissions error the next time SocialSecretary tries to write it. Fix with:

sudo chown -R secretary:secretary /opt/socialsecretary/actors

Data files should not have execute bits. If ls -l shows -rwxrwxr-x on a file under log/ or on last_broadcast.txt, they were created under an unusual umask; sudo chmod 664 $ACTOR/*.txt $ACTOR/*.json $ACTOR/log/*.jsonl is harmless and tidy. log/ itself should be 2750 (the setgid bit keeps new files group-owned); sudo chmod 2750 $ACTOR/log if it is not.

Upgrading from before September 2026: the first start after upgrading moves the old activity_log.jsonl into log/, named for its first line's timestamp, unread and unmodified. You do not need to do this yourself, and you will see it happen once in the journal: [log] migrated … → ….

2.2 The journal on a normal day

[inbound] received Delete from https://mastodon.social/users/doozy
[inbound] dropped unverifiable Delete(actor) — account deleted: https://mastodon.social/users/doozy
[inbound] received Delete from https://mastodon.social/ap/users/116112797909806658
[inbound] dropped unverifiable Delete(actor) — account deleted: https://mastodon.social/ap/users/116112797909806658

This is the bulk of what you will see, and it is nothing. Large Mastodon instances tell every server that has ever seen an account when that account is deleted — spam purges, mostly. Once one of their users follows you, you are on the list. Expect tens to hundreds of these a day, more from mastodon.social than anywhere else. They are verified as far as they can be (the deleted account's key is gone, so the signature cannot be checked) and dropped before routing.

As of 1.0.1 each one is still one line in the activity log (activity.rejected, reason signature) — the log's job is to be comprehensive, and dropping something is not the same as it never having happened. If this volume bothers you, logUnverifiableSelfDeletes: false in socialsecretary_config.mjs turns off logging for this one specific, routine case; it never affects a genuine signature failure. Default is on. Either way, nothing about the journal output above changes — this is a change to the log file, not to what you see with your own eyes.

Post deletions, as opposed to account deletions, only come from servers that received the post. You will see far fewer, and a Delete for a post you actually hold (a reply in your mailbox) is filed to deletions/.

Real traffic looks like:

[inbound] received Follow from https://mastodon.social/users/someone
[inbound] added follower: https://mastodon.social/users/someone (sharedInbox: https://mastodon.social/inbox)
[inbound] sent Accept to https://mastodon.social/inbox
[outbound] 1 envelope(s) in outbox
[outbound] processing create: my-post-20260907120000
[outbound] delivering Create to 3 inbox(es)
[outbound] my-post-20260907120000: 3/3 delivered
[outbound] watermark advanced to 2026-09-07T12:00:00Z
[inbound] received Like from https://mastodon.social/users/someone
[inbound] routed Like to mailboxes/likes/ (object: https://YOUR_DOMAIN.COM/posts/my-post)

A post with a mention that resolved, and one that did not, looks like:

[outbound] resolved @alice@mastodon.social → https://mastodon.social/users/alice
[outbound] could not resolve @typo@mastodon.social: GET https://mastodon.social/.well-known/webfinger?resource=acct:typo@mastodon.social returned 404

The post still goes out; the person behind the typo simply is not notified. Both lines are also in the activity log in full (handle.resolved / handle.unresolved), which is the only place that keeps a permanent record of which mention failed and why — the journal is not kept forever, the log is (§2.3).

2.3 What grows, and how fast

Path	Grows when	Rate (one actor, modest use)	Bound
`archive/`	every post, update, delete, like, boost, or undo you send	~5 KB per object	none — this is your permanent record
`log/`	every verified activity, both directions, filed or not — the third contract (§5.4)	a few hundred bytes per event; a quiet account is a few MB a year, a busy one more	self-managing — rolls to a new file at `logRollSizeMB` (default 50), never overwrites
`mailboxes/*/`	replies, likes, boosts, mentions arrive	proportional to engagement	bounded only by the client consuming it
`mailboxes/processed/`	the client moves consumed files here	same	clear at will (§5.5)
`index/received/`	every mailbox write	one ~200-byte file per remote object, actor, or engagement	proportional to engagement; never needs clearing
`index/by-url/`	every archive write for an object with a `url`	one small file per object you publish	same as archive
`index/by-envelope-id/`	every archive write for an envelope with an `id` — as of 1.0.1, that is all of them	one small file per envelope	same as archive
`followers.json`	follows and unfollows	~250 bytes per follower	—
`retry.json`	a delivery soft-fails	—	should return to `[]`
`failed/`	an envelope's broadcast throws	—	should stay empty

Nothing here will fill a disk in any timeframe you need to plan for. On a 1 GB VPS, the first thing to run short is not disk.

3. Journal Lines and What They Mean

Everything SocialSecretary says goes to the journal (sudo journalctl -u socialsecretary). Each line is prefixed with the module that wrote it. The ones that call for attention:

Line	Meaning	Action
`[outbound] WARNING: N stale envelope(s) in processing/`	A previous run did not finish. The files are named on the following lines.	§4.3
`[outbound] <type> <id> failed: … — moving to failed/`	An envelope could not be processed at all — any type, including a like, boost, or undo whose target post could not be fetched (new in 1.0.1; previously this was a skip with no `failed/` entry).	§4.4
`[outbound] cannot parse <file>: … — left in outbox/`	A file in `outbox/` is not valid JSON. Left in place; seen again every run until removed.	Fix or remove the file
`[outbound] soft failure from <inbox>: … — queuing retry`	A follower's server timed out or returned 5xx. Normal in small numbers.	Nothing, unless it persists — §4.5
`[outbound] retry <id> → <inbox> failed N times — giving up (follower kept)`	That inbox stayed down through every retry. The delivery is abandoned; the follower stays.	Nothing
`[outbound] retry <id> → <inbox>: no stored body (pre-September-2026 entry) — dropping`	A retry queued by pre-1.0.0 code, which cannot be replayed faithfully. Seen once, after upgrading.	Nothing
`[outbound] hard failure <status> from <inbox> — pruning`	A follower's inbox returned 404/410. They are gone; removed from followers.	Nothing
`[outbound] could not resolve @h@d: …`	A mention or recipient handle could not be resolved; the post goes out without that mention tag (or, for a `direct` post with no other recipient, the send fails — see `failed/` above).	Check the handle
`[outbound] update for unknown object <id> — skipped, not converted to a create`	An update named an `id` Secretary never sent. Not sent in any form — see the Handoff Envelope Specification, Update Envelopes, for why this is a strict failure rather than a fallback.	Check your client's records: a bad `id`, drifted local state, or an original create that failed earlier without your noticing
`[outbound] delete for unknown object <id> — nothing to unsend`	A delete named an `id` Secretary never sent, or already deleted.	Nothing
`[outbound] undo <id>: no record for envelope id <ref>` or `… is a <kind>, not a like or boost`	An undo named something that is not a like or boost Secretary sent. Not sent.	Nothing, unless the reference was meant to match — check your client's records
`[inbound] signature verification failed for <actor>`	A request claiming to be from someone could not be verified. Occasional ones are noise; a stream of them from one domain is either a broken sender or a clock problem on your side. Also written to the activity log as `activity.rejected` (reason `signature`).	Check `timedatectl`; your clock must be within a few minutes of true
`[inbound] could not fetch public key for verification: …`	The sender's server was unreachable when we tried to verify. Returned 200; the sender will retry. The routine case — an account deleting itself — has its own line and its own explanation in §2.2.	Nothing unless persistent for one domain
`[inbound] blocked domain: <domain>`	Denylist working as configured. Also written to the activity log as `activity.rejected` (reason `denylist`), corrected from earlier drafts of this guide, which said denylist hits were not logged.	Nothing
`[inbound] follower deleted their account — pruned and filed`	A follower's account was deleted. Removed from followers, filed to `deletions/`.	Nothing
`[inbound] failed to write mailbox file …`	Disk or permissions problem. The activity is in the log but not the mailbox.	Check disk space and ownership (§2.1)
`[index] Port 3000 is already in use`	Something else is on 3000. Under systemd this should not happen.	`sudo ss -ltnp \| grep 3000`
`Error: private key not found` (startup)	Key not at the derived path.	INSTALL.md Step 5
`[log] migrated … → … (contents unchanged)`	The one-time move of the old `activity_log.jsonl` into `log/` after upgrading to 1.0.1.	Nothing

Lines you will see and can ignore: received <Type> from … (every inbound request, logged before verification), the dropped family (§2.2), routed <Type> to mailboxes/… (a mailbox write), N/N delivered, resolved @h@d → … (a mention that resolved fine), and no handler for activity type: <Type> (an activity of a kind SocialSecretary does not act on — Update, Flag, and so on — logged and nothing else). The console is a running narrative for whoever is watching it; the full, structured account of any one of these is in the activity log (§2.3, and the Activity Log Specification) — that is where you go to answer "what exactly happened to that one post," not the journal.

4. Procedures

4.1 It is crash-looping

sudo journalctl -u socialsecretary -n 100 --no-pager

The last lines before each Started socialsecretary.service are the error. Almost always one of: a syntax error in socialsecretary_config.mjs after an edit (fix the edit; node --check socialsecretary_config.mjs finds it), a file under /opt/socialsecretary not owned by secretary (§2.1), or the private key missing after a restore.

systemd does not give up. With RestartSec=5 the attempts are spaced too far apart to trip its burst limit, so it retries every five seconds indefinitely — the restart counter in the journal will climb into the thousands. This is deliberate: the moment you fix the cause, the next attempt succeeds and the service is back with no further action. The cost is a noisy journal while it is broken, and the fact that nothing stops to get your attention. That is what the external check in §5.6 is for. Once the cause is fixed, you need do nothing; watch systemctl status turn green.

A config that still contains YOUR_DOMAIN.COM — the stub installed over the real file — fails at startup with a message saying exactly that (from 1.0.1; 1.0.0 reports it as a missing private key at a placeholder path). It has happened. Restore the real config from backup (§5.1).

4.2 Envelopes sit in outbox/ and nothing happens

The outbox watcher fires on filesystem events, which are unreliable on some filesystems (NFS, some container mounts). The trigger endpoint exists for this reason. Try it:

curl -X POST http://localhost:3000/trigger

If that broadcasts, tell your publishing client to ping the trigger after every write. If it does not, the journal will say why.

4.3 Stale files in processing/

SocialSecretary moves each envelope from outbox/ to processing/ before broadcasting it and removes it afterwards. A file still there when a run starts was left by a run that did not finish — a crash, a power loss, a systemctl stop at the wrong moment. SocialSecretary refuses to guess whether it was delivered, because re-sending a delivered post duplicates it on every follower's timeline. It warns, by filename, and leaves the decision to you.

This applies to any envelope type stuck in processing/ — a delete, like, boost, or undo included, now that all of them really do pass through it (before 1.0.1, a delete never reached this far at all; see the Handoff Envelope Specification). To decide, look the object up in the archive:

ACTOR=/opt/socialsecretary/actors/YOUR_DOMAIN.COM/YOUR_HANDLE
sudo cat $ACTOR/processing/<file>.json | grep '"url"'
sudo grep -rl '<that url>' $ACTOR/archive/ | head -1 | xargs sudo cat | grep transportState

"transportState": "broadcast" — it was delivered. Delete the file from processing/.
"transportState": "queued" or no archive record — it was not delivered. Move it back: sudo mv $ACTOR/processing/<file>.json $ACTOR/outbox/ and trigger. The watermark will not stop it: its published is newer than last_broadcast.txt if it was never sent.

4.4 Something in failed/

Each file in failed/ has a sibling <name>.json.error.txt with the exception. Read it. The envelope is exactly as your client wrote it. The usual causes:

A malformed envelope — bad JSON, missing id, published not ISO 8601.
A direct post whose only recipient handle could not be resolved. There is no fallback address for a direct post; Secretary refuses to send it publicly by mistake, so it fails instead.
A like, boost, or undo whose target post could not be fetched, or whose author's actor document has no inbox. New in 1.0.1 — before, this was a skip with a console warning and no trace in failed/; now it is visible the same way a bad content envelope always was.

Fix the cause, then either move the envelope back to outbox/ or delete it. SocialSecretary never retries failed/ on its own.

4.5 retry.json is not empty

Each entry is a delivery to one inbox that soft-failed (timeout, 5xx) and will be retried the next time a broadcast runs. Retries are attempted only during a broadcast — if you have not posted, they wait. Each entry stores the exact activity that failed, and a retry sends exactly that; the attempts field counts tries, and after retryMaxAttempts (default 12) SocialSecretary gives up on that delivery and logs it. The follower stays.

retryMaxAttempts has been documented since 1.0.0 but was not actually read from your config until 1.0.1 — a value you set had no effect; the built-in default of 12 always applied regardless. If you had set it to something else, it takes effect for the first time on this upgrade.

You will rarely need to touch this file. If you want to abandon a pending delivery sooner, clear it:

sudo -u secretary sh -c 'echo "[]" > $ACTOR/retry.json'

(a publishing client may offer an equivalent command of its own). The follower stays in followers.json; if their server comes back, they receive your next post normally.

4.6 Restart

sudo systemctl restart socialsecretary

Required after editing socialsecretary_config.mjs or deploying new code. Not required after editing denylist.txt (read on every request), actor.json, webfinger, or avatar.png (static files, never read by SocialSecretary).

4.7 Deploy a new release

cd /tmp
curl -O https://socialsecretary.pub/dist/socialsecretary-latest.tar.gz
curl -O https://socialsecretary.pub/dist/socialsecretary-latest.tar.gz.sha256
curl -O https://socialsecretary.pub/dist/socialsecretary-latest.tar.gz.asc
gpg --auto-key-locate clear,wkd --locate-keys rob@socialsecretary.pub        # fingerprint below
gpg --verify socialsecretary-latest.tar.gz.asc socialsecretary-latest.tar.gz   # Good signature
sha256sum -c socialsecretary-latest.tar.gz.sha256                            # OK
sudo tar -xzf socialsecretary-latest.tar.gz -C /opt/socialsecretary --strip-components=1
sudo chown -R secretary:secretary /opt/socialsecretary
sudo systemctl restart socialsecretary
sudo journalctl -u socialsecretary -n 20

Before trusting the signature, confirm --locate-keys printed this exact fingerprint (INSTALL.md's Step 3 explains what each of the three lines above proves; this section just gives the commands):

735C CBB0 C5C2 73DD E2C5 0FA1 A84B 1564 762B 3085

--locate-keys fetches the key from socialsecretary.pub over WKD and adds it to this machine's gpg keyring — a one-time thing per machine, not per release, and harmless to re-run (importing an already-known key is a no-op). This line was missing from this section through 1.0.1's first real run of this whole sequence against a server: Rob's own machine already had the key from building the release, so the gap went unnoticed until it hit a server gpg had never touched, and gpg --verify failed with no explanation beyond "I don't have the key" (L-032). Without this line, gpg --verify cannot succeed on any machine that hasn't separately imported the key some other way — it silently depends on that having happened, rather than being part of the deploy procedure itself.

A release never contains socialsecretary_config.mjs, keys/, or actors/, so extracting over the installation cannot touch your settings, key, or data. It does replace stubs/, which is correct — the stubs are part of the release. Read the release notes first; a release that changes config fields says so.

The same line holds outside this exact procedure, however these files are being moved around — reviewing them, copying them by hand, anything other than this tarball. Every .mjs module and everything under stubs/ is disposable: overwrite it, upgrade it, replace it wholesale, always. socialsecretary_config.mjs, denylist.txt, keys/, and actors/ never are — they hold this installation's own settings and state, not the project's code, and nothing about a release, a stub, or a fresh copy of either should ever land on top of them. L-021 in the project ledger is what this line is for: a stub config, handled alongside real deployable files during a review, was copied over the live one and took the service down for a day. config.mjs now refuses to start on a config that still carries the stub's placeholder values (§4.1), which catches the mistake even when this paragraph doesn't — but the paragraph is the one that stops it from happening at all.

4.8 Block a domain

Add it to denylist.txt in the actor directory, one domain per line. Takes effect on the next request. Blocked senders receive 403 and are logged to the activity log as activity.rejected (reason denylist) — corrected from an earlier version of this guide, which said the opposite. Existing followers from that domain are not removed automatically; delete their entries from followers.json (§4.9) if you want them gone.

4.9 Edit followers.json by hand

Rarely necessary — follows and unfollows maintain it — but if you must: stop SocialSecretary first (it holds a lock on writes but not on your editor), edit as secretary or fix ownership afterwards, keep it valid JSON, start again. Back it up before you begin. It is the only record of who follows you; there is no way to rebuild it.

5. Maintenance

5.1 Back up

Four things. Everything else is regenerable or replaceable.

What	Why
`keys/`	Your identity. Lost key = lost identity; every follower must re-follow
`socialsecretary_config.mjs`	Your settings
`actors/…/followers.json`	The only record of who follows you
`actors/…/archive/`	Every object you have published; cannot be re-derived

mailboxes/ is worth including if engagement matters to you. log/ is an audit trail; back it up if you would want it after a disaster. index/ regenerates from archive/ and mailboxes/.

A nightly tar of /opt/socialsecretary minus index/ and mailboxes/processed/ is under 50 MB for a small account and covers all of it.

5.2 Key rotation

Not supported in v1. The public key is cached in every follower's instance; there is no standard way to tell them it changed. If you must rotate, followers re-follow. Announce it first.

5.3 Upgrade Node

SocialSecretary uses only built-in modules and runs on any Node 18 or later. Upgrade Node with apt as normal and restart SocialSecretary.

5.4 The activity log

log/ is the third contract, alongside the Handoff Envelope and Mailbox Specifications: a complete, append-only account of what SocialSecretary sent and received, both directions, one JSON line per event. Unlike 1.0.0's activity_log.jsonl, you do not rotate it — do not point logrotate at it. SocialSecretary manages its own growth: it appends to the current file and starts a new one, named for the moment it started, whenever the current file would pass logRollSizeMB (default 50, set in socialsecretary_config.mjs). A quiet account keeps one file for years; a busy one accumulates several. Nothing is ever truncated or overwritten in place.

Deleting an old file is always safe — this is a property of the format, not a convenience claim. Every line carries every id it needs; no line refers back to an earlier one; SocialSecretary never writes a "continued from" marker. If you want the disk space back:

ls -1 $ACTOR/log/*.jsonl               # oldest names sort first
rm $ACTOR/log/activity_log-2026042*.jsonl   # example: everything from April

Do not delete or truncate the current file (the one with the greatest name) while forming an opinion about what "current" means from a stale ls — if you are unsure, check again right before deleting; Secretary handles a deleted current file gracefully (it starts a new one on its next write) but there is no reason to test that path on purpose. Never rename a file in log/: filename order is the only ordering guarantee a reader has, and this specifically is not safe. See the Activity Log Specification for the full contract, including how a client (or you, with jq and tail) reads it efficiently without loading the whole directory.

Nothing reads the log but you and whatever publishing client you point at it — SocialSecretary itself never reads its own log back.

5.5 Clear processed mailboxes

sudo rm -rf $ACTOR/mailboxes/processed/*

Safe at any time. It removes only the ability to reprocess old mail and to rebuild the received index from scratch with tools/triage_deletions.mjs --seed-index. If you expect to need either, keep it.

5.6 Monitor from outside

The single most useful external check is whether your actor document is being served:

curl -sf https://YOUR_DOMAIN.COM/actor.json > /dev/null && echo OK || echo DOWN

That tests DNS, TLS, and Caddy. It does not test SocialSecretary itself; for that, curl -sf https://YOUR_DOMAIN.COM/followers exercises the process. Either line drops into any uptime monitor.

6. Tools

`tools/triage_deletions.mjs`

Written for a one-time cleanup and kept because the situation it handles — a deletions/ mailbox that has grown beyond what the client can usefully process — can recur if the received index is ever lost. Read its header. Report by default; --seed-index and --purge are explicit.

sudo -u secretary node /opt/socialsecretary/tools/triage_deletions.mjs --actor-dir $ACTOR

7. Removing SocialSecretary

sudo systemctl disable --now socialsecretary
sudo rm /etc/systemd/system/socialsecretary.service
sudo systemctl daemon-reload
# keep or remove — this is your identity and history:
sudo tar -czf ~/socialsecretary-final.tar.gz /opt/socialsecretary
sudo rm -rf /opt/socialsecretary
sudo userdel secretary

Your followers' instances will keep trying to fetch your actor document for a while and then give up. There is no "delete account" activity to send that every instance honours; the Fediverse forgets you by attrition.

SocialSecretary Operator Guide — Version 1 — September 2026