erik/slugkit.com
1
0
Fork 0
Code Issues 9 Pull requests Projects Releases Packages Wiki Activity Actions

Troubleshoot pending Mastodon follow #6

New issue
Closed
opened 2026-06-25 23:58:20 -05:00 by erik · 7 comments
erik commented 2026-06-25 23:58:20 -05:00
Owner
Copy link

Goal

Diagnose and fix why following @slug@slugkit.com from Mastodon remains pending and does not appear in the slugkit.com followers collection.

Requirements

  • Reproduce or inspect the failed follow from a Mastodon account.
  • Check production logs around the follow attempt for POST requests to:
    • https://slugkit.com/users/slug/inbox
    • https://slugkit.com/inbox
  • Verify whether the signed ActivityPub Follow activity reaches the app.
  • Determine whether the failure is due to request routing, HTTP signatures, actor lookup, database writes, delivery of the Accept, or Mastodon processing the response.
  • Confirm the followers table/state after a follow attempt.
  • Add or improve logging/error reporting for ActivityPub inbox processing if needed.
  • Add regression coverage where practical.

Observed data

  • Actor document resolves successfully:
    • https://slugkit.com/users/slug returns HTTP 200 with Content-Type: application/activity+json.
  • WebFinger resolves successfully:
    • https://slugkit.com/.well-known/webfinger?resource=acct:slug@slugkit.com
    • subject: acct:slug@slugkit.com
    • self link: https://slugkit.com/users/slug
  • Actor document advertises:
    • inbox: https://slugkit.com/users/slug/inbox
    • outbox: https://slugkit.com/users/slug/outbox
    • followers: https://slugkit.com/users/slug/followers
    • following: https://slugkit.com/users/slug/following
    • featured: https://slugkit.com/users/slug/featured
    • shared inbox: https://slugkit.com/inbox
    • public key ID: https://slugkit.com/users/slug#main-key
  • Followers collection currently returns:
    • https://slugkit.com/users/slug/followers
    • type: OrderedCollection
    • totalItems: 0
  • Mastodon UI for @slug@slugkit.com shows Cancel request, which indicates Mastodon considers the follow request pending rather than accepted.
  • A basic unsigned POST test to both inbox endpoints returns 400 Missing actor, showing the routes are reachable but does not validate signed ActivityPub delivery.
  • GET /users/slug/inbox returns 404; this may be acceptable for an inbox, but should be checked against expected Fedify/ActivityPub behavior.

Acceptance criteria

  • Root cause is identified and documented.
  • Following @slug@slugkit.com from Mastodon results in an accepted follow instead of a pending request.
  • https://slugkit.com/users/slug/followers reports the Mastodon follower after the follow is accepted.
  • The app sends or queues the expected ActivityPub Accept response for valid Follow activities.
  • Relevant logs/tests make future inbox/follow failures easier to diagnose.

Dependencies

  • None
## Goal Diagnose and fix why following `@slug@slugkit.com` from Mastodon remains pending and does not appear in the `slugkit.com` followers collection. ## Requirements - Reproduce or inspect the failed follow from a Mastodon account. - Check production logs around the follow attempt for POST requests to: - `https://slugkit.com/users/slug/inbox` - `https://slugkit.com/inbox` - Verify whether the signed ActivityPub `Follow` activity reaches the app. - Determine whether the failure is due to request routing, HTTP signatures, actor lookup, database writes, delivery of the `Accept`, or Mastodon processing the response. - Confirm the followers table/state after a follow attempt. - Add or improve logging/error reporting for ActivityPub inbox processing if needed. - Add regression coverage where practical. ## Observed data - Actor document resolves successfully: - `https://slugkit.com/users/slug` returns `HTTP 200` with `Content-Type: application/activity+json`. - WebFinger resolves successfully: - `https://slugkit.com/.well-known/webfinger?resource=acct:slug@slugkit.com` - subject: `acct:slug@slugkit.com` - self link: `https://slugkit.com/users/slug` - Actor document advertises: - inbox: `https://slugkit.com/users/slug/inbox` - outbox: `https://slugkit.com/users/slug/outbox` - followers: `https://slugkit.com/users/slug/followers` - following: `https://slugkit.com/users/slug/following` - featured: `https://slugkit.com/users/slug/featured` - shared inbox: `https://slugkit.com/inbox` - public key ID: `https://slugkit.com/users/slug#main-key` - Followers collection currently returns: - `https://slugkit.com/users/slug/followers` - `type: OrderedCollection` - `totalItems: 0` - Mastodon UI for `@slug@slugkit.com` shows `Cancel request`, which indicates Mastodon considers the follow request pending rather than accepted. - A basic unsigned POST test to both inbox endpoints returns `400 Missing actor`, showing the routes are reachable but does not validate signed ActivityPub delivery. - `GET /users/slug/inbox` returns `404`; this may be acceptable for an inbox, but should be checked against expected Fedify/ActivityPub behavior. ## Acceptance criteria - [ ] Root cause is identified and documented. - [ ] Following `@slug@slugkit.com` from Mastodon results in an accepted follow instead of a pending request. - [ ] `https://slugkit.com/users/slug/followers` reports the Mastodon follower after the follow is accepted. - [ ] The app sends or queues the expected ActivityPub `Accept` response for valid `Follow` activities. - [ ] Relevant logs/tests make future inbox/follow failures easier to diagnose. ## Dependencies - None
erik commented 2026-06-26 14:03:06 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-26T19:01:22.866Z

Deployment update

Deployed a slugkit.com test fix via GitOps for Mastodon follow verification.

Changes in slugkit.com commit 4457983:

  • Start the Fedify ActivityPub queue worker during app startup.
  • Add ActivityPub Follow diagnostics for follower persistence and Accept queueing.
  • Add regression coverage that a queued signed Follow is processed into a follower row and queues an Accept.

Release/deploy details:

  • Pushed branch task-becaf72b-activitypub-follow.
  • Published prerelease tag web-v0.1.2-rc.1.
  • Forgejo Actions task #6 succeeded and pushed image evcraddock/slugkit-com:0.1.2-rc.1.
  • Updated homelab Flux manifest in commit f0dc699 to use that image and set ACTIVITYPUB_ENABLED=true.
  • Flux reconciled main@sha1:f0dc699059e83e1f8c6fea16d9763ab87b501161.
  • Current pod is slugkit-com-58d7fb58d5-cszf7 and rollout succeeded.

Post-deploy checks:

  • Startup logs include slugkit ActivityPub queue worker started.
  • Actor document still advertises https://slugkit.com/users/slug/inbox and shared inbox https://slugkit.com/inbox.
  • Followers collection currently reports totalItems: 0 before the new verification follow.

Verification needed from Mastodon: cancel the existing pending request if necessary, then follow @slug@slugkit.com again and check whether the app logs follower persistence/Accept queueing and whether the followers collection increments.

_Synced from todu comment by @todu on 2026-06-26T19:01:22.866Z_ ## Deployment update Deployed a slugkit.com test fix via GitOps for Mastodon follow verification. Changes in slugkit.com commit `4457983`: - Start the Fedify ActivityPub queue worker during app startup. - Add ActivityPub Follow diagnostics for follower persistence and Accept queueing. - Add regression coverage that a queued signed `Follow` is processed into a follower row and queues an `Accept`. Release/deploy details: - Pushed branch `task-becaf72b-activitypub-follow`. - Published prerelease tag `web-v0.1.2-rc.1`. - Forgejo Actions task `#6` succeeded and pushed image `evcraddock/slugkit-com:0.1.2-rc.1`. - Updated homelab Flux manifest in commit `f0dc699` to use that image and set `ACTIVITYPUB_ENABLED=true`. - Flux reconciled `main@sha1:f0dc699059e83e1f8c6fea16d9763ab87b501161`. - Current pod is `slugkit-com-58d7fb58d5-cszf7` and rollout succeeded. Post-deploy checks: - Startup logs include `slugkit ActivityPub queue worker started`. - Actor document still advertises `https://slugkit.com/users/slug/inbox` and shared inbox `https://slugkit.com/inbox`. - Followers collection currently reports `totalItems: 0` before the new verification follow. Verification needed from Mastodon: cancel the existing pending request if necessary, then follow `@slug@slugkit.com` again and check whether the app logs follower persistence/Accept queueing and whether the followers collection increments.
erik commented 2026-06-26 14:30:48 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-26T19:29:00.111Z

Verification success

After deploying evcraddock/slugkit-com:0.1.2 with ACTIVITYPUB_ENABLED=true, Mastodon follow verification succeeded.

Observed production logs after retrying the follow:

  • Incoming request reached path: '/users/slug/inbox'.
  • ActivityPub Follow accepted and follower saved logged with follower https://mastodon.online/users/evcraddock.
  • ActivityPub Follow Accept queued logged.

Production DB state:

  • followers contains one active follower:
    • profile: https://mastodon.online/users/evcraddock
    • handle: @evcraddock@mastodon.online
    • inbox: https://mastodon.online/users/evcraddock/inbox
    • shared inbox: https://mastodon.online/inbox
    • unfollowed_at: null

Public checks:

  • https://slugkit.com/users/slug/followers returns totalItems: 1 and includes https://mastodon.online/users/evcraddock.
  • https://slugkit.com/feed now renders Followers 1.

Root cause confirmed: generated app was not starting Fedify's queue worker, so queued inbox activities were not processed. Enabling federation in deployment was also required (ACTIVITYPUB_ENABLED=true).

_Synced from todu comment by @todu on 2026-06-26T19:29:00.111Z_ ## Verification success After deploying `evcraddock/slugkit-com:0.1.2` with `ACTIVITYPUB_ENABLED=true`, Mastodon follow verification succeeded. Observed production logs after retrying the follow: - Incoming request reached `path: '/users/slug/inbox'`. - `ActivityPub Follow accepted and follower saved` logged with follower `https://mastodon.online/users/evcraddock`. - `ActivityPub Follow Accept queued` logged. Production DB state: - `followers` contains one active follower: - profile: `https://mastodon.online/users/evcraddock` - handle: `@evcraddock@mastodon.online` - inbox: `https://mastodon.online/users/evcraddock/inbox` - shared inbox: `https://mastodon.online/inbox` - `unfollowed_at: null` Public checks: - `https://slugkit.com/users/slug/followers` returns `totalItems: 1` and includes `https://mastodon.online/users/evcraddock`. - `https://slugkit.com/feed` now renders `Followers 1`. Root cause confirmed: generated app was not starting Fedify's queue worker, so queued inbox activities were not processed. Enabling federation in deployment was also required (`ACTIVITYPUB_ENABLED=true`).
erik commented 2026-06-26 18:24:16 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-26T23:21:40.055Z

Update

  • Reverted the standalone /main-key signing experiment in slugkit.com with commit 833f450 (Revert "Sign follow accepts with standalone key URL").
  • Published rollback image evcraddock/slugkit-com:0.1.7 from the reverted code.
  • Applied the rollback through GitOps in homelab commit 9faeb91 (Roll slugkit-com back to 0.1.7) and reconciled Flux.
  • Verified production deployment is available on image 0.1.7.
  • Verified externally:
    • GET https://slugkit.com/users/slug returns content-type: application/activity+json.
    • Actor publicKey.id is back to https://slugkit.com/users/slug#main-key.
    • GET https://slugkit.com/users/slug/main-key now returns 404.
  • Created slugkit template follow-up task task-58e3d3cb (Align ActivityPub key IDs with actor document) with the Mastodon error details and slugkit.com code/commit context for a different clanker to implement correctly.
_Synced from todu comment by @todu on 2026-06-26T23:21:40.055Z_ ### Update - Reverted the standalone `/main-key` signing experiment in `slugkit.com` with commit `833f450` (`Revert "Sign follow accepts with standalone key URL"`). - Published rollback image `evcraddock/slugkit-com:0.1.7` from the reverted code. - Applied the rollback through GitOps in homelab commit `9faeb91` (`Roll slugkit-com back to 0.1.7`) and reconciled Flux. - Verified production deployment is available on image `0.1.7`. - Verified externally: - `GET https://slugkit.com/users/slug` returns `content-type: application/activity+json`. - Actor `publicKey.id` is back to `https://slugkit.com/users/slug#main-key`. - `GET https://slugkit.com/users/slug/main-key` now returns `404`. - Created slugkit template follow-up task `task-58e3d3cb` (`Align ActivityPub key IDs with actor document`) with the Mastodon error details and slugkit.com code/commit context for a different clanker to implement correctly.
erik commented 2026-06-27 08:19:17 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-27T13:17:43.249Z

Update

  • Deployed forward fix d748155 (Use default Fedify delivery for follow accepts) to production.
  • Published image evcraddock/slugkit-com:0.1.8.
  • Applied through GitOps in homelab commit 40df236 (Roll slugkit-com forward to 0.1.8) and reconciled Flux.
  • Verified deployment is available on image 0.1.8.
  • Verified externally:
    • GET https://slugkit.com/users/slug returns content-type: application/activity+json.
    • Actor publicKey.id remains https://slugkit.com/users/slug#main-key.
    • GET https://slugkit.com/users/slug/followers reports totalItems: 1.
  • This change matches the known-working erikcraddock.me follow-accept delivery path by using Fedify's default ctx.sendActivity(...) behavior instead of forcing { preferSharedInbox: true, immediate: true }.
_Synced from todu comment by @todu on 2026-06-27T13:17:43.249Z_ ### Update - Deployed forward fix `d748155` (`Use default Fedify delivery for follow accepts`) to production. - Published image `evcraddock/slugkit-com:0.1.8`. - Applied through GitOps in homelab commit `40df236` (`Roll slugkit-com forward to 0.1.8`) and reconciled Flux. - Verified deployment is available on image `0.1.8`. - Verified externally: - `GET https://slugkit.com/users/slug` returns `content-type: application/activity+json`. - Actor `publicKey.id` remains `https://slugkit.com/users/slug#main-key`. - `GET https://slugkit.com/users/slug/followers` reports `totalItems: 1`. - This change matches the known-working `erikcraddock.me` follow-accept delivery path by using Fedify's default `ctx.sendActivity(...)` behavior instead of forcing `{ preferSharedInbox: true, immediate: true }`.
erik commented 2026-06-27 08:24:53 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-27T13:21:48.162Z

Handoff update: ActivityPub follow still unresolved

Current status: not fixed. Mastodon still does not complete the follow for @slug@slugkit.com; UI reportedly remains wrong/pending after the latest deployment.

Production state

  • Current production image: evcraddock/slugkit-com:0.1.8
  • GitOps homelab commit: 40df236 (Roll slugkit-com forward to 0.1.8)
  • slugkit.com branch: task-becaf72b-activitypub-follow
  • Latest slugkit.com code commit deployed: d748155 (Use default Fedify delivery for follow accepts)
  • Production checks after deploy:
    • GET https://slugkit.com/users/slug returns application/activity+json
    • actor publicKey.id is https://slugkit.com/users/slug#main-key
    • GET https://slugkit.com/users/slug/followers returns totalItems: 1

What has been fixed/verified

  1. Queue worker issue was real and fixed

    • Original generated app used Fedify with manuallyStartQueue: true but did not start the queue worker.
    • Added queue startup in slugkit.com:
      • src/federation/routes.ts
      • src/index.ts
    • After deployment, Mastodon Follow reached /users/slug/inbox.
    • Local follower row was saved for https://mastodon.online/users/evcraddock.
    • /users/slug/followers and /feed showed follower count 1.
    • This was ported to slugkit template under completed task task-677d2a8f.

  2. Generic actor fetch was added

    • GET https://slugkit.com/users/slug now returns ActivityPub JSON even without an ActivityPub Accept header.
    • This was intended to help Mastodon resolve https://slugkit.com/users/slug#main-key by fetching the actor URL.

  3. Standalone /main-key experiment was tried and reverted

    • Commit b0fd7a6 tried signing outbound follow Accept with https://slugkit.com/users/slug/main-key and added a standalone key endpoint.

    • Mastodon rejected it with:

      {"error":"publicKey id for https://slugkit.com/users/slug does not correspond to https://slugkit.com/users/slug/main-key"}

    • Root cause: actor document still advertised publicKey.id as https://slugkit.com/users/slug#main-key, so Mastodon correctly rejected the /main-key key ID mismatch.

    • Reverted in commit 833f450 and deployed as 0.1.7.

    • Follow-up slugkit template task created: task-58e3d3cb (Align ActivityPub key IDs with actor document).

  4. Follow Accept delivery was changed to match known-working site

    • Compared slugkit.com to known-working repo: /home/erik/Private/code/github/evcraddock/erikcraddock.me.

    • Core Follow handler shape was similar.

    • Differences found:

      • erikcraddock.me uses plain ctx.sendActivity({ identifier }, followerActor, accept).
      • slugkit.com was forcing { preferSharedInbox: true, immediate: true }.

    • Commit d748155 changed slugkit.com to Fedify default delivery:

      await options.context.sendActivity({ identifier: options.identifier }, followerActor, accept);

    • Tests/build passed and this was deployed as 0.1.8.

    • This still did not resolve the real Mastodon follow state.

Important observed Mastodon errors

Earlier outbound Accept failures included:

{"error":"Public key not found for key https://slugkit.com/users/slug#main-key"}

and during the /main-key experiment:

{"error":"publicKey id for https://slugkit.com/users/slug does not correspond to https://slugkit.com/users/slug/main-key"}

The second error is understood and reverted. The remaining issue is likely still around outbound signed Accept delivery/signature verification, but that is not proven.

Comparison with known-working erikcraddock.me

Known-working site:

  • Repo: /home/erik/Private/code/github/evcraddock/erikcraddock.me

  • Main federation file: src/federation/setup.ts

  • Follow handler:

    • saves follower
    • creates new Accept({ actor: ctx.getActorUri("erik"), object: follow })
    • sends with await ctx.sendActivity({ identifier: "erik" }, followerActor, accept)

  • Uses @fedify/fedify 1.10.2

  • Uses persistent Fedify KV via SqliteKvStore

  • Exports a single long-lived federation instance:

    export const federation = createFedifyFederation();

slugkit.com differs:

  • Uses @fedify/fedify 2.3.0 from lockfile.
  • Uses new MemoryKvStore().
  • Recreates federation instances in route handlers and queue startup via createActivityPubFederation(options).
  • Queue and route share the same message queue, but not a single federation instance or persistent Fedify KV.

Do not downgrade Fedify unless explicitly approved. The next clanker should port working runtime shape forward where possible instead of reverting to old dependency versions.

Strong next leads

  1. Capture exact outbound request to Mastodon

    • Need headers/body of slugkit.com outbound Accept to compare against erikcraddock.me.
    • Especially inspect:
      • Signature / Signature-Input
      • Digest / Content-Digest
      • Date
      • Host
      • Content-Type
      • HTTP signature keyId
      • body actor/object fields
    • Current logs show Fedify delivery errors but do not dump enough request details.

  2. Make slugkit.com runtime more like erikcraddock.me

    • Use one long-lived federation instance per process instead of recreating it per request/queue start.
    • Use persistent Fedify KV rather than fresh MemoryKvStore.
    • Keep one shared queue for route enqueueing and worker processing.

  3. Check Fedify 2 HTTP signature behavior against Mastodon

    • Fedify 2 uses a newer “double knock” sender: RFC 9421 first, then draft-cavage fallback.
    • Mastodon may respond differently depending on which request attempt it accepts/rejects.
    • Need to inspect actual Mastodon response and whether fallback is happening.

  4. Avoid more /main-key changes unless actor document is changed consistently

    • If using /users/slug/main-key, actor document must advertise publicKey.id = https://slugkit.com/users/slug/main-key too.
    • Current production is back to #main-key.

  5. Private key rotation still needed after troubleshooting

    • The production ActivityPub private key was accidentally printed during investigation.
    • Rotate only after federation is understood/fixed to avoid adding another variable mid-debug.

Files changed in slugkit.com during this investigation

  • src/federation/routes.ts
  • src/federation/follow.ts
  • src/federation/__tests__/routes.test.ts
  • src/index.ts

Test status before latest deploy

  • npm test passed: 265 tests
  • npm run build passed

Bottom line

The site now accepts and persists the incoming Follow locally, but Mastodon still does not treat the follow as accepted. The remaining problem is almost certainly outbound Accept delivery/signature compatibility or Fedify runtime setup, not local follower persistence.

_Synced from todu comment by @todu on 2026-06-27T13:21:48.162Z_ ### Handoff update: ActivityPub follow still unresolved Current status: **not fixed**. Mastodon still does not complete the follow for `@slug@slugkit.com`; UI reportedly remains wrong/pending after the latest deployment. #### Production state - Current production image: `evcraddock/slugkit-com:0.1.8` - GitOps homelab commit: `40df236` (`Roll slugkit-com forward to 0.1.8`) - slugkit.com branch: `task-becaf72b-activitypub-follow` - Latest slugkit.com code commit deployed: `d748155` (`Use default Fedify delivery for follow accepts`) - Production checks after deploy: - `GET https://slugkit.com/users/slug` returns `application/activity+json` - actor `publicKey.id` is `https://slugkit.com/users/slug#main-key` - `GET https://slugkit.com/users/slug/followers` returns `totalItems: 1` #### What has been fixed/verified 1. **Queue worker issue was real and fixed** - Original generated app used Fedify with `manuallyStartQueue: true` but did not start the queue worker. - Added queue startup in slugkit.com: - `src/federation/routes.ts` - `src/index.ts` - After deployment, Mastodon `Follow` reached `/users/slug/inbox`. - Local follower row was saved for `https://mastodon.online/users/evcraddock`. - `/users/slug/followers` and `/feed` showed follower count `1`. - This was ported to slugkit template under completed task `task-677d2a8f`. 2. **Generic actor fetch was added** - `GET https://slugkit.com/users/slug` now returns ActivityPub JSON even without an ActivityPub `Accept` header. - This was intended to help Mastodon resolve `https://slugkit.com/users/slug#main-key` by fetching the actor URL. 3. **Standalone `/main-key` experiment was tried and reverted** - Commit `b0fd7a6` tried signing outbound follow `Accept` with `https://slugkit.com/users/slug/main-key` and added a standalone key endpoint. - Mastodon rejected it with: ```json {"error":"publicKey id for https://slugkit.com/users/slug does not correspond to https://slugkit.com/users/slug/main-key"} ``` - Root cause: actor document still advertised `publicKey.id` as `https://slugkit.com/users/slug#main-key`, so Mastodon correctly rejected the `/main-key` key ID mismatch. - Reverted in commit `833f450` and deployed as `0.1.7`. - Follow-up slugkit template task created: `task-58e3d3cb` (`Align ActivityPub key IDs with actor document`). 4. **Follow Accept delivery was changed to match known-working site** - Compared slugkit.com to known-working repo: `/home/erik/Private/code/github/evcraddock/erikcraddock.me`. - Core Follow handler shape was similar. - Differences found: - erikcraddock.me uses plain `ctx.sendActivity({ identifier }, followerActor, accept)`. - slugkit.com was forcing `{ preferSharedInbox: true, immediate: true }`. - Commit `d748155` changed slugkit.com to Fedify default delivery: ```ts await options.context.sendActivity({ identifier: options.identifier }, followerActor, accept); ``` - Tests/build passed and this was deployed as `0.1.8`. - This still did not resolve the real Mastodon follow state. #### Important observed Mastodon errors Earlier outbound `Accept` failures included: ```json {"error":"Public key not found for key https://slugkit.com/users/slug#main-key"} ``` and during the `/main-key` experiment: ```json {"error":"publicKey id for https://slugkit.com/users/slug does not correspond to https://slugkit.com/users/slug/main-key"} ``` The second error is understood and reverted. The remaining issue is likely still around outbound signed `Accept` delivery/signature verification, but that is **not proven**. #### Comparison with known-working erikcraddock.me Known-working site: - Repo: `/home/erik/Private/code/github/evcraddock/erikcraddock.me` - Main federation file: `src/federation/setup.ts` - Follow handler: - saves follower - creates `new Accept({ actor: ctx.getActorUri("erik"), object: follow })` - sends with `await ctx.sendActivity({ identifier: "erik" }, followerActor, accept)` - Uses `@fedify/fedify` `1.10.2` - Uses persistent Fedify KV via `SqliteKvStore` - Exports a single long-lived federation instance: ```ts export const federation = createFedifyFederation(); ``` slugkit.com differs: - Uses `@fedify/fedify` `2.3.0` from lockfile. - Uses `new MemoryKvStore()`. - Recreates federation instances in route handlers and queue startup via `createActivityPubFederation(options)`. - Queue and route share the same message queue, but not a single federation instance or persistent Fedify KV. Do **not** downgrade Fedify unless explicitly approved. The next clanker should port working runtime shape forward where possible instead of reverting to old dependency versions. #### Strong next leads 1. **Capture exact outbound request to Mastodon** - Need headers/body of slugkit.com outbound `Accept` to compare against erikcraddock.me. - Especially inspect: - `Signature` / `Signature-Input` - `Digest` / `Content-Digest` - `Date` - `Host` - `Content-Type` - HTTP signature `keyId` - body actor/object fields - Current logs show Fedify delivery errors but do not dump enough request details. 2. **Make slugkit.com runtime more like erikcraddock.me** - Use one long-lived federation instance per process instead of recreating it per request/queue start. - Use persistent Fedify KV rather than fresh `MemoryKvStore`. - Keep one shared queue for route enqueueing and worker processing. 3. **Check Fedify 2 HTTP signature behavior against Mastodon** - Fedify 2 uses a newer “double knock” sender: RFC 9421 first, then draft-cavage fallback. - Mastodon may respond differently depending on which request attempt it accepts/rejects. - Need to inspect actual Mastodon response and whether fallback is happening. 4. **Avoid more `/main-key` changes unless actor document is changed consistently** - If using `/users/slug/main-key`, actor document must advertise `publicKey.id = https://slugkit.com/users/slug/main-key` too. - Current production is back to `#main-key`. 5. **Private key rotation still needed after troubleshooting** - The production ActivityPub private key was accidentally printed during investigation. - Rotate only after federation is understood/fixed to avoid adding another variable mid-debug. #### Files changed in slugkit.com during this investigation - `src/federation/routes.ts` - `src/federation/follow.ts` - `src/federation/__tests__/routes.test.ts` - `src/index.ts` #### Test status before latest deploy - `npm test` passed: 265 tests - `npm run build` passed #### Bottom line The site now accepts and persists the incoming Follow locally, but Mastodon still does not treat the follow as accepted. The remaining problem is almost certainly outbound `Accept` delivery/signature compatibility or Fedify runtime setup, not local follower persistence.
erik commented 2026-06-27 08:58:09 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-27T13:52:57.047Z

Update

  • Implemented and deployed evcraddock/slugkit-com:0.1.9 from slugkit.com commit 9f5e309 (Sign ActivityPub sends with multikey id).
  • Root cause: Mastodon 4.7 stores the actor RSA key from Fedify 2's assertionMethod as https://slugkit.com/users/slug#multikey-1, but outbound Accept delivery was signed with Fedify's default legacy #main-key, so Mastodon rejected it with Public key not found for key https://slugkit.com/users/slug#main-key.
  • Fix: direct outbound ActivityPub delivery now signs with #multikey-1 while keeping the current Fedify version.
  • Deployed through homelab commit 80c5f48 (Roll slugkit-com forward to 0.1.9) and reconciled Flux successfully.
  • Verified visibly in Chromium: withdrew the pending Mastodon request, followed @slug@slugkit.com again, refreshed the Mastodon profile, and it remained Unfollow instead of reverting to Cancel request.
  • Verified production logs show Follow accepted and Accept delivered without the prior outbound 401 error.
  • Verified https://slugkit.com/users/slug/followers reports totalItems: 1 with https://mastodon.online/users/evcraddock.
_Synced from todu comment by @todu on 2026-06-27T13:52:57.047Z_ ### Update - Implemented and deployed `evcraddock/slugkit-com:0.1.9` from slugkit.com commit `9f5e309` (`Sign ActivityPub sends with multikey id`). - Root cause: Mastodon 4.7 stores the actor RSA key from Fedify 2's `assertionMethod` as `https://slugkit.com/users/slug#multikey-1`, but outbound Accept delivery was signed with Fedify's default legacy `#main-key`, so Mastodon rejected it with `Public key not found for key https://slugkit.com/users/slug#main-key`. - Fix: direct outbound ActivityPub delivery now signs with `#multikey-1` while keeping the current Fedify version. - Deployed through homelab commit `80c5f48` (`Roll slugkit-com forward to 0.1.9`) and reconciled Flux successfully. - Verified visibly in Chromium: withdrew the pending Mastodon request, followed `@slug@slugkit.com` again, refreshed the Mastodon profile, and it remained `Unfollow` instead of reverting to `Cancel request`. - Verified production logs show Follow accepted and Accept delivered without the prior outbound 401 error. - Verified `https://slugkit.com/users/slug/followers` reports `totalItems: 1` with `https://mastodon.online/users/evcraddock`.
erik 2026-06-27 10:05:04 -05:00
erik commented 2026-06-27 10:05:05 -05:00
Author
Owner
Copy link

Synced from todu comment by @todu on 2026-06-27T15:01:40.232Z

Completed

  • Root cause identified: Mastodon 4.7 stored the Fedify 2 assertionMethod RSA key as #multikey-1, while outbound ActivityPub delivery was signed with the legacy #main-key, producing Public key not found for key https://slugkit.com/users/slug#main-key.
  • Implemented and deployed fixes for Follow Accept delivery and follower fanout delivery so outbound requests sign with #multikey-1.
  • Verified in Chromium that following @slug@slugkit.com remains accepted after refresh (Unfollow instead of Cancel request).
  • Verified https://slugkit.com/users/slug/followers reports the Mastodon follower.
  • Verified publishing a new note fans out to Mastodon and appears on https://mastodon.online/@slug@slugkit.com.
  • Added regression coverage for Follow Accept, direct delivery, shared inbox delivery, and follower fanout key ids.
  • Merged the completed changes to main at d6007d0.
_Synced from todu comment by @todu on 2026-06-27T15:01:40.232Z_ ### Completed - Root cause identified: Mastodon 4.7 stored the Fedify 2 `assertionMethod` RSA key as `#multikey-1`, while outbound ActivityPub delivery was signed with the legacy `#main-key`, producing `Public key not found for key https://slugkit.com/users/slug#main-key`. - Implemented and deployed fixes for Follow `Accept` delivery and follower fanout delivery so outbound requests sign with `#multikey-1`. - Verified in Chromium that following `@slug@slugkit.com` remains accepted after refresh (`Unfollow` instead of `Cancel request`). - Verified `https://slugkit.com/users/slug/followers` reports the Mastodon follower. - Verified publishing a new note fans out to Mastodon and appears on `https://mastodon.online/@slug@slugkit.com`. - Added regression coverage for Follow `Accept`, direct delivery, shared inbox delivery, and follower fanout key ids. - Merged the completed changes to `main` at `d6007d0`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
web-v0.1.34
web-v0.1.33
web-v0.1.32
web-v0.1.31
web-v0.1.30
web-v0.1.29
web-v0.1.28
web-v0.1.27
web-v0.1.26
web-v0.1.25
web-v0.1.24
web-v0.1.23
web-v0.1.22
web-v0.1.21
web-v0.1.20
web-v0.1.19
web-v0.1.18
web-v0.1.17
web-v0.1.16
web-v0.1.15
web-v0.1.14
web-v0.1.13
web-v0.1.12
web-v0.1.11
web-v0.1.10
web-v0.1.9
web-v0.1.8
web-v0.1.7
web-v0.1.6
web-v0.1.5
web-v0.1.4
web-v0.1.3
web-v0.1.2
web-v0.1.2-rc.2
web-v0.1.2-rc.1
web-v0.1.1
web-v0.1.0
Labels
Clear labels   
activitypub

   
actor

   
admin

   
ai-safety

   
article

   
articles

   
atproto

   
bug

   
cli

   
configuration

   
content

   
dev-environment

   
docker

   
enhancement

   
feature

   
federation

   
fediverse

   
forgejo

   
garage

   
health

   
how-to

   
logging

   
markdown

   
mastodon

   
media

   
navigation

   
opengraph

   
passkeys

   
preview

   
priority:medium

   
production

   
release

   
rendering

   
s3

   
settings

   
setup

   
status:active

   
status:done

   
status:inprogress

   
technical

   
webauthn
No labels
activitypub
actor
admin
ai-safety
article
articles
atproto
bug
cli
configuration
content
dev-environment
docker
enhancement
feature
federation
fediverse
forgejo
garage
health
how-to
logging
markdown
mastodon
media
navigation
opengraph
passkeys
preview
priority:medium
production
release
rendering
s3
settings
setup
status:active
status:done
status:inprogress
technical
webauthn
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
erik/slugkit.com#6
No description provided.