Import link sources and authors from frontmatter #270

Merged

erik merged 5 commits from feat/task-66375e77-link-source-authors into main 2026-07-24 14:13:14 -05:00

Conversation 4 Commits 5 Files changed 12 +1706 -103

erik commented 2026-07-24 12:42:40 -05:00

Owner

Copy link

Summary

Extend link Markdown imports to discover missing source and author metadata from linked pages, while supporting explicit overrides and idempotently preserving source/contact/post relationships.

Task

Task: #task-66375e77

Changes

• Replace the handwritten post-frontmatter parser with the CLI's existing YAML parser.
• Support explicit nested source and authors overrides with contextual validation.
• Fetch missing relationship metadata from bounded remote HTML without sending Slugkit credentials.
• Discover sources from JSON-LD publisher or site-name metadata, with deterministic hostname/origin fallback.
• Discover authors from JSON-LD objects, arrays, references, and standard author metadata; missing authors do not block import.
• Limit discovery to HTML, a five-second timeout, and 1 MiB response bodies.
• Match sources/contacts by canonical URL or unique normalized name and reject ambiguity.
• Create missing contacts and sources, associate contacts with source and post credits, and preserve existing relationships.
• Deduplicate discovered authors and make repeated imports idempotent.
• Show associated published links on source and contact detail pages while keeping drafts private.
• Document precedence, discovery, fallback, matching, mutation order, and partial failures.

Testing

• ./scripts/pre-pr.sh
• 127 CLI tests
• Explicit overrides, HTML/JSON-LD discovery, hostname fallback, unavailable authors, fetch bounds, credential isolation, creation, reuse, idempotency, ambiguity, partial failure, and preservation covered
• Manual Ars Technica import discovered its source and author metadata
• Source/contact related-link rendering covered by route tests and verified manually; no matching visual baseline exists

Checklist

• Unit tests added/updated
• Documentation updated
• Explicit metadata remains authoritative
• No unrelated changes included

## Summary Extend link Markdown imports to discover missing source and author metadata from linked pages, while supporting explicit overrides and idempotently preserving source/contact/post relationships. ## Task Task: #task-66375e77 ## Changes - Replace the handwritten post-frontmatter parser with the CLI's existing YAML parser. - Support explicit nested `source` and `authors` overrides with contextual validation. - Fetch missing relationship metadata from bounded remote HTML without sending Slugkit credentials. - Discover sources from JSON-LD publisher or site-name metadata, with deterministic hostname/origin fallback. - Discover authors from JSON-LD objects, arrays, references, and standard author metadata; missing authors do not block import. - Limit discovery to HTML, a five-second timeout, and 1 MiB response bodies. - Match sources/contacts by canonical URL or unique normalized name and reject ambiguity. - Create missing contacts and sources, associate contacts with source and post credits, and preserve existing relationships. - Deduplicate discovered authors and make repeated imports idempotent. - Show associated published links on source and contact detail pages while keeping drafts private. - Document precedence, discovery, fallback, matching, mutation order, and partial failures. ## Testing - [x] `./scripts/pre-pr.sh` - [x] 127 CLI tests - [x] Explicit overrides, HTML/JSON-LD discovery, hostname fallback, unavailable authors, fetch bounds, credential isolation, creation, reuse, idempotency, ambiguity, partial failure, and preservation covered - [x] Manual Ars Technica import discovered its source and author metadata - [x] Source/contact related-link rendering covered by route tests and verified manually; no matching visual baseline exists ## Checklist - [x] Unit tests added/updated - [x] Documentation updated - [x] Explicit metadata remains authoritative - [x] No unrelated changes included

erik added 1 commit 2026-07-24 12:42:40 -05:00

feat: import link source relationships ... 227d2f258c

Parse nested link frontmatter with YAML, then idempotently resolve or create sources and author contacts while preserving existing relationships.\n\nTask: #task-66375e77

erik commented 2026-07-24 12:45:30 -05:00

Author

Owner

Copy link

Review summary

PR #270 adds explicit nested source and author metadata to link imports using the existing YAML parser. The implementation validates relationship fields before mutation, resolves records by canonical URL or unique normalized name, creates missing contacts and sources in documented order, unions relationships with existing source contacts and post credits, and preserves post-only behavior for legacy files.

Acceptance criteria

• Met: Link imports accept and validate nested source and authors frontmatter.
• Met: Missing sources are created and assigned through post sourceId.
• Met: Missing authors become contacts associated with source contacts and post credits.
• Met: Existing records are reused by canonical URL or unique normalized-name matching.
• Met: Repeated imports reuse records and avoid duplicate relationships.
• Met: Existing unrelated source contacts and post credits are preserved through ordered ID unions.
• Met: Ambiguous matches and relationship mutation failures return contextual errors.
• Met: Existing link files without relationship metadata retain post-only behavior.
• Met: CLI documentation includes complete source-and-authors examples and matching/failure semantics.
• Met: Tests cover validation, creation, reuse, multiple authors, missing URLs, idempotency, ambiguity, partial failure, preservation, and backward compatibility.

Blocking issues

• None.

Warnings

• None.

Verification

• ./scripts/pre-pr.sh: passed
• CLI tests: 120 passed
• Forgejo CI: passed
• Visual tests: not applicable; no public UI changed

Verdict

Approved.

### Review summary PR #270 adds explicit nested source and author metadata to link imports using the existing YAML parser. The implementation validates relationship fields before mutation, resolves records by canonical URL or unique normalized name, creates missing contacts and sources in documented order, unions relationships with existing source contacts and post credits, and preserves post-only behavior for legacy files. ### Acceptance criteria - **Met:** Link imports accept and validate nested `source` and `authors` frontmatter. - **Met:** Missing sources are created and assigned through post `sourceId`. - **Met:** Missing authors become contacts associated with source contacts and post credits. - **Met:** Existing records are reused by canonical URL or unique normalized-name matching. - **Met:** Repeated imports reuse records and avoid duplicate relationships. - **Met:** Existing unrelated source contacts and post credits are preserved through ordered ID unions. - **Met:** Ambiguous matches and relationship mutation failures return contextual errors. - **Met:** Existing link files without relationship metadata retain post-only behavior. - **Met:** CLI documentation includes complete source-and-authors examples and matching/failure semantics. - **Met:** Tests cover validation, creation, reuse, multiple authors, missing URLs, idempotency, ambiguity, partial failure, preservation, and backward compatibility. ### Blocking issues - None. ### Warnings - None. ### Verification - `./scripts/pre-pr.sh`: passed - CLI tests: 120 passed - Forgejo CI: passed - Visual tests: not applicable; no public UI changed ### Verdict **Approved.**

erik added 1 commit 2026-07-24 13:40:45 -05:00

feat: discover link relationship metadata ... 390aad4646

Fetch bounded remote HTML when source or author frontmatter is absent, using structured metadata with a deterministic hostname fallback while keeping explicit overrides authoritative.\n\nTask: #task-66375e77

erik added 1 commit 2026-07-24 13:43:56 -05:00

test: cover link discovery timeout ... d8a10afbae

Task: #task-66375e77

erik commented 2026-07-24 13:46:46 -05:00

Author

Owner

Copy link

Review summary

The corrected PR now implements the intended automatic behavior: missing source and author fields trigger bounded remote HTML discovery, while explicit frontmatter overrides remain authoritative. The importer handles JSON-LD objects, arrays, and references plus standard HTML metadata, falls back deterministically to hostname/origin for sources, tolerates unavailable authors, and then performs idempotent source/contact/post association without exposing API credentials.

Acceptance criteria

• Met: Nested explicit source and authors frontmatter is parsed and validated.
• Met: Missing sources are discovered from page metadata or hostname/origin fallback.
• Met: Reliable discovered authors become contacts associated with source and post credits.
• Met: Missing, malformed, or unavailable author metadata does not block source or post import.
• Met: Explicit source and author overrides independently suppress discovery for their field.
• Met: Discovery is restricted to HTML, five seconds, and 1 MiB, and remote requests receive no API authorization.
• Met: Missing records are created and assigned through sourceId, source contacts, and creditContactIds.
• Met: Canonical matching, author deduplication, and repeated-import tests demonstrate idempotency.
• Met: Existing unrelated source contacts and post credits are preserved.
• Met: Ambiguous matches and relationship mutation failures return actionable contextual errors.
• Met: Documentation includes explicit and automatic-discovery examples, precedence, fallback, matching, and failure behavior.
• Met: Tests cover JSON-LD/HTML discovery, references, overrides, fallback, missing authors, malformed metadata, response size, timeout, credential isolation, validation, creation, reuse, idempotency, ambiguity, partial failure, and preservation.

Blocking issues

• None.

Warnings

• None.

Verification

• ./scripts/pre-pr.sh: passed
• CLI tests: 127 passed
• Manual Ars Technica import: source and authors discovered and associated
• Forgejo CI: passed
• Visual tests: not applicable; no public UI changed

Verdict

Approved.

### Review summary The corrected PR now implements the intended automatic behavior: missing source and author fields trigger bounded remote HTML discovery, while explicit frontmatter overrides remain authoritative. The importer handles JSON-LD objects, arrays, and references plus standard HTML metadata, falls back deterministically to hostname/origin for sources, tolerates unavailable authors, and then performs idempotent source/contact/post association without exposing API credentials. ### Acceptance criteria - **Met:** Nested explicit `source` and `authors` frontmatter is parsed and validated. - **Met:** Missing sources are discovered from page metadata or hostname/origin fallback. - **Met:** Reliable discovered authors become contacts associated with source and post credits. - **Met:** Missing, malformed, or unavailable author metadata does not block source or post import. - **Met:** Explicit source and author overrides independently suppress discovery for their field. - **Met:** Discovery is restricted to HTML, five seconds, and 1 MiB, and remote requests receive no API authorization. - **Met:** Missing records are created and assigned through `sourceId`, source contacts, and `creditContactIds`. - **Met:** Canonical matching, author deduplication, and repeated-import tests demonstrate idempotency. - **Met:** Existing unrelated source contacts and post credits are preserved. - **Met:** Ambiguous matches and relationship mutation failures return actionable contextual errors. - **Met:** Documentation includes explicit and automatic-discovery examples, precedence, fallback, matching, and failure behavior. - **Met:** Tests cover JSON-LD/HTML discovery, references, overrides, fallback, missing authors, malformed metadata, response size, timeout, credential isolation, validation, creation, reuse, idempotency, ambiguity, partial failure, and preservation. ### Blocking issues - None. ### Warnings - None. ### Verification - `./scripts/pre-pr.sh`: passed - CLI tests: 127 passed - Manual Ars Technica import: source and authors discovered and associated - Forgejo CI: passed - Visual tests: not applicable; no public UI changed ### Verdict **Approved.**

erik added 1 commit 2026-07-24 13:52:25 -05:00

fix: ignore empty optional import fields ... 376adbc167

Task: #task-66375e77

erik commented 2026-07-24 13:55:04 -05:00

Author

Owner

Copy link

Follow-up review

The latest change preserves prior import behavior for empty optional string fields: excerpt: "" is treated as omitted, while non-string optional values still produce contextual validation errors and required fields still reject empty values.

Verification:

• ./scripts/pre-pr.sh: passed
• CLI tests: 127 passed, including empty optional excerpt coverage
• Manual import of chatgpts-memory-dossier.md: created successfully with excerpt: null
• Discovered relationship: Simon Willison’s Weblog with Simon Willison associated to both source and post credit
• Forgejo CI: passed
• Diff check: clean

No blocking issues or warnings found. Approved.

### Follow-up review The latest change preserves prior import behavior for empty optional string fields: `excerpt: ""` is treated as omitted, while non-string optional values still produce contextual validation errors and required fields still reject empty values. Verification: - `./scripts/pre-pr.sh`: passed - CLI tests: 127 passed, including empty optional excerpt coverage - Manual import of `chatgpts-memory-dossier.md`: created successfully with `excerpt: null` - Discovered relationship: `Simon Willison’s Weblog` with `Simon Willison` associated to both source and post credit - Forgejo CI: passed - Diff check: clean No blocking issues or warnings found. **Approved.**

erik added 1 commit 2026-07-24 14:07:09 -05:00

fix: show related links on relationship pages ... e8f518a302

Task: #task-66375e77

erik commented 2026-07-24 14:12:35 -05:00

Author

Owner

Copy link

Independent review summary

PR #270 implements nested YAML source/author overrides, bounded metadata discovery, deterministic matching and relationship mutation, and published related-link rendering on source/contact detail pages. The full diff and tests were reviewed against task task-66375e77.

Acceptance criteria

• Met: Link imports accept and contextually validate nested source and authors frontmatter.
• Met: Missing source metadata is discovered from JSON-LD/site metadata or falls back to normalized hostname/origin.
• Met: Reliable discovered authors are resolved as contacts and associated with source and post.
• Met: Missing, malformed, unavailable, non-HTML, oversized, or timed-out author discovery does not block source/post import.
• Met: Explicit source and author fields independently suppress discovery for that field.
• Met: Discovery is HTML-only, limited to five seconds and 1 MiB, and uses a separate credential-free request.
• Met: Missing records are created and assigned through sourceId, source contacts, and creditContactIds.
• Met: Canonical URL/unique normalized-name matching, ID unions, and repeated-import coverage demonstrate idempotency.
• Met: Existing unrelated source contacts and post credits are preserved.
• Met: Ambiguity and relationship mutation failures return contextual, actionable errors.
• Met: CLI documentation covers explicit/automatic examples, precedence, fallback, matching, ordering, and partial failures.
• Met: Automated tests cover discovery, overrides, hostname fallback, missing/malformed authors, fetch bounds, validation, creation/reuse, idempotency, ambiguity, partial failure, and preservation.

Focused findings

• Empty optional string frontmatter such as excerpt: "" is treated as omitted; required strings and present relationship objects remain strictly validated.
• Source/contact detail routes filter to published link posts and render the existing FeedPost; route tests verify the related published link appears and the draft does not.

Blocking issues

• None.

Warnings

• None.

Verification

• ./scripts/pre-pr.sh: passed locally (all formatting, lint, typecheck, and tests; CLI 127 tests, template site 305 tests).
• Working tree remained clean.
• Forgejo CI status could not be re-read through fj pr status because Forgejo CLI 0.5.0 failed to parse the returned job URL; this does not change the code-review verdict.

Verdict

Approved. No blocking issues or warnings.

### Independent review summary PR #270 implements nested YAML source/author overrides, bounded metadata discovery, deterministic matching and relationship mutation, and published related-link rendering on source/contact detail pages. The full diff and tests were reviewed against task `task-66375e77`. ### Acceptance criteria - **Met:** Link imports accept and contextually validate nested `source` and `authors` frontmatter. - **Met:** Missing source metadata is discovered from JSON-LD/site metadata or falls back to normalized hostname/origin. - **Met:** Reliable discovered authors are resolved as contacts and associated with source and post. - **Met:** Missing, malformed, unavailable, non-HTML, oversized, or timed-out author discovery does not block source/post import. - **Met:** Explicit source and author fields independently suppress discovery for that field. - **Met:** Discovery is HTML-only, limited to five seconds and 1 MiB, and uses a separate credential-free request. - **Met:** Missing records are created and assigned through `sourceId`, source contacts, and `creditContactIds`. - **Met:** Canonical URL/unique normalized-name matching, ID unions, and repeated-import coverage demonstrate idempotency. - **Met:** Existing unrelated source contacts and post credits are preserved. - **Met:** Ambiguity and relationship mutation failures return contextual, actionable errors. - **Met:** CLI documentation covers explicit/automatic examples, precedence, fallback, matching, ordering, and partial failures. - **Met:** Automated tests cover discovery, overrides, hostname fallback, missing/malformed authors, fetch bounds, validation, creation/reuse, idempotency, ambiguity, partial failure, and preservation. ### Focused findings - Empty optional string frontmatter such as `excerpt: ""` is treated as omitted; required strings and present relationship objects remain strictly validated. - Source/contact detail routes filter to published link posts and render the existing `FeedPost`; route tests verify the related published link appears and the draft does not. ### Blocking issues - None. ### Warnings - None. ### Verification - `./scripts/pre-pr.sh`: passed locally (all formatting, lint, typecheck, and tests; CLI 127 tests, template site 305 tests). - Working tree remained clean. - Forgejo CI status could not be re-read through `fj pr status` because Forgejo CLI 0.5.0 failed to parse the returned job URL; this does not change the code-review verdict. ### Verdict **Approved.** No blocking issues or warnings.

erik merged commit a92acc76e7 into main 2026-07-24 14:13:14 -05:00

erik deleted branch feat/task-66375e77-link-source-authors 2026-07-24 14:13:14 -05:00

erik referenced this pull request from a commit 2026-07-24 14:13:16 -05:00

Merge pull request 'Import link sources and authors from frontmatter' (#270) from feat/task-66375e77-link-source-authors into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

activitypub

  

admin

  

api

  

architecture

  

articles

  

auth

  

bug

  

cleanup

  

cli

  

comments

  

compatibility

  

config

  

contacts

  

content

  

database

  

deployment

  

design

  

dev-env

  

diagnostics

  

docs

  

documentation

  

email

  

enhancement

  

feature

  

federation

  

feed

  

health

  

homepage

  

implementation

  

integration

  

logging

  

mastodon

  

media

  

navigation

  

npm-packages

  

openapi

  

opengraph

  

packages

  

passkeys

  

preview

  

priority:high

  

priority:low

  

priority:medium

  

production

  

proof

  

proxy

  

public-routes

  

public-ui

  

release

  

safety

  

slug-api

  

slug-auth

  

slug-core

  

slug-federation

  

slug-media

  

slug-template

  

social

  

sources

  

status:active

  

status:canceled

  

status:done

  

status:inprogress

  

status:waiting

  

syndication

  

tailwind

  

template

  

test

  

web

  

webauthn

No labels

activitypub

admin

api

architecture

articles

auth

bug

cleanup

cli

comments

compatibility

config

contacts

content

database

deployment

design

dev-env

diagnostics

docs

documentation

email

enhancement

feature

federation

feed

health

homepage

implementation

integration

logging

mastodon

media

navigation

npm-packages

openapi

opengraph

packages

passkeys

preview

priority:high

priority:low

priority:medium

production

proof

proxy

public-routes

public-ui

release

safety

slug-api

slug-auth

slug-core

slug-federation

slug-media

slug-template

social

sources

status:active

status:canceled

status:done

status:inprogress

status:waiting

syndication

tailwind

template

test

web

webauthn

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

erik/slugkit!270

No description provided.