wan
f5625cde98
The `wiki.siblings` bill of materials is gone. A related co-publication is now nominated with an ordinary relative Markdown link — `[label](notes.md)` — resolved at publish time against the target file's own `title` frontmatter. No macro namespace, no per-file declaration, and the link still works in Gitea's blob view and any editor preview. The macro could not be made safe. Substitution was `str.replace(tag, …)` over the whole rendered page: no word boundary, so a tag also occurring in lyrics or a code block got rewritten; no collision check, so overlapping tags corrupted each other in declaration order. SCHEMA.md documented the placeholder as `[[TAG]]` while the code replaced the bare tag, so following the schema produced `[[[[Title|label]]]]`. Fixes a live corruption bug in the process. `sibling_map` was built only from the files in the publish batch, and an unresolved sibling degraded to plain text — so pushing `ses/pulse.md` alone republished it with its wikilink silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the workflow "always runs --all", which it never did on an ordinary push. The repository is now always scanned in full, whatever a run publishes, so link resolution and title-collision detection can never depend on which files a push touched. Writes stay bounded to the push diff plus one hop in each link direction: the page a changed page links to (so a link cannot point at an unpublished page) and the pages linking to it (so an edited `title` does not leave stale inbound links). Publishing is not instant; reading is cheap. An unresolvable `.md` link is now a hard error, never graceful degradation, and linking to a file still does not publish it — `wiki.publish: true` remains the only gate. A leftover-link guard fails the run on any link shape the resolver did not recognise, including a Markdown link written inside a verbatim ```columns fence, so a pandoc upgrade cannot silently publish literal text. Also: - render before write: every page in a run is rendered before the first is published, so a broken page leaves the wiki untouched instead of half-updated. `publish_one` is split into `render_page` (pure) and the write. - pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside subprocess's reader thread, `stdout` came back None, and the real error was lost behind a TypeError. Latent on the Debian runner via PEP 538 only. - title collisions are detected repo-wide, not just within a push, and before any wiki contact. - tests/, run by .gitea/workflows/test.yml in the publish runner image. The pandoc output contract the resolver depends on is now pinned by a test instead of by prose — which is how SCHEMA.md came to assert something false. - drop an f-string with nothing to interpolate. - quote the SOURCE_REF interpolation in the workflow instead of pasting an input straight into shell. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
108 lines
7.7 KiB
Markdown
108 lines
7.7 KiB
Markdown
# pipelines/songs
|
|
|
|
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
|
|
|
|
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
|
|
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
|
|
|
|
## v1 scope
|
|
|
|
Implemented:
|
|
|
|
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
|
|
- Pandoc-based markdown → wikitext body rendering
|
|
- First-line h1 stripping
|
|
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
|
|
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set); additional categories via `categories` list
|
|
- LRC file upload (idempotent SHA1 check) + footer link
|
|
- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched
|
|
- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
|
|
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
|
|
- Files without `wiki.publish: true` silently skipped
|
|
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
|
|
|
|
Not in scope (intentional):
|
|
|
|
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
|
|
- Auto-generated album index pages — user writes album category-page descriptions freeform
|
|
|
|
## Source repo and branch convention
|
|
|
|
Source repo: `novoyuuparosk-wiki/ncmr-songs`. The pipeline triggers on push to `master`.
|
|
|
|
**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/<song-slug>` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle.
|
|
|
|
**Excluded paths** (won't trigger the pipeline):
|
|
|
|
- `wip/**`
|
|
- `README.md`, `.gitignore`, `.claude/**`
|
|
|
|
## Invocation
|
|
|
|
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
|
|
|
|
Triggers:
|
|
- `push` to `master` (path-filtered as above)
|
|
- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit
|
|
|
|
### Environment (secrets and variables)
|
|
|
|
| Name | Kind | Source | Purpose |
|
|
|---|---|---|---|
|
|
| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
|
|
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
|
|
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Mikkeli@giteaBot`) |
|
|
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
|
|
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Gitea instance base URL for cloning |
|
|
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
|
|
| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
|
|
| `GITEA_REPO_URL` | env (set in workflow) | composed from vars | Source-file URL base for the banner link |
|
|
|
|
## Dependencies
|
|
|
|
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
|
|
|
|
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
|
|
|
|
## Modes
|
|
|
|
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
|
|
|
|
- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`.
|
|
- `--files <paths…>` — publish these files and their one-hop link neighbours. Used for an ordinary push.
|
|
|
|
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
|
|
|
|
## One-off wiki setup
|
|
|
|
- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory.
|
|
- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working.
|
|
|
|
## Decisions log (songs pipeline)
|
|
|
|
| Decision | Value | Date |
|
|
|---|---|---|
|
|
| Metadata source | YAML frontmatter inside each `.md` (see [SCHEMA.md](SCHEMA.md)) | 2026-06-09 |
|
|
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
|
|
| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
|
|
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
|
|
| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 |
|
|
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
|
|
| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
|
|
| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 |
|
|
| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 |
|
|
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
|
|
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
|
|
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
|
|
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
|
|
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
|
|
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
|
|
| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 |
|
|
| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
|
|
| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |
|
|
| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `<pre class="columns">`, expanded post-Pandoc into a flex `<div>` of `<poem>` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 |
|
|
|
|
## Status
|
|
|
|
v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org.
|