novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: READMEs describe current behaviour, not history
tests / pytest (push) Successful in 6s
Details

Browse Source 
Removes the decision logs, the dated rows, the strikethrough retirements and
the setup progress checkboxes from all four READMEs. A decision log records how
the system came to be; a reader needs to know how it is. The two had drifted
far enough apart that the docs asserted things the shipped config contradicted.

Corrected, all of which were stale rather than merely verbose:

- the architecture section described the source-repo workflow as self-contained
  and called the reusable workflows here "not actively called; may be stale".
  They are the pipelines; the source-repo workflow is a thin caller.
- `workflow_call` was recorded as abandoned, and the instance as having no orgs.
  Both pipelines run through cross-repo `workflow_call` under an org.
- tech-blogs was described as not yet initialised. It has published.
- ses and tech invocation sections claimed a self-contained workflow too.

Kept, rewritten as behaviour rather than as dated decisions, because they are
live operational constraints and not history:

- the callee workflow cache, why a `publish-*.yml` edit appears not to take,
  and the flush command with its `--no-deps` and `--pull` caveats
- why this repo has to stay public
- the `URL_TO_GITEA` / `PUBLIC_URL_TO_GITEA` split and what validates it
- host network mode, and that `kaniko-act` cannot be rebuilt by build-image.yml

Also documents what the renderer, its failure modes and its publish scope
actually are, which the decision logs had been standing in for, and adds the
architecture note that renderer code is cloned fresh per run while workflow
YAML is cached — the distinction that governs whether a change takes effect.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Wan Ziyu
2026-09-09 17:39:28 +09:00
parent f19538b607
commit 3ab59b493e
5 changed files with 147 additions and 210 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+61 -71
View File
@@ -10,28 +10,36 @@ CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from u
| [`pipelines/ses/`](pipelines/ses/) | `novoyuuparosk-wiki/ses-light-novel` | SES light novel pages | v1 in development |
| [`pipelines/tech/`](pipelines/tech/) | `novoyuuparosk-wiki/tech-blogs` | Tech blog posts | v1 live |
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime, decisions). This root README covers only what's cross-cutting.
Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime). This root README covers only what's cross-cutting.
## Architecture
Hybrid layout. The Gitea Actions trigger must live in the source repo (Gitea only fires workflows from `.gitea/workflows/` of the pushed-to repo); the rendering logic lives here. The source-repo workflow is self-contained but clones this repo at runtime to get the renderer.
Split across two repos, because Gitea only fires workflows from `.gitea/workflows/` of the repo that was pushed to, while the rendering logic belongs here.
```
<source-repo>/
.gitea/workflows/<name>.yml <- workflow: clones this repo, runs renderer (deps pre-baked in job image)
.gitea/workflows/publish.yml <- thin caller: on: push, uses: the reusable workflow below
novoyuuparosk-auto-wiki/ <- this repo
.gitea/workflows/<pipeline>.yml <- reusable workflow stubs (kept for reference; not actively called; may be stale)
pipelines/<pipeline>/ <- per-pipeline code, schema, templates
lib/ <- shared modules (MediaWiki client, etc.)
.gitea/workflows/publish-<p>.yml <- on: workflow_call. Clones both repos, runs the renderer
.gitea/workflows/test.yml <- on: push. Runs tests/ in the publish runner image
.gitea/workflows/build-image.yml <- builds and pins the shared job container image
pipelines/<pipeline>/ <- per-pipeline renderer, schema, templates
lib/ <- shared render and MediaWiki helpers
tests/ <- pytest suite for lib/ and the pipelines
```
Note: `workflow_call` across private repos was abandoned — the auto-generated run token is scoped to the triggering repo only and cannot clone a private callee. The source-repo workflow clones this repo directly using `FAPAT`.
The two halves reach the runner by different routes, which matters when a change appears not to take effect:
- **Renderer code** (`pipelines/`, `lib/`) is a plain `git clone` of this repo's **default branch** on every run. Always current, never cached.
- **Workflow YAML** (`publish-*.yml`) is resolved through the caller's `uses:` and **is cached** — see *Callee workflow cache* below.
**This repo must stay public.** `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied, and shared ownership does not satisfy the read requirement. A private callee 404s with `repository not found`. The alternative, untested here, is Settings → Actions → General → collaborative owners (Gitea 1.26+).
## Wiki
- Base URL: https://wiki.novoyuuparosk.org
- MediaWiki API: https://wiki.novoyuuparosk.org/api.php *(confirmed)*
- MediaWiki API: https://wiki.novoyuuparosk.org/api.php (classic action API)
## Bot identity
@@ -39,90 +47,72 @@ MediaWiki BotPassword issued for user `Mikkeli`, bot name `giteaBot`. Login form
`Mikkeli` holds the `bot` right, so pipeline edits are flagged as bot edits and stay out of default Recent Changes — `mwclient`'s `page.save()` requests the bot flag and the wiki honours it.
This section previously named `Dubrowski@giteaAutomaton`, but every bot-flagged revision in the wiki's history is attributed to `Mikkeli`, so that had been stale for some time — the live secret never matched the doc.
Credentials are stored in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
Credentials live in the Gitea org-scope secret vault under `novoyuuparosk-wiki` (`WIKI_BOT_USER`, `WIKI_BOT_PASSWORD`). Not stored in this repo.
## Runner infrastructure
One `act_runner` instance serves all pipelines. Runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) inside the same `docker-compose` stack that hosts the Gitea instance. Job execution is via the host Docker socket — runner is a container, jobs spawn as sibling containers.
One `act_runner` instance serves all pipelines. It runs on a Pi 5 (Raspberry Pi OS Bookworm, `aarch64`) as a service in the same `docker-compose` stack as Gitea, built `linux-arm64` from the `gitea/act_runner` image. Job execution is via the host Docker socket — the runner is a container and jobs spawn as sibling containers.
`act_runner` build: `linux-arm64`, from the `gitea/act_runner` Docker image.
Container network mode is `host`, so job containers can reach Gitea on `localhost:3005` and resolve mDNS hostnames.
Container network mode: `host` — required so job containers can reach `localhost:3005` (Gitea) and resolve mDNS hostnames.
### Callee workflow cache
`act_runner` caches each resolved cross-repo callee at `/root/.cache/act/<owner>-<repo>@<ref>` inside the runner container, and **never refreshes it per run**. The key contains no commit SHA, so pushing to `master` does not invalidate it: edits to `publish-*.yml` — including the pin job's own commits — keep executing the stale cached copy indefinitely.
Two consequences. When a pipeline misbehaves after a workflow edit, suspect the cache before suspecting permissions. And after any `publish-*.yml` change, flush it:
```
docker compose up -d --no-deps --force-recreate act_runner
```
`--no-deps` is required: `act_runner` declares `depends_on: gitea`, so without it the recreate cascades and bounces the Gitea instance. Pass no `--pull`, or the flush silently adopts whatever `act_runner:latest` has become. The cache sits in the container layer while `.runner` is in the `/data` volume, so registration survives the recreate.
Changing a repo's owner also changes the cache key, which flushes it as a side effect.
### Job container image
All pipelines share a single pre-built Docker image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The `Dockerfile` is at the repo root. It bakes in system deps (git, pandoc, ca-certificates) and all pipeline Python packages so job containers start instantly with no install steps.
All pipelines share one pre-built image, served from the Gitea registry at `pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner`. The root `Dockerfile` bakes in system deps (git, pandoc, ca-certificates) and every pipeline's Python packages, so job containers start with no install step.
The image builds automatically via [`.gitea/workflows/build-image.yml`](.gitea/workflows/build-image.yml), which triggers on pushes that touch the `Dockerfile`, any pipeline `requirements.txt`, or that workflow itself. It uses kaniko (daemonless, unprivileged) to build and push two tags: an immutable `:<short-sha>` and a moving `:latest`.
[`build-image.yml`](.gitea/workflows/build-image.yml) rebuilds it on pushes touching the `Dockerfile`, any pipeline `requirements.txt`, or itself. It uses kaniko (daemonless, unprivileged) to push an immutable `:<short-sha>` and a moving `:latest`. A follow-up `pin` job rewrites the `image:` pin in each `publish-*.yml` to the new short SHA and commits it back to `master` using `FAPAT`, so the publish workflows always reference an immutable tag with no manual bump. That pin commit touches only `workflow_call` files, so it triggers no further runs — but it does need a cache flush to take effect.
A follow-up `pin` job then rewrites the `image:` pin in each `publish-*.yml` to the new `:<short-sha>` and commits it back to `master` (using `FAPAT` for contents write). The publish workflows therefore always reference an immutable tag, kept current automatically — no manual bump. The pin commit only touches `workflow_call` files, so it triggers no further runs.
The builder image `kaniko-act` lives under `mikkeli`, not the org, and **cannot be rebuilt by `build-image.yml`**: kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so the resulting image has no `/tmp` and `wget -O /tmp/...` fails. Bumping it needs a host `docker build` and push, or a Dockerfile that forces the dir to materialise by writing a file inside it. [`build-kaniko-act.yml`](.gitea/workflows/build-kaniko-act.yml) is dispatch-only and runs in the old `mikkeli` copy.
## Gitea Actions setup (cross-cutting)
Secrets and variables are scoped to the `novoyuuparosk-wiki` org, inherited by all repos under it. Runs belong to the *caller* repo, so a source repo calling a reusable workflow here resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.
Secrets and variables are scoped to the `novoyuuparosk-wiki` org and inherited by every repo under it. A run belongs to the *caller* repo, so a source repo resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.
**Secrets:**
- `WIKI_BOT_USER` = `Mikkeli@giteaBot`
- `WIKI_BOT_PASSWORD` = the value from *Bot identity* above
- `FAPAT` = Full-Access PAT owned by `mikkeli`, used by source-repo workflows to clone this repo at runtime. The PAT stays personal; only its storage scope moved to the org — hence the `mikkeli:` basic-auth username in the clone URLs.
- `PKGRW_PAT` = package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry
| Name | Value / purpose |
|---|---|
| `WIKI_BOT_USER` | `Mikkeli@giteaBot` |
| `WIKI_BOT_PASSWORD` | the BotPassword from *Bot identity* above |
| `FAPAT` | Full-Access PAT owned by `mikkeli`, used by the publish workflows to clone repos at runtime. The PAT is personal; only its storage scope is the org — hence the `mikkeli:` basic-auth username in clone URLs |
| `PKGRW_PAT` | package read/write PAT owned by `mikkeli`, used by `build-image.yml` to push to the container registry |
**Variables:**
- `WIKI_BASE_URL` = `https://wiki.novoyuuparosk.org`
- `WIKI_API_URL` = `https://wiki.novoyuuparosk.org/api.php`
- `URL_TO_GITEA` = **runner-internal** Gitea base URL, used only for cloning (e.g. `http://localhost:3005` — the runner uses host networking). Named with `URL_TO_` prefix — Gitea blocks variable names starting with `GITEA_` or `GITHUB_`.
- `PUBLIC_URL_TO_GITEA` = **public** Gitea base URL (`https://git.novoyuuparosk.org`), used only for the source link in each published page's banner.
These two must stay separate. `URL_TO_GITEA` is a loopback address, so composing the banner from it published every page — across all three pipelines — citing `localhost:3005`, which resolves for nobody reading the wiki. `lib/wiki.public_source_base()` now rejects a loopback, mDNS, or private-range value, so that misconfiguration fails the run instead of reaching a page.
| Name | Value | Used for |
|---|---|---|
| `WIKI_BASE_URL` | `https://wiki.novoyuuparosk.org` | wiki base |
| `WIKI_API_URL` | `https://wiki.novoyuuparosk.org/api.php` | MediaWiki action API |
| `URL_TO_GITEA` | `http://localhost:3005` | **runner-internal** address, cloning only |
| `PUBLIC_URL_TO_GITEA` | `https://git.novoyuuparosk.org` | **public** address, page banner source links only |
Names avoid a `GITEA_`/`GITHUB_` prefix because Gitea blocks those, hence `URL_TO_GITEA` rather than `GITEA_URL`.
The two URL variables must stay separate. `URL_TO_GITEA` has to be an address the runner container can clone from, and host networking makes that a loopback. `PUBLIC_URL_TO_GITEA` is baked into the `{{Auto-generated}}` banner of every published page and has to resolve for a reader on the internet. `lib/wiki.public_source_base()` validates the banner value and aborts the run on an unset, relative, loopback, mDNS, or private-range address, so collapsing them back into one fails loudly instead of publishing dead links.
## Branch naming
- This repo: `automation/<pipeline-name>` for pipeline-development branches (e.g., `automation/songs`).
- Source repos: each pipeline's README defines the source-side branch convention (e.g., `autowiki/<song-slug>` in `ncmr-songs`).
- This repo: `automation/<pipeline-name>` for pipeline-development branches (e.g. `automation/songs`).
- Source repos: each pipeline's README defines its own convention (e.g. `autowiki/<song-slug>` in `ncmr-songs`).
## Decisions log (cross-cutting)
## Tests
| Decision | Value | Date |
|---|---|---|
| Architecture | Hybrid: source-repo workflow clones this repo at runtime for the renderer | 2026-06-09 |
| Workflow pattern | Self-contained (not `workflow_call`) — cross-repo `workflow_call` blocked by token scoping on private repos | 2026-06-09 |
| Runner execution | Docker, added as a service to the existing Gitea docker-compose | 2026-06-09 |
| Runner network mode | `host` — job containers need to reach Gitea on localhost | 2026-06-09 |
| Secret/runner scope | User-level on `mikkeli` (no orgs on this instance) | 2026-06-09 |
| Ownership | This repo and all three source repos moved to the `novoyuuparosk-wiki` org. Secrets/variables re-created at org scope; runner re-registered instance-level so it serves org-owned runs | 2026-08-11 |
| **This repo must stay public** | `act_runner` resolves a cross-repo `uses:` by cloning the callee **anonymously** — the job token is not applied. A private callee therefore 404s with `repository not found`, regardless of the caller sharing its owner. Shared ownership does **not** satisfy the read requirement. Alternative if it must be private again: Settings → Actions → General → collaborative owners (Gitea 1.26+), untested here | 2026-08-11 |
| Runner cache masks this | The resolved callee is cached at `/root/.cache/act/<owner>-<repo>@<ref>`. Changing owner changes the key, so a working pipeline can break on a clone that had been served from cache for months. Suspect the cache before suspecting permissions | 2026-08-11 |
| **Callee edits need a cache flush** | The cache is *not* refreshed per run. Edits to `publish-*.yml` on `master` — including the pin job's own commits — keep executing the stale cached copy until the key changes or the cache is cleared. Observed directly: a run used the previous namespace and tag despite `master` being current. Flush with `docker compose up -d --no-deps --force-recreate act_runner`; the cache is in the container layer, while `.runner` is in the `/data` volume, so registration survives. `--no-deps` is required — `act_runner` declares `depends_on: gitea`, so without it the recreate cascades and bounces the Gitea instance itself. Pass no `--pull`, or the flush silently adopts whatever `act_runner:latest` has become | 2026-08-11 |
| kaniko cannot rebuild kaniko-act | The org rebuild exits 0 but the resulting image has no `/tmp` — kaniko does not persist the empty dir from `mkdir -p -m 1777 /tmp`, so `wget -O /tmp/...` fails. The bootstrap note's claim that version bumps "can be built by the build-image workflow itself" does not hold as written. `kaniko-act` therefore still lives under `mikkeli`; a real bump needs a host `docker build` + push, or a Dockerfile that forces the dir to materialise (e.g. writing a file inside it) | 2026-08-11 |
| Container registry | Gitea cannot transfer packages, so images were *rebuilt* into `pi5-16.local:3005/novoyuuparosk-wiki/*` rather than moved. `kaniko-act` bootstrapped via the new dispatch-only `build-kaniko-act.yml`, running in the old `mikkeli` copy; `novoyuuparosk-wiki-runner` came from a normal `build-image.yml` run. Registry auth is still the `mikkeli`-owned `PKGRW_PAT`, hence the `mikkeli:` username in the auth blob | 2026-08-11 |
| Old images left in place | The `mikkeli/*` package versions are orphaned but retained — nothing references them, and deleting a container version is irreversible. Safe to purge once the org images have proven themselves | 2026-08-11 |
| MediaWiki API path | `api.php` (classic action API) | 2026-06-09 |
| Branch naming (this repo) | `automation/<pipeline>` for pipeline-development branches | 2026-06-09 |
| Variable naming | `URL_TO_GITEA` not `GITEA_URL` — Gitea blocks `GITEA_`/`GITHUB_` prefixes | 2026-06-09 |
| **Clone URL ≠ banner URL** | Split into `URL_TO_GITEA` (runner-internal, for cloning) and `PUBLIC_URL_TO_GITEA` (public, for the page banner). One variable served both, and since host networking forces the clone address to `localhost:3005`, every auto-published page in every pipeline cited a link no reader could open. The banner value is validated by `public_source_base()` — loopback, mDNS and private-range addresses abort the run. Pages published before this carry the bad URL in their wikitext and need a full re-publish per pipeline to clear | 2026-09-09 |
| Source URL separators | `rel.as_posix()` in the banner URL, so the path separator never depends on the OS the publisher happens to run on | 2026-09-09 |
| Columns shorthand | Side-by-side columns authored as a ` ```columns ` fenced block, expanded post-Pandoc in shared `lib/wiki.py` (universal across pipelines). No wiki template or PHP extension — runs Pi-side before the API call | 2026-06-14 |
```
pip install -r requirements-dev.txt
python -m pytest tests -q
```
Per-pipeline decisions live in each pipeline's README.
## Setup checklist (cross-cutting)
Via the Gitea web UI logged in as `mikkeli` (an owner of `novoyuuparosk-wiki`):
- [x] Org-scoped secrets and variables set per *Gitea Actions setup* above
- [x] `WIKI_BOT_USER`
- [x] `WIKI_BOT_PASSWORD`
- [x] `FAPAT` (Full-Access PAT — value not stored in this README; saved directly into the Gitea secret. Regenerate if lost.)
- [x] `WIKI_BASE_URL`
- [x] `WIKI_API_URL`
- [x] `URL_TO_GITEA`
- [ ] `PUBLIC_URL_TO_GITEA` = `https://git.novoyuuparosk.org` — **required**; the publishers abort without it
With Pi access:
- [x] Add `act_runner` service to the existing Gitea docker-compose
- [x] Generate a runner registration token at `/-/admin/actions/runners`, bake into the compose env, `docker compose up -d act_runner`, confirm "online" in the Gitea UI
Per-pipeline setup lives in each pipeline's README. Start with [`pipelines/songs/`](pipelines/songs/).
Pure string-in/string-out coverage of `lib/` and the pipelines' discovery, validation and publish-scoping logic — no wiki, no network. The `TestPandocContract` cases shell out to real pandoc and are skipped if it isn't installed; they pin the pandoc output shape the link resolver depends on, so a pandoc upgrade fails a test instead of publishing broken links. [`test.yml`](.gitea/workflows/test.yml) runs the suite on push in the same image the pipelines publish from.
pipelines/ses/README.md
+12 -27
View File
@@ -5,7 +5,7 @@ Publishes SES light novel pages from `novoyuuparosk-wiki/ses-light-novel` to `ht
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
## v1 scope
## What the renderer does
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
- `SES:` title prefix — all pages live in the SES MediaWiki namespace
@@ -26,7 +26,7 @@ Not in scope (intentional):
Source repo: `novoyuuparosk-wiki/ses-light-novel`. The pipeline triggers on push to `master`.
No branch convention enforced — work directly on `master` or use whatever branch workflow suits.
No branch convention enforced — work directly on `master` or use whatever branch workflow suits. There is no `wip/` exclusion; `wiki.publish: true` is the only publishing gate.
**Excluded paths** (won't trigger the pipeline):
@@ -34,12 +34,18 @@ No branch convention enforced — work directly on `master` or use whatever bran
## Invocation
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ses-light-novel` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
`novoyuuparosk-wiki/ses-light-novel/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-ses.yml`](../../.gitea/workflows/publish-ses.yml) here, which clones both repos and runs `publish.py`.
Editing `publish-ses.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
Triggers:
- `push` to `master` (path-filtered as above)
- `workflow_dispatch` — manual trigger from the Gitea Actions UI
### Publish scope
Every run publishes every file with `wiki.publish: true`; there is no diff detection. Writes are idempotent — the banner carries the commit SHA rather than a timestamp, so a re-run at the same commit rewrites nothing.
### Environment (secrets and variables)
| Name | Kind | Source | Purpose |
@@ -56,29 +62,8 @@ Triggers:
## Dependencies
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. Shared render and MediaWiki helpers live in `lib/wiki.py` at the repo root.
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
## Wiki-side prerequisites
Shared code lives in `lib/wiki.py` (repo root).
## One-off wiki setup
- [ ] **`Template:Auto-generated`** — must exist on the wiki (shared with songs pipeline; already created).
## Decisions log (ses pipeline)
| Decision | Value | Date |
|---|---|---|
| Namespace | All pages prefixed `SES:` — auto-applied by renderer, not written in source `title:` | 2026-06-10 |
| Type resolution | `type:` frontmatter field; falls back to immediate parent directory name | 2026-06-10 |
| Categories | `Category:SES` + `Category:SES:<type>` on every page | 2026-06-10 |
| Sibling links | Not implemented — write `[[SES:Page title]]` wikilinks directly in the body | 2026-06-10 |
| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 |
| Publish mode | Always `--all` | 2026-06-10 |
| Idempotency | Commit SHA in banner, not timestamp | 2026-06-10 |
| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 |
## Status
v1 in development.
**`Template:Auto-generated`** must exist on the wiki. It is shared across all three pipelines; the wikitext source lives in [`pipelines/songs/template_auto_generated.wikitext`](../songs/template_auto_generated.wikitext).
pipelines/songs/README.md
+59 -81
View File
@@ -1,108 +1,86 @@
# pipelines/songs
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`.
Publishes lyric pages from `novoyuuparosk-wiki/ncmr-songs` to `https://wiki.novoyuuparosk.org`. v1, live.
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
## v1 scope
## What the renderer does
Implemented:
- Parses YAML frontmatter per [SCHEMA.md](SCHEMA.md); files without `wiki.publish: true` are skipped silently
- Strips the body's first-line `# Heading`, then renders markdown → wikitext with Pandoc
- Expands ` ```columns ` fenced blocks into a flex row of `<poem>` columns (shared transform in `lib/wiki.py`)
- Resolves relative Markdown links to sibling `.md` files into `[[wiki page title|label]]`, using the target's own `title` frontmatter as the page name
- Prepends the `{{Auto-generated|source=…|commit=<sha>}}` banner
- Uploads a declared `lrc` companion file when its SHA1 has changed, and appends a `[[Media:…]]` footer link
- Injects `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set, plus one `[[Category:…]]` per `categories` entry. Category pages themselves are user-written and never overwritten
- Writes via the MediaWiki bot API, skipping the write when the generated text already matches the page
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Auto-generated banner (`{{Auto-generated|source=...|commit=<sha>}}`)
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set); additional categories via `categories` list
- LRC file upload (idempotent SHA1 check) + footer link
- Inter-page link resolution: a relative Markdown link (`[label](notes.md)`) becomes `[[wiki page title|label]]`, resolved against a repo-wide index so it never depends on which files a push touched
- Repo-wide page-title collision detection, and render-before-write, both ahead of any wiki contact
- MediaWiki bot API write with idempotency — no-op if wiki content matches generated output
- Files without `wiki.publish: true` silently skipped
- Pages with existing non-auto-generated content skipped (logged to stderr) — delete the page or add the banner to hand ownership to the bot
Deliberately not in scope: multi-language metadata (the body conveys what languages exist; the renderer doesn't introspect), and auto-generated album index pages (album category-page descriptions are written freehand).
Not in scope (intentional):
## Failure modes
- Multi-language metadata (the body conveys what languages exist; renderer doesn't introspect)
- Auto-generated album index pages — user writes album category-page descriptions freeform
Everything below aborts the run with a non-zero exit and the wiki untouched. Validation and rendering both complete before the first write, so a broken page never leaves the wiki half-updated.
| Condition | Result |
|---|---|
| Missing/empty `title`, non-boolean `wiki.publish`, malformed `release_date`, non-list `categories` | validation error |
| Two publishable files claiming the same `title` | validation error, checked repo-wide |
| A `.md` link whose target is missing, or exists but isn't published | render error |
| A `.md` link that survives rendering unresolved (unrecognised shape, or written inside a `columns` fence) | render error |
| `GITEA_REPO_URL` resolving to a loopback, mDNS or private-range host | run aborts before any wiki contact |
Two things degrade rather than fail: a missing or non-UTF-8 `lrc` file is skipped with a warning and the page still publishes; and a page whose existing wikitext does not start with `{{Auto-generated` is skipped with a warning, to avoid overwriting hand-written content. Delete the page or add the banner to hand it to the bot.
## Source repo and branch convention
Source repo: `novoyuuparosk-wiki/ncmr-songs`. The pipeline triggers on push to `master`.
Source repo: `novoyuuparosk-wiki/ncmr-songs`, triggering on push to `master`.
**Branch convention in `ncmr-songs`**: per-song short-lived branches named `autowiki/<song-slug>` (e.g., `autowiki/pulse`). Create when staging edits, merge to master when ready to publish, delete after one cycle.
**Branch convention there**: per-song short-lived branches named `autowiki/<song-slug>` (e.g. `autowiki/pulse`). Create when staging edits, merge to `master` when ready to publish, delete after one cycle.
**Excluded paths** (won't trigger the pipeline):
- `wip/**`
- `README.md`, `.gitignore`, `.claude/**`
**Paths that don't trigger a run**: `wip/**`, `README.md`, `.gitignore`, `.claude/**`. Note that `wip/**` and `README.md` are excluded by the renderer too, so they can never publish even on a run triggered by something else.
## Invocation
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/ncmr-songs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
`novoyuuparosk-wiki/ncmr-songs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-songs.yml`](../../.gitea/workflows/publish-songs.yml) here, which clones both repos and runs `publish.py`.
Triggers:
- `push` to `master` (path-filtered as above)
- `workflow_dispatch` — manual trigger from the Gitea Actions UI; useful after renderer or template changes without needing a dummy commit
### Environment (secrets and variables)
- `push` to `master`, path-filtered as above
- `workflow_dispatch` — manual run from the Gitea Actions UI. Passes an empty base ref, which the pipeline reads as "publish everything", so this is the way to re-render every page after a renderer or template change
| Name | Kind | Source | Purpose |
|---|---|---|---|
| `WIKI_API_URL` | variable | Gitea (cross-cutting) | MediaWiki action API endpoint |
| `WIKI_BASE_URL` | variable | Gitea (cross-cutting) | Wiki base URL |
| `WIKI_BOT_USER` | secret | Gitea (cross-cutting) | Bot login (`Mikkeli@giteaBot`) |
| `WIKI_BOT_PASSWORD` | secret | Gitea (cross-cutting) | BotPasswords value |
| `URL_TO_GITEA` | variable | Gitea (cross-cutting) | Runner-internal Gitea base URL, for cloning only |
| `PUBLIC_URL_TO_GITEA` | variable | Gitea (cross-cutting) | Public Gitea base URL, for the banner source link only |
| `FAPAT` | secret | Gitea (cross-cutting) | Full-Access PAT for cloning private repos |
| `SOURCE_REF` | env (set in workflow) | `github.sha` | Pushed commit SHA, used in banner and edit summary |
| `GITEA_REPO_URL` | env (set in workflow) | `PUBLIC_URL_TO_GITEA` + repo path | Source-file URL base for the banner link. Validated by `public_source_base()`: a loopback, mDNS or private-range host aborts the run rather than publishing an unreachable link |
Editing `publish-songs.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
### Publish scope
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every publishable file, because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff plus one hop in each link direction: any page a changed page links to (so a link can't point at an unpublished page), and any page that links to a changed page (so an edited `title` doesn't leave stale inbound links).
| Invocation | Publishes |
|---|---|
| `--all` | every publishable file. Used on first push and `workflow_dispatch` |
| `--files <paths…>` | those files plus their one-hop link neighbours. Used for an ordinary push |
Because the banner carries the commit SHA, every page in a run's write set is rewritten on a new commit; a page pulled in only by the link closure is a no-op API read when the run repeats at the same SHA.
### Environment
| Name | Kind | Purpose |
|---|---|---|
| `WIKI_API_URL` | variable, org-scoped | MediaWiki action API endpoint |
| `WIKI_BASE_URL` | variable, org-scoped | Wiki base URL |
| `WIKI_BOT_USER` | secret, org-scoped | Bot login (`Mikkeli@giteaBot`) |
| `WIKI_BOT_PASSWORD` | secret, org-scoped | BotPassword value |
| `URL_TO_GITEA` | variable, org-scoped | Runner-internal Gitea base URL, for cloning only |
| `PUBLIC_URL_TO_GITEA` | variable, org-scoped | Public Gitea base URL, for the banner source link only |
| `FAPAT` | secret, org-scoped | Full-Access PAT for cloning at runtime |
| `SOURCE_REF` | env, set in workflow from `github.sha` | Commit SHA, used in the banner and the edit summary |
| `GITEA_REPO_URL` | env, set in workflow from `PUBLIC_URL_TO_GITEA` + repo path | Base of the banner source link. Validated by `public_source_base()` |
## Dependencies
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild.
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
## Wiki-side prerequisites
## Modes
**Reading is always repo-wide; writing is scoped to the push.** Every run scans every file with `wiki.publish: true` (not under `wip/`), because link resolution and title-collision detection have to see the whole tree. What gets *written* is the push diff, plus one hop in each link direction: any page a changed page links to, and any page that links to a changed page.
- `--all` — publish every publishable file. Used on the first push and on `workflow_dispatch`.
- `--files <paths…>` — publish these files and their one-hop link neighbours. Used for an ordinary push.
Writes only happen when content has changed (commit SHA in the banner drives idempotency), so a page pulled in by the link closure is normally a no-op API read.
## One-off wiki setup
- [x] **`Template:Auto-generated`** — exists on the wiki. Wikitext source: `template_auto_generated.wikitext` in this directory.
- [x] **Bot user in `bot` group** — grants `skipcaptcha`; verified working.
## Decisions log (songs pipeline)
| Decision | Value | Date |
|---|---|---|
| Metadata source | YAML frontmatter inside each `.md` (see [SCHEMA.md](SCHEMA.md)) | 2026-06-09 |
| Body rendering | Pandoc-based, with a thin Python pre/post-processor | 2026-06-09 |
| Excluded paths | `wip/**`, plus repo-meta files | 2026-06-09 |
| Page template | Designed from scratch (no existing wiki pages to mirror) | 2026-06-09 |
| Album categories | Auto-injected `[[Category:Songs]]` + `[[Category:Album:<name>]]` when `album` is set; category pages are user-written and not overwritten | 2026-06-09 |
| Additional categories | Optional `categories` list in frontmatter; each entry → `[[Category:<value>]]` | 2026-06-10 |
| `album` optional | Companion pages don't need an album; omitting the field emits no album category | 2026-06-10 |
| ~~Sibling resolution~~ | ~~Parent declares siblings as a bill of materials (`path` + `tag` + optional `label`); `[[TAG]]` substituted post-Pandoc~~ — **retired 2026-09-09**, see below | 2026-06-10 |
| Inter-page links | Plain relative Markdown link (`[label](notes.md)`); target's own `title` frontmatter is the page name. No frontmatter declaration, no macro namespace, and the link still works in Gitea's blob view. Replaces the `wiki.siblings` bill of materials, which substituted a bare tag by unanchored string replace — no word boundary, no collision check, and it degraded a link to plain text whenever the target fell outside the batch | 2026-09-09 |
| Unresolvable link | Hard error, never graceful degradation. A link to a missing or unpublished file fails the run; linking to a file does not publish it | 2026-09-09 |
| Render before write | All pages in a run are rendered before the first is written, so a broken page leaves the wiki untouched rather than half-updated | 2026-09-09 |
| Source-repo branch convention | `autowiki/<song-slug>` short-lived branches in `ncmr-songs` | 2026-06-09 |
| Idempotency | Commit SHA in banner (`commit=<sha>`), not timestamp — stable across identical re-runs | 2026-06-09 |
| Publish mode | Scan repo-wide, write the push diff plus one-hop link neighbours. Publishing is not instant, so the write set is bounded; resolution never is, which is what makes bounding the write set safe | 2026-09-09 |
| Title collisions | Detected across the whole repository, before any wiki contact — not only among the files one push happened to touch | 2026-09-09 |
| Tests | `tests/` run by `.gitea/workflows/test.yml` in the publish runner image, so the pandoc-output contract the link resolver depends on is pinned by a test rather than by prose | 2026-09-09 |
| Manual-page protection | Bot skips pages without the `{{Auto-generated` banner to avoid overwriting hand-written content | 2026-06-09 |
| `workflow_call` abandoned | Cross-repo `workflow_call` fails — run token scoped to triggering repo; cannot clone private callee | 2026-06-09 |
| Side-by-side columns | Authored as a ` ```columns ` fenced block (separator `===`); Pandoc emits `<pre class="columns">`, expanded post-Pandoc into a flex `<div>` of `<poem>` columns. No wiki template or extension — all Pi-side before the API write. See [SCHEMA.md](SCHEMA.md#columns-side-by-side-shorthand) | 2026-06-14 |
## Status
v1 live. Pipeline active and publishing to https://wiki.novoyuuparosk.org.
- **`Template:Auto-generated`** must exist on the wiki. Wikitext source: [`template_auto_generated.wikitext`](template_auto_generated.wikitext) in this directory.
- The bot user must be in the `bot` group, which grants `skipcaptcha`.
pipelines/songs/SCHEMA.md
+1 -1
View File
@@ -268,4 +268,4 @@ Files without a `wiki.publish: true` frontmatter field are silently skipped —
Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.
Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing.
Frontmatter keys this schema does not define are ignored rather than rejected.
pipelines/tech/README.md
+13 -29
View File
@@ -5,7 +5,7 @@ Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki.
- Source-file contract (frontmatter and body conventions): [SCHEMA.md](SCHEMA.md)
- Cross-cutting setup (wiki URL, bot identity, runner, Gitea secrets/variables): [repo root README](../../README.md)
## v1 scope
## What the renderer does
- YAML frontmatter parsing per [SCHEMA.md](SCHEMA.md)
- Pandoc-based markdown → wikitext body rendering (h1 elements preserved)
@@ -18,9 +18,9 @@ Publishes tech blog posts from `novoyuuparosk-wiki/tech-blogs` to `https://wiki.
## Source repo
Source repo: `novoyuuparosk-wiki/tech-blogs` (not yet initialised — pipeline code is ready).
Source repo: `novoyuuparosk-wiki/tech-blogs`, triggering on push to `master`, with `workflow_dispatch` for manual runs.
The pipeline triggers on push to `master` and supports `workflow_dispatch` for manual runs.
No branch convention enforced. There is no `wip/` exclusion — `wiki.publish: true` is the only publishing gate.
**Excluded paths** (won't trigger the pipeline):
@@ -28,12 +28,18 @@ The pipeline triggers on push to `master` and supports `workflow_dispatch` for m
## Invocation
The pipeline runs as a self-contained Gitea Actions workflow in `novoyuuparosk-wiki/tech-blogs` (`.gitea/workflows/publish.yml`). It clones this repo at runtime to get the renderer.
`novoyuuparosk-wiki/tech-blogs/.gitea/workflows/publish.yml` is a thin caller: it triggers, then `uses:` [`publish-tech.yml`](../../.gitea/workflows/publish-tech.yml) here, which clones both repos and runs `publish.py`.
Editing `publish-tech.yml` requires a runner cache flush before it takes effect; the renderer itself is cloned fresh every run. See *Callee workflow cache* in the [root README](../../README.md).
Triggers:
- `push` to `master` (path-filtered as above)
- `workflow_dispatch` — manual trigger from the Gitea Actions UI
### Publish scope
Every run publishes every file with `wiki.publish: true`; there is no diff detection. Writes are idempotent — the banner carries the commit SHA rather than a timestamp, so a re-run at the same commit rewrites nothing.
### Environment (secrets and variables)
| Name | Kind | Source | Purpose |
@@ -50,30 +56,8 @@ Triggers:
## Dependencies
Job container: `novoyuuparosk-wiki-runner:latest` (pre-built, stored in the local Docker daemon on the runner host). Bakes in Python 3.12, Pandoc, and all pipeline Python packages — no install steps at job runtime.
Job container image and its build are described in the [root README](../../README.md#job-container-image). Python packages for this pipeline are in `requirements.txt` here; changing it triggers an image rebuild. Shared render and MediaWiki helpers live in `lib/wiki.py` at the repo root.
See the `Dockerfile` at the repo root and `requirements.txt` in this directory. Rebuild the image after changes to either.
## Wiki-side prerequisites
Shared code lives in `lib/wiki.py` (repo root).
## One-off wiki setup
- [ ] **`Template:Auto-generated`** — must exist on the wiki (shared with all pipelines; already created).
## Decisions log (tech pipeline)
| Decision | Value | Date |
|---|---|---|
| No title prefix | Tech posts live at bare titles — no namespace needed | 2026-06-10 |
| Fixed category | Every post gets `Category:Tech blog` | 2026-06-10 |
| Year category | `Category:Blog:<year>` from `date` field; falls back to the execution year if no date | 2026-06-10 |
| Tag categories | Each tag → `[[Category:<tag>]]` with no prefix | 2026-06-10 |
| WIP exclusion | None — `wiki.publish: false` is the only gate | 2026-06-10 |
| Publish mode | Always `--all` | 2026-06-10 |
| Idempotency | Commit SHA in banner | 2026-06-10 |
| Manual-page protection | Bot skips pages without `{{Auto-generated` banner | 2026-06-10 |
| Source repo | `novoyuuparosk-wiki/tech-blogs` — pipeline ships before repo is initialised | 2026-06-10 |
## Status
v1 pipeline ready. Source repo (`novoyuuparosk-wiki/tech-blogs`) not yet initialised.
**`Template:Auto-generated`** must exist on the wiki. It is shared across all three pipelines; the wikitext source lives in [`pipelines/songs/template_auto_generated.wikitext`](../songs/template_auto_generated.wikitext).