novoyuuparosk-auto-wiki

CI/CD pipelines that auto-apply commits to https://wiki.novoyuuparosk.org from upstream content repos.

Pipelines

Path	Source repo	Purpose	Status
pipelines/songs/	novoyuuparosk-wiki/ncmr-songs	Song lyric pages	v1 live
pipelines/ses/	novoyuuparosk-wiki/ses-light-novel	SES light novel pages	v1 in development
pipelines/tech/	novoyuuparosk-wiki/tech-blogs	Tech blog posts	v1 live

Per-pipeline READMEs cover everything specific to that pipeline (source schema, renderer, runtime). This root README covers only what's cross-cutting.

Architecture

Split across two repos, because Gitea only fires workflows from .gitea/workflows/ of the repo that was pushed to, while the rendering logic belongs here.

<source-repo>/
  .gitea/workflows/publish.yml      <- thin caller: on: push, uses: the reusable workflow below

novoyuuparosk-auto-wiki/            <- this repo
  .gitea/workflows/publish-<p>.yml  <- on: workflow_call. Clones both repos, runs the renderer
  .gitea/workflows/test.yml         <- on: push. Runs tests/ in the publish runner image
  .gitea/workflows/build-image.yml  <- builds and pins the shared job container image
  pipelines/<pipeline>/             <- per-pipeline renderer, schema, templates
  lib/                              <- shared render and MediaWiki helpers
  tests/                            <- pytest suite for lib/ and the pipelines

The two halves reach the runner by different routes, which matters when a change appears not to take effect:

• Renderer code (pipelines/, lib/) is a plain git clone of this repo's default branch on every run. Always current, never cached.
• Workflow YAML (publish-*.yml) is resolved through the caller's uses: and is cached — see Callee workflow cache below.

This repo must stay public. act_runner resolves a cross-repo uses: by cloning the callee anonymously — the job token is not applied, and shared ownership does not satisfy the read requirement. A private callee 404s with repository not found. The alternative, untested here, is Settings → Actions → General → collaborative owners (Gitea 1.26+).

Wiki

• Base URL: https://wiki.novoyuuparosk.org
• MediaWiki API: https://wiki.novoyuuparosk.org/api.php (classic action API)

Bot identity

MediaWiki BotPassword issued for user Mikkeli, bot name giteaBot. Login form: Mikkeli@giteaBot.

Mikkeli holds the bot right, so pipeline edits are flagged as bot edits and stay out of default Recent Changes — mwclient's page.save() requests the bot flag and the wiki honours it.

Credentials live in the Gitea org-scope secret vault under novoyuuparosk-wiki (WIKI_BOT_USER, WIKI_BOT_PASSWORD). Not stored in this repo.

Runner infrastructure

One act_runner instance serves all pipelines. It runs on a Pi 5 (Raspberry Pi OS Bookworm, aarch64) as a service in the same docker-compose stack as Gitea, built linux-arm64 from the gitea/act_runner image. Job execution is via the host Docker socket — the runner is a container and jobs spawn as sibling containers.

Container network mode is host, so job containers can reach Gitea on localhost:3005 and resolve mDNS hostnames.

Callee workflow cache

act_runner caches each resolved cross-repo callee at /root/.cache/act/<owner>-<repo>@<ref> inside the runner container, and never refreshes it per run. The key contains no commit SHA, so pushing to master does not invalidate it: edits to publish-*.yml — including the pin job's own commits — keep executing the stale cached copy indefinitely.

Two consequences. When a pipeline misbehaves after a workflow edit, suspect the cache before suspecting permissions. And after any publish-*.yml change, flush it:

docker compose up -d --no-deps --force-recreate act_runner

--no-deps is required: act_runner declares depends_on: gitea, so without it the recreate cascades and bounces the Gitea instance. Pass no --pull, or the flush silently adopts whatever act_runner:latest has become. The cache sits in the container layer while .runner is in the /data volume, so registration survives the recreate.

Changing a repo's owner also changes the cache key, which flushes it as a side effect.

Job container image

All pipelines share one pre-built image, served from the Gitea registry at pi5-16.local:3005/novoyuuparosk-wiki/novoyuuparosk-wiki-runner. The root Dockerfile bakes in system deps (git, pandoc, ca-certificates) and every pipeline's Python packages, so job containers start with no install step.

build-image.yml rebuilds it on pushes touching the Dockerfile, any pipeline requirements.txt, or itself. It uses kaniko (daemonless, unprivileged) to push an immutable :<short-sha> and a moving :latest. A follow-up pin job rewrites the image: pin in each publish-*.yml to the new short SHA and commits it back to master using FAPAT, so the publish workflows always reference an immutable tag with no manual bump. That pin commit touches only workflow_call files, so it triggers no further runs — but it does need a cache flush to take effect.

The builder image kaniko-act lives under mikkeli, not the org, and cannot be rebuilt by build-image.yml: kaniko does not persist the empty dir from mkdir -p -m 1777 /tmp, so the resulting image has no /tmp and wget -O /tmp/... fails. Bumping it needs a host docker build and push, or a Dockerfile that forces the dir to materialise by writing a file inside it. build-kaniko-act.yml is dispatch-only and runs in the old mikkeli copy.

Gitea Actions setup (cross-cutting)

Secrets and variables are scoped to the novoyuuparosk-wiki org and inherited by every repo under it. A run belongs to the caller repo, so a source repo resolves secrets and variables from its own owner — which is why they live at org scope rather than on this repo.

Secrets:

Name	Value / purpose
WIKI_BOT_USER	Mikkeli@giteaBot
WIKI_BOT_PASSWORD	the BotPassword from Bot identity above
FAPAT	Full-Access PAT owned by mikkeli, used by the publish workflows to clone repos at runtime. The PAT is personal; only its storage scope is the org — hence the mikkeli: basic-auth username in clone URLs
PKGRW_PAT	package read/write PAT owned by mikkeli, used by build-image.yml to push to the container registry

Variables:

Name	Value	Used for
WIKI_BASE_URL	https://wiki.novoyuuparosk.org	wiki base
WIKI_API_URL	https://wiki.novoyuuparosk.org/api.php	MediaWiki action API
URL_TO_GITEA	http://localhost:3005	runner-internal address, cloning only
PUBLIC_URL_TO_GITEA	https://git.novoyuuparosk.org	public address, page banner source links only

Names avoid a GITEA_/GITHUB_ prefix because Gitea blocks those, hence URL_TO_GITEA rather than GITEA_URL.

The two URL variables must stay separate. URL_TO_GITEA has to be an address the runner container can clone from, and host networking makes that a loopback. PUBLIC_URL_TO_GITEA is baked into the {{Auto-generated}} banner of every published page and has to resolve for a reader on the internet. lib/wiki.public_source_base() validates the banner value and aborts the run on an unset, relative, loopback, mDNS, or private-range address, so collapsing them back into one fails loudly instead of publishing dead links.

Branch naming

• This repo: automation/<pipeline-name> for pipeline-development branches (e.g. automation/songs).
• Source repos: each pipeline's README defines its own convention (e.g. autowiki/<song-slug> in ncmr-songs).

Tests

pip install -r requirements-dev.txt
python -m pytest tests -q

Pure string-in/string-out coverage of lib/ and the pipelines' discovery, validation and publish-scoping logic — no wiki, no network. The TestPandocContract cases shell out to real pandoc and are skipped if it isn't installed; they pin the pandoc output shape the link resolver depends on, so a pandoc upgrade fails a test instead of publishing broken links. test.yml runs the suite on push in the same image the pipelines publish from.