novoyuuparosk-wiki/novoyuuparosk-auto-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f5625cde98204c5663bfbede6944b586148d679f
novoyuuparosk-auto-wiki/pipelines/songs/SCHEMA.md
T
wan f5625cde98 songs: replace the wiki.siblings macro with plain Markdown links 
The `wiki.siblings` bill of materials is gone. A related co-publication is
now nominated with an ordinary relative Markdown link — `[label](notes.md)` —
resolved at publish time against the target file's own `title` frontmatter.
No macro namespace, no per-file declaration, and the link still works in
Gitea's blob view and any editor preview.

The macro could not be made safe. Substitution was `str.replace(tag, …)` over
the whole rendered page: no word boundary, so a tag also occurring in lyrics or
a code block got rewritten; no collision check, so overlapping tags corrupted
each other in declaration order. SCHEMA.md documented the placeholder as
`[[TAG]]` while the code replaced the bare tag, so following the schema
produced `[[[[Title|label]]]]`.

Fixes a live corruption bug in the process. `sibling_map` was built only from
the files in the publish batch, and an unresolved sibling degraded to plain
text — so pushing `ses/pulse.md` alone republished it with its wikilink
silently stripped to the bare words "Dev gibberish". SCHEMA.md asserted the
workflow "always runs --all", which it never did on an ordinary push.

The repository is now always scanned in full, whatever a run publishes, so
link resolution and title-collision detection can never depend on which files
a push touched. Writes stay bounded to the push diff plus one hop in each link
direction: the page a changed page links to (so a link cannot point at an
unpublished page) and the pages linking to it (so an edited `title` does not
leave stale inbound links). Publishing is not instant; reading is cheap.

An unresolvable `.md` link is now a hard error, never graceful degradation, and
linking to a file still does not publish it — `wiki.publish: true` remains the
only gate. A leftover-link guard fails the run on any link shape the resolver
did not recognise, including a Markdown link written inside a verbatim
```columns fence, so a pandoc upgrade cannot silently publish literal text.

Also:

- render before write: every page in a run is rendered before the first is
  published, so a broken page leaves the wiki untouched instead of
  half-updated. `publish_one` is split into `render_page` (pure) and the write.
- pin pandoc's stdout to UTF-8. `text=True` decoded it with the ambient locale
  encoding; on any non-UTF-8 locale the Japanese and Chinese lyrics died inside
  subprocess's reader thread, `stdout` came back None, and the real error was
  lost behind a TypeError. Latent on the Debian runner via PEP 538 only.
- title collisions are detected repo-wide, not just within a push, and before
  any wiki contact.
- tests/, run by .gitea/workflows/test.yml in the publish runner image. The
  pandoc output contract the resolver depends on is now pinned by a test
  instead of by prose — which is how SCHEMA.md came to assert something false.
- drop an f-string with nothing to interpolate.
- quote the SOURCE_REF interpolation in the workflow instead of pasting an
  input straight into shell.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 16:24:36 +09:00

272 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# SCHEMA — song source files
Source-file contract for the songs pipeline. Files in `novoyuuparosk-wiki/ncmr-songs` must follow this contract to be picked up by the auto-publisher.
The schema lives entirely in the YAML frontmatter block at the top of each `.md` file. The body below the frontmatter is plain markdown, rendered to MediaWiki wikitext by Pandoc — no special body conventions are enforced beyond the standard markdown grammar. There is no macro syntax and nothing to declare: a link to another song page is an ordinary relative Markdown link, resolved to a wikilink at publish time.
## File layout
```markdown
---
title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
release_date: 2026-06-18
lrc: pulse.lrc
---
# pulse under latent semantic envelopes
... body markdown; a related page is nominated as [development notes](pulse_dev.md) ...
```
The renderer parses the frontmatter, validates required fields, hands the body (with the first `# Heading` line removed) to Pandoc, then resolves `.md` links in the wikitext output to the wiki page titles of their targets.
## Fields
### Required
#### `title` (string)
The MediaWiki page name that the bot writes or updates. MediaWiki capitalises the first letter automatically; lowercase input is fine.
Example: `title: pulse under latent semantic envelopes` → wiki page `Pulse under latent semantic envelopes` at URL `/wiki/Pulse_under_latent_semantic_envelopes`.
The body should contain a matching `# Title` first-line heading for human readability when viewing the source file directly (in an editor, on Gitea's blob view, etc.). The renderer strips this first-line heading during conversion — the wiki page already supplies its own title.
#### `wiki.publish` (boolean)
Publishing gate. `true` means the pipeline writes this file to the wiki. `false` means the file is ignored.
The `wiki` key is a mapping rather than a flat field so future toggles (`wiki.protected`, `wiki.summary_template`, etc.) can be added without restructuring.
Files under `wip/` are also skipped, regardless of `wiki.publish`.
### Optional
#### `album` (string)
The full album name. Drives `[[Category:Songs]]` and `[[Category:Album:<album name>]]` injection at the bottom of the rendered wiki page.
Example: `album: Shirakaba Express Service` → category tags `[[Category:Songs]]` and `[[Category:Album:Shirakaba Express Service]]`.
Pages without an album (companion documents, placeholders, songs without an assigned album) simply omit this field — neither category is injected unless `categories` provides one.
#### `categories` (list of strings)
Additional wiki categories to inject, beyond the album category. Each entry becomes `[[Category:<value>]]`.
```yaml
categories:
- Song development notes
- Work in progress
```
`album` and `categories` are additive — a page can have both.
#### `release_date` (ISO 8601 date)
The song's release date in `YYYY-MM-DD` format. Skipped if absent. Currently informational; may drive sort order on future auto-generated indexes.
#### `lrc` (string, relative path)
Pointer to a companion LRC file, relative to the `.md` file's location. The pipeline uploads the file to the wiki as `<title>.lrc` (if the content has changed) and appends a `[[Media:<title>.lrc|Synced lyrics (.lrc)]]` link to the page footer.
Example: in `ses/pulse.md`, `lrc: pulse.lrc` declares that `ses/pulse.lrc` is the companion synced-lyrics file.
If the LRC file is missing or not valid UTF-8, the upload is skipped with a warning — the page still publishes.
## Body
Plain markdown. The renderer applies these transformations before and after Pandoc:
1. **Pre-Pandoc**: strip the first-line `# Heading` if present.
2. **Post-Pandoc**: expand `columns` fenced blocks into side-by-side wikitext (see below).
3. **Post-Pandoc**: resolve links to sibling `.md` files into wikilinks (see below).
Do not embed raw wikitext-specific syntax (`{{Template}}`, raw `[[Wikilink]]`, etc.) in the body unless you intend the literal output. Use markdown idioms; the renderer adds the metadata-derived bits (banner, LRC link, categories, link resolution) around Pandoc's output.
### Nominating a related co-publication
Link to it the ordinary way, with a relative Markdown link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
There is nothing to declare in the frontmatter. The link stays a working link in Gitea's blob view, in an editor preview, and anywhere else the raw file is read — and on the wiki it becomes a wikilink to the target's page:
```wikitext
For more behind-the-scene shenanigans, see [[pulse under latent semantic envelopes - development notes|dev gibberish]].
```
**How the target becomes a page name.** The path is resolved relative to the linking file's directory, and the target file's own `title` frontmatter field is used verbatim as the wikilink target — the same string the bot writes that page to. There is no second name format to keep in sync: rename a page by editing its `title`, and every link aimed at it follows on the next publish. MediaWiki handles the first-letter capitalisation and the space-to-underscore in URLs itself.
| Written in the body | Rendered wikitext |
|---|---|
| `[dev gibberish](pulse_dev.md)` | `[[<target's title>\|dev gibberish]]` |
| `[pulse_dev.md](pulse_dev.md)` (label same as path) | `[[<target's title>]]` |
| `[notes](../ses/pulse_dev.md)` | `[[<target's title>\|notes]]` |
| `[notes](pulse_dev.md#Section)` | `[[<target's title>#Section\|notes]]` |
| `[site](https://example.com)` | `[https://example.com site]` (untouched) |
**A `.md` link that cannot be resolved fails the run.** Both cases abort with a non-zero exit before anything is published:
| Condition | Result |
|---|---|
| Target file does not exist | **error** — `file does not exist` |
| Target exists but is not published (no `wiki.publish: true`, or under `wip/`) | **error** — `file exists but is not published` |
Linking to a file does **not** publish it. Publication is always the target's own explicit `wiki.publish: true`; nothing is published as a side effect of being linked. If you want the link, opt the target in.
**Scope of a run.** A push publishes the files it changed, plus — one hop in each direction — any page a changed page links to, and any page that links to a changed page. The second is what keeps inbound links correct when a page's `title` moves. Regardless of how few pages a run publishes, the whole repository is always scanned first, so link resolution and title-collision checks see every publishable file.
**Anchors** (`pulse_dev.md#Section`) are passed through verbatim. MediaWiki derives its own section anchors from heading text, so an anchor that doesn't match a heading on the target page lands the reader at the top of it rather than at the section. Nothing validates this.
**Links inside a `columns` fence do not work.** The fence is verbatim by design (see below); Pandoc does not process Markdown in it, so a link written there would reach the wiki as literal text. The renderer detects this and fails the run rather than publishing it. Put the link in ordinary body prose.
### Columns (side-by-side) shorthand
For parallel content — e.g. an original and its translation — write a fenced code block tagged `columns` and separate the columns with a line containing only `===`:
````markdown
```columns
**原題**
一行目
二行目
===
**Title**
first line
second line
```
````
The renderer turns this into a flexbox row of `<poem>` columns (one per `===`-delimited section). Notes:
- **Line breaks and blank lines are preserved verbatim** — that's the point of using a fenced block; Pandoc passes the body through untouched, and each column is wrapped in `<poem>` so MediaWiki keeps the line breaks.
- **N columns**: use N−1 `===` separators. Two is the common case (original / translation).
- **Emphasis in Markdown**: write `*italic*`, `**bold**`, or `***bold-italic***` (asterisk style, single line) — these are converted to wikitext. Raw inline HTML such as `<b>…</b>` or `<br>` also still works (it's HTML-unescaped on the way out), but you shouldn't need it.
- This is a shared transform (`lib/wiki.py`), so it works for any pipeline, not just songs. Column width/gap styling currently lives in that transform.
## Renderer behaviour
A run happens in three phases, in this order, and the wiki is not contacted until the third. Anything wrong in phase 1 or 2 aborts with a non-zero exit and the wiki untouched — never half-updated.
**Phase 1 — scan and validate the whole repository.** For every `.md` outside `wip/` (and excluding `README.md`):
1. Parse the frontmatter; skip the file silently unless `wiki.publish: true`.
2. Validate the fields (see *Validation errors* below).
3. Register `title` and fail on a collision with any other publishable file, anywhere in the tree.
4. Build the path → page-title index every `.md` link resolves through.
**Phase 2 — render every page that this run will publish.** For each selected file:
1. Strip the leading `# Heading` from the body if present.
2. Pipe the body through `pandoc -f markdown -t mediawiki`, then expand any `columns` fenced blocks into side-by-side `<poem>` columns.
3. Resolve `.md` links to `[[Page title|label]]` wikilinks; fail on any that cannot be resolved, and on any link shape that survived rendering unresolved.
4. Prepend the auto-generated banner: `{{Auto-generated|source=<source URL>|commit=<sha>}}`.
5. If `lrc` is declared, append a `[[Media:…]]` link.
6. Append category tags: `[[Category:Songs]]` and `[[Category:Album:<album>]]` if `album` is set; `[[Category:<cat>]]` for each entry in `categories`.
**Phase 3 — write.** For each rendered page:
1. If `lrc` is declared: upload the LRC file if its SHA1 has changed.
2. Read the current wiki page content via the MediaWiki API; if identical to the generated output, skip the write (idempotency — keeps the wiki history clean).
3. If the page exists and does not start with `{{Auto-generated`, skip with a warning (manual page protection).
4. Otherwise, write the page with an edit summary referencing the source commit.
## Implemented scope
- Frontmatter parsing and validation
- Repo-wide page-title collision detection, before any wiki write
- Pandoc-based markdown → wikitext body rendering
- First-line h1 stripping
- Banner template injection
- LRC file upload and footer link
- Relative `.md` link resolution (`[label](notes.md)` → `[[wiki page title|label]]`), repo-wide and batch-independent
- Album category injection (`[[Category:Songs]]` + `[[Category:Album:<name>]]`) when `album` is set
- Additional categories via `categories` list
- Render-before-write: a broken page fails the run instead of half-updating the wiki
- Idempotent writes (no-op skip when content matches)
- Manual-page protection (skip pages without auto-gen banner)
- Files without `wiki.publish: true` silently skipped
## Not in scope (intentional)
- Multi-page aggregation: no album index pages generated; each file → its own wiki page. Album landing pages are wiki category pages, written freeform by the user.
- Language-aware rendering: the body conveys what languages exist; the renderer doesn't introspect.
## Examples
### A released song
```yaml
---
title: pulse under latent semantic envelopes
album: Shirakaba Express Service
wiki:
publish: true
release_date: 2026-06-18
lrc: pulse.lrc
---
```
Its body nominates the companion page with a plain relative link:
```markdown
For more behind-the-scene shenanigans, see [dev gibberish](pulse_dev.md).
```
### A companion document (no album)
```yaml
---
title: pulse under latent semantic envelopes - development notes
wiki:
publish: true
categories:
- Song development notes
---
```
### A placeholder (not published)
```yaml
---
title: (placeholder)
album: kairo
wiki:
publish: false
---
```
### A WIP file (in `wip/`, skipped regardless)
```yaml
---
title: 海淀
wiki:
publish: false
---
```
## Validation errors the renderer must produce
Files without a `wiki.publish: true` frontmatter field are silently skipped — no error. The pipeline fails fast and loud only on files that are opted in (`wiki.publish: true`) with invalid data:
- Missing or empty `title` on a file with `wiki.publish: true`
- `wiki.publish` present but not a boolean
- `release_date` (if present) does not parse as YYYY-MM-DD
- `categories` (if present) is not a list
- Two publishable source files declare the same `title` (page-name collision) — checked across the whole repository, not just the files a push touched
- A `.md` link whose target file does not exist
- A `.md` link whose target exists but is not published
- A `.md` link that survives rendering unresolved — a link shape the renderer does not recognise, or a Markdown link written inside a `columns` fence
Validation and rendering both run before any wiki write. A failure at either stage aborts the run with a non-zero exit and leaves the wiki untouched, including when only some of the run's pages are broken.
Frontmatter keys the schema does not define are ignored, so a leftover `wiki.siblings` block from the retired macro system is inert rather than an error. Delete them; they do nothing.
Reference in New Issue View Git Blame Copy Permalink