82 lines
11 KiB
Markdown
82 lines
11 KiB
Markdown
|
---
|
||
|
|
title: Generative (and not) AI use in making the song 'pulse under latent semantic envelopes'
|
||
|
|
date: 2026-06-14
|
||
|
|
tags:
|
||
|
|
- GenAI
|
||
|
|
- Music production
|
||
|
|
- Songwriting
|
||
|
|
wiki:
|
||
|
|
publish: true
|
||
|
|
---
|
||
|
|
|
||
|
|
Whether you like it or not, this is the age of general purpose generative AIs. There of course are still bulwarks of the human civilisation strongly opposing their infiltration into all walks of life, and the music industry is actually one of the stronger ones.
|
||
|
|
Quality and creativity are very subjective properties, and I see complete end-to-end AI music generation (e.g. Suno/Studio, Udio, and sloppy budget options) as still lacking a bit of oomph in both. That is a very hot topic and I won't delve deep into it.
|
||
|
|
The important thing is I don't and don't plan to use end-to-end (text prompt to audio file) generative AI in what I see as my own recreational creative activities, like songwriting. Instead, I use AI technology in many supportive roles that simply put, makes life easier.
|
||
|
|
I feel it's probably a good idea to disclose how and where I use what sorts of AI. Not that these are classified or confidential or whatever to begin with.
|
||
|
|
|
||
|
|
**Disclaimer**: this article is not brought to you by any of the projects / services / individuals / organisations mentioned below.
|
||
|
|
|
||
|
|
## Overview
|
||
|
|
|
||
|
|
The most controversial, I would imagine, application of modern generative AI in this song is the homebrew English TTS that kind of sounds like Yuzuki Yukari. I never said that is Yukari mind you.
|
||
|
|
The exact repository used for this song (and the previous one, Tunnels) is [this one](https://github.com/RVC-Boss/GPT-SoVITS). I can even specify the commit hash I technically used throughout (haven't really updated since I first cloned) but apparently no one cares about that.
|
||
|
|
The moral and legal issues around this topic is very messy, but thankfully at least I'm amoral so only the legal concerns are a real thing.
|
||
|
|
|
||
|
|
My day job supposedly is about digital infrastructure maintenance, AI integration into DevOps, and stuff like that. I don't like the job, but I do like the idea of letting machines do the hard labour for me.
|
||
|
|
Much of my generative AI applications, especially in the example of the very song [pulse under latent semantic envelopes](https://wiki.novoyuuparosk.org/wiki/Pulse_under_latent_semantic_envelopes), are that sort of stuff. To be more exact, I throw specs and reqs at Claude and let it make diagnostic and automation tools for me. At first glance some of them might reek of anything but music.
|
||
|
|
|
||
|
|
Another type of help is more directly related to something usually described as songwriting. When I write the lyrics I engage heavily with chatbots. Almost exclusively Claude now (driven by Sonnet 4.6 / Opus 4.7 this time). Obviously I don't tell it to just write everything for me; the finalised version of lyrics is obviously bad like human authored lyrics should be. More details later.
|
||
|
|
|
||
|
|
At the final stages of production my faithful illustrator friend hako told me about [React-Remotion](https://www.remotion.dev/) which is really a thunderstorm that totally changed how I make my music videos. I don't know if I'm going to elaborate on that but it deserves a shoutout for sure. A shoutout to [hako](https://x.com/lxcombox), too. Support your local illustrators who know a thing or two about computers and laser diodes!
|
||
|
|
|
||
|
|
## Voice synthesis
|
||
|
|
|
||
|
|
The motive is dead simple: Yukari doesn't have any English TTS package and I so want to write in English and have her do the songs in English. I know VOCALOID 6 is an option but it is not a practical option.
|
||
|
|
|
||
|
|
My requirements for a Yukari English TTS is kind of simple yet niche. The simplicity is that I don't really need a lot of emotional or timbre variations, just the flat Shizuku voice would do; the niche is that I want a very unique accent. Somewhere else I described that as a mix of rallienglanti, Bristolian, and Aussie. In reality what I speak and what I actually want is probably more than that. I speak what I hear (and consider pleasing) and I hear a lot of stuff.
|
||
|
|
|
||
|
|
Very roughly and factually wrong all over the place, the theory for [GPT-SoVITS](https://github.com/RVC-Boss/GPT-SoVITS) is that the procedure of converting text into speech is further broken down to two steps. **GPT** takes the text and embeds it as pronunciation markers (something like phonemes, but not exactly) and emotions (controls the rises and falls, maybe). **SoVITS** applies the vocal texture based on the intermediate embeddings. The reality is more complicated than that, but the important thing is you can separate accents (handled by GPT) and texture/timbre (handled by SoVITS) into very individual concerns.
|
||
|
|
This setup is what makes possible patching speaker A's voice with speaker B's accents. For the very least you just train the SoVITS weights with A's data, and GPT weights with B's accents. Conceptually concise, innit.
|
||
|
|
Theoretically, it is even possible to blend samples from different speakers into one training set. I haven't really done that but I suspect the results would rather be amalgated than homogeneously blended. Apparently cherrypicking samples based on dominant features (e.g. Räikkönen for the tapped 'R's, shouty Aussie for the 'ey-ay' bias, and Mixu Paatelainen for the plosives) could do something good. But I don't know how good that might be and even if it gives me what I have in mind I don't think I have the time (patience) to do the cherrypicking. I just wanted a quickie [sic].
|
||
|
|
|
||
|
|
Disclosure here is that for the voice that dubs the opening two verses and the introduction to signals and noises in the song *pulse*, I used David Cameron Walker's voice for the GPT part. And I haven't bought my Dreamland subscription which I probably should do for my sins.
|
||
|
|
*Anyone that listens to DCW's pods should notice the voice in my song bears almost no similarity to the real Dave Walker, which is kind of a relief to me.*
|
||
|
|
I can at least leave a link to the most dedicated podcast about the (dominantly English) language of football [here](https://www.footballcliches.com/).
|
||
|
|
|
||
|
|
***Update:*** *I have subscribed to Dreamland now. For my sins.*
|
||
|
|
|
||
|
|
## Vibrator coded utilities
|
||
|
|
|
||
|
|
FYI: I don't like the buzzword 'vibe coding'. I think I've only really done it on pure vibes for a very short period. If you produce human-compatible specs almost following the SPICE model of your SWE ones and twos, I think 'vibrator' coding is an understatement.
|
||
|
|
|
||
|
|
But naming aside, that is basically how I came to finish my long-stalled mastering toolkit, which is the primary by-product of this song. You actually can find it on [GitHub](https://github.com/mikkelimatlock/uj-mastering-master). This is a Python-Qt tool that analyses several metrics I find useful when mastering / prepping for release a song. Most importantly LUFS.
|
||
|
|
The dissatisfaction with Youlean Loudness Meter is that when used in FL Studio it wants me to play over the whole song in 1x speed to calculate the loudness across the whole song. Or I've been doing it wrong. And this is a hassle when the whole song is almost 13 minutes long. So I had to make a tool that spits out results for the 13 minutes in less than 13 minutes, simple as.
|
||
|
|
The basis of the *UJ mastering master* was actually mostly hand-coded by myself ages ago, when it was a very, very crude command line non-interactive script that suddenly pops `matplotlib` plot windows. Peak UX that was, and we have come a long way.
|
||
|
|
|
||
|
|
Another very indirectly associated stack of utilities is my personal Gitea instance that runs on my Pi 5 at home. I felt uneasy about hosting my half-arsed songs on GitHub not because they are a fucking evil corporate, but because my songs are half-arsed. We all need a bit of privacy, no?
|
||
|
|
*Of course I know there is private repo, I ain't dumb. It's psychologically more assuring to save something like your porn colletion in a physically private location.*
|
||
|
|
The point is I would never get this done properly xor in a way I can remember 30 minutes after I set it all up. So I let Claude Code configurate docker compose yamls and that's mostly it.
|
||
|
|
I even set up a Gitea action workflow (several workflows with shared bits) for publishing to my wiki from document/writing repos hosted there. That's how you're seeing this page. Or maybe not this exact page.
|
||
|
|
|
||
|
|
### A tiny reflection on the whole vibrator thing
|
||
|
|
|
||
|
|
I am always afraid of being rusty of code-writing. But I am equally always anxious about diving too deep into code-writing and failing to see the bigger picture.
|
||
|
|
Not one of the first 92 people to say this but leaving the implementations to agents frees me for more tactical and strategical decisions, and that really is a weakness of the agents.
|
||
|
|
Them models and agents are getting better but still, they run into troubles. That is another thing in the agent coding experience that makes me feel I'm still knocking around.
|
||
|
|
|
||
|
|
## A new age for making glorified slideshows as your music videos
|
||
|
|
|
||
|
|
For quite long a time I've been actively (or passively, depends) refusing to make videos that have motions for my songs. One of the reasons is that I thought that making those elements move in sync to the audio is something not practically automatable and would have to be reduced to human flesh proficiency. I don't want to have that proficiency, partly for another reason I don't like music videos: I myself always switch my visual perception off halfway or as soon as 1 minute into a song.
|
||
|
|
|
||
|
|
But Remotion really is a game changer for me personally. I actually had some experience animating stuff with the HTML/CSS/JS stack. And it's actually amazing how it's almost capable of editing 'real' videos in real time. I haven't really done anything heavy, though, just your usual static assets floating around and dynamically (but deterministically) calculated visual effects.
|
||
|
|
|
||
|
|
The point here is that a Remotion 'video' project is described in code, so it's light and agent-friendly. Modern HTML capabilities are mindboggling, and frankly maybe even the Mythos won't be creative an imaginative enough to cook it to its full. So as a human I still need to know the ones and twos of what HTML/CSS is capable of and more importantly what code implementations to call on for the mental images. Or at least try to be fluent in a language so the coding agent gets you.
|
||
|
|
|
||
|
|
## How to have a conversation with an LLM, in the year of 2026, like it was 2019
|
||
|
|
|
||
|
|
I consider myself open to new technologies and ideas yet slow to make the initial adaptation. My reasoning is that some of those on the technical level are no more than makeshifts that won't be required that much once a paradigm shift or significant breakthrough is there one a higher level.
|
||
|
|
An example would be the so-called prompt engineering. To hell with that is all I'll say.
|
||
|
|
|
||
|
|
I'm still not fully embracing clustered multi-agent tasking, skills-based enforcing era of agentic working. Nor that my personal uses really need that much overhead. These might happen if I ever switch to a self-hosted agent framework like Hermes, but Claude Code, in honesty, does most of the memory/context stuff for me.
|
||
|
|
Songwriting, the literal part, is even less suited to being made into a model, a set of procedures, a paradigm, or anything rigid. For really localised sub-tasks like typesetting or grammar checks it's probably OK to use a routine skill, but for the softer, more creative parts, I believe simple and natural conversations is the way to go.
|
||
|
|
That, and actively instructing about the inner mechanism of an LLM agent rather than truly simple and natural conversations. It's just in the blood at this point now.
|