Thanks for reading everything! I want to thank my advisor Adam Finkelstein, the National Science Foundation, the Faust community (especially Stéphane Letz, Yann Orlarey, @JOSmithIII).
Chain retrieval listening demo: dirt.design/PEACE-map/chain-…
and on Desktop (not mobile), check out the cross-modal embedding explorer: dirt.design/PEACE-map/
We also train both code encoders to enable a new kind of effect chain retrieval that ignores the parameters of the effects. For T5, we mask tokens as in standard language modeling. For BoxGraph, we randomly replace parameter nodes with learnable embeddings. At inference, either model can "mask" the parameters to create embeddings that (hopefully) represent chain topology alone. For 21 effects, this means 420 permutations of length-2 programs, so a gallery of 420 code embeddings. BoxGraph beats T5 and approaches the performance of a supervised AFx-Rep architecture trained on the same data that uses logits to select the likeliest chain. This gives me hope for "zero-shot chain retrieval": give out-of-domain audio trained on VSTs but retrieve Faust programs that produce similar audio. This will require more high-quality Faust effects to be included in the training data of models such as PEACE, so this is a high-priority for me. Check out the Faust Libraries for more context: github.com/grame-cncm/faustl….
We use ReverbFX (@JuliusRichter13 et al., 2025) to apply 1831 out-of-domain professional reverbs to held out dry a cappella singing. Each reverb is applied to two random but different singing clips. From clip A, we retrieve clip B among 1831 candidates based on audio embedding similarity. The coolest finding: attaching SLAP heads to a frozen AFx-Rep improves retrieval. DSP code offers useful supervision! Second, training end-to-end from scratch (including the audio encoder) with BoxGraph offers the best performance.
Our first eval is cross-modal retrieval. Both T5 and BoxGraph reach around 78% audio-to-code recall@10 from a gallery of 4096 code embeddings. However, this eval is very "in-domain" of the training data. Next, we quantify whether the audio encoder is good out-of-domain.
Our audio encoder is the architecture from AFx-Rep (@csteinmetz1 et al., ISMIR 2024). For code, we use two complementary encoders. First, T5 is a classic "vanilla" transformer that tokenizes Faust source. Second, BoxGraph is a graph neural network operating on an intermediate representation of the Faust compiler. This is the second paper of my PhD where I used a GNN with shared weights across message passing blocks. See SynAPSE (DAFx 2026): x.com/DoItRealTime/status/20… To encode large graphs, we simply increase the number of layers (reusing the weights) and cover the diameter of the graph. This allows BoxGraph to have far fewer parameters than T5.
You know CLAP and may know SLAP. Allow me to introduce SynAPSE: FM Synthesizer Audio-Parameter Shared Embeddings. It's a joint embedding of Yamaha DX7 presets and audio. It also investigates a quirky question: is it possible to encode FM topologies not seen during training? 🧵
We curate 21 effects written in Faust (expect PEACE v2 to have even better effects). We create chains of length 1-3 (expect richer mixing graphs in v2) and create a train/val/eval split of 200K/4096/4096 pairs of programs and their "effected" audio. This is why PEACE stands for Pairwise Effected Audio Code Embeddings.
The training objective is SLAP (@Juj_Guinot et al., ISMIR 2025).
To be presented at ISMIR 2026 ☮️ PEACE, a joint embedding of audio effects programs written in Faust and their corresponding "wet"/"processed" audio.
It enables audio-to-code retrieval when given a library of programs and their embeddings.
Joint embeddings have applications beyond retrieval, and the paper couldn't address all of them experimentally. Think about CLIP/CLAP and make the analogy to rich understanding of digital signal processing code (DSP) and music.
Read the paper: arxiv.org/abs/2610.03405
and codebase: github.com/DBraun/PEACE
More findings from the paper 🧵
Our method uses the non-contrastive training objective from SLAP (h/t @Juj_Guinot@howariou@elio_elioo). Like SLAP, we observe a small modality gap. We know text-to-audio relied on CLAP. What's next for SynAPSE? 🤔
We use two baseline architectures, one with a Transformer and one with a Highway Network. When all 32 algorithms are seen during training, all methods perform retrieval well on held out presets. When algorithms are held out during training, the DX7-GNN leaps ahead of baselines.
A synth is a graph running over thousands of steps, producing one or two channels of information. Audio encoders transform this into a representation for retrieval. We let a GNN encode DX7 presets using more channels, in fewer steps, going directly to the embedding space.
Yes! We use use a graph neural network (GNN) that imitates the signal flow of an FM synthesizer. Sinusoid operators are nodes, and modulation/feedback are message passing. We train on only odd-numbered algorithms and test on 8 of the even ones.
You know CLAP and may know SLAP. Allow me to introduce SynAPSE: FM Synthesizer Audio-Parameter Shared Embeddings. It's a joint embedding of Yamaha DX7 presets and audio. It also investigates a quirky question: is it possible to encode FM topologies not seen during training? 🧵