Mirelo AI and Kyutai have released MuScriptor, an open model that converts a finished music recording directly into per-instrument MIDI.
That distinction matters. Traditional transcription tools often perform best on a single instrument or require a source-separation system to create audio stems before note transcription begins. MuScriptor is designed for the full mix: vocals, drums, bass, keys, guitar, and other parts can be identified and returned as separate symbolic tracks.
The result is not clean studio stems. It is an editable representation of what notes were played, when they started and stopped, and which instrument group produced them.
From waveform to symbolic music
Audio and MIDI represent different things.
An audio recording stores the resulting pressure waveform. MIDI stores performance events such as pitch, timing, duration, and instrument. Converting a dense recording into MIDI requires the model to infer overlapping notes that may share frequencies, mask one another, or include effects and room sound.
MuScriptor follows a sequence-to-sequence transcription design:
mixed audio
-> 16 kHz mono waveform
-> mel spectrogram
-> decoder-only Transformer
-> MIDI-like note-event tokens
-> per-instrument MIDI tracksThe model processes audio in five-second chunks. Its tokenizer represents note onsets, offsets, pitch, and an instrument category. The open medium checkpoint has about 300 million parameters, while small and large variants are approximately 100 million and 1.3 billion parameters.
Why the full-mix input changes the workflow
A stem-first pipeline typically looks like this:
song -> source separation -> isolated audio stems -> transcription -> MIDIEvery stage can introduce errors. A separator may leak vocals into the guitar stem or remove quiet harmonics. The transcription model then receives an already altered signal.
MuScriptor attempts to infer instrument identity and notes jointly from the original mix. That does not guarantee higher accuracy in every case, but it removes an operational dependency and makes one-click tools more practical.
Mirelo’s application adds a review layer: users can inspect piano-roll tracks, solo or mute parts, correct notes, and export MIDI. It also derives musical context such as chords, key, and tempo.
What developers can build with it
The obvious product is automatic transcription, but the model is more valuable as a component.
Arrangement analysis: Compare instrumentation, register, density, and harmonic rhythm across a catalog.
Practice tools: Remove or mute a symbolic part, slow the tempo, and generate a guide for the remaining instruments.
Remix preparation: Extract an approximate bass line, drum pattern, or chord voicing and edit it inside a digital audio workstation.
Music search: Index songs by note sequences, chord movement, instrumentation, or recurring motifs instead of relying only on text metadata.
Dataset creation: Produce draft symbolic annotations for large audio collections, followed by confidence filtering and human correction.
Education: Let students inspect the relationship between a recording, a score-like representation, and harmonic labels.
The open release has an important license boundary
The code is released under the MIT License, but the model weights are published under CC BY-NC. That non-commercial restriction matters for startups and paid products.
A developer can experiment, research, and prototype with the open checkpoints, but a commercial deployment needs a careful license review or a separate agreement. Mirelo also says the version used in its own studio application was trained on a larger dataset than the public release.
“Open model” should therefore not be interpreted as unrestricted commercial infrastructure.
Known limitations
The model card is unusually useful because it documents representational constraints.
- Velocity and dynamics are not produced.
- Two simultaneous notes with the same pitch and instrument cannot be represented.
- Drum events contain onsets rather than sustained note durations.
- Exact offsets can be less reliable than note starts.
- Dense mixes, heavy processing, unusual timbres, and rare instruments remain difficult.
- Training data leans toward pop and Western classical music.
- The instrument taxonomy maps 128 MIDI programs into 36 broader groups.
These constraints mean the output should be treated as an editable draft, not a definitive score.
A sensible evaluation plan
Teams should not evaluate only whether the exported MIDI “sounds close.” A useful test set should measure:
- Note onset precision and recall by instrument family.
- Pitch accuracy in dense polyphonic passages.
- Instrument assignment accuracy.
- Consistency across five-second chunk boundaries.
- Chord, key, and tempo accuracy.
- Correction time for a musician compared with manual transcription.
The last metric determines product value. A slightly imperfect model can still be useful if it removes most of the repetitive work and produces predictable errors.
The signal
Generative music receives most of the attention, but transcription models may create more immediate professional utility. MuScriptor turns recorded music into structured, editable data that can feed existing tools and workflows.
The larger shift is from AI that creates a finished artifact to AI that recovers the latent structure inside one.
