ensomi

Open music research · R2 is playable now

ensomi — Play the music around you.

Choose a song you love, and R2 charts it for you here in your browser. Watch the chart run against the music, then take it into osu! and play it yourself. The song never leaves your computer.

The chart generator needs a desktop browser. Open this page on a computer to load the model, choose a song and generate a chart.

1Model

About 120 MB from sed-i/ensomi-r2 on the first visit; later visits read it from this browser's cache. Uses WebGPU for the beat tracker when the browser offers it and WebAssembly threads for the rest. With the model cached, a 3:20 song took about 6 seconds in Chrome on an Apple M5, and about 13 seconds without WebGPU.

2Song

Any format this browser can decode. The file stays on your computer.

3Conditions

A generated chart can come out easier or harder than the rating asked for; difficulty control will be refined.

Decodes the song at its own sample rate and converts it as the Python release does, so beat times match the release to within a few thousandths of a millisecond instead of up to about 1.3 ms. Adds about half a second per song, more for high-sample-rate files (about 1.3 s at 96 kHz).

Sections optional

Ask for a style or a star target in one span of the song. Use mm:ss. Each section lasts 8–32 seconds, stays inside the song and does not overlap another; one may start where another ends.

  1. Prepare
  2. Timing
  3. Rows
  4. Arrangement
  5. Export

Load the model and choose a song to start.

Why ensomi

Most of the time, music is something we listen to. A rhythm game lets you step inside a song: your hands follow the beat, and listening turns into playing. But a song becomes playable only after someone charts it by hand, note by note, and most of the music in your life never gets a chart.

We are changing that. ensomi learns from charts that mappers have made and writes new ones from the audio, so that the music you love, and the music already around you, can become something you play.

And that is only the start. What becomes possible when a model truly understands music, and everyone can build on it? New ways to play, to create, to work with sound. We are exploring that in the open, with the community that made all of this possible.

Our mission

Turn music understanding and rhythm-game creation into open tools for players, mappers and developers, until listening to music, playing it and creating with it become one activity. These are the four frontiers we are exploring.

Ambient play

The music around you becomes something you play.

A song is already playing nearby. ensomi recognises it, finds where it is, and starts a chart, generated in real time, that follows the music without interrupting it.

Open music understanding

Your ear leads; the timing follows.

A mapper drops in a song and ensomi proposes its beats and tempo, even where the meter shifts or the pulse is hard to read. Where their ear disagrees, they mark what they hear, and the timing is worked out again around that mark. Open by design, so DAWs, games and tools nobody has imagined yet can build on the same understanding of music.

Training

Practise on music you love.

A player picks songs they love and a movement they want to get better at, and the chart is shaped around that movement.

New forms of 4K multiplayer

Your best play lands in your opponent's chart.

Like clearing lines in versus Tetris, a clean passage or a long combo sends an attack: a burst of notes or a harder pattern, dropped into what your opponent is about to play. In co-op you coordinate instead, and what one of you plays shapes the other's chart.

R2 is our first step: a whole song, charted while you wait, and you can already ask for a style in one part of it. From here, we are going after charts that play along in real time, change as you play and respond to what you hear.

R2: from a song to a chart you can play

R2 is our first publicly playable release. It works from the audio alone, and between your song and your chart, three things happen.

  1. It finds the beat.R2 listens for the beats, then lines each tempo section up with where the song's sounds begin.
  2. It chooses where notes fall.Beat by beat, it decides where notes start, from what it hears.
  3. It arranges the chart.It takes a character, such as how many long notes and chords, from human charts whose notes fall in a similar way, then lays out lanes, taps and long notes. This step does not hear the audio yet.

What comes out is a complete osu!mania 4K chart, made in your browser, that you can watch against the music and open in osu!.

R2 and the rows model were trained on osu!'s ranked and loved 4K charts from 2 to 6 stars. We do not distribute those charts or their audio. See data provenance for how they were used and for corrections.

Model on Hugging Face Read the report Code

Next: N1, R3

Two models, one direction. N1 learns to understand music. R3 turns that understanding into charts that feel written for the song.

N1 · open music understanding

Before the first note, a mapper has to find every beat.

It is slow, careful work, and today's tools fall apart on tempo changes, odd meters and songs with a pulse that is hard to read. N1 is the model we are building to truly hear music, starting with timing.

  1. Start from a tempo map

    Drop in a song and get an .osu with its red lines already in place.

  2. Fix it by ear

    Mark what you hear and N1 reworks the timing around it. The rest stays put.

  3. Build on it

    MIT-licensed with its code, so DAWs, games and other tools can use it too.

R3 · the next chart system

R2 arranges a chart without hearing the song.

So long notes drift from the music, and a chart can come out easier or harder than you asked. R3 is the next chart system, built to fix what R2 showed us.

  1. Notes that follow the music

    N1's timing puts notes on what you hear, and the arrangement finally listens, so long notes follow the song.

  2. Difficulty you can trust

    Ask for a star rating and get that star rating.

  3. In real time

    Charts generated as the music plays, so the music around you can become playable.

Follow the research, and help shape it, on GitHub.

Run it yourself

The generator above runs R2 in your browser with @ensomi/runtime: the networks in ONNX Runtime Web, everything else in ensomi's Rust runtime compiled to WebAssembly. The browser decodes the song and draws its own random numbers, so a seed here does not give the same chart as the same seed on the command line.

To run the release from the command line (we checked it on macOS with Apple silicon; Linux and CUDA were not run):

git clone https://github.com/ensomi-labs/ensomi-model.git
cd ensomi-model
git checkout --detach r2-1.0.1
uv sync --python 3.10 --extra mps
hf download sed-i/ensomi-r2 --revision r2-1.0.1 --local-dir artifacts/hf-r2

uv run --python 3.10 --extra mps python -m ensomi_model.system.make \
  --model-dir artifacts/hf-r2 --audio /path/to/song.mp3 --star 3.5 \
  --title "Song title" --artist "Artist" --seed 0 --out artifacts/my-chart.osz

The first run also downloads BeatThis's checkpoint (81 MB) from its authors. Decoding audio other than WAV needs ffmpeg. --help lists every request. Full instructions are on the Hugging Face card.

Join the community!

Map, play, make music or write code? Come and explore with us. Tell us what you hear and what you want to build.

1130614129