About
This App allows you to map body movement to MIDI notes with the help of your webcam and MediaPipe’s AI. It’s intended use is as a Note/Control message controller.
I revisited my motion synth project but with a focus on generating MIDI notes instead of sound on the browser.
This project is still on early stages but it has proved to work pretty well and be really fun to use. It’s made with Astro and SolidJS.
Challenges faced during the development:
- Mediapipe’s solutions are in CJS.
- Re-implemented Mediapipe’s camera solution but with the chance to update the params.
- Astro-Solid integration is kind of broken during SSR (this means that static rendering breaks as well) when handling children, this app will probably be migrated to Solid Start in the future.
- Performance in different devices.
Current features:
- Select different MIDI controllers to send info to them.
- Locally stored state.
- Directional mappers and note range settings for each keypoint in poses.
- Custom camera solution to fix issues with @mediapipe solutions.
Planned Features:
- Export config.
- Map to different controllers/MIDI actions.
- Multiple mappers per keypoint.
- Rethinking the UI and the user flow.
- In app quantization.
- Performance optimizations.
- Demo patches/videos to integrate with other softwares.
- Modify the config via MIDI.
- i18n.
The App is made with:
- Astro .
- SolidJS .
- Mediapipe (AI) .
- Vercel (Hosting) .