• Stars
    star
    109
  • Rank 319,077 (Top 7 %)
  • Language
    TypeScript
  • License
    GNU General Publi...
  • Created over 6 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A collection of web interfaces for AI-assisted interactive music creation

Pianoto-logo

DOI

Interactive Music Inpainting for the Web

PIANOTO NOTONO NONOTO
A screenshot of the PIANOTO interface A screenshot of the NOTONO interface A screenshot of the NONOTO interface
Expressive piano performances creation Time-frequency inpainting of instrument spectrograms Inpainting of music sheet (with Ableton-Link support)

This repository holds the source code for the web-based, AI-assisted interactive music creation apps developed by Thรฉis Bazin at Sony CSL Paris. These apps are all based on inpainting, that is, the use of AI-models to transform images, sounds or sheets in a local fashion: just tell the model which zones of the media you'd like to transform (because you don't like it or because you'd like to see a variation of it!) and it will regenerate it for you. Thanks to the intuitive interactions offered by inpainting, these apps remove the need for cumbersome micro-level edits and work nicely on mobile as well!

โš ๏ธ BYOB (Bring Your Own Backend)

Note that the music-inpainting.ts interfaces each require connection to an AI model for inference! Don't panic though:

  • ๐ŸŽ‰ We provide compatible models as Docker images (GPU recommended).
  • โš—๏ธ The apps are model-agnostic: try to use them in conjunction with your own models!

PIANOTO

PIANOTO helps you be the piano maestro that you deserve to be!

It is an A.I.-powered interactive MIDI piano roll, for mobile and desktop. Swipe on the musical representation to regenerate zones, always staying coherent with the context.

20221001-pianoto_usage_demo.mp4

NOTONO

NOTONO lets you use your visual brain to create instrument sounds.

Your guitar sample is lacking some depth? Paint some subs from an organ into it! Generate new sounds from scratch or drag-and-drop sounds of your choice into NOTONO to edit them.

2021_06_28-notono-final_fantasy_piano-short_demo.mp4

NONOTO

NONOTO is an interactive interface for symbolic sheet music generation and transformation by inpainting. Along with the DeepBach model by Gaรซtan Hadjeres, NONOTO lets you generate 4-part chorale-like music in the style of J.S. Bach. Synchronize the app with Ableton Live via the included Ableton Link support and route the output via MIDI to get a smart, melodic and harmonic 4-channel sequencer. Get creative with the chorales by routing the different voices to custom synthesizers!

Here is a demo video of using NONOTO in sync with Ableton Live via Ableton-Link.

deepbach_acid_crew.mp4

These apps can be used either as standard web applications or as a packaged Electron app (which brings a few perks such as native drag-out or the support for Ableton-Link). They all share the same audio engine, based on the Web Audio API through Yotam Mann's Tone.js. They all support MIDI In and Out for interoperability with external software or hardware, via Jean-Philippe Cรดtรฉ's webmidi.js helper library over the Web MIDI API.

Usage

You can access the interfaces here!

Packaged Electron applications (Desktop)

You can download MacOS and Linux standalone applications here.

Manual installation

We recommended using the nvm installation manager for Node.js, available here. The music-inpainting.ts apps are currently developed with node v18.5.0 and npm v8.15.0.

We use the standard npm package manager.

The apps can be installed as follows:

git clone https://github.com/SonyCSLParis/music-inpainting-ts.git
cd music-inpainting-ts
npm install

Once this is done, the music-inpainting.ts Electron app (with live-reloading enabled for hacking) can be started with:

npm start

Running the models locally

We strongly recommend running these images on a machine equipped with an NVIDIA CUDA-compatible GPU.

Application Model Docker image
PIANOTO PIAv3 public.ecr.aws/csl-music-team/piano_inpainting_app:v3
NOTONO NOTONO public.ecr.aws/csl-music-team/notono:pytorch-1.11.0-cuda11.3-cudnn8-runtime
NONOTO DeepBach public.ecr.aws/csl-music-team/deepbach:latest

Sample commands (with recommended arguments and parameters)

Reminders
  • The docker run -p parameter takes a pair of ports of the form CONTAINER_PORT:LOCALHOST_PORT.
  • You might need to run the following commands as root depending on your installation of Docker.

The following commands start an inference server with access to the GPU with index 0 and listening on port 5005 (adapt to your own convenience).

Choose the appropriate model, run the command and leave the server running in the background. Then launch the music-inpainting.ts interface as described above, set the Inpainting API address input field to http://localhost:5005 and select the appropriate mode, that's it!

PIAv3

docker run -p 5000:5005 --rm --gpus 0 public.ecr.aws/csl-music-team/piano_inpainting_app:v3 serve

โš ๏ธ Note the (required) additional serve command!

NOTONO (the model)

docker run -p 8000:5005 --rm --gpus 0 public.ecr.aws/csl-music-team/notono:pytorch-1.11.0-cuda11.3-cudnn8-runtime

DeepBach

docker run -p 5000:5005 --rm --gpus 0 public.ecr.aws/csl-music-team/deepbach:latest --num_iterations_per_quarter=25 --num_iterations_per_quarter_initial_generate=10

Credits

Some icons used were made by Freepik from [www.flaticon.com].

Acknowledgements

We thank EPITECH students Martin Leblancs, Matthieu Queru and Sandro Sgro for their help on a preliminary version of PIANOTO during a development project at Sony CSL Paris.

This work was supported by the French ANRT through the CIFRE Industrial PhD grant Nr. 2019/0472.

More Repositories

1

pesto

Self-supervised learning for fast pitch estimation
Python
171
star
2

music2latent

Encode and decode audio samples to/from compressed latent representations!
Python
129
star
3

DrumGAN

Synthesis of Drum Sounds With Perceptual Timbral Conditioning Using Generative Adversarial Networks
Python
97
star
4

vqcpc-bach

Vector Quantized Contrastive Predictive Coding for Template-based Music Generation
Python
79
star
5

pesto-full

Full models and training code for PESTO
Python
48
star
6

ssl-singer-identity

Python
47
star
7

Comparing-Representations-for-Audio-Synthesis-using-GANs

This repo contains code for comparing audio representation sin the task of audio synthesis wth Generative Adversarial Networks (GAN)
Python
37
star
8

interactive-spectrogram-inpainting

Implementation of the framework described in the paper Spectrogram Inpainting for Interactive Generation of Instrument Sounds published at the 2020 Joint Conference on AI Music Creativity.
Python
37
star
9

cae-invar

Learning Complex Basis Functions for Invariant Signal Representations with the Complex Autoencoder
Python
33
star
10

cgae-invar

Convolutional gated autoencoder for learning transposition-invariant features from audio
Python
23
star
11

AutoFX

Algorithms to automatically recognize guitar effects and retrieve their parameters for timbre reproduction
Python
21
star
12

Stem-JEPA

Joint Embedding Predictive Architecture for Musical Stem Compatibility Estimation
Python
19
star
13

vqcpc-gan

VQCPC-GAN: Variable-length Adversarial Audio Synthesis using Vector-Quantized Contrastive Predictive Coding
14
star
14

cl-pcp

Common Lisp Package for Parallel Corpus Processing
Common Lisp
12
star
15

audio-representations

JEPAs for audio representation learning
Python
12
star
16

audio-metrics

Python
7
star
17

fcg-server

Software package for running an FCG server.
Common Lisp
6
star
18

BENG

Basic ENglish Grammar, a hybrid FCG grammar
Common Lisp
6
star
19

fcg-hybrids

Library for Fluid Construction Grammar that allows the implementation of FCG grammars that interface with other NLP tools.
Common Lisp
6
star
20

NeuralDrumMachine

SCSS
5
star
21

Motorized-stage

This repository explains how to turn a manual microscope stage into a motorized one.
G-code
5
star
22

fcg-dev-tools

Library of developer tools for grammar engineering in Fluid Construction Grammar.
Common Lisp
4
star
23

grammar-configurator

A library for Fluid Construction Grammar that allows to inspect and modify the configuration of a grammar through FCG's interactive web interface.
Common Lisp
3
star
24

opensheetmusicdisplay

fork of opensheetmusicdisplay with some private methods made public
TypeScript
3
star
25

DarkGAN

wefew
2
star
26

graph_search_framework

Python
2
star
27

diffariff-companion

This is the accompanying website to 'Diff-a-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models'
2
star
28

fcg-libraries

Open Source Libraries for Fluid Construction Grammar
Common Lisp
2
star
29

GlobalSummerSchool18

Jupyter Notebook
2
star
30

building-fr-narrative-from-wikidata

Demonstration Paper Submitted at the Workshop on semantic techniques for narrative-based understanding
Python
2
star
31

sgc

A garbage collector for C/C++
C
1
star
32

MF_artis

Python
1
star
33

Causal-Inpainting-Application

Causal Inpainting Application
Python
1
star
34

frame-extractor

Babel library for extracting semantic frames
Common Lisp
1
star
35

website

The public interface to Penelope
JavaScript
1
star
36

web-services

Creation of a package to request data from different APIs in Common Lisp.
Common Lisp
1
star
37

Deep_SRDA

Pytorch Implementation of Deep Streaming Regularized Discriminant Analysis for Online Continual Learning.
Python
1
star
38

PIA.No3

PIAnยฐ3 - Multi-scale perceiver (Tower of Read/Write)
Python
1
star
39

kglab

Python
1
star