Audio Notebooks

A collection of Jupyter Notebooks related to audio processing.

The notebooks act like interactive utility scripts for converting between different representations, usually stored in data/project/ where project is the dataset you're working with. Generally, if you change data_root near the top of the notebook and run the reset of the notebook, it will do something useful.

Setup

librosa currently needs some extra help on OS X, make sure to follow the instructions here first.

$ brew install ffmpeg # for loading and saving audio
$ git clone https://github.com/kylemcdonald/AudioNotebooks.git
$ cd AudioNotebooks.git
$ pip install -r requirements.txt
$ jupyter notebook

Terminology

Here are some words used in the names of the notebooks, and what they mean:

• Samples refers to one-shot sounds, usually less than 1-2 seconds long. These can be loaded from a directory, like data/project/samples/ or from a precomputed numpy matrix like data/project/samples.npy. When they are stored in a .npy file, all the samples are necessarily concatenated or expanded to be the same length.
• Multisamples refers to audio that needs to be segmented into samples.
• Fingerprints refer to small images, usually 32x32 pixels, representing a small chunk of time like 250ms or 500ms. These are either calculated with CQT, STFT, or another frequency domain analysis technique. They are useful for running t-SNE or training neural nets.
• Spritesheets are single files with multiple sounds, either visually as fingerprints or sonically as a sequence of sounds, organized carefully so they can be chopped up again later.

Some formats in use:

• .npy are numpy matrices. Numpy can load and save these very quickly, even for large datasets.
• .tsv are tab separated files referring to one sample per line, usally with normalized numbers in each column. These are good for loading into openFrameworks apps, or into the browser.
• .txt are like .tsv but only have one item per line, usually a single string. Also good for loading into openFrameworks apps, or into the browser.
• .pkl are Pickle files, which is the native Python serialization format, and is used for saving and loading datastructures that have lists of objects with lots of different kinds of values (not just numbers or strings).
• .h5 is the way the Keras saves the weights for a neural net.
• .json is good for taking what would usually go into a Pickle file, and saving it in a format that can be loaded onto the web. It's also one of the formats used by Keras, part of a saved model.

Example Workflows

Audio spritesheet

1. Collect Samples
2. Samples to Audio Spritesheet

t-SNE embedding for samples

1. Collect Samples
2. Samples to Fingerprints
3. Fingerprints to t-SNE (with mode = "fingerprints")

The standard workflow is to create a t-SNE embedding from fingerprints, but it's also possible to create an embedding after learning a classifier:

1. Collect Samples
2. Samples to Fingerprints
3. Collect Metadata
4. Metadata to Labels
5. Fingerprints and Labels to Classifier
6. Fingerprints to t-SNE (with mode = "combined")

t-SNE embedding for phonemes

Right this only really works with extracting phonemes from transcribed speech, using Gentle.

1. Gentle to Samples (with save_wav = True)
2. Samples to Fingerprints
3. Fingerprints to t-SNE

It's also possible to use Sphinx for speech that does not have transcriptions, but it can be very significantly slower:

1. Sphinx to Samples
2. Collect Samples
3. Samples to Fingerprints
4. Fingerprints to t-SNE

t-SNE grid fingerprints spritesheet

By virtue of creating a rectangular grid, you may lose some points. This technique will only work on 10-20k points maximum

1. Collect Samples
2. Samples to Fingerprints
3. Fingerprints to t-SNE
4. Run the example-data app from ofxAssignment or use CloudToGrid to convert a 2d t-SNE embedding to a grid embedding.
5. Fingerprints to Spritesheet

If you only want a spritesheet without any sorting, skip step 4 and only run step 5 partially.

Predict tags given tagged audio

1. Collect Samples
2. Samples to Fingerprints
3. Collect Metadata
4. Metadata to Labels
5. Fingerprints and Labels to Classifier