NoiSQL โ Generating Music With SQL Queries
NoiSQL (named after Robert Noyce) shows how to play sound and music with declarative SQL queries.
It contains oscillators for basic waves, envelopes, sequencers, arpeggiators, effects (distortion, delay), noise generators, AM and FM, and LFO, ... Sometimes it can generate something nice, but usually not.
Table of Contents
Quick Start - Linux
Clone the Repository.
Download clickhouse-local:
curl https://clickhouse.com/ | sh
Install (Linux):
sudo ./clickhouse install
Check the installation:
clickhouse-local --version
Quick Start - MacOS
Before beginning, please ensure that you have homebrew installed.
Clone the Repository.
Change to the Repository folder (i.e. cd NoiSQL)
Install ClickHouse (macOS):
mkdir -p bin
curl https://clickhouse.com/ | sh
mv clickhouse bin/clickhouse-local
export PATH="$(pwd)/bin:$PATH"
NOTE: If you want this to live past a terminal restart add to your profile. That may look something like the below or .bash_profile or .zshrc depending on your terminal of choice.
echo 'export PATH="'$(pwd)'/bin:$PATH"' >> .profile
In order to playback audio from the terminal (via STDIN) we use an open source project (with a convenient brew recipe) called 'sox' (note that Xcode developer tools is a dependency, and if you don't have it installed, you will need to install it first via xcode-select --install)
brew install sox
Bleeps and Bloops - a demonstration
Now, ClickHouse Local is setup and it is time to make our first noises.
Demo (Linux):
./music2.sql.sh | aplay -f cd
Demo (macOS):
./music2.sql.sh | play -t raw -b 16 -e signed -c 2 -r 44100 -v .75 -
Live editing (Linux):
sudo apt install inotifytools
./play.sh music0.sql
You can edit the music0.sql file, and the changes will automatically apply while playing. On macOS this is not possible due to the lack of inotifytools BUT the play script can be used to play any of the samples music*.sql files provided
Examples
If, you are unable to use it yourself. You can still hear the output as .mp4 here.
music3.mp4
music2.mp4
music1.mp4
music0.mp4
How It Works
An SQL query selects from the system.numbers table, containing a sequence of natural numbers, and transforms this sequence into a PCM audio signal in the CD format. This output signal is piped to the aplay -f cd tool (on Linux) to play it.
The CD (Compact Disc Audio) format is a 44100 Hz 16-bit stereo PCM signal.
- 44100 Hz is the "sampling frequency", meaning that the signal has a value 44 100 times every second;
- 16 bit is the precision of every value, and the value is represented by
Int16(signed, two's complement, little endian) integer; - stereo means that we have two values at every sample - for the left channel and for the right channel;
The signal is represented in binary, corresponding to the ClickHouse's RowBinary format. Therefore, the bit rate of the signal is 16 * 2 * 44100 = 1411 kBit.
Although we could also use the classic 8-bit mono 9 kHz format, the CD format gives more possibilities.
To get the idea, run this:
clickhouse-local --query "
SELECT
(number % 44100)::Int16 AS left,
((number + 22050) % 44100)::Int16 AS right
FROM system.numbers
FORMAT RowBinary" | aplay -f cd
It will give you an uninteresting clicky sound.
clickhouse-local --query "
WITH number * 2 * pi() / 44100 AS time
SELECT
(sin(time * 200) * 0x7FFF)::Int16 AS left,
(sin(time * 400) * 0x7FFF)::Int16 AS right
FROM system.numbers
FORMAT RowBinary" | aplay -f cd
It will give you uninteresting waves.
Making Something Interesting
Here is a query from music0.sql, that generates something at least listenable. Let's walk through this SQL query.
The WITH clause in the SQL query allows defining expressions for further use.
We can define both constants and functions (in form of lambda expressions).
We use the allow_experimental_analyzer setting to make this query possible.
Input
Let's define the time column to be a floating point value representing the number of seconds.
WITH
44100 AS sample_frequency
, number AS tick
, tick / sample_frequency AS time
Output Control
Let us work with the signal in the floating point format, with values in the [-1, 1] range.
Here are the functions to convert it to the output PCM CD signal.
A knob for the volume:
, 1 AS master_volume
If the signal exceeds the boundaries, it will be clipped:
, level -> least(1.0, greatest(-1.0, level)) AS clamp
Floating point values are converted to Int16 for the output:
, level -> (clamp(level) * 0x7FFF * master_volume)::Int16 AS output
If we don't care about stereo - we can simply output a tuple (pair) of identical values:
, x -> (x, x) AS mono
Basic waves
We define oscillators as functions of time, with the period adjusted to be one second.
You can modify the frequency simply by multiplying the time argument.
For example, sine_wave(time * 400) - a sine wave of 400 Hz frequency.
The sine wave gives the cleanest and most boring sound.
, time -> sin(time * 2 * pi()) AS sine_wave
The square wave gives a very harsh sound; it can be imagined as a maximally-distorted sine wave.
, time -> time::UInt64 % 2 * 2 - 1 AS square_wave
Sawtooth wave
, time -> (time - floor(time)) * 2 - 1 AS sawtooth_wave
Triangle wave
, time -> abs(sawtooth_wave(time)) * 2 - 1 AS triangle_wave
Helpers and LFO
LFO means "Low-Frequency Oscillation," and it is used to control a parameter of one signal with another low-frequency signal. It can have multiple interesting effects.
For example, AM - amplitude modulation is modulating the volume of one signal with another signal, and it will give a trembling effect, making your sine waves sound more natural.
Another example, FM - frequency modulation, is making the frequency of one signal change with another frequency. See the explanation.
We take the wave and map it to the [from, to] interval:
, (from, to, wave, time) -> from + ((wave(time) + 1) / 2) * (to - from) AS lfo
Here is a discrete version of an LFO. It allows changing the signal as a step function:
, (from, to, steps, time) -> from + floor((time - floor(time)) * steps) / steps * (to - from) AS step_lfo
For some unknown reason, the frequencies of musical notes ("musical scale") are exponentially distributed, so sometimes we have to apply computations in the logarithmic coordinates to make a more pleasant sound:
, (from, to, steps, time) -> exp(step_lfo(log(from), log(to), steps, time)) AS exp_step_lfo
Noise
Generating noise is easy. We just need random numbers.
But we want the noise to be deterministic (determined by the time) for further processing.
That's why we use cityHash64 instead of a random and erf instead of randNormal.
All the following are variants of white noise. Although we can also generate brown noise with the help of runningAccumulate.
, time -> cityHash64(time) / 0xFFFFFFFFFFFFFFFF AS uniform_noise
, time -> erf(uniform_noise(time)) AS white_noise
, time -> cityHash64(time) % 2 ? 1 : -1 AS bernoulli_noise
Distortion
Distortion alters the signal in various ways to make it sound less boring.
The harshest distortion - clipping - amplifies the signal, then clips what's above the range. It adds higher harmonics and makes sound more metallic, and makes sine waves more square.
, (x, amount) -> clamp(x * amount) AS clipping
For a milder version, it makes sense to apply the pow function, such as square root to the [-1, 1] signal:
, (x, amount) -> clamp(x > 0 ? pow(x, amount) : -pow(-x, amount)) AS power_distortion
We can reduce the number of bits in the values of the signal, making it more coarse. It adds some sort of noise, making it sound worn out.
, (x, amount) -> round(x * exp2(amount)) / exp2(amount) AS bitcrush
Lowering the sample rate makes the signal dirty. Try to apply it for white noise.
, (time, sample_frequency) -> round(time * sample_frequency) / sample_frequency AS desample
Something like compressing the periodic components in time and adding empty space - no idea why I need it:
, (time, wave, amount) -> (time - floor(time) < (1 - amount)) ? wave(time * (1 - amount)) : 0 AS thin
Skewing the waves in time making sine ways more similar to sawtooth waves:
, (time, wave, amount) -> wave(floor(time) + pow(time - floor(time), amount)) AS skew
Envelopes
The envelope is a way to make the signal sound like a note of a keyboard musical instrument, such as a piano.
It modulates the volume of the signal by:
- attack - time for the sound to appear;
- hold - time for the sound to play at maximum volume;
- release - time for the sound to decay to zero;
This is a simplification of what typical envelopes are, but it's good enough.
, (time, offset, attack, hold, release) ->
time < offset ? 0
: (time < offset + attack ? ((time - offset) / attack)
: (time < offset + attack + hold ? 1
: (time < offset + attack + hold + release ? (offset + attack + hold + release - time) / release
: 0))) AS envelope
We can make the musical note sound periodically to define a rhythm. For convenience, we define "bpm" as "beats per minute" and make it sound once in every beat.
, (bpm, time, offset, attack, hold, release) ->
envelope(
time * (bpm / 60) - floor(time * (bpm / 60)),
offset,
attack,
hold,
release) AS running_envelope
Sequencers
To create a melody, we need a sequence of notes. A sequencer generates it.
In the first example, we simply take it from an array and make it repeat indefinitely:
, (sequence, time) -> sequence[1 + time::UInt64 % length(sequence)] AS sequencer
But the obvious way to generate a melody is to take a Sierpinski triangle.
Sierpinski triangles sound delicious:
, time -> bitAnd(time::UInt8, time::UInt8 * 8) AS sierpinski
Another way is to map the number of bits in a number to a musical note:
, time -> bitCount(time::UInt64) AS bit_count
Calculating the number of trailing zero bits gives us a nice arpeggiator:
, time -> log2(time::UInt64 > 0 ? bitXor(time::UInt64, time::UInt64 - 1) : 1) AS trailing_zero_bits
Delay
If you ever wanted to generate dub music, you cannot go without a delay effect:
, (time, wave, delay, decay, count) -> arraySum(n -> wave(time - delay * n) * pow(decay, n), range(count)) AS delay
Components
Here is a kick:
sine_wave(time * 50) * running_envelope(120, time, 0, 0.01, 0.01, 0.025) AS kick,
Here is a snare:
white_noise(time) * running_envelope(120, time, 0.5, 0.01, 0.01, 0.05) AS snare,
Let's also define five melodies.
What is the idea? Let's take a Sierpinski triangle, put it into a sine wave, add FM to make it even fancier, and apply a few LFOs over the place.
sine_wave(
time * (100 * exp2(trailing_zero_bits(time * 8) % 12 / 6))
+ sine_wave(time * 3 + 1/4)) * 0.25 * lfo(0.5, 1, sine_wave, time * 11)
* running_envelope(480, time, 0, 0.01, 0.1, 0.5)
* lfo(0, 1, sine_wave, time / 12
) AS melody1,
sine_wave(time * (200 * exp2(bit_count(time * 8) % 12 / 6))
+ sine_wave(time * 3 + 1/4)) * 0.25 * lfo(0.5, 1, sine_wave, time * 11)
* running_envelope(480, time, 0, 0.11, 0.1, 0.5) * lfo(0, 1, sine_wave, time / 24
) AS melody2,
sine_wave(time * (400 * exp2(sierpinski(time * 8) % 12 / 6))
+ sine_wave(time * 3 + 1/4)) * 0.25 * lfo(0.5, 1, sine_wave, time * 11)
* running_envelope(480, time, 0, 0.21, 0.1, 0.5) * lfo(0, 1, sine_wave, time / 32
) AS melody3,
sine_wave(time * (800 / exp2(trailing_zero_bits(time * 8) % 12 / 6))
+ sine_wave(time * 3 + 1/4)) * 0.25 * lfo(0.5, 1, sine_wave, time * 11)
* running_envelope(480, time, 0, 0.31, 0.1, 0.5) * lfo(0, 1, sine_wave, time / 16
) AS melody4
Combine It
So, what will happen if we mix together some garbage and listen to it?
SELECT
mono(output(
1.0 * melody1
+ 0.5 * melody2
+ 0.25 * melody3
+ 1.0 * melody4
+ 1 * kick
+ 0.025 * snare))
FROM table;
Additional Commands
Generate five minutes of audio and write to a .pcm file:
clickhouse-local --format RowBinary --query "SELECT * FROM system.numbers LIMIT 44100 * 5 * 60" \
| clickhouse-local --allow_experimental_analyzer 1 --format RowBinary --structure "number UInt64" --queries-file music0.sql \
> music0.pcm
Convert pcm to wav:
ffmpeg -f s16le -ar 44.1k -ac 2 -i music0.pcm music0.wav
Convert pcm to mp4:
ffmpeg -f s16le -ar 44.1k -ac 2 -i music0.pcm music0.mp4
Limitations
I haven't, yet, found a good way to implement filters (low-pass, high-pass, band-pass, etc.). It does not have Fourier transform, and we cannot operate on the frequency domain. However, the moving average can suffice as a simple filter.
Further Directions
You can use ClickHouse as a sampler - storing the prepared musical samples in the table and arranging them with SELECT queries. For example, the Mod Archive can help.
You can use ClickHouse as a vocoder. Just provide the microphone input signal instead of the system.numbers as a table to clickhouse-local.
You can make the queries parameterized, replacing all the hundreds of constants with parameters. Then attach a device with hundreds of knobs and faders to your PC and provide their values of them as a streaming input table. Then you can control your sound like a pro.
Real-time video generation can be added as well.
Motivation
This is a fun project and neither a good nor convenient solution to a problem. Better solutions exist.
There is not much sense in this project, although it can facilitate testing ClickHouse.
You could argue that modern AI, for example, Riffusion, can do a better job. The counterargument is - if you enjoy what you are doing, it's better not to care if someone does it better but with less pleasure.
Contributing
If you want to share new interesting examples, please make a pull request, adding them directly to this repository!
Reading Corner
- Sound by Bartosz Ciechanowski;
- Bytebeat;
- Color of Noise;
- Glitchmachine;
- Audio Synthesis;
- 44,100 Hz;