Better synthio

[jepler]

• Allow >2 notes at a time (increased to 12 on mimxrt10xx including metro m7, left unchanged elsewhere)
• allow specifying a single-cycle waveform instead of the built in square wave

a couple of videos:

• https://octodon.social/@stylus/110142773566079748
• https://youtube.com/shorts/Q_Cbmx3uvco

also closes #7837

[jepler]

For some reason this doesn't play via the mixer (weird), haven't determined why.

[tannewt]

This looks good to me! I'm excited to see you improve synth stuff. Let me know when it is ready for approval.

[jepler]

I think this is ready, it's nice increment in the functionality. I showcased it in a quick youtube video: https://youtube.com/shorts/Q_Cbmx3uvco

[jepler]

import time
import array
import math
import audiopwmio
import audiobusio
import audiomixer
import board
import synthio
import ulab.numpy as np
import board
from wiichuck.nunchuk import Nunchuk


#  lists of modal intervals (relative to root). Customize these if you want other scales/keys
major = (0, 2, 4, 5, 7, 9, 11)
minor = (0, 2, 3, 5, 7, 8, 10)
dorian = (0, 2, 3, 5, 7, 9, 10)
phrygian = (0, 1, 3, 5, 7, 8, 10)
lydian = (0, 2, 4, 6, 7, 9, 11)
mixolydian = (0, 2, 4, 5, 7, 9, 10)
locrian = (0, 1, 3, 5, 6, 8, 10)

scale = [i for i in major] + [i + 12 for i in major] + [i + 24 for i in major] + [i+36 for i in major] + [i + 48 for i in major]

# Centering factors for the stick
CX = CY = 128
# Stick distance from center to record any movement
MAG = 70
# Amount of X/Y movement to register U/D/L/R
YT = XT = 30

controller = Nunchuk(board.STEMMA_I2C())
def get_controller_position(j):
    x = j.x - CX
    y = j.y - CY
    if abs(x) + abs(y) < MAG:
        result = "C"
    elif x > XT:
        if y > YT:
            result = "UR"
        elif y < -YT:
            result = "DR"
        else:
            result = "R"
    elif x < -XT:
        if y > YT:
            result = "UL"
        elif y < -YT:
            result = "DL"
        else:
            result = "L"
    elif y > YT:
        result = "U"
    elif y < -YT:
        result = "D"
    else:
        result = "C"
    return result

def chord(root_note, scale, mod1, mod2):
    print(root_note)
    offsets = [0, 2, 4]
    if mod1: offsets.append(6) # add7
    if mod2: offsets.append(8) # add9
    result = [KEY + scale[root_note+i] for i in offsets]
    result.append(result[0] - 12) # root, octave down
    return result
dac = audiobusio.I2SOut(board.D10, board.D9, board.D12)

SAMPLE_SIZE=128
assert SAMPLE_SIZE % 4 == 0
VOLUME=4000
sine = np.array(np.sin(np.linspace(0, 2*np.pi, SAMPLE_SIZE, endpoint=False)) * VOLUME, dtype=np.int16)

square = np.concatenate((
    np.ones(SAMPLE_SIZE//2, dtype=np.int16) * VOLUME,
    np.ones(SAMPLE_SIZE//2, dtype=np.int16) * -VOLUME
))

sawtooth = np.linspace(VOLUME, -VOLUME, num=SAMPLE_SIZE, dtype=np.int16)

triangle = np.concatenate((
    np.linspace(0, VOLUME, num=SAMPLE_SIZE//4, dtype=np.int16, endpoint=False),
    np.linspace(VOLUME, -VOLUME, num=SAMPLE_SIZE//2, dtype=np.int16, endpoint=False),
    np.linspace(-VOLUME, 0, num=SAMPLE_SIZE//4, dtype=np.int16, endpoint=False),
))

waveform = np.zeros(SAMPLE_SIZE, dtype=np.int16)

assert len(sine) == len(square) == len(sawtooth) == len(triangle)

def mix(f1, f2, f3):
    result = sine * f1 + square * f2
    mag = np.sum(result ** 2) ** .5
    return result * (f3 / mag)

KEY = 60  # C
roots = {
    'U': False,  # Up turns chord off
    'C': None,   # Center sustains
    'UR': 5,
    'R': 1,
    'DR': 4,
    'D': 7,      # Down is the tonic chord
    'DL': 10,     # 
    'L': 6,       # 
    'UL': 2,      # 
}
print([v%7 for v in roots.values() if v])
synth = synthio.Synthesizer(sample_rate=48000, waveform=waveform)
dac.play(synth)

last_signature = None
pressed=set()
while True:
    v = controller.values
    #print(v)
    ax = min(1, max(-1, (v.acceleration.x - 500) / 200))
    ay = min(1, max(-1, (v.acceleration.y - 500) / 200))
    
    sine_weight = (ax + 1) / 2
    square_weight = 1 - sine_weight
    volume = ay * 8000 + 8000
    
    #print((ax, ay, f1, f2, f3, f4))
    waveform[:] = mix(sine_weight, square_weight, volume)
    #print(waveform)
    c = get_controller_position(v.joystick)
    signature = c, v.buttons.C, v.buttons.Z
    if signature == last_signature:
        continue
    print(signature)
    last_signature = signature
    root = roots.get(c, None)
    if root is False:
        synth.release_all()
    elif root is not None:
        cc = set(chord(roots[c], scale, v.buttons.C, v.buttons.Z))
        pressed = set(synth.pressed)
        to_release = pressed - cc
        to_press = cc - pressed
        print(cc, to_release, to_press)
        synth.release_then_press(release=to_release, press=to_press)

somewhat messy test code using wii nunchuk style controller on the stemma qt port

[todbot]

This is so great @jepler! With the waveform variable being able to be changed after passing it off to synthio.Synthesizer, does this mean we could have similar functionality to audiocore.RawSample? This could enable real-time changing of a playing sample, which would be incredibly useful.

[jepler]

If I understand what you're asking -- yes. the waveform can be updated at any time. This is used in my demo code to morph between square & sine wave, and to implement volume control:

waveform[:] = mix(sine_weight, square_weight, volume)

waveform has to be updated in-place by the [:] slice assignment.

[todbot]

Actually I guess what I'm really asking is: what's the difference between synthio.Synthesizer and audiocore.RawSample (wrt to their input buffers)?

A long while ago I did some explorations in allowing one to replace the buffer given to RawSample while it was running. I'm very excited to see similar (and better) capability in synthio.Synthesizer.

Another variation of the question: does the waveform buffer to synthio.Synthesizer need to be a small single-cycle waveform, or can it be a longer arbitrary waveform?

[jepler]

One thing synthesizer does is stretch or compress the waveform according to the note pitch. While rawsample just always plays the same. It also plays multiple notes, without requiring a Mixer involved (though if you want multiple different waveforms simultaneously you would presently need multiple synthesizer objects), including starting and stopping multiple notes at the same instant.

Right now the length of the waveform is limited to 1024 samples and can only be mono. The limit has to do with enabling use of a fixed point DDS in a 32 bit register.

Since "general" audio sources like WaveFile wouldn't work with synthesizer, and because dealing with just one format simplified code, I made waveform be a buffer of type 'h' rather than accepting a RawSample.

I hope this answers your question but if not please feel free to re-ask because I'm not sure I caught the essence of it.

[jepler]

I also should add I'm not a strong audio person and I want to be open to having primitives in circuitpython to better serve those who are; so especially if you think something here is going in a wrong direction or even a "not quite right" direction the feedback absolutely would be welcome.

[todbot]

I also should add I'm not a strong audio person and I want to be open to having primitives in circuitpython to better serve those who are; so especially if you think something here is going in a wrong direction or even a "not quite right" direction the feedback absolutely would be welcome.

This is all brilliant and I think it's awesome. It's something I am very excited to try. (and hope we can up MAX_CHANNELS to more than 2 for rp2040)

I had assumed any sort of actual synthesis wouldn't' be accepted. So I had been focused on doing simple buffer replacements on RawSample(*) or adding .pause()/.resume()/.loop(start=pos1,end=pos2) to WaveFile

(* which now Synthesizer does, while also doing proper phase accumulation for pitch shifting)

[jepler]

Do you want a build with more channels on rp2040?I can try increasing it in this PR but I wouldn't be testing it myself

[todbot]

I'm getting my environment setup to pull this PR and try exactly that

[jepler]

I appreciate that! please let us know your findings.

[todbot]

Hmm, currently does not work on RP2040 at all as far as I can tell. I made a simpler version of your test code above that has no user input. The I2S DAC I'm using is verified working (using RawSample example code), but synthio doesn't appear to output anything. Am I doing something obviously dumb?

import time, math
import board, audiobusio
import synthio
import ulab.numpy as np
import random

# pimoroni pico dv board
lck_pin, bck_pin, dat_pin  = board.GP28, board.GP27, board.GP26

audio = audiobusio.I2SOut(bit_clock=bck_pin, word_select=lck_pin, data=dat_pin)

SAMPLE_SIZE=128
VOLUME=4000

sine = np.array(np.sin(np.linspace(0, 2*np.pi, SAMPLE_SIZE, endpoint=False)) * VOLUME, dtype=np.int16)
sawtooth = np.linspace(VOLUME, -VOLUME, num=SAMPLE_SIZE, dtype=np.int16)
waveform = np.zeros(SAMPLE_SIZE, dtype=np.int16)  # intially all zeros (silence)
waveforms = (sine, sawtooth)

synth = synthio.Synthesizer(sample_rate=24000, waveform=waveform)
audio.play(synth)

while True:
    time.sleep(0.3)
    waveform[:] = random.choice(waveforms) # pick a waveform randomly
    time.sleep(0.3)
    notes = (random.randint(30,60), random.randint(30,60) ) # pick notes randomly
    print("playing notes:", notes)
    synth.release_all_then_press( notes )

[jepler]

I'll investigate; it could be related to why synthio + audiomixer seemed not to work, and which I glossed over assuming my test code was broken.

[jepler]

I think that there's an original problem with synthio. I ran the following in 8.0.5 on a pico w and i get an incorrect repeated block of sound instead of the whole melody:

import board
import synthio
import audiobusio
lck_pin, bck_pin, dat_pin  = board.GP28, board.GP27, board.GP26
audio = audiobusio.I2SOut(bit_clock=bck_pin, word_select=lck_pin, data=dat_pin)
melody = synthio.MidiTrack(b"\0\x90H\0*\x80H\0\6\x90J\0*\x80J\0\6\x90L\0*\x80L\0\6\x90J\0" +
    b"*\x80J\0\6\x90H\0*\x80H\0\6\x90J\0*\x80J\0\6\x90L\0T\x80L\0" +
    b"\x0c\x90H\0T\x80H\0\x0c\x90H\0T\x80H\0", tempo=640)
audio.play(melody)

In the case of the synthio.Synthesizer code, it's forever playing the initial, no-note frame of data.

[jepler]

I want to see this fixed but I'm also going on vacation starting Friday, so I'll file a separate issue for this and declare it out-of-scope for this PR. I'm sorry the rp2040 part didn't work out right now.

[todbot]

No worries, not time-critical. Very excited this is on deck at all

[jepler]

@todbot maybe see if the synthio+audiomixer workaround can get you going with this on rp2040? (see #4783)

import audiomixer
synth = synthio.Synthesizer(sample_rate=24000, waveform=waveform)
mixer = audiomixer.Mixer(sample_rate=synth.sample_rate, channel_count=1)
audio.play(mixer)
mixer.voice[0].play(synth)

[tannewt]

One copy-pasta to fix. Looks good otherwise! Thanks for improving this! Video looks awesome.

[jepler]

@tannewt I incorporated your fixes but still have a CI failure to deal with. hopefully the next round will be green.

[tannewt]

Thank you!

[todbot]

Yes! These fixes work on RP2040! Sounds pretty great. Thank you @jepler!

(I'm now looking into how to add a simple amplitude envelope to Synthesizer, but will emulate it now with waveform hacking)

[chachagsedaro]

This looks incredible, thank you!