SongSmith is a senior project created by three UF students.
It is a Django web app that takes in lyrics and generates a corresponding MIDI melody based on the lyrics. It uses a TransGAN neural network written in PyTorch that was trained on the Lakh MIDI Dataset.
We also wrote up a related blog post/article going over how our model works and the machine learning concepts underlying it. The article is meant to be accessible to any intelligent lay reader (no ML or programming experience required). Link here.
There are several steps to run the project.
In order to create MIDI files, you must install some peices of software:
- Musescore3 (Musescore4 breaks this) - https://musescore.org/en/3.6.2
- Fluidsynth - https://www.fluidsynth.org/download/
- Any MIDI instrument (we used https://www.nerds.de/en/ipmidi.html, the trial version didn't seem to cause issues).
On top of this, you will need several python libraries:
- pretty_midi
- music21
- soundfile
- torch
- torchtext
- torchdata
- numpy
- pyphen
- IPython
- flask
- Pyfluidsynth (make sure you don't have fluidsynth installed before doing this, having both breaks the whole thing due to an empty project on pip and python not knowing how to fix itself)
Last, you will have to go into the generate_song.py located in the ui folder on line 20, there is a line: music21.environment.set("musescoreDirectPNGPath", ".."). You need to change the second argument to be the absolute directory to your musescore3 executable.
After this is done, you can run main.py in the ui folder which will launch the webapp, and everything should work perfect.
If you have any issues, pyfluidsynth tends to be the issue. https://github.com/nwhitehead/pyfluidsynth is where you can find the readme to verify install was correct.
