Jump here for information on usage.

Description

dse is made up of two components.

• First, a highly-compatible .SWD and .SMD file parser/writer based on Psy_Commando's ppmdu_2 parser, while also extending it with multi-levelled compat exporting and importing and thus, in many cases, binary-perfect imports and exports. This is the library section of the repo.
• Then, built upon that are 2 fully-featured converters for creating .SMD sequence files from MIDI, and for creating .SWD soundbanks from the industry-standard SF2 Soundfont format (write-up), with the parameter mapping and audio processing for sample compatibility necessary to make the files compatible with the DSE sound system all built-in. This is the binaries section of the repo.

This repo should be pretty stable.

Goal

To make custom music for EoS!

To achieve this, there are two sub-goals:

1. To map the functionality of the MIDI and SF2 file formats so that custom music can be imported

2. To be as compatible as possible and guarantee write validity

For those using this as a library, here is a bit of an elaboration on how dse ensures the latter. (You don't have to worry about it if you're using the binaries, it's taken care of for you :)

This was written from the ground up to try to be as compatible as possible with the .SWD and .SMD file formats while also remaining as writable as possible. To achieve this, first all automatically generated parameters are written in on the fly based on specifications (things like chunklen), and secondly for the other values directly affecting the sound, decisions about the preservation of values of unknown meaning had to be made. There are currently 3 possible ways to use the parser:

• .SWD => memory => .SWD
  Binary perfect recreation of the original file in most cases with slight differences in certain known byte values, not stripping any of the unknown bytes.
• .SWD => memory => .SWD.XML => memory => .SWD
  Still perfect, but first converting the .SWD file into a readable .SWD.XML format to allow for editing.
• .SWD => memory => .SWD.XML (with serde_use_common_values_for_unknowns set to true) => memory => .SWD
  The intended use case for most scenarios. In this mode, much of the unknown bytes are stripped during the serialization process to XML, leaving only the parameters we know the meaning of. The stripped values are then automatically filled in with the most common defaults during the deserialization process. All of this gives meaning to the tool, as it means that much of the unknown bytes can be stripped with no issues! P.s. the XML file generated by this method is also the simplest

This also guarantees that, as long as the writer picks the appropriate level of verboseness they would like in their editing of these files, that whatever file comes out the other end of the pipe should be absolutely correctly formatted no matter what. (Please report it if it does not! Especially if you're turning the XML files into .SWD or .SMD files. That's probably a bug.)

Important note about .SMD files: All of the above is true for .SMD files as well, but there's a caveat. .SMD files contain DSEEvents, and particularly PlayNote events that allows for variable-length numbers as its parameters. The crate assumes that the original DSE engine compresses the keydownduration value to the least amount of bytes necessary to store the value, which means for example, that 0x0000000A would just be stored as 0x0A, and that 0x00000000 wouldn't even be stored, with the corresponding NbParamBytes set to zero. This is the pattern found in most .SMD files. HOWEVER! I've come across certain .SMD files that have stored a 0x00000000 as one byte 0x00, even though that's completely unnecessary. The notable example is bgm0016.smd. In those cases there's just no way for the crate to properly generate the NbParamBytes to match the file in a binary-perfect way. While the crate could read the .SMD and just write it out again based on data in memory (because internally when the file is first read, not a single byte of the file is ever discarded), that'd just be a file copy in File Explorer! By deviating ever so slightly from the original file's formatting, it's possible to maintain writability, and to ensure that this is the only reason that things don't line up I've tested quite a bunch to see that the exported files from these problematic files also work perfectly fine in the original ROMs.

Usage

While this can be used as a Rustlang crate for those who are feeling adventurous, the intended way to use it is to use the binaries the crate generates.

Build process:

1. Install Rustlang
2. git clone https://github.com/adakite1/dse.git
3. cd dse
4. cargo build or cargo build --release for the release optimized version

The binaries will be in target/[debug or release].

Examples

swdl_tool.exe

.\swdl_tool.exe to-xml .\NDS_UNPACK\data\SOUND\BGM\*.swd -o unpack
.\swdl_tool.exe from-xml .\unpack\*.swd.xml -o .\NDS_UNPACK\data\SOUND\BGM\
.\swdl_tool.exe add-sf2 ./*.sf2 ./bgm.swd -t 20000 -S 20000 -l 3

smdl_tool.exe

.\smdl_tool.exe to-xml .\NDS_UNPACK\data\SOUND\BGM\*.smd -o unpack
.\smdl_tool.exe from-xml .\unpack\*.smd.xml -o .\NDS_UNPACK\data\SOUND\BGM\
.\smdl_tool.exe from-midi .\midi_export.mid ./bgm0043.swd --midi-prgch --generate-optimized-swdl

Notes on the MIDI conversion functionality:

• First thing to do if there's an error or the result sounds weird: Try importing the MIDI file into a DAW and reexporting it! Sometimes MIDI files have quirks in them that trip the parser up, but a DAW usually helps straighten this out. (The one I tested with was Reaper, so it should work the best)
• The MIDI file must be of type smf0 or, in the case of smf1, be composed of 16 MIDI tracks or lower! (Not counting any meta event tracks at the very start if your music composition software exports those)
• Currently, the instruments are mapped automatically to the entries in the prgi chunk of the provided SWD file, channel number <=> index in the prgi chunk
• ↳ Alternatively, a new flag was introduced to smdl_tool recently, -M/--midi-prgch. Using this flag, instead of mapping by index, the programs will be mapped using MIDI program change and bank select commands. The formula is as follows: final_mapping = midi_bank * 128 + midi_program_change
• P.s., for certain commands, you can simply put in a swd.xml file and it would still work as intended. Consult --help for the subcommand in question to find out.
• CC07 Volume, CC10 Pan Position, and CC11 Expression are the currently supported MIDI CC controllers; they're mapped to their SMDL equivalents
• If you want a track to loop and go back to an earlier position after reaching the end, you need to add a MIDI Meta event of the Marker type, and set it to "LoopStart" (this is in parity with ppmdu_2 MIDI exports too!)
• ... That's all I can think of for now, if there's something else I'll update this. Ping me about any weirdness that happens!

Notes on the SF2 conversion functionality:

• First thing to do if there's an error or the result sounds weird: Try importing the SF2 file into Polyphone and reexporting it! This is for the same reason as with the MIDI file, but it helps straighten the file out so that the parser can read it more easily.
• In general, Polyphone is your best friend.
• Common Error - TryFromIntError: For example, an error that could occur here is TryFromIntError. If you get this, then the soundfont is just too big to contain in the swdl bank. Simply open the soundfont in Polyphone, create new soundfont, and copy the instruments you want into it, and voila! You can load it in swdl_tool now.
• Side note/rule of thumb: Sample rate and lookahead > # of samples. A lot of new sf2's have like a million samples in them for a single piano, which is amazing when you have a computer from the last 10 years, but the NDS is not that. Always prioritize quality of the few samples you have over the # of samples! If you look at a sample bank from EoS, you'll see lots of instruments with just 1 or 2 samples.
• Quick tip: The main bank can store a lot of unused samples, but you can only load so many samples at one time! So, for each track, try to minmax the amount of samples you can load in memory at one time, keeping in mind the amount you'll need for the other stuff too. Probably best to do this last. (P.s. if there's not enough memory, it'll just show a black screen)
• Pro tip! (It's called pro tip because you don't really have to worry about this until much later): Higher lookahead means better samples! However, longer lookahead takes longer. Thus, a tip is to work with less lookahead at the start, and when it's time to ship your hack, leave it over night on a high setting so that it can do the number crunching while you're asleep! (The difference might be minimal so really this is a last step before publishing)
• Common Error - Game shows black screen after adding in the generated files: The NDS has limited memory, and you have reached it! Remember, you can store as many samples as you want in the main bank (well, practically), but in individual tracks you can only load so many samples at a time. (The limit is around 100kB in uncompressed audio in my testing) Use --generate-optimized-swdl to prevent loading unused samples for individual tracks, and, especially for modern soundfonts with huge numbers of samples in them, try to cut them down in Polyphone.

Finally, if you're ever confused about how to use the CLI tools, do ./smdl_tool.exe --help and ./swdl_tool.exe --help. There's explanations for everything these tools can do in there.

Important note: DON'T RUN dse.exe!! It shouldn't do much but it's the file I use for doing very specific things like reading specifically named files and stuff for testing purposes. Use the other two :)

Thanks to...

• Psy_Commando for the brilliant documentation of all the .SWD and .SMD files, and for the creation of the ppmdu_2 parsers. It was invaluable in the creation of this parser, and laid the groundwork for my understanding of much of DSE's internal structures.
• CodaHighland from the Halley'sCometSoftware discord and Psy_Commando for the documentation of various new DSE events that, together, pretty much completely decipher the SMDL file format. While the absolute specifics of these events might need some more exploring, the fact that we know what all these events do has been invaluable to the parser, and also gives me a lot of hope that we can definitely make something that works reaallly well to convert MIDI.
• nazberrypie, creator of Trezer, for documenting the effects of the link bytes. This was invaluable in finally finishing the .SMD parser, and also helped me understand the linking structure of .SMD and .SWD files much better.
• The authors of the byteorder, bevy_reflect, serde, quick-xml, base64, clap, glob, phf, midly, chrono, colored, soundfont, adpcm-xq, and r8brain libraries for making them. It has helped me immensely as I attempted many times from various angles to get this to finally work.
• And finally, the #reverse-engineering channel on the SkyTemple discord! I know I'm somewhat new in the community, but Adex, Nazberry, Psy_Commando, Mond, and others have helped me so much in getting the information and boost I needed to make it this far. I honestly can't imagine finishing this so quickly if I had done it alone. Thanks guys!