1st and 2nd endings overlap in DCML conversions

[malcolmsailor]

There's already an issue that I created at the music21 repo for this problem, but I think there should be one here since it means that many of the DCML conversions are presently corrupt.

The basic problem is that the parser doesn't handle 1st and 2nd endings, meaning the chord changes from both are put into the same measure, leading to incorrect results that crash the music21 parser. For example, from Corpus/Quartets/Beethoven%2C_Ludwig_van/Op018_No3/4/analysis.txt

m117 I6 I6
m118 viio6 viio6
m120 I iiio6 b1.33 IV b2.33 ii6

I think the best (and really only tractable) solution for music21 at the moment is to put the 1st endings into metadata and only write out the 2nd endings. (See the discussion between Myke and me at the music21 issue.) I plan to implement this myself sometime soon. This will at least result in valid romantext output. Somebody could write an additional script that compares the measures files from the ABC repo to add the first endings back in, but I think Myke's argument that this should not be a part of music21 itself is persuasive (and in any case what goes into music21 is up to him), and I won't be able to do it myself.

[MarkGotham]

Thanks @malcolmsailor.

While I think we can think/talk more about a best practice, it's clear to me that having this functionality available is all good/positive.

Are you content to implement? Perhaps you have already?

See also initial work over here: https://github.com/MarkGotham/bar-measure

[MarkGotham]

Looking again at this, I'm reminded that the issue isn't with unambiguously deducing first and second time bars, but rather the start repeat.

That being the case, @malcolmsailor, I'd be inclined to go with your option 1: include first and second time measures, despite missing the start repeat. This is incomplete, but it's a much more complete solution than effectively removing first time measures together (including relegating to metadata or notes).

We already have a script (alignStreams) that handles some of these gaps between score and analysis (or indeed any two streams) and there are actively plans to develop that (e.g., https://github.com/MarkGotham/bar-measure - alignment issues go way beyond the scope of what's discussed here). So I'd envisage using these DCML-rntxt conversions as test cases for wether enhanced functionality of the alignStreams / bar-measure kind can successfully deal with common issues like missing start repeat.

There's also (probably) already music21 functionality for ignoring 1st time bars / repeat information. If not, I'm happy to provide that (here or there as appropriate).

So then we have analysis files that get as close as possible, and multiple options for closing any remaining gaps:

1. ignore first time / ignore all repeats.
2. insert the missing repeat information from the score (here, the start repeats).
3. insert the missing repeat information from the DCML measures file.

What do you think @malcolmsailor? Thanks in advance for your thoughts.

[malcolmsailor]

option 1: include first and second time measures, despite missing the start repeat.

I think I already have a branch where this is implemented, so I'm happy to go with this option.

This is incomplete, but it's a much more complete solution than effectively removing first time measures together (including relegating to metadata or notes).

I definitely hear where you're coming from, but I think there's an argument to be had for the other case as well, since it produces "syntactically correct output" (in the sense that every measure proceeds to another measure that it should actually proceed to, rather than having the end of the first ending proceed directly to the start of the second ending, which, besides the fact that it may lead to implausible chord transitions, could lead to more serious problems, e.g., if there is a modulation in the first ending). I don't really see there as being a good solution.

To avoid the issue of modulations in the first ending I mentioned in the previous paragraph it occurs to me I can just add an explicit key to the start of each ending.

Another issue that occurred to me just now is that it may not only be the forward repeat bar that is hard to locate but also possibly where the endings start. Imagine, for example, that there is a passage that goes like this (in romanText):

m8 I
m9a I
m10a V
m9b I
m10 b I

Since there is no chord change at m. 9, or at 10b, the DCML annotator might omit the chord changes in mm. 9a and 9b as well as 10b altogether, so the result after conversion would look like

m8 I
m10a V

I'm not sure whether the ms3 tool used to make the ABC and Mozart files always puts chord changes at the start of endings or not, but if not we risk this kind of issue regardless of which strategy we use for dealing with the endings.

[MarkGotham]

Great thanks @malcolmsailor, let’s do it!

• You finalise the rntxt output
• I’ll make a start on advancing the post-hoc fixes (mapping start repeats over and removing first time bars).

Thanks too for raising the potential issue wrt chord at the start of first time. Some thoughts:

• best to simply read the information of the tabular file directly where possible. First step is to check whether this is an issue. @johentsch, whether you do this already or not, do you think it would be best for DCML to include this info? Any thoughts?
• failing that, the missing element is clear, at least in this case, right? Given 10a followed by 9b we know there must be a 9a. (I.e., if measure number doesn’t increase). I'm just talking about the example you give of course, quite possible that there are cases in which this doesn’t work (e.g., all tonic throughout both 1st and 2nd times).
• failing all of that, it's likely still solvable at fix time (i.e., correct the incorrect first time bar location)

[malcolmsailor]

failing that, the missing element is clear, at least in this case, right? Given 10a followed by 9b we know there must be a 9a. (I.e., if measure number doesn’t increase). I'm just talking about the example you give of course, quite possible that there are cases in which this doesn’t work (e.g., all tonic throughout both 1st and 2nd times).

In the example I gave, there is no 9b, since there is no chord change there. As I say, I don't know whether such examples actually exist.

[MarkGotham]

Yes, let's find out whether there's a problem to solve first. The easiest way to do this debug this might be simply to:

• implement your new writer
• re-run the conversion
• see what issues remain

FWIW, what I'm saying is that in your example:

m8 I
m9a I
m10a V
m9b I
m10 b I

I could imagine m9a I causing issues because it continues m8 in all senses, but I'd be very surprised if m9b I were not specified because although the music follows m8, by the measure count style logic of how it's set out on score and there, there's a sense to which it "follows" m10a. Just a guess though. @johentsch may know.

[malcolmsailor]

I could imagine m9a I causing issues because it continues m8 in all senses, but I'd be very surprised if m9b I were not specified because although the music follows m8, by the measure count style logic of how it's set out on score and there, there's a sense to which it "follows" m10a. Just a guess though. @johentsch may know.

I just tried deleting the harmonies in the first measure of both the first and second ending and the corresponding measures are indeed missing from the harmonies tsv files, as I expected. ms3 only writes harmonies where there are annotations. Whether anyone ever omits the harmony annotation at the beginning of a second ending is a different question.

[MarkGotham]

Thanks for checking. A different question and also a testable one (more easily, I suspect, than the start of the 1st time).

[johentsch]

The way I've been thinking of this in ms3 was, as has come up above, that you can have two versions:

• reduced playthrough without any repeats, where you always skip into the seconda volta and skip over any 3rd... endings.
• full playthrough (unfold feature), including D.C., reflecting the proportions when playing everything, such that the endings appear in their correct environment.
  This distinction plays out in the so-called quarterbeats column (offset from piece's beginning): in the default case, it has missing values for all but second endings; but if you get the "unfolded" table there are no missing values.

This weekend I released a couple of 1.0.x updates in conjunction with upgrading the datasets. For checking consistency of phrase annotations we're now unfolding the annotations because you often have an ending in each volta so the counts don't add up and checking consistency is complicated. We will have to the same at one point for the pedal points but are not there yet.

I'm not sure this information is of any help for your discussion?