The C tracing deadlocks for larger programs

[cmnrd]

See discussion in #481.

[Soroosh129]

Just as a note, I fixed one or two bugs in the C target's tracing framework in #532. I don't think this deadlock is caused by those fixes since the issue seems to be older than the fixes by a wide margin.

The benchmarks branch is now even with master thanks to @cmnrd, so a git pull will bring in my fixes.

[edwardalee]

This is now fixed. The problem was that with tracing turned on, there is an upper bound on the number of distinct kinds of tracing events you can have. This bound was being exceeded, and the code was neither issuing a warning nor releasing a mutex lock, hence causing a deadlock. Now it does both. Everyone will have to do the git submodule update --init --recursive song and dance.

[Soroosh129]

I don't know if this helps, but I have created a git-pull.sh locally in my lingua-franca folder that contains the following lines:

#!/bin/bash
git submodule update --init --recursive
git fetch --all
git pull

[lhstrh]

I don't know if this helps, but I have created a git-pull.sh locally in my lingua-franca folder that contains the following lines:

#!/bin/bash
git submodule update --init --recursive
git fetch --all
git pull

I don't recommend using this script. I think the script reflects some misconceptions around the workings of submodules and might do more harm than good (not to discount the good intentions behind sharing this). It's been too often that I hear about submodules being "complicated" and requiring "magic incantations." Let me try to demystify the "magic incantations" and offer some advice on how to work with submodules. Some of this information I have already shared on the lf-dev list, but I'll add this here for the sake of clarity.

What is a submodule?

A submodule is a repository that is mounted in the directory tree of another repository. Once initialized, the submodule can be interacted with as any other Git repository. Changes made in the submodule are pushed to its own remote, not to the remote of the repository that it is a submodule of. It should be noted that, after initialization, the submodule will have checked out a particular commit, not the HEAD of a particular branch.

I hear I need to use git submodule update --init --recursive. What problem does it solve?

The git submodule update --init --recursive comment tells Git to 1) recursively find submodules; 2) initialize them; and 3) check out the commit specified by a hash recorded on the branch of the repo that includes the submodule. If you have a checkout of your main repository and have any uninitialized submodules (e.g., because you did a normal git clone instead of a git clone --recursive), then this is the command to use. When you have a checkout with initialized submodules, however, then there is no reason to use this command (particularly the --initialize flag). It is not a "fix all" command for when "things don't work." In fact, using this command without understanding what it does is a perfect recipe for ending up in a "things don't work" situation. Suppose you are working on the reactor-c submodule. You're on a different branch in reactor-c and several commits ahead. Then you pull changes from the lingua-franca repo with the git-pull.sh script above. Now, all of a sudden, your submodule is no longer on a branch and checked out whatever commit was specified in the lingua-franca branch that you just updated. This might come as a surprise, and can mess up your workflow. This is why git pull doesn't do what git-pull.sh does.

OK, so what should I do instead?

Here is some general advise on how to deal with submodules.

Modifying the submodule pointer

Recall that a checked-out submodule can be interacted with as any clone; you can make changes to it, commit them, and push them. The only difference is that the HEAD might not always be attached because the submodule is locked to a particular commit, not the head of a particular branch. In the submodule, you can check out any branch you like, do pulls and pushes, etc. The important thing to keep in mind is that whenever you want to record the hash to pin the submodule to, you have to pop out of the submodule tree, back into the lingua-franca tree, git add the submodule, git commit, and git push. You do this only once you are ready to let other users of your branch (including CI) update their submodule to the same version that you are using. E.g.: git add org.lflang/src/lib/c/reactor-c and git commit -m 'Updated submodule' and git push. Unless you do this, the lingua-franca CI, (and everyone else with the latest on master) will still have their submodule point some earlier established commit hash. To find out whether your local submodule checkout matches the hash that is specified in the branch that you're on in the main repository, you can run git submodule status (if it has a + in front of the hash, then it means your submodule is at a different version; git status will also tell you.

Updating your submodule after the submodule pointer changed

Whenever you do a git switch, git checkout, or git pull, your submodule pointer might change. Again, git submodule status will tell you whether your current submodule checkout matches the updated pointer. Assuming it was the reactor-c submodule that had its pointer changed, you can update the local submodule checkout to the updated pointer using git submodule update org.lflang/src/lib/c/reactor-c. You can also decide to move the pointers of all submodules, using git submodule update if you are sure that it's fine to update all submodules (the --recursive flag is only necessary if you submodules also have submodules, but we don't have that situation currently).

Do I need to do git fetch --all before doing a git pull?

No, but it does no harm either. The git fetch --allcommand downloads commits, files, and refs from a remote repository into your local repo. Fetching is what you do when you want to see what everybody else has been working on. It doesn't update any branches, however. The git pull command, on the other hand, automatically fetches from branches that you have checked out earlier, including your current branch, which it will also apply updates to.

tl;dr: To update a branch in your main repo, use git pull. If a submodule pointer changed and you want your submodule to check out the commit corresponding to the updated pointer, use git submodule update. If you want to change the submodule pointer to correspond to the current state of your submodule checkout, use git add <submodule>.

[Soroosh129]

Thanks Marten. This is really helpful.

[edwardalee]

Maybe this should go on one of our wiki pages? Also, does the Oomph setup clone with --recursive? It seems not, because I've been needing to do submodule update --init --recursive.

[lhstrh]

I believe it does initialize submodules. Were the reactor-ts and reactor-c directories empty?

[lhstrh]

I'll create a wiki page as per your suggestion. There are a few more things to mention here, such as contributing to submodules while they are checked out using https.