Gradle witness for 'jtorctrl' not updated for Tor 0.4.5

[bfix]

Description

Bisq does not build due to checksum mismatch in gradle witness

Version

v1.7.5

Steps to reproduce

Build from scratch

Expected behaviour

Build successfull

Actual behaviour

Stops on error:

Checksum failed for com.github.JesusMcCloud:jtorctl:389d61b1b5a85eb2f23c582c3913ede49f80c9f2b553e4762382c836270e57e5

Additional info

Change in gradle-witness.gradle from:

com.github.JesusMcCloud:jtorctl:389d61b1b5a85eb2f23c582c3913ede49f80c9f2b553e4762382c836270e57e5

to:

com.github.JesusMcCloud:jtorctl:6465e0b22b921344a065a6e82a5390ab2f9dac5ab293c38bd8a76baddead6492

[boring-cyborg]

Thanks for opening your first issue here!

Be sure to follow the issue template. Your issue will be reviewed by a maintainer and labeled for further action.

[chimp1984]

@cd2357 Can you comment?

[cd2357]

TLDR:

This happens because jitpack is serving one of two different versions of the jtorctl jar, depending on the caller's IP.

I came to this based on:

• Own tests. I downloaded the jtorctl jar using VPNs to different locations and I sometimes got one version of the file, sometimes the other.
• The fact that current GitHub builds are successful (i.e. the jtorctl retrieved by GH matches the hash in our gradle-witness), while at the same time the build fails for some people like OP (i.e. jitpack serves them a jtorctl jar which doesn't match the gradle-witness hash).

Both versions of the jar are attached. Change extension from zip to jar:

jtorctl-1.5-389.zip -- hash 389d61b1b5a85eb2f23c582c3913ede49f80c9f2b553e4762382c836270e57e5
jtorctl-1.5-646.zip -- hash 6465e0b22b921344a065a6e82a5390ab2f9dac5ab293c38bd8a76baddead6492

---

Root-cause analysis

What I suspect happened is this:

• Beginning-Mid October: jitpack had an issue and lost some build artefacts (jtorctl)

  • this resulted in Bisq builds failing with:
    Could not find artifact com.github.JesusMcCloud:jtorctl:jar:1.5 in jitpack.io (https://jitpack.io)

• 17 Oct: jitpack issue partially fixed, but artefacts still lost

  • A (new) build of jitpack was triggered
    • Same commit, same code version (as per build log)
    • hash 6465e0b22b921344a065a6e82a5390ab2f9dac5ab293c38bd8a76baddead6492

• Turns out Jitpack uses Cloudflare to distribute its artefacts

  • Cloudflare mirrors the different artefacts in different regions and serves them to the caller from the closest-reachable CDN location
  • Depending on the SLAs jitpack has with Cloudflare, new artefact versions can take a while to propagate
    • This is where we are now
      • New version of the jitpack artefact is available (hash 6465..)
      • This newer jtorctl is distributed to some Cloudflare nodes, not to others (yet?)
      • Depending by which Cloudflare server the download request is served, you can get one file or the other

Q: If the artefact was simply re-built by jitpack, why does it have a different hash?

@ripcurlx pointed out it could be because the new build used a different JDK version (1.8.0_212 vs. 1.8.0_252).

I checked and this could indeed be the case. I built jtorctl from source using those specific JDK versions (from here). Indeed, the resulting jars had different hashes. They were also different than the two hashes from jitpack.

Q: Ok so Jitpack now serves a different jtorctl. How do we know this is entirely due to a different JDK? Can we trust the new jar to not contain code modifications?

I would say yes we can trust it.

I used http://java-decompiler.github.io to decompile both jars and compare the sources. See File > Save all sources for getting an export of all sources.

Then using diff --recursive jtorctl-1.5-389.jar.src jtorctl-1.5-646.jar.src I can confirm the sources are the same.

Q: What now? What's the next step?

The only option I can think of is wait until Cloudflare propagates the new jar to all its regional servers.

In other words, wait until downloading the jar from jitpack reliably results in the same jar, no matter from which location its downloaded.

Until then, we can alternatively work with either of the two jtorctl hashes in gradle-witness, depending which one is pulled. This means manually updating the hash in gradle-witness locally, in case the build complains.

(Not the best solution, but can't think of a better one.)

---

There are other open questions to consider for medium / long-term:

• Why did jitpack lose the artefacts?
  • Temporary outage maybe, but even after they got back online the previous jar is lost? No backups?
    • Will they lose artefacts again?
• Why is jitpack using such an old JDK even for newer builds?
  • Java version: 1.8.0_252, vendor: Private Build ( as per jtorctl build log)
• What alternatives are there, in case jitpack / maven central / etc go down, or experience similar issues?
  • One idea is we host the dependencies ourselves in a Git LFS repo. There are already some possible solutions for this.

---

What do you guys think?

Do you see another way to tackle this at the moment?

cc @bisq-network/bisq-devs

[cd2357]

@ripcurlx should I revert #5805 to force GH to always download the dependencies on new PR builds? Until this jtorctl thing is solved?

[chimp1984]

Thanks @cd2357 for the details report!

I checked and this could indeed be the case. I built jtorctl from source using those specific JDK versions (from here). Indeed, the resulting jars had different hashes. They were also different than the two hashes from jitpack.

We use a tool for removing meta data which cause diff. hashes in jars in the build script. That was an intent to get to deterministic builds. At least for the Bisq jar file that should work. If we would include that step in those binaries (if those are not hosted from Bisq we should do that so we can adjust the build process) the hash of the jar should be the same (I assume the diff. hash is mostly because of those metadata which includes timestamps not because of the jdk change - though that might have differences as well, but then we should freeze the jdk version in the build as well).

So in short we should try to get a step further to deterministic builds.
And I think the benefit from the cache feature might not be worth the additional complexity and risk.

vendor: Private Build

That does not sound good as well. We should control which jdk is used for builds.

As said I think we should fork that repo anyway and host it on Bisq. There might be another sub dependency in netlayer (socksproxy) - if so that should also be moved to Bisq IMO.

[zarathustra19]

May i run and use Bisq without installing on Arch ?

[cd2357]

Fixed as per discussion above.