Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Construct dependency fingerprint from canonicalized hash tree #90

Merged
merged 1 commit into from
Jul 6, 2022

Conversation

greglook
Copy link
Collaborator

@greglook greglook commented Jul 5, 2022

During some debugging, I found that the current dependency fingerprinting code is hashing the dependency structure basically as-is, leaving it brittle against changes in things like dependency ordering and map iteration order. Instead, we should canonicalize each dependency coordinate and roll things up into a tree of hashes.

A little debug printing shows that a dependency calculation for the current code is hashing lots of raw data structures:

Hashed input:                                                                                                                                                                                                                                        
v2/dependencies                                                                                                                                                                                                                                      
:lein-monolith.task.fingerprint/default {:dependencies ([org.clojure/clojure "1.11.1"] [amperity/aws "MONOLITH-SNAPSHOT" :exclusions ([com.amazonaws/*])] ,,, [mvxcvi/puget nil] [mvxcvi/whidbey nil])}
:monolith/leaky {:managed-dependencies [[aleph "0.4.6" :exclusions [byte-streams]] [amperity/envoy "1.0.1"] ,,, [yada/lean "1.2.16" :exclusions [byte-streams]]]}
:provided       {:dependencies ([org.apache.spark/spark-core_2.12 "3.1.2" :exclusions ([org.apache.hadoop/hadoop-client])] [org.apache.spark/spark-sql_2.12 "3.1.2"] [com.amazonaws/aws-java-sdk-bundle "1.11.1026"] ,,,)}
    => 595wMrKcf8UPCOpWyl032pohkh0

After this change, the same debugging shows the tree rollup:

...

Hashed input:
v2/dep-coordinate
:exclusions	([com.amazonaws/*])
:version	"MONOLITH-SNAPSHOT"
    => IvYsPCjF3X2QDGuKqFMvJOM6tLs

Hashed input:
v2/dep-coordinate
:version	"MONOLITH-SNAPSHOT"
    => WnEIDZX_gfYyKNWTg1EwmZTXPhg

Hashed input:
v2/dep-coordinate
:version	nil
    => 6h06eUyN4JXUfcpzZFS-zXwmZsE

...

Hashed input:
v2/dep-coordinate
:version	"0.6.0"
    => AekLXJrkY2zBH8TGKcsGTd1IYRE

Hashed input:
v2/dep-coordinate
:version	"3.1.2"
    => t16552m11qHRfTSpB5sVKnGoTH0

Hashed input:
v2/dep-coordinate
:version	"1.11.1"
    => x8Jetn5-OcsZWy59Spfvr2I7n7Y

Hashed input:
v2/profile-dependencies
amperity/aws	IvYsPCjF3X2QDGuKqFMvJOM6tLs
amperity/domain-core	WnEIDZX_gfYyKNWTg1EwmZTXPhg
amperity/envoy	6h06eUyN4JXUfcpzZFS-zXwmZsE
amperity/hadoop	tj1lCqROWc2_Z_JXwOpUAqTCocg
amperity/spark-base	qoKQMZNzrGA1wNeC81DluNOpBSA
amperity/spark-env	WnEIDZX_gfYyKNWTg1EwmZTXPhg
amperity/storage-coordinates	WnEIDZX_gfYyKNWTg1EwmZTXPhg
amperity/storage-spark	WnEIDZX_gfYyKNWTg1EwmZTXPhg
cider/cider-nrepl	ixE63Qpr_sc5kjv97PQNRW6vDaw
com.stuartsierra/component	6h06eUyN4JXUfcpzZFS-zXwmZsE
mvxcvi/puget	6h06eUyN4JXUfcpzZFS-zXwmZsE
mvxcvi/whidbey	6h06eUyN4JXUfcpzZFS-zXwmZsE
nrepl/nrepl	AekLXJrkY2zBH8TGKcsGTd1IYRE
org.apache.spark/spark-hadoop-cloud_2.12	t16552m11qHRfTSpB5sVKnGoTH0
org.clojure/clojure	x8Jetn5-OcsZWy59Spfvr2I7n7Y
    => tBPKOsU5ejRVwjDlYD8MIUlq35g

Hashed input:
v3/dependencies
:lein-monolith.task.fingerprint/default	:dependencies	tBPKOsU5ejRVwjDlYD8MIUlq35g
:monolith/leaky	:managed-dependencies	XDQPF6EzKm2Vg-NKeC9AXDcGg9c
:provided	:dependencies	wOCcXLakP6MwREX_PbOUNnT7iaI
    => fQZgHkrH2g-Ap-GQtv_pU0Sbbxo

I tested and reordering dependencies within a project no longer changes the dependency fingerprint for that project.

@greglook greglook requested a review from a team July 5, 2022 22:58
@greglook greglook merged commit e0ce4bf into main Jul 6, 2022
@greglook greglook deleted the canonical-dependency-hashing branch July 6, 2022 05:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants