-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[u] Define sources within catalog JSON (#3114) #3476
Conversation
Codecov Report
@@ Coverage Diff @@
## develop #3476 +/- ##
========================================
Coverage 81.86% 81.86%
========================================
Files 123 123
Lines 14291 14291
========================================
Hits 11699 11699
Misses 2592 2592 Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you got the chain labels wrong.
And remember to note the PR provenance in the description. See my edit to the description. |
Added chain label to this PR, blocked #3277 on this PR. |
Because of it's disruptive effect on #3441 I'm pulling this from Approved. |
957282d
to
13aeec0
Compare
self note: 13aeec0 IT passes |
deployments/dev/environment.py
Outdated
plugins=dict(metadata=dict(name='hca'), | ||
repository=dict(name='tdr')), | ||
sources=sources) | ||
for name, atlas, sources in [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No point in looping over a singleton.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was hoping to have this in a state where we can easily add the lungmap
catalogs back in when we work on #3489 but i don't think we can have this and each snapshot on a line.
deployments/dev/environment.py
Outdated
'dcp2', | ||
'hca', | ||
list(filter(None, [ | ||
mksrc('datarepo-dev-a9252919', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Each snapshot must be a single line or we'll go nuts. With my diff, the lines are longer than 80 trimmed but don't exceed the 120 margin so they survive reformatting with PyCharm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just did the first three.
Index: UPGRADING.rst
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/UPGRADING.rst b/UPGRADING.rst
--- a/UPGRADING.rst (revision 13aeec00ab57f58d5b1dea76b5fbafdf018e458e)
+++ b/UPGRADING.rst (date 1633393162333)
@@ -15,9 +15,9 @@
========================================
The ``AZUL_TDR_SOURCES`` and ``AZUL_…_SOURCES`` environment variables have been
-removed. Sources must be defined within the catalog configuration as a
-list of sources. As always, use the sandbox deployment's ``environment.py`` as
-a model when upgrading personal deployments.
+removed. Sources must be defined within the catalog configuration as a list of
+sources. As always, use the sandbox deployment's ``environment.py`` as a model
+when upgrading personal deployments.
HumanCellAtlas/dcp2#17 TDR dev dataset is stale
Index: deployments/dev/environment.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/deployments/dev/environment.py b/deployments/dev/environment.py
--- a/deployments/dev/environment.py (revision 13aeec00ab57f58d5b1dea76b5fbafdf018e458e)
+++ b/deployments/dev/environment.py (date 1633393139401)
@@ -55,338 +55,326 @@
'AZUL_DEPLOYMENT_STAGE': 'dev',
'AZUL_CATALOGS': json.dumps({
- f'{prefix}{name}': dict(atlas=atlas,
- internal=internal,
- plugins=dict(metadata=dict(name='hca'),
- repository=dict(name='tdr')),
- sources=sources)
- for name, atlas, sources in [
- (
- 'dcp2',
- 'hca',
- list(filter(None, [
- mksrc('datarepo-dev-a9252919',
- 'hca_dev_005d611a14d54fbf846e571a1f874f70__20210827_20210903',
- 7),
- mksrc('datarepo-dev-c148d39c',
- 'hca_dev_027c51c60719469fa7f5640fe57cbece__20210827_20210902',
- 8),
- mksrc('datarepo-dev-e2ab8487',
- 'hca_dev_03c6fce7789e4e78a27a664d562bb738__20210902_20210907',
- 1530),
- mksrc('datarepo-dev-37639c56',
- 'hca_dev_05657a599f9d4bb9b77b24be13aa5cea__20210827_20210928',
- 185),
- mksrc('datarepo-dev-9f4012c9',
- 'hca_dev_05be4f374506429bb112506444507d62__20210827_20210902',
- 1544),
- mksrc('datarepo-dev-baa2812f',
- 'hca_dev_0792db3480474e62802c9177c9cd8e28__20210827_20210903',
- 1450),
- mksrc('datarepo-dev-38e08b5c',
- 'hca_dev_08b794a0519c4516b184c583746254c5__20210901_20210903',
- 2),
- mksrc('datarepo-dev-2749da57',
- 'hca_dev_091cf39b01bc42e59437f419a66c8a45__20210830_20210903',
- 20),
- mksrc('datarepo-dev-eab7fa76',
- 'hca_dev_0c3b7785f74d40918616a68757e4c2a8__20210827_20210903',
- 177),
- mksrc('datarepo-dev-fef02a92',
- 'hca_dev_0d4b87ea6e9e456982e41343e0e3259f__20210827_20210903',
- 8),
- mksrc('datarepo-dev-78bae095',
- 'hca_dev_0fd8f91862d64b8bac354c53dd601f71__20210830_20210903',
- 10),
- mksrc('datarepo-dev-ff0a2fe7',
- 'hca_dev_116965f3f09447699d28ae675c1b569c__20210827_20210903',
- 8),
- mksrc('datarepo-dev-4de1b9fd',
- 'hca_dev_16ed4ad8731946b288596fe1c1d73a82__20210830_20210903',
- 28),
- mksrc('datarepo-dev-135f340c',
- 'hca_dev_1c6a960d52ac44eab728a59c7ab9dc8e__20210827_20210928',
- 10),
- mksrc('datarepo-dev-86c60513',
- 'hca_dev_1cd1f41ff81a486ba05b66ec60f81dcf__20210901_20210903',
- 18),
- mksrc('datarepo-dev-f5321179',
- 'hca_dev_1ce3b3dc02f244a896dad6d107b27a76__20210827_20210903',
- 421),
- mksrc('datarepo-dev-76de829d',
- 'hca_dev_2043c65a1cf84828a6569e247d4e64f1__20210831_20210907',
- 1734),
- mksrc('datarepo-dev-c808badb',
- 'hca_dev_2086eb0510b9432bb7f0169ccc49d270__20210827_20210903',
- 10),
- mksrc('datarepo-dev-23782220',
- 'hca_dev_23587fb31a4a4f58ad74cc9a4cb4c254__20210827_20210909',
- 1476),
- mksrc('datarepo-dev-4c3e6011',
- 'hca_dev_248fcf0316c64a41b6ccaad4d894ca42__20210907_20210907',
- 2958),
- mksrc('datarepo-dev-1c2c69d9',
- 'hca_dev_24c654a5caa5440a8f02582921f2db4a__20210830_20210903',
- 3),
- mksrc('datarepo-dev-38f08cd8',
- 'hca_dev_2a64db431b554639aabb8dba0145689d__20210830_20210903',
- 10),
- mksrc('datarepo-dev-4cf05ce2',
- 'hca_dev_2a72a4e566b2405abb7c1e463e8febb0__20210901_20210903',
- 2290),
- mksrc('datarepo-dev-3041c2cf',
- 'hca_dev_2af52a1365cb4973b51339be38f2df3f__20210830_20210903',
- 10),
- mksrc('datarepo-dev-311340f6',
- 'hca_dev_2d8460958a334f3c97d4585bafac13b4__20210902_20210907',
- 3589),
- mksrc('datarepo-dev-766bfb76',
- 'hca_dev_2ef3655a973d4d699b4121fa4041eed7__20210827_20210903',
- 8),
- mksrc('datarepo-dev-1720b3c5',
- 'hca_dev_379ed69ebe0548bcaf5ea7fc589709bf__20210827_20210902',
- 4),
- mksrc('datarepo-dev-ac6efd3f',
- 'hca_dev_38449aea70b540db84b31e08f32efe34__20210830_20210903',
- 42),
- mksrc('datarepo-dev-40283c27',
- 'hca_dev_3a69470330844ece9abed935fd5f6748__20210901_20210903',
- 125),
- mksrc('datarepo-dev-b08233fa',
- 'hca_dev_3cfcdff5dee14a7ba591c09c6e850b11__20210827_20210903',
- 8),
- mksrc('datarepo-dev-bdc9f342',
- 'hca_dev_3e329187a9c448ec90e3cc45f7c2311c__20210901_20210903',
- 1001),
- mksrc('datarepo-dev-ec07c8d8',
- 'hca_dev_4037007b0eff4e6db7bd8dd8eec80143__20210831_20210903',
- 39),
- mksrc('datarepo-dev-c0ec174a',
- 'hca_dev_403c3e7668144a2da5805dd5de38c7ff__20210827_20210903',
- 63),
- mksrc('datarepo-dev-31b3553a',
- 'hca_dev_414accedeba0440fb721befbc5642bef__20210827_20210903',
- 4),
- mksrc('datarepo-dev-b4789901',
- 'hca_dev_41fb1734a121461695c73b732c9433c7__20210830_20210903',
- 12),
- mksrc('datarepo-dev-4e5ffd52',
- 'hca_dev_42d4f8d454224b78adaee7c3c2ef511c__20210830_20210903',
- 9),
- mksrc('datarepo-dev-5ef7f2e2',
- 'hca_dev_455b46e6d8ea4611861ede720a562ada__20210901_20210903',
- 74),
- mksrc('datarepo-dev-a6c6b953',
- 'hca_dev_4bec484dca7a47b48d488830e06ad6db__20210830_20210903',
- 14),
- mksrc('datarepo-dev-f31edbc2',
- 'hca_dev_4d6f6c962a8343d88fe10f53bffd4674__20210901_20210903',
- 12),
- mksrc('datarepo-dev-bb8fbae4',
- 'hca_dev_51f02950ee254f4b8d0759aa99bb3498__20210827_20210928',
- 6),
- mksrc('datarepo-dev-71de019e',
- 'hca_dev_520afa10f9d24e93ab7a26c4c863ce18__20210827_20210928',
- 649),
- mksrc('datarepo-dev-ffcf8b00',
- 'hca_dev_52b29aa4c8d642b4807ab35be94469ca__20210830_20210903',
- 467),
- mksrc('datarepo-dev-f76414c8',
- 'hca_dev_52d10a60c8d14d068a5eaf0d5c0d5034__20210827_20210902',
- 176),
- mksrc('datarepo-dev-319b80f7',
- 'hca_dev_577c946d6de54b55a854cd3fde40bff2__20210827_20210903',
- 7),
- mksrc('datarepo-dev-279f1986',
- 'hca_dev_5ee710d7e2d54fe2818d15f5e31dae32__20210901_20210903',
- 41),
- mksrc('datarepo-dev-0abea017',
- 'hca_dev_6072616c87944b208f52fb15992ea5a4__20210827_20210902',
- 603),
- mksrc('datarepo-dev-da221b1a',
- 'hca_dev_60ea42e1af4942f58164d641fdb696bc__20210827_20210903',
- 1145),
- mksrc('datarepo-dev-c3d623dc',
- 'hca_dev_63b5b6c1bbcd487d8c2e0095150c1ecd__20210830_20210903',
- 11),
- mksrc('datarepo-dev-ecb9c129',
- 'hca_dev_67a3de0945b949c3a068ff4665daa50e__20210827_20210903',
- 732),
- mksrc('datarepo-dev-3545971c',
- 'hca_dev_71436067ac414acebe1b2fbcc2cb02fa__20210827_20210928',
- 4),
- mksrc('datarepo-dev-12b7a9e1',
- 'hca_dev_7880637a35a14047b422b5eac2a2a358__20210901_20210903',
- 366),
- mksrc('datarepo-dev-7913b094',
- 'hca_dev_78b2406dbff246fc8b6120690e602227__20210827_20210902',
- 216),
- mksrc('datarepo-dev-4747d8fe',
- 'hca_dev_7adede6a0ab745e69b67ffe7466bec1f__20210830_20210903',
- 1601),
- mksrc('datarepo-dev-aba01389',
- 'hca_dev_7b947aa243a74082afff222a3e3a4635__20210831_20210907',
- 7),
- mksrc('datarepo-dev-bf3a4c8a',
- 'hca_dev_8185730f411340d39cc3929271784c2b__20210830_20210903',
- 12),
- # FIXME: https://github.com/HumanCellAtlas/dcp2/issues/48
- None and mksrc('datarepo-dev-a86993e6',
- 'hca_dev_83f5188e3bf749569544cea4f8997756__20210929_20210929',
- 1612),
- mksrc('datarepo-dev-bd995e95',
- 'hca_dev_842605c7375a47c59e2ca71c2c00fcad__20210830_20210903',
- 8),
- mksrc('datarepo-dev-d0772077',
- 'hca_dev_8787c23889ef4636a57d3167e8b54a80__20210827_20210903',
- 3),
- mksrc('datarepo-dev-8eb2ffd1',
- 'hca_dev_87d52a86bdc7440cb84d170f7dc346d9__20210830_20210903',
- 16),
- mksrc('datarepo-dev-0c5c20b5',
- 'hca_dev_8c3c290ddfff4553886854ce45f4ba7f__20210902_20210907',
- 6640),
- mksrc('datarepo-dev-a198b032',
- 'hca_dev_90bd693340c048d48d76778c103bf545__20210827_20210903',
- 2245),
- mksrc('datarepo-dev-59d37b9a',
- 'hca_dev_946c5add47d1402a97bba5af97e8bce7__20210831_20210903',
- 149),
- mksrc('datarepo-dev-788c3b52',
- 'hca_dev_955dfc2ca8c64d04aa4d907610545d11__20210831_20210903',
- 13),
- mksrc('datarepo-dev-4b88b45b',
- 'hca_dev_962bd805eb894c54bad2008e497d1307__20210830_20210903',
- 28),
- mksrc('datarepo-dev-02c59b72',
- 'hca_dev_99101928d9b14aafb759e97958ac7403__20210830_20210903',
- 1190,
- ma=1),
- mksrc('datarepo-dev-a6312a94',
- 'hca_dev_992aad5e7fab46d9a47ddf715e8cfd24__20210830_20210903',
- 41),
- mksrc('datarepo-dev-75589244',
- 'hca_dev_996120f9e84f409fa01e732ab58ca8b9__20210827_20210903',
- 26),
- mksrc('datarepo-dev-d4b988d6',
- 'hca_dev_a004b1501c364af69bbd070c06dbc17d__20210830_20210903',
- 16,
- ma=1),
- mksrc('datarepo-dev-9ec7beb6',
- 'hca_dev_a29952d9925e40f48a1c274f118f1f51__20210827_20210902',
- 26),
- mksrc('datarepo-dev-d3d5bbfa',
- 'hca_dev_a39728aa70a04201b0a281b7badf3e71__20210830_20210903',
- 33),
- mksrc('datarepo-dev-7b7daff7',
- 'hca_dev_a96b71c078a742d188ce83c78925cfeb__20210827_20210902',
- 6),
- mksrc('datarepo-dev-58610528',
- 'hca_dev_a9c022b4c7714468b769cabcf9738de3__20210827_20210903',
- 23),
- mksrc('datarepo-dev-1dce87e5',
- 'hca_dev_ae71be1dddd84feb9bed24c3ddb6e1ad__20210916_20210916',
- 3515),
- mksrc('datarepo-dev-b2004d1c',
- 'hca_dev_b4a7d12f6c2f40a39e359756997857e3__20210831_20210903',
- 24),
- mksrc('datarepo-dev-0b465564',
- 'hca_dev_b51f49b40d2e4cbdbbd504cd171fc2fa__20210830_20210903',
- 193),
- mksrc('datarepo-dev-cd97e83a',
- 'hca_dev_b963bd4b4bc14404842569d74bc636b8__20210827_20210928',
- 2),
- mksrc('datarepo-dev-376d3f4a',
- 'hca_dev_bd40033154b94fccbff66bb8b079ee1f__20210901_20210903',
- 18),
- mksrc('datarepo-dev-aa783adb',
- 'hca_dev_c5f4661568de4cf4bbc2a0ae10f08243__20210827_20210928',
- 1),
- mksrc('datarepo-dev-990234a8',
- 'hca_dev_c6ad8f9bd26a4811b2ba93d487978446__20210827_20210903',
- 639),
- mksrc('datarepo-dev-61b8d081',
- 'hca_dev_c715cd2fdc7c44a69cd5b6a6d9f075ae__20210827_20210902',
- 9),
- mksrc('datarepo-dev-71926fdc',
- 'hca_dev_c893cb575c9f4f26931221b85be84313__20210901_20210903',
- 20),
- mksrc('datarepo-dev-2f4bfe5d',
- 'hca_dev_ccd1f1ba74ce469b9fc9f6faea623358__20210827_20210902',
- 222),
- mksrc('datarepo-dev-aa8357fb',
- 'hca_dev_ccef38d7aa9240109621c4c7b1182647__20210831_20210903',
- 1314),
- mksrc('datarepo-dev-24e672db',
- 'hca_dev_cddab57b68684be4806f395ed9dd635a__20210831_20210907',
- 2545),
- mksrc('datarepo-dev-f4cb2365',
- 'hca_dev_ce33dde2382d448cb6acbfb424644f23__20210827_20210928',
- 189),
- mksrc('datarepo-dev-0d6f73ac',
- 'hca_dev_d012d4768f8c4ff389d6ebbe22c1b5c1__20210827_20210903',
- 8),
- mksrc('datarepo-dev-5674b4eb',
- 'hca_dev_d2111fac3fc44f429b6d32cd6a828267__20210830_20210903',
- 735),
- mksrc('datarepo-dev-b3632667',
- 'hca_dev_d3446f0c30f34a12b7c36af877c7bb2d__20210901_20210903',
- 40),
- mksrc('datarepo-dev-92c3a1de',
- 'hca_dev_d3a4ceac4d66498497042570c0647a56__20210831_20210903',
- 14),
- mksrc('datarepo-dev-4e88d60b',
- 'hca_dev_d3ac7c1b53024804b611dad9f89c049d__20210827_20210928',
- 11),
- mksrc('datarepo-dev-dbc582d9',
- 'hca_dev_dbcd4b1d31bd4eb594e150e8706fa192__20210827_20210902',
- 84),
- mksrc('datarepo-dev-848e2d4f',
- 'hca_dev_dbd836cfbfc241f0983441cc6c0b235a__20210827_20210902',
- 1),
- mksrc('datarepo-dev-d7517bce',
- 'hca_dev_dc1a41f69e0942a6959e3be23db6da56__20210827_20210902',
- 5),
- mksrc('datarepo-dev-27ad01e5',
- 'hca_dev_df88f39f01a84b5b92f43177d6c0f242__20210827_20210928',
- 1),
- mksrc('datarepo-dev-b839d6c7',
- 'hca_dev_e526d91dcf3a44cb80c5fd7676b55a1d__20210902_20210907',
- 606),
- mksrc('datarepo-dev-3faef568',
- 'hca_dev_e5d455791f5b48c3b568320d93e7ca72__20210827_20210903',
- 8),
- mksrc('datarepo-dev-e304a8fe',
- 'hca_dev_e77fed30959d4fadbc15a0a5a85c21d2__20210830_20210903',
- 333),
- mksrc('datarepo-dev-6fdac3db',
- 'hca_dev_e8808cc84ca0409680f2bba73600cba6__20210902_20210907',
- 898),
- mksrc('datarepo-dev-dbc3e131',
- 'hca_dev_eaefa1b6dae14414953b17b0427d061e__20210827_20210903',
- 385),
- # FIXME: https://github.com/HumanCellAtlas/dcp2/issues/45
- None and mksrc('datarepo-dev-6883f2a5',
- 'hca_dev_f48e7c39cc6740559d79bc437892840c__20210830_20210929',
- 14),
- mksrc('datarepo-dev-10f0610a',
- 'hca_dev_f81efc039f564354aabb6ce819c3d414__20210827_20210903',
- 4),
- mksrc('datarepo-dev-24e9529e',
- 'hca_dev_f83165c5e2ea4d15a5cf33f3550bffde__20210901_20210908',
- 7663),
- mksrc('datarepo-dev-67240cf2',
- 'hca_dev_f86f1ab41fbb4510ae353ffd752d4dfc__20210901_20210903',
- 20),
- mksrc('datarepo-dev-e8e0a59a',
- 'hca_dev_f8aa201c4ff145a4890e840d63459ca2__20210901_20210903',
- 384),
- mksrc('datarepo-dev-96d8e08c',
- 'hca_dev_faeedcb0e0464be7b1ad80a3eeabb066__20210831_20210903',
- 62),
- ])),
- ),
- ] for prefix, internal in [
+ f'{prefix}dcp2': {
+ 'atlas': 'hca',
+ 'internal': internal,
+ 'plugins': {'metadata': {'name': 'hca'}, 'repository': {'name': 'tdr'}},
+ 'sources': list(filter(None, [
+ mksrc('datarepo-dev-a9252919', 'hca_dev_005d611a14d54fbf846e571a1f874f70__20210827_20210903', 7),
+ mksrc('datarepo-dev-c148d39c', 'hca_dev_027c51c60719469fa7f5640fe57cbece__20210827_20210902', 8),
+ mksrc('datarepo-dev-e2ab8487', 'hca_dev_03c6fce7789e4e78a27a664d562bb738__20210902_20210907', 1530),
+ mksrc('datarepo-dev-37639c56',
+ 'hca_dev_05657a599f9d4bb9b77b24be13aa5cea__20210827_20210928',
+ 185),
+ mksrc('datarepo-dev-9f4012c9',
+ 'hca_dev_05be4f374506429bb112506444507d62__20210827_20210902',
+ 1544),
+ mksrc('datarepo-dev-baa2812f',
+ 'hca_dev_0792db3480474e62802c9177c9cd8e28__20210827_20210903',
+ 1450),
+ mksrc('datarepo-dev-38e08b5c',
+ 'hca_dev_08b794a0519c4516b184c583746254c5__20210901_20210903',
+ 2),
+ mksrc('datarepo-dev-2749da57',
+ 'hca_dev_091cf39b01bc42e59437f419a66c8a45__20210830_20210903',
+ 20),
+ mksrc('datarepo-dev-eab7fa76',
+ 'hca_dev_0c3b7785f74d40918616a68757e4c2a8__20210827_20210903',
+ 177),
+ mksrc('datarepo-dev-fef02a92',
+ 'hca_dev_0d4b87ea6e9e456982e41343e0e3259f__20210827_20210903',
+ 8),
+ mksrc('datarepo-dev-78bae095',
+ 'hca_dev_0fd8f91862d64b8bac354c53dd601f71__20210830_20210903',
+ 10),
+ mksrc('datarepo-dev-ff0a2fe7',
+ 'hca_dev_116965f3f09447699d28ae675c1b569c__20210827_20210903',
+ 8),
+ mksrc('datarepo-dev-4de1b9fd',
+ 'hca_dev_16ed4ad8731946b288596fe1c1d73a82__20210830_20210903',
+ 28),
+ mksrc('datarepo-dev-135f340c',
+ 'hca_dev_1c6a960d52ac44eab728a59c7ab9dc8e__20210827_20210928',
+ 10),
+ mksrc('datarepo-dev-86c60513',
+ 'hca_dev_1cd1f41ff81a486ba05b66ec60f81dcf__20210901_20210903',
+ 18),
+ mksrc('datarepo-dev-f5321179',
+ 'hca_dev_1ce3b3dc02f244a896dad6d107b27a76__20210827_20210903',
+ 421),
+ mksrc('datarepo-dev-76de829d',
+ 'hca_dev_2043c65a1cf84828a6569e247d4e64f1__20210831_20210907',
+ 1734),
+ mksrc('datarepo-dev-c808badb',
+ 'hca_dev_2086eb0510b9432bb7f0169ccc49d270__20210827_20210903',
+ 10),
+ mksrc('datarepo-dev-23782220',
+ 'hca_dev_23587fb31a4a4f58ad74cc9a4cb4c254__20210827_20210909',
+ 1476),
+ mksrc('datarepo-dev-4c3e6011',
+ 'hca_dev_248fcf0316c64a41b6ccaad4d894ca42__20210907_20210907',
+ 2958),
+ mksrc('datarepo-dev-1c2c69d9',
+ 'hca_dev_24c654a5caa5440a8f02582921f2db4a__20210830_20210903',
+ 3),
+ mksrc('datarepo-dev-38f08cd8',
+ 'hca_dev_2a64db431b554639aabb8dba0145689d__20210830_20210903',
+ 10),
+ mksrc('datarepo-dev-4cf05ce2',
+ 'hca_dev_2a72a4e566b2405abb7c1e463e8febb0__20210901_20210903',
+ 2290),
+ mksrc('datarepo-dev-3041c2cf',
+ 'hca_dev_2af52a1365cb4973b51339be38f2df3f__20210830_20210903',
+ 10),
+ mksrc('datarepo-dev-311340f6',
+ 'hca_dev_2d8460958a334f3c97d4585bafac13b4__20210902_20210907',
+ 3589),
+ mksrc('datarepo-dev-766bfb76',
+ 'hca_dev_2ef3655a973d4d699b4121fa4041eed7__20210827_20210903',
+ 8),
+ mksrc('datarepo-dev-1720b3c5',
+ 'hca_dev_379ed69ebe0548bcaf5ea7fc589709bf__20210827_20210902',
+ 4),
+ mksrc('datarepo-dev-ac6efd3f',
+ 'hca_dev_38449aea70b540db84b31e08f32efe34__20210830_20210903',
+ 42),
+ mksrc('datarepo-dev-40283c27',
+ 'hca_dev_3a69470330844ece9abed935fd5f6748__20210901_20210903',
+ 125),
+ mksrc('datarepo-dev-b08233fa',
+ 'hca_dev_3cfcdff5dee14a7ba591c09c6e850b11__20210827_20210903',
+ 8),
+ mksrc('datarepo-dev-bdc9f342',
+ 'hca_dev_3e329187a9c448ec90e3cc45f7c2311c__20210901_20210903',
+ 1001),
+ mksrc('datarepo-dev-ec07c8d8',
+ 'hca_dev_4037007b0eff4e6db7bd8dd8eec80143__20210831_20210903',
+ 39),
+ mksrc('datarepo-dev-c0ec174a',
+ 'hca_dev_403c3e7668144a2da5805dd5de38c7ff__20210827_20210903',
+ 63),
+ mksrc('datarepo-dev-31b3553a',
+ 'hca_dev_414accedeba0440fb721befbc5642bef__20210827_20210903',
+ 4),
+ mksrc('datarepo-dev-b4789901',
+ 'hca_dev_41fb1734a121461695c73b732c9433c7__20210830_20210903',
+ 12),
+ mksrc('datarepo-dev-4e5ffd52',
+ 'hca_dev_42d4f8d454224b78adaee7c3c2ef511c__20210830_20210903',
+ 9),
+ mksrc('datarepo-dev-5ef7f2e2',
+ 'hca_dev_455b46e6d8ea4611861ede720a562ada__20210901_20210903',
+ 74),
+ mksrc('datarepo-dev-a6c6b953',
+ 'hca_dev_4bec484dca7a47b48d488830e06ad6db__20210830_20210903',
+ 14),
+ mksrc('datarepo-dev-f31edbc2',
+ 'hca_dev_4d6f6c962a8343d88fe10f53bffd4674__20210901_20210903',
+ 12),
+ mksrc('datarepo-dev-bb8fbae4',
+ 'hca_dev_51f02950ee254f4b8d0759aa99bb3498__20210827_20210928',
+ 6),
+ mksrc('datarepo-dev-71de019e',
+ 'hca_dev_520afa10f9d24e93ab7a26c4c863ce18__20210827_20210928',
+ 649),
+ mksrc('datarepo-dev-ffcf8b00',
+ 'hca_dev_52b29aa4c8d642b4807ab35be94469ca__20210830_20210903',
+ 467),
+ mksrc('datarepo-dev-f76414c8',
+ 'hca_dev_52d10a60c8d14d068a5eaf0d5c0d5034__20210827_20210902',
+ 176),
+ mksrc('datarepo-dev-319b80f7',
+ 'hca_dev_577c946d6de54b55a854cd3fde40bff2__20210827_20210903',
+ 7),
+ mksrc('datarepo-dev-279f1986',
+ 'hca_dev_5ee710d7e2d54fe2818d15f5e31dae32__20210901_20210903',
+ 41),
+ mksrc('datarepo-dev-0abea017',
+ 'hca_dev_6072616c87944b208f52fb15992ea5a4__20210827_20210902',
+ 603),
+ mksrc('datarepo-dev-da221b1a',
+ 'hca_dev_60ea42e1af4942f58164d641fdb696bc__20210827_20210903',
+ 1145),
+ mksrc('datarepo-dev-c3d623dc',
+ 'hca_dev_63b5b6c1bbcd487d8c2e0095150c1ecd__20210830_20210903',
+ 11),
+ mksrc('datarepo-dev-ecb9c129',
+ 'hca_dev_67a3de0945b949c3a068ff4665daa50e__20210827_20210903',
+ 732),
+ mksrc('datarepo-dev-3545971c',
+ 'hca_dev_71436067ac414acebe1b2fbcc2cb02fa__20210827_20210928',
+ 4),
+ mksrc('datarepo-dev-12b7a9e1',
+ 'hca_dev_7880637a35a14047b422b5eac2a2a358__20210901_20210903',
+ 366),
+ mksrc('datarepo-dev-7913b094',
+ 'hca_dev_78b2406dbff246fc8b6120690e602227__20210827_20210902',
+ 216),
+ mksrc('datarepo-dev-4747d8fe',
+ 'hca_dev_7adede6a0ab745e69b67ffe7466bec1f__20210830_20210903',
+ 1601),
+ mksrc('datarepo-dev-aba01389',
+ 'hca_dev_7b947aa243a74082afff222a3e3a4635__20210831_20210907',
+ 7),
+ mksrc('datarepo-dev-bf3a4c8a',
+ 'hca_dev_8185730f411340d39cc3929271784c2b__20210830_20210903',
+ 12),
+ # FIXME: https://github.com/HumanCellAtlas/dcp2/issues/48
+ None and mksrc('datarepo-dev-a86993e6',
+ 'hca_dev_83f5188e3bf749569544cea4f8997756__20210929_20210929',
+ 1612),
+ mksrc('datarepo-dev-bd995e95',
+ 'hca_dev_842605c7375a47c59e2ca71c2c00fcad__20210830_20210903',
+ 8),
+ mksrc('datarepo-dev-d0772077',
+ 'hca_dev_8787c23889ef4636a57d3167e8b54a80__20210827_20210903',
+ 3),
+ mksrc('datarepo-dev-8eb2ffd1',
+ 'hca_dev_87d52a86bdc7440cb84d170f7dc346d9__20210830_20210903',
+ 16),
+ mksrc('datarepo-dev-0c5c20b5',
+ 'hca_dev_8c3c290ddfff4553886854ce45f4ba7f__20210902_20210907',
+ 6640),
+ mksrc('datarepo-dev-a198b032',
+ 'hca_dev_90bd693340c048d48d76778c103bf545__20210827_20210903',
+ 2245),
+ mksrc('datarepo-dev-59d37b9a',
+ 'hca_dev_946c5add47d1402a97bba5af97e8bce7__20210831_20210903',
+ 149),
+ mksrc('datarepo-dev-788c3b52',
+ 'hca_dev_955dfc2ca8c64d04aa4d907610545d11__20210831_20210903',
+ 13),
+ mksrc('datarepo-dev-4b88b45b',
+ 'hca_dev_962bd805eb894c54bad2008e497d1307__20210830_20210903',
+ 28),
+ mksrc('datarepo-dev-02c59b72',
+ 'hca_dev_99101928d9b14aafb759e97958ac7403__20210830_20210903',
+ 1190,
+ ma=1),
+ mksrc('datarepo-dev-a6312a94',
+ 'hca_dev_992aad5e7fab46d9a47ddf715e8cfd24__20210830_20210903',
+ 41),
+ mksrc('datarepo-dev-75589244',
+ 'hca_dev_996120f9e84f409fa01e732ab58ca8b9__20210827_20210903',
+ 26),
+ mksrc('datarepo-dev-d4b988d6',
+ 'hca_dev_a004b1501c364af69bbd070c06dbc17d__20210830_20210903',
+ 16,
+ ma=1),
+ mksrc('datarepo-dev-9ec7beb6',
+ 'hca_dev_a29952d9925e40f48a1c274f118f1f51__20210827_20210902',
+ 26),
+ mksrc('datarepo-dev-d3d5bbfa',
+ 'hca_dev_a39728aa70a04201b0a281b7badf3e71__20210830_20210903',
+ 33),
+ mksrc('datarepo-dev-7b7daff7',
+ 'hca_dev_a96b71c078a742d188ce83c78925cfeb__20210827_20210902',
+ 6),
+ mksrc('datarepo-dev-58610528',
+ 'hca_dev_a9c022b4c7714468b769cabcf9738de3__20210827_20210903',
+ 23),
+ mksrc('datarepo-dev-1dce87e5',
+ 'hca_dev_ae71be1dddd84feb9bed24c3ddb6e1ad__20210916_20210916',
+ 3515),
+ mksrc('datarepo-dev-b2004d1c',
+ 'hca_dev_b4a7d12f6c2f40a39e359756997857e3__20210831_20210903',
+ 24),
+ mksrc('datarepo-dev-0b465564',
+ 'hca_dev_b51f49b40d2e4cbdbbd504cd171fc2fa__20210830_20210903',
+ 193),
+ mksrc('datarepo-dev-cd97e83a',
+ 'hca_dev_b963bd4b4bc14404842569d74bc636b8__20210827_20210928',
+ 2),
+ mksrc('datarepo-dev-376d3f4a',
+ 'hca_dev_bd40033154b94fccbff66bb8b079ee1f__20210901_20210903',
+ 18),
+ mksrc('datarepo-dev-aa783adb',
+ 'hca_dev_c5f4661568de4cf4bbc2a0ae10f08243__20210827_20210928',
+ 1),
+ mksrc('datarepo-dev-990234a8',
+ 'hca_dev_c6ad8f9bd26a4811b2ba93d487978446__20210827_20210903',
+ 639),
+ mksrc('datarepo-dev-61b8d081',
+ 'hca_dev_c715cd2fdc7c44a69cd5b6a6d9f075ae__20210827_20210902',
+ 9),
+ mksrc('datarepo-dev-71926fdc',
+ 'hca_dev_c893cb575c9f4f26931221b85be84313__20210901_20210903',
+ 20),
+ mksrc('datarepo-dev-2f4bfe5d',
+ 'hca_dev_ccd1f1ba74ce469b9fc9f6faea623358__20210827_20210902',
+ 222),
+ mksrc('datarepo-dev-aa8357fb',
+ 'hca_dev_ccef38d7aa9240109621c4c7b1182647__20210831_20210903',
+ 1314),
+ mksrc('datarepo-dev-24e672db',
+ 'hca_dev_cddab57b68684be4806f395ed9dd635a__20210831_20210907',
+ 2545),
+ mksrc('datarepo-dev-f4cb2365',
+ 'hca_dev_ce33dde2382d448cb6acbfb424644f23__20210827_20210928',
+ 189),
+ mksrc('datarepo-dev-0d6f73ac',
+ 'hca_dev_d012d4768f8c4ff389d6ebbe22c1b5c1__20210827_20210903',
+ 8),
+ mksrc('datarepo-dev-5674b4eb',
+ 'hca_dev_d2111fac3fc44f429b6d32cd6a828267__20210830_20210903',
+ 735),
+ mksrc('datarepo-dev-b3632667',
+ 'hca_dev_d3446f0c30f34a12b7c36af877c7bb2d__20210901_20210903',
+ 40),
+ mksrc('datarepo-dev-92c3a1de',
+ 'hca_dev_d3a4ceac4d66498497042570c0647a56__20210831_20210903',
+ 14),
+ mksrc('datarepo-dev-4e88d60b',
+ 'hca_dev_d3ac7c1b53024804b611dad9f89c049d__20210827_20210928',
+ 11),
+ mksrc('datarepo-dev-dbc582d9',
+ 'hca_dev_dbcd4b1d31bd4eb594e150e8706fa192__20210827_20210902',
+ 84),
+ mksrc('datarepo-dev-848e2d4f',
+ 'hca_dev_dbd836cfbfc241f0983441cc6c0b235a__20210827_20210902',
+ 1),
+ mksrc('datarepo-dev-d7517bce',
+ 'hca_dev_dc1a41f69e0942a6959e3be23db6da56__20210827_20210902',
+ 5),
+ mksrc('datarepo-dev-27ad01e5',
+ 'hca_dev_df88f39f01a84b5b92f43177d6c0f242__20210827_20210928',
+ 1),
+ mksrc('datarepo-dev-b839d6c7',
+ 'hca_dev_e526d91dcf3a44cb80c5fd7676b55a1d__20210902_20210907',
+ 606),
+ mksrc('datarepo-dev-3faef568',
+ 'hca_dev_e5d455791f5b48c3b568320d93e7ca72__20210827_20210903',
+ 8),
+ mksrc('datarepo-dev-e304a8fe',
+ 'hca_dev_e77fed30959d4fadbc15a0a5a85c21d2__20210830_20210903',
+ 333),
+ mksrc('datarepo-dev-6fdac3db',
+ 'hca_dev_e8808cc84ca0409680f2bba73600cba6__20210902_20210907',
+ 898),
+ mksrc('datarepo-dev-dbc3e131',
+ 'hca_dev_eaefa1b6dae14414953b17b0427d061e__20210827_20210903',
+ 385),
+ # FIXME: https://github.com/HumanCellAtlas/dcp2/issues/45
+ None and mksrc('datarepo-dev-6883f2a5',
+ 'hca_dev_f48e7c39cc6740559d79bc437892840c__20210830_20210929',
+ 14),
+ mksrc('datarepo-dev-10f0610a',
+ 'hca_dev_f81efc039f564354aabb6ce819c3d414__20210827_20210903',
+ 4),
+ mksrc('datarepo-dev-24e9529e',
+ 'hca_dev_f83165c5e2ea4d15a5cf33f3550bffde__20210901_20210908',
+ 7663),
+ mksrc('datarepo-dev-67240cf2',
+ 'hca_dev_f86f1ab41fbb4510ae353ffd752d4dfc__20210901_20210903',
+ 20),
+ mksrc('datarepo-dev-e8e0a59a',
+ 'hca_dev_f8aa201c4ff145a4890e840d63459ca2__20210901_20210903',
+ 384),
+ mksrc('datarepo-dev-96d8e08c',
+ 'hca_dev_faeedcb0e0464be7b1ad80a3eeabb066__20210831_20210903',
+ 62),
+ ]))}
+ for prefix, internal in [
('', False),
('it0', True)
]
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hannes-ucsc the source that pass in ma=1
arguments and 2 problematic None and mksrc()
lines still wont fit under the 120 char limit with the suggested changes, should we just make an exception to the 4 lines that go over this 120 char length limit?
make format pep8
does not attempt to automatically format or throw a warning for these lines.
When using the Pycharm formatter it attempts to use aligned wraps, but not for all arguments (against contributing guidelines)...
# FIXME: https://github.com/HumanCellAtlas/dcp2/issues/48
None and mksrc('datarepo-dev-a86993e6',
'hca_dev_83f5188e3bf749569544cea4f8997756__20210929_20210929', 1612),
mksrc('datarepo-dev-bd995e95', 'hca_dev_842605c7375a47c59e2ca71c2c00fcad__20210830_20210903', 8),
mksrc('datarepo-dev-d0772077', 'hca_dev_8787c23889ef4636a57d3167e8b54a80__20210827_20210903', 3),
mksrc('datarepo-dev-8eb2ffd1', 'hca_dev_87d52a86bdc7440cb84d170f7dc346d9__20210830_20210903', 16),
mksrc('datarepo-dev-0c5c20b5', 'hca_dev_8c3c290ddfff4553886854ce45f4ba7f__20210902_20210907', 6640),
mksrc('datarepo-dev-a198b032', 'hca_dev_90bd693340c048d48d76778c103bf545__20210827_20210903', 2245),
mksrc('datarepo-dev-59d37b9a', 'hca_dev_946c5add47d1402a97bba5af97e8bce7__20210831_20210903', 149),
mksrc('datarepo-dev-788c3b52', 'hca_dev_955dfc2ca8c64d04aa4d907610545d11__20210831_20210903', 13),
mksrc('datarepo-dev-4b88b45b', 'hca_dev_962bd805eb894c54bad2008e497d1307__20210830_20210903', 28),
mksrc('datarepo-dev-02c59b72', 'hca_dev_99101928d9b14aafb759e97958ac7403__20210830_20210903', 1190,
ma=1),
13aeec0
to
535db5d
Compare
self note : 648b58b passes IT, used sandbox config. |
648b58b
to
0d00fb6
Compare
deployments/prod/environment.py
Outdated
( | ||
'dcp9', | ||
( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rule of thumb: lists for repeating elements of a homogeneous type, tuples for non-repeating structures of heterogeneous types.
This can be compressed a bit:
Index: deployments/prod/environment.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/deployments/prod/environment.py b/deployments/prod/environment.py
--- a/deployments/prod/environment.py (revision 0d00fb640605b9b84aa35920906615eaa52c71b4)
+++ b/deployments/prod/environment.py (date 1634017810974)
@@ -41,52 +41,40 @@
'AZUL_CATALOGS': json.dumps({
f'{name}': dict(atlas=atlas,
- internal=internal,
+ internal=bool(i),
plugins=dict(metadata=dict(name='hca'),
repository=dict(name='tdr')),
sources=sources)
- for catalog_type, atlas, sources in [
+ for atlas, names, sources in [
(
- (
- ('dcp9', False),
- ('it9', True)
- ),
'hca',
+ ['dcp9', 'it9'],
[
'tdr:tdr-fp-546ade29:snapshot/hca_prod_20201120_dcp2___20210910_dcp9:',
]
),
(
- (
- ('dcp10', False),
- ('it10', True)
- ),
'hca',
+ ['dcp10', 'it10'],
[
'tdr:tdr-fp-43194825:snapshot/hca_prod_20201120_dcp2___20211004_dcp10:'
]
),
(
- (
- ('dcp1', False),
- ('it1', True)
- ),
'hca',
+ ['dcp1', 'it1'],
[
'tdr:broad-datarepo-terra-prod-hca2:snapshot/hca_prod_20201118_dcp1___20201209:',
]
),
(
- (
- ('lungmap', False),
- ('it0lungmap', True)
- ),
'lungmap',
+ ['lungmap', 'it0lungmap'],
[
'tdr:tdr-fp-9f58860b:snapshot/hca_prod_00f056f273ff43ac97ff69ca10e38c89__20211004_lungmap_20211005:'
]
)
- ] for name, internal in catalog_type
+ ] for i, name in enumerate(names)
}),
'AZUL_PARTITION_PREFIX_LENGTH': '2',
0d00fb6
to
920b665
Compare
ec4a0f9
to
29405b8
Compare
29405b8
to
3e9e48e
Compare
This PR is |
3e9e48e
to
b4075d4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pro-forma approval in order to merge into develop
Split off from #3277
#3114
Author
Fix:
for bugs) or there is a good reason why they're differentAuthor (reindex)
r
tag to commit title or this PR does not require reindexingreindex
label to PR or this PR does not require reindexingAuthor (freebies & chains)
chain
label to the blocking PR or this PR is not chained to another PRAuthor (upgrading)
u
tag to commit title or this PR does not require upgradingupgrade
label to PR or this PR does not require upgradingAuthor (requirements, before every review)
make requirements_update
or this PR leaves requirements*.txt, common.mk and Makefile untouchedR
tag to commit title or this PR leaves requirements*.txt untouchedreqs
label to PR or this PR leaves requirements*.txt untouchedAuthor (before every review)
make integration_test
passes in personal deployment or this PR does not touch functionality that could break the ITdevelop
, squashed old fixupsPrimary reviewer (after approval)
no demo
no sandbox
Operator (before pushing merge the commit)
reindex
label andr
commit title tagno demo
sandbox
label or PR is labeledno sandbox
no sandbox
sandbox
or this PR does not require reindexingsandbox
sandbox
or this PR does not require reindexingsandbox
Operator (after pushing the merge commit)
N reviews
labelling is accurateOperator (reindex)
dev
or this PR does not require reindexing or does not targetdev
dev
or this PR does not require reindexing or does not targetdev
prod
or this PR does not require reindexing or does not targetprod
prod
or this PR does not require reindexing or does not targetprod
Operator