Y25-078 - Set up traction training environment to connect with MLWH training

[dasunpubudumal]

User story

As developers, we need to set up traction messaging setup in the training environment. As of now, no application is pushing data to the newly set up MLWH training. Therefore, as our first step, we would like Traction training to use MLWH training.

Because we did not have a messaging infrastructure set up in our traction training environment, and because we did not have a MLWH training database, we could not have certain events such as volume tracking (see "Additional Context") in Traction traced through MLWH. Therefore, we would need to set the end-to-end messaging configuration in traction training using the infrastructure that has now been set up.

When the dedicated Rabbit broker was set up¹, we have updated the traction configuration in training environment to use the dedicated broker (managed by PSD) to be used for volume tracking purposes. However, there seem to be some missing link and because of that, tol-lab-share in training environment fails to consume the messages. This will have to be investigated and fixed as part of this story.

Tip

Seems like Traction fails to push messages to the dedicated broker in training. Note that the same setup works fine in UAT, that uses the same dedicated broker. Check if enabled in bunny.yml is properly being set in training.

Also, change the following configs to use the dedicated broker only in the training environment (see roles/deploy_traction_service/templates/bunny.yml.j2):

  ...
  {% if rails_env != "training" %}
  enabled: true
  {% else %}
  enabled: false
  {% endif %}
  broker_host: localhost
  broker_port: 5672
  {% if rails_env == "production" %}
  broker_username: ...
  broker_password: ...
  vhost: production
  routing_key: production.traction.message.#
  {% else %}
  ...

Important

This story concerns only with the training environment. No configs of other environments should be affected.

Who are the primary contacts for this story

Dasun P, Stephen I, Seena N, Ben T,

Who is the nominated tester for UAT

PSD

Acceptance Criteria

• Volume tracking messages going through from traction training to MLWH should be persisted in training MLWH aliquot table.
• Other messages going through from traction (through psd.traction.to-warehouse queue) to MLWH should be persisted in the correct tables in training MLWH.

Additional Context

If messages are going through RabbitMQ but aren't getting consumed, chances are that there is a setup misconfiguration in the new unified_warehouse instance hosted at ware-training.psd.sanger.ac.uk. It might be worth having a look at the configuration in the deployment project if that is the case.

Note

Volume Tracking

Events such as library, pool and run creation and updates are recorded in MLWH's aliquot table. We aggregate these records accordingly to inquire certain queries such as how much volume is left in a certain library. This information is made visible to the other teams and stakeholders through MLWH. For more context, please refer to the volume tracking documentation.

Footnotes

1. The DNS for the link contains -uat suffix, which is a bit misleading. It suggests that the dedicated broker is being used in UAT. However, this is to be used in both the environments UAT and Training. We will update the DNS record later with a suffix like -dev or -test. ↩

[dasunpubudumal]

I kept seeing the following message in training instance of unified_warehouse when I updated a library:

Feb 20 15:18:20 ware-training-jammy unified_warehouse[3225232]: (thread-6200) [WIP:training]  WARN -- : 🦊 training_MultiLimsWarehouseConsumer_3225232: Delay Exception: No study with uuid 'c6ba2a7e-7763-11ef-8efc-02423c45b5b8'
Feb 20 15:18:50 ware-training-jammy unified_warehouse[3225232]: (thread-5800) [WIP:training]  WARN -- : 🦊 training_MultiLimsWarehouseConsumer_3225232: Delay: {"lims":"Traction","pac_bio_run":{"last_updated":"2025-02-20T15:12:19.779Z","id_pac_bio_run_lims":"TRACTION-RUN-1780","pac_bio_run_uuid":"954803cf-7e4b-4635-b515-6c053207f4c6","pac_bio_run_name":"TRACTION-RUN-1780","wells":[{"plate_number":1,"plate_uuid_lims":"f540e2eb-30bb-4a1c-b77f-da20c6b100bd","well_label":"A1","well_uuid_lims":"825fd59b-b927-4fde-b723-0e34cdf5ee15","samples":[{"cost_code":"S10691","pac_bio_library_tube_id_lims":15125,"pac_bio_library_tube_uuid":"","pac_bio_library_tube_name":"7877STDY15386487","sample_uuid":"f1c86c6c-e2ed-11ef-b6ac-024250f2bd16","study_uuid":"c6ba2a7e-7763-11ef-8efc-02423c45b5b8","tag_sequence":"CATCGC","tag_set_id_lims":13,"tag_identifier":"bcM0004","tag_set_name":"MAS_SMRTbell_barcoded_adapters_(v2)","pipeline_id_lims":"PacBio_IsoSeq_mplx","pac_bio_library_tube_barcode":"TRAC-2-13436"}]},{"plate_number":1,"plate_uuid_lims":"f540e2eb-30bb-4a1c-b77f-da20c6b100bd","well_label":"B1","well_uuid_lims":"21ae2bca-b472-4d34-bf4a-101e3de17ae4","samples":[{"cost_code":"S10691","pac_bio_library_tube_id_lims":15126,"pac_bio_library_tube_uuid":"","pac_bio_library_tube_name":"7877STDY15386488","sample_uuid":"f1d9890c-e2ed-11ef-b6ac-024250f2bd16","study_uuid":"c6ba2a7e-7763-11ef-8efc-02423c45b5b8","tag_sequence":"ACAGTC","tag_set_id_lims":13,"tag_identifier":"bcM0001","tag_set_name":"MAS_SMRTbell_barcoded_adapters_(v2)","pipeline_id_lims":"PacBio_IsoSeq_mplx","pac_bio_library_tube_barcode":"TRAC-2-13437"}]},{"plate_number":1,"plate_uuid_lims":"f540e2eb-30bb-4a1c-b77f-da20c6b100bd","well_label":"C1","well_uuid_lims":"44667da8-fc25-481d-a457-44143630ac4e","samples":[{"cost_code":"S10691","pac_bio_library_tube_id_lims":15127,"pac_bio_library_tube_uuid":"","pac_bio_library_tube_name":"7877STDY15386489","sample_uuid":"f1dc42be-e2ed-11ef-b6ac-024250f2bd16","study_uuid":"c6ba2a7e-7763-11ef-8efc-02423c45b5b8","tag_sequence":"ATGACG","tag_set_id_lims":13,"tag_identifier":"bcM0002","tag_set_name":"MAS_SMRTbell_barcoded_adapters_(v2)","pipeline_id_lims":"PacBio_IsoSeq_mplx","pac_bio_library_tube_barcode":"TRAC-2-13438"}]},{"plate_number":1,"plate_uuid_lims":"f540e2eb-30bb-4a1c-b77f-da20c6b100bd","well_label":"D1","well_uuid_lims":"f5e9386b-bbb0-496b-b5c8-50d40a9aea18","samples":[{"cost_code":"S10967","pac_bio_library_tube_id_lims":15173,"pac_bio_library_tube_uuid":"","pac_bio_library_tube_name":"K_PPI_t_c15413873","sample_uuid":"c2c36192-e4a5-11ef-b656-0242e38ee9b0","study_uuid":"c7f945fc-5ae7-11ef-a265-024293460e78","tag_sequence":null,"tag_set_id_lims":null,"tag_identifier":null,"tag_set_name":null,"pipeline_id_lims":"Pacbio_HiFi","pac_bio_library_tube_barcode":"TRAC-2-13502"}]}]}}

And this same message kept popping up into the queue again and again, which means that unified_warehouse re-queued the message even though there was an issue with the message. This might have to be fixed in a separate story (as it is not unique to the training instance). This is something we have seen in sanger/lab-share-lib#296 as well.