Milestone - Define EnvO value sets for supported environmental extensions (3.2)

[ssarrafan]

A key part of the schema is the allocation of different metadata elements to different environmental
packages (e.g., ‘depth’ is a required metadata element for soil and sediment samples, and conversely
‘altitude’ is required for aerial samples). In the Pilot, we directly adopted the MIxS environment packages,
and extended them with fields required by EMSL and JGI. While this provided a foundation, we identified
many areas where the MIxS environmental packages are too rigid, or are at suboptimal levels of
granularity. In collaboration with the GSC and the broader research community, we will support the
development of more specific packages for a variety of ecosystems (e.g., environments like wetlands,
mangroves or complex riparian systems should have their own package extensions, and the schema allows
for progressive refinement or crossing of packages), and continue to improve existing packages based on
community feedback. To address a common community challenge in navigating ontologies, each of these
environmental packages will be supported by defined EnvO value sets (cross-sections of the ontology with
key terms relevant for a specific environment) such that data submitters can provide precise and accurate
descriptive terms through a simple dropdown, without having to navigate the whole EnvO structure
(Submission Portal, Milestone 3.2).
Page 28

see #469 #470 #471

[mslarae13]

This should be environmental extensions. We need to make this correction across lots of things.

Do we have target extensions? Is it all the ones currently on the subport? (Which is basically all)

[ssarrafan]

This should be environmental extensions. We need to make this correction across lots of things.

Do we have target extensions? Is it all the ones currently on the subport? (Which is basically all)

@cmungall can you respond to Montana's questions please.

[cmungall]

We should prioritize BER-relevant ones

[ssarrafan]

This is a Q4 milestone. Updating issue to Q4.

[aclum]

@mslarae13 and @turbomam will get together to discuss this.

[mslarae13]

We should prioritize BER-relevant ones

So,
HIGH

• SoilInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin SoilMixsInspiredMixin]
• PlantAssociatedInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]
• SedimentInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]
• WaterInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]

LOW

• AirInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]
• BiofilmInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]
• BuiltEnvInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]
• HcrCoresInterface [ DhMultiviewCommonColumnsMixin]
• HcrFluidsSwabsInterface [ DhMultiviewCommonColumnsMixin]
• HostAssociatedInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]
• MiscEnvsInterface [ DhMultiviewCommonColumnsMixin]
• WastewaterSludgeInterface [ DhMultiviewCommonColumnsMixin SampIdNewTermsMixin]

[aclum]

We can exclude the user facility interface tabs.

[turbomam]

Thanks @aclum and @mslarae13 for tending to this. I have been thinking about different ways to keep track of our intentions, the implementations, and whether a value set is complete. There's probably no one perfect way of doing it.

I think we should decide

• do we have some value sets that can be considered exemplars to guide the ones we haven't done yet
• will we need to use classes from any ontologies other than EnvO and PO for the environments that have been marekd as high priority?
• do we anticipate that we may need to request some classes in EnvO in order to build value sets with adequate coverage for our collaborators?
  • where can we find the kinds of values we will need to cover? Values that are already in MongoDB? Values that appear in submissions but haven't made their way into MongoDB yet? Our Postgres representation of NCBI Biosamples?
• beyond the exemplars above, what technologies will we use to generate these value sets (including expert human review)
• what practices will we use to refine them in the future
• what file in what repo will serve as the source of truth for the value sets
• (somewhat independent) how will the value sets will be presented to Submission Portal users, or how the input form the submitters will be validated

I have created a table that relates @mslarae13 's recent prioritization list with some other knowledge about the environments/Extensions/DH Interfaces. I would like to include most of this information in whatever progress tracking system we use. Since the table is wide, maybe we should move it to a Google Sheet or a repo-checked-in TSV, instead of embedding it in an issue like this.

MixS Environment name	submission portal DhInterface name	harmonizerApi.ts status	priority in #468 (comment)	env_broad_scale	env_local_scale	env_medium
HumanAssociated		disabled
HumanGut		disabled
HumanOral		disabled
HumanSkin		disabled
HumanVaginal		disabled
PlantAssociated	PlantAssociatedInterface	published	high
Sediment	SedimentInterface	published	high
Soil	SoilInterface	published	high
Water	WaterInterface	published	high
Air	AirInterface	published	low
BuiltEnvironment	BuiltEnvInterface	published	low
HostAssociated	HostAssociatedInterface	published	low
HydrocarbonResourcesCores	HcrCoresInterface	published	low
HydrocarbonResourcesFluidsSwabs	HcrFluidsSwabsInterface	published	low
MicrobialMatBiofilm	BiofilmInterface	published	low
MiscellaneousNaturalOrArtificialEnvironment	MiscEnvsInterface	published	low
WastewaterSludge	WastewaterSludgeInterface		low
Agriculture
FoodAnimalAndAnimalFeed
FoodFarmEnvironment
FoodFoodProductionFacility
FoodHumanFoods
SymbiontAssociated

[turbomam]

@pkalita-lbl you can see that I have tracked the DhInterface name from submission-schema/schemasheets/tsv_in/classes.tsv and the status from harmonizerApi.ts in my table above

I didn't include the excel_worksheet_name annotations form your new

• Add excel_worksheet_name annotations to DhInterface subclasses submission-schema#215

but the table is intended to do some of the mapping that we have been talking about.

I'm a little surprised that WastewaterSludgeInterface appears many places in the submission-schema repo (and @mslarae13 included it in her prioritization list, albeit as low) but it doesn't appear in harmonizerApi.ts

[aclum]

will we need to use classes from any ontologies other than EnvO and PO for the environments that have been marekd as high priority?

• I'd say no. UBERON was the other ontology Chris wanted to allow but it is not applicable to the high priority environments.

[mslarae13]

Updating @turbomam 's table (IN PROGRESS)

MixS Environment name	submission portal DhInterface name	harmonizerApi.ts status	priority in #468 (comment)	env_broad_scale	env_local_scale	env_medium
HumanAssociated		disabled
HumanGut		disabled
HumanOral		disabled
HumanSkin		disabled
HumanVaginal		disabled
PlantAssociated	PlantAssociatedInterface	published	high
Sediment	SedimentInterface	published	high
Soil	SoilInterface	published	high
Water	WaterInterface	published	high
Air	AirInterface	published	low
BuiltEnvironment	BuiltEnvInterface	published	low
HostAssociated	HostAssociatedInterface	published	low
HydrocarbonResourcesCores	HcrCoresInterface	published	low
HydrocarbonResourcesFluidsSwabs	HcrFluidsSwabsInterface	published	low
MicrobialMatBiofilm	BiofilmInterface	published	low
MiscellaneousNaturalOrArtificialEnvironment	MiscEnvsInterface	published	low
WastewaterSludge	WastewaterSludgeInterface	unpublished (need to add)	low
Agriculture			high
FoodAnimalAndAnimalFeed			low
FoodFarmEnvironment			low
FoodFoodProductionFacility			low
FoodHumanFoods			low
SymbiontAssociated			low

[mslarae13]

The following extensions are NOT in NMDC. I'm not sure why, and we need to check what version of MIxS we're using. I'll make a separate issue for that. but for this milestone & the squad addressing it, we'll skip these extensions

Agriculture
FoodAnimalAndAnimalFeed
FoodFarmEnvironment
FoodFoodProductionFacility
FoodHumanFoods
SymbiontAssociated

Edit, this issue exists, which is similar. nmdc submission-schema and nmdc-schema don't seem to be aware of slots that are unique to these extensions. Making me conclude we don't use v6.

• Does nmdc-server know about all current MIxS environmetal extensions, which COULD be HARMONIZER_TEMPLATES? nmdc-server#1238

[ssarrafan]

@mslarae13 @aclum @cmungall thanks for all the updates on this issue. Will this be done by September? This is due this quarter.

[mslarae13]

Will this be done by September? This is due this quarter.

@ssarrafan that's the goal

[ssarrafan]

Per @cmungall Patrick is not needed for this issue. Discussed at meeting today with Alicia, Emiley, Chris.
FYI @mslarae13

[mslarae13]

@ssarrafan due date for this is still end of September, right?

[ssarrafan]

@ssarrafan due date for this is still end of September, right?

Yes so far.

[mslarae13]

We discussed env_broad_scale for soil this week.
We'll work on medium next week.

I would like to add some clarity around what we might mean by done :) Because complete and in production is no longer likely by Monday (since that's when we're pushing to production)

• By end of september, Env_broad, local, and medium terms will be identified for SOIL from manual evaluation of OAK queries
• Broad and medium OAK queries will be updated for SOIL based on the evaluation (@turbomam does that sound doable? By Sept end)

Early October

• Env local will be evaluated for SOIL & OAK query updated
• ENV triad enums will be updated to match those that are provided by the query for SOIL
• OCTOBER production release will include these queries.

EARLY NOVEMBER

• start query eval for sediment

@ssarrafan @emileyfadrosh @cmungall @lamccue
Unfortunately, this does mean the milestone won't be complete by September. IF we can crank through local by the end of the month, then it's kinda done, just isn't on production.

Can we reschedule this milestone?

[ssarrafan]

Discussed this milestone with @turbomam today and he said he would bring it up at the squad meeting on Wednesday. Montana has already created great sub-issues that can be linked here. They will discuss how much more review is needed on Wednesday and then update this milestone with the new estimated timeline.
FYI @mslarae13 @cmungall

[mslarae13]

@ssarrafan @turbomam @cmungall
See above comment for the new time line. I already sorted this out. Unless early october isn't still do-able, but the release is at the end of october, and we wouldn't make the change until post berkeley-rollout any way.

[sierra-moxon]

Updates for DOE report, replicated in part from https://github.com/microbiomedata/issues/blob/ADR-define-envo-value-sets/decisions/0015-env-triad-terms.md

• env_broad_scale will consist of subclasses of biome [ENVO:00000428], with the exception of host-associated samples, including plant-associated samples.
• env_medium will consist of subclasses environmental material [ENVO:00010483]
• env_local_scale will consist of subclasses of material entity [BFO:0000040], minus biome [ENVO:00000428] and environmental material [ENVO:00010483]
  • It's expected that additional branches of the is_a hierarchy will subtracted, like chemical entity [CHEBI:24431]
• obsolete terms will be excluded

For soil environment specifically: (MIxS Extension)

• env_broad_scale will exclude aquatic biome [ENVO:00002030]
• env_local_scale (in progress)
• env_medium (in progress) - will descend from "soil" term

In addition to these programmatic extractions, we collate metadata, including whether the term has been used in NCBI or GOLD biosamples and whether it is currently used at NMDC directly via biosample submission. Our team of ENVO oncologists, data analysts, and subject matter experts in the field then review the resulting lists generated programmatically. They use their knowledge to constrain the lists further to value sets small enough for users to navigate in submissions.

[ssarrafan]

DOE report today update and moving to Q2:
The soil value sets have been completed and are implemented in the Submission Portal. Sediment and water values are near complete, while the plant value sets require additional terms from the Plant Ontology (PO) that require some modifications to the process. As part of the work this quarter, we refactored the code that automatically generated candidate value sets from queries and mappings, making the overall process more transparent and reproducible. Anticipated competition FY25 Q2

[sierra-moxon]

This is slated to be complete on Friday Jan 31, 2025