Test and release 5.x data model epic

[hslh]

Decision register

What version of EMX2 are you using (see footer)
v10.109.5-SNAPSHOT on preview server 4114

Describe the bug
I went through the manual tests for the flat data model on the preview server for #4114 . This is the resulting list of things to fix.

• ‘Variables’ is missing from the top ribbon for a network of networks. See $testCatalogue/testNetworkofNetworks. (test 1 step 8) #4234

• LATER ??? Eleanor design: Network page within a network of networks used to show information on Partners, Cohorts/Collections, Variables and Data Sources (or is that now Datasets?). EDIT: things in prod are also different to the testplan... so we need to agree on what we want here. See: $testCatalogue/testNetworkofNetworks/networks/testNetwork1. (test 1 step 11/test 2 step 15) #4340
  --- loslaten, networks laat zien gewoon wat er ingevuld is

• some types are shown incorrectly on catalogue startpage as 'resources' #4342

• fix(catalogue): resource details page not rendering when resource.peopleInvolved is empty #4347

• Eleanor design: ‘About’-page for a network has only website and description, most content is missing (e.g. partners, funding, variables, cohorts).
  See: $testCatalogue/ssr-catalogue/testNetworkofNetworks/about-catalogue. (test 1 step 13/test 2 step 11) To be fixed in fix(catalogue): fix resource detail view to use correct type; fix catalogue about page should show network page but navigate as about #4296 EDIT: The fields we see on the 'About' page for a network(of networks) still need to be decided upon. 'Population' doesn't strike me as something you can generalise on over a whole network, eg.
  --- loslaten, network of networks laat zien gewoon wat er ingevuld is

• fix(catalogue): menu on the left hand side of the cohort detail page doesn't always match the blocks on the right hand side - please bring to acc #4320 NOT BLOCKING BUT FOR NEXT SPRINT

• fix(catalogue): the ‘repeats’-icon is missing from variables (e.g. the little +10).  #4244

• On variable page, the ‘Definition’-section is called ‘Description’ in the sidebar and ‘Harmonisation Details Per Collection’ is called ‘Harmonisation Details Per Data Source’ in sidebar.
  See: $testCatalogue/ssr-catalogue/testNetwork1/variables/testVarRepeats_-testNetwork1-cdm_1-testNetwork1?keys={"name":"testVarRepeats_","collection":{"id":"testNetwork1"},"dataset":{"name":"cdm_1","collection":{"id":"testNetwork1”}}}. (test 4 step 10)

• fix(catalogue): On variable page, when you click a source variable under ‘Harmonisation Details Per Collection’, nothing happens, no dialog pops up.  #4246

• On cohort page, in the ‘General Design’-section, the ‘Cohort Design’, in this case ‘longitudinal’, isn’t displayed.
  See: $testCatalogue/ssr-catalogue/all/cohorts/testCohort2. (test 5 step 18)

• fix(catalogue): On variables overview page, filtering on variable name ‘asthma’ and selecting the 4 test cohorts doesn’t display any variables.  #4247

• On the network variables page, the harmonisation matrix is displayed even though the selected topic and cohort have no mappings. #4343

• On the catalogue contents page, the button on the top ribbon shows ‘Collections (0)’, even though there are 105 collections (the button works fine, though). See: $testCatalogue/ssr-catalogue/all/collections. (test 8 step 11) #4256

ISSUES FOUND IN VERSION v11.11.1-SNAPSHOT

• On the details page for a network (eg in preview: https://emx2.dev.molgenis.org/catalogue-demo/ssr-catalogue/testNetwork1/networks/testNetwork1) the 'chapter' headings on the left hand side are completely different to what we had before. (See comment below) (test 2 step 15) #4254

• Update remaining uses of 'cohort' to 'sources/resources/etc.' #4341

• The harmonisation matrix and the info per cohort don't match, for some reason. testvarrepeats in preview (here) gives different results in the matrix to what you see below when you click on testcohort3 in the harmonisation details per data source. Test plan 4 step 12, test plan 6 step 6. See comment below.
  --> test data fixen en dan controleren

• The contact page (behind button 'Contact' on the cohort detail page) doesn't seem to have the updates we made in the big release in July, i.e. pop-up with "Name of test cohort 2", Contact, Name entry field, Email entry field, Organisation entry field, Topic dropdown, Message entry field, "or contact us at: molgenis-support@umcg.nl", Send button" Test plan 5, step 6
  --> testen in acc (emailsettings nodig)
  EDIT: 16-10-24 this doesn't work in acc. Assume the email settings are still the same as before?
  We also need to change the email address in the test data to support@molgenis.org.

• The start/end year of cohort testcohort2 should be '1955-ongoing', not 'not available' Test plan 5, step 24, and test plan 9 step 10 (different years for cohort 1 but the same issue)
  --> bewust veranderd ivm DCAT

• The headings on the left hand side of the cohort page are missing 'Variables & Topics' in between Contributors and Available Data & Samples. eg on https://emx2.dev.molgenis.org/catalogue-demo/ssr-catalogue/all/cohorts/testCohort2 Test plan 5, step 11 --> Blok Variables & Topics wordt eerst weggehaald en daarna goed gedefinieerd en gespecificeerd.

• The text under 'Variables & Topics' is still a placeholder text and needs to be updated or removed. Test plan 5 (no test step for this new block yet)
  --> Blok Variables & Topics wordt eerst weggehaald en daarna goed gedefinieerd en gespecificeerd.

• The Main medical condition and Comorbidity items in the pop-up for a subcohort are presented in random order #4287

• Clicking on the arrow or the network name for information about a network opens page in the same window rather than in a new window. #4300

• Data access fee and prelinked fields not shown under Access conditions Test plan 5 step 57 (compare test plan 9 step 28, where they ARE shown because they are 'true'). --> TBD, see https://github.com/molgenis/GCC/issues/475

• Linkage options field is not shown under Access conditions on resource information page #4303

ISSUES FOUND IN ACC WITH V11.20.0

• network logos not shown on resource detail page: https://data-catalogue-acc.molgeniscloud.org/catalogue/ssr-catalogue/EHEN/cohorts/ABCD
• logo's networks not show on network homepage

ISSUES FOUND IN ACC WITH V11.16.2

• fix (catalogue)menu options aren't consistent with the buttons available on network-of-networks page #4322

ISSUES FOUND IN ACC WITH V11.20.6

• fix(catalogue): Menu ribbon is inconsistent across the different resource types. #4366

• On the variable detail page, the headings are 'harmonisation status per data source' and 'harmonisation details per data source'. Use of the words 'data source' strikes me as a little confusing (data sources are just one of the resource types which can have harmonised variables). Suggestion: change this to "harmonisation status per source of data" and "harmonisation details per source of data". Small difference but subtly quite important! eg https://data-catalogue-acc.molgeniscloud.org/testCatalogue/ssr-catalogue/testNetwork1/variables/testVarRepeats_-testNetwork1-cdm_1-testNetwork1?keys={%22name%22:%22testVarRepeats_%22,%22resource%22:{%22id%22:%22testNetwork1%22},%22dataset%22:{%22name%22:%22cdm_1%22,%22resource%22:{%22id%22:%22testNetwork1%22}}} . See also In variable detail view, 'harmonisation status per data source' should be changed to 'harmonisation status per source' #4386
  NOT BLOCKING BUT FOR NEXT SPRINT

• fix(catalogue): correction of the yy-mm-dd date format into yyyy-yyyy for start/end years needs to be carried out on the collection event, too #4367 NOT blocking - and also closed without change, the collection event dates can stay as they are

• fix(catalogue): the alignment of the fields under Access Conditions is weird and we've had comments from others about it. #4368 NOT blocking

• We have dropped the 'variables and topics' tile in the cohort (no bad thing) but we must implement a link to the cohort's source variables very soon. We've discussed and decided before, see spec here: https://umcgonline.sharepoint.com/:w:/r/sites/SGCohortcatalogues/Gedeelde%20documenten/Test%20Activities/Specs%20for%20feature%20requests/20240924%20Variables%20%26%20Topics.docx?d=w53fdf3aa9eb14212a2f8ef948c8197e9&csf=1&web=1&e=WU6Tce
  NOT BLOCKING BUT FOR NEXT SPRINT see stories GCC no. 144 and 711

ISSUES FOUND IN V11.25.0

• bug(catalogue): pagination doesn't work on the datasets tile on the cohort page #4413 NOT blocking
• design(catalogue): Network information page doesn't show resources or variables - spec required #4415 NOT BLOCKING BUT FOR NEXT SPRINT

ISSUES FOUND IN V11.25.1

• bug(catalogue): the network page no longer has participants/samples/longitudinal total figures (it does have total subpopulations) #4422 not cosmetically blocking but we do need to check that adding the layer 'collections' hasn't upset other functionality (as it seems to have done here)

ACTIONS FOR HESSEL, JORIS, ELEANOR AND OTHERS SEPARATE FROM SOFTWARE CHANGES AND NOT LISTED ABOVE

• get clarity about how (eg) the cohort page does or does not look the same as the network page --> which differences between resources do we need to test?
  --- we houden het bij cohorten pagina test met cohort dat zoveel mogelijk ingevuld is
• sort out the email settings in acc so that we can test the Contact page on a cohort
  --- doet Hessel <- note that this is a schema-level setting, so we'll have to repeat this every time we load a schema with a fresh set of demo data
• include design schematic in test data
  --- wait for merging and deployment of fix: add design schematic to demo data #4411
• decide on final design of All Resources page
  --- need a few suggestions for Morris to choose from
  --- 3 CTAs on the All Resources page: 1) Resources 2) Networks 3) Variables
  --- text under Resoures moet het uitleggen
  --- clicking on Resources --> pagina van bv cohort studies maar dan met extra filter om resource type te kiezen
  --- PR aanmaken
  --- Eleanor maakt opzet: here
• finish test plan (and test data) for All Resources page
• check test data for variable asthma_0 and Pelagie, whether it's correct (Test plan 7)
• finalise variable numbers displayed etc in test plans (reinstate old variables or not, is it worth it or do we test everything sufficiently as it is?)
  --- Eleanor checkt of de extra variabel nodig is DONE: the variable testVarLang2 which had been removed is not crucial: we have another variable with no mappings (testVarCategorical_) which tests when variables without mappings are/aren't shown.
  --- update the test plans using variables to remove mentions of testVarLang2, change mentions of testVarLang1 into mentions of testVarLang and revise the numbers of variables shown at various stages
• check data model of the test data in testCatalogue (warning messages about columns when uploading)
• test data for test plan 9: fill in multiple values where this hasn't happened yet
• Update testCatalogue schemata on dev and acc to reflect changes in test data
• Edit all the test plans to include the new navigation ("Collection CTA")
• go through all the test plans again once we think everything crucial has been fixed
• communicate with users about timing of the release (issue 747 in GCC repo)
• check that there are no blocking issues for UMCG before we go live

ISSUES in v25.0:

• fix(catalogue): There is a large white space underneath repeated variable mappings in the harmonisation matrix on the variable detail page #4419 not blocking: ugly white space in variable details page
• Table explorer in network staging cannot display Resources table #4264 networks staging cannot display resources table
• Year 0 harmonisation details check mark is displayed while there is no data for this mapping #4428: harmonisation details (syntax and such) for Year 0 are not shown NOT BLOCKING BUT FOR NEXT SPRINT
• fix(catalogue): attribute order and names of attributes in UI do not always match order and names in data model #4429: order and names of attributes do not always match the data model NOT BLOCKING BUT FOR NEXT SPRINT

[EleanorHyde-UMCG]

Going through the test plans again in preview to see if I can tick off any of the points made above (running into test data issues, though, for some points). I've found something else which I will add here below and at the bottom of the list above:

• On the details page for a network (eg in preview: https://preview-emx2-pr-4178.dev.molgenis.org/catalogue-demo/ssr-catalogue/testNetwork1/networks/testNetwork1) the 'chapter' headings on the left hand side are completely different to what we had before. What we had before was:

ACRONYM FOR TESTNETWORK1
Description
Partners
Funding & Acknowledgements
Cohorts
Data Sources
Variables

What we have now is:
ACRONYM FOR TESTNETWORK1
Description
General Design
Population
Organisations
Datasets
Networks
Publications
Funding & Acknowledgements

Items like 'general design' and 'population' sound more like the cohort details page, is this deliberate? Not sure we can make such sweeping statements about networks, which are often a combination of various sets of populations / designs etc.

[EleanorHyde-UMCG]

This item has been partially solved:

• On cohort page, the ‘Partners’-section is missing. The ‘Available Data & Samples’-section is there, but it is missing from the sidebar. ’Networks’-section is called ‘Networks’ in sidebar, but ‘Part of Networks’ in the main content screen.

See: $testCatalogue/ssr-catalogue/testNetwork1/cohorts/testCohort2. (test 3 step 6b/test 5 step 11, 53/test 9 step 26)

Before the big release in July, we absorbed Partners into Contributors (but unfortunately omitted to adjust this test plan 3). However, Organisations should also have been absorbed into Contributors. What we currently have in prod is:

Description
General Design
Population
Contributors
Available Data & Samples
Subpopulations
Collection Events
Datasets
Networks
Publications
Access Conditions
Funding & Acknowledgements
Documentation

So what still needs to be fixed for this item is:

• remove Organisations as a separate heading on the cohort detail page and absorb it into Contributors (as it is in prod)
• make the heading/menu item for Networks / Part of Networks consistent ('part of networks' is new and may have been decided upon within the flat data model group?)

@BrendaHijmans

[EleanorHyde-UMCG]

Pop-up for 'about statuses' in the harmonisation matrix (eg on a variable detail page such as this in preview) needs to be brought in line with the new labels for what used to be labelled 'cohorts'. The current text is this:

Pop up should show 'completed: cohort was able to fully map to the harmonised variables'; 'partially: cohort was able to partially map to the harmonised variable'; 'No data: no harmonisation data is available'.

Test plan 4 step 11

I've added a new item in the list above.

[EleanorHyde-UMCG]

The harmonisation matrix and the info per cohort don't match, for some reason. testvarrepeats in preview (here) gives different results in the matrix to what you see below when you click on testcohort3 in the harmonisation details per data source. Test plan 4 step 12, test plan 6 step 6.

Have added this to the list above.

[EleanorHyde-UMCG]

Data banks and data sources are included in the harmonisation matrix for a network. Do we expect them to have harmonised variables? (test plan 4 step 15, test plan 7 step 33)

@BrendaHijmans

[EleanorHyde-UMCG]

The number of variables for a network is incorrect (I suspect a change made for the network-of-networks works has something to do with this). The test plan has been changed since the big release in July, also for the network of networks, to include another variable ga_lmp (under testcohort2) but this is not picked up in preview.

Overview page of network --> number of variables should be 8, not 7. Clicking on the variables should also give 8 variables, including ga_lmp, which has been mapped by testcohort2 (which belongs to testnetwork1).

Test plan 4, step 5

Let's discuss and check what should be happening here (I wasn't involved in the change of test data). @BrendaHijmans ?

[EleanorHyde-UMCG]

The contact page (behind button 'Contact') doesn't seem to have the updates we made in the big release in July, i.e. pop-up with "Name of test cohort 2", Contact, Name entry field, Email entry field, Organisation entry field, Topic dropdown, Message entry field, "or contact us at: molgenis-support@umcg.nl", Send button"

Test plan 5, step 6

Have added to list above.

[EleanorHyde-UMCG]

The start/end year of cohort testcohort2 should be '1955-ongoing', not 'not available' Test plan 5, step 24, and test plan 9 step 10 (different years for cohort 1 but the same issue)

Have added to list above.

[EleanorHyde-UMCG]

The headings on the left hand side of the cohort page are missing 'Variables & Topics' in between Contributors and Available Data & Samples. eg on https://preview-emx2-pr-4178.dev.molgenis.org/catalogue-demo/ssr-catalogue/all/cohorts/testCohort2
Test plan 5, step 11

Have added to the list above.

[EleanorHyde-UMCG]

The text under 'Variables & Topics' is still a placeholder text and needs to be updated or removed. Test plan 5 (no test step for this new block yet)

Have added to the list above.

[EleanorHyde-UMCG]

What do we expect to happen when pressing the 'Harmonised variables' button on the cohort page? eg testcohort2 And why does testcohort2 have 299 variables? Need an explanation of what's changed here :-) @BrendaHijmans

[EleanorHyde-UMCG]

The comorbidity items in the pop-up for a subcohort are presented in random order but should be in alphabetical/numerical order. Test plan 5 step 45

Have added to the list above.

[EleanorHyde-UMCG]

When clicking on the arrow or the network name (rather than 'website') for information about a network to which a cohort is linked (Eg name for test network2) the link opens in the same window rather than in a new window. (We decided to have all external links open in a new window to avoid 'losing' the catalogue in your tabs) Clicking on 'website' gives the correct behaviour. Test plan 5, around step 53, test plan 9 around step 27.

Have added this to the list.

[EleanorHyde-UMCG]

Data access fee and prelinked fields not shown under Access conditions (bug https://github.com/orgs/molgenis/projects/15/views/1?filterQuery=data+access+fee&pane=issue&itemId=68839761) Test plan 5 step 57 (compare test plan 9 step 28, where they ARE shown because they are 'true').

Linkage options field is also not shown (but this is not a true/false field). Test plan 5 step 57 and test plan 9 step 28.

Have added to list.

[EleanorHyde-UMCG]

When browsing 'all resources', there's no difference in the text under 'Cohort studies' and 'Studies' (both have "cohorts & biobanks" as explanatory text). Is this deliberate?

@BrendaHijmans

[EleanorHyde-UMCG]

The harmonisation matrix and the info per cohort don't match, for some reason. testvarrepeats in preview (here) gives different results in the matrix to what you see below when you click on testcohort3 in the harmonisation details per data source. Test plan 4 step 12, test plan 6 step 6.

Have added this to the list above.

Fixed by fixing the test data. The migration script meant for production data worked differently to what was intended on the (randomly filled-in) test repeated variable, meaning the number of repeats was smaller than meant to be, which in turn meant that fewer repeats were shown in the harmonisation matrix. No software fix required.

[EleanorHyde-UMCG]

All open items are now under this milestone: https://github.com/molgenis/molgenis-emx2/milestone/40