Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decoupling TSV metadata schema parsing from API endpoint #8085

Open
poikilotherm opened this issue Sep 1, 2021 · 1 comment
Open

Decoupling TSV metadata schema parsing from API endpoint #8085

poikilotherm opened this issue Sep 1, 2021 · 1 comment
Assignees
@poikilotherm

Comments

@poikilotherm
Copy link
Contributor

Currently, any TSV file loaded into a Dataverse instance flows through DatasetFieldServiceApi.loadDatasetFields().
This method isn't just the endpoint, it also contains all the code to parse and convert the TSV into living model objects.

This needs to be torn apart:
@poikilotherm poikilotherm self-assigned this Sep 1, 2021
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 3, 2021
@poikilotherm
build(deps): add Univocity Parsers for TSV file parsing IQSS#8085
6b8e431
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 3, 2021
@poikilotherm
feat(schema): introduce placeholders for references in IQSS#8085
24345c2
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 3, 2021
@poikilotherm
feat(schema): make MetadataBlock data-binding possible IQSS#8085
b516482 
- Use the Univocity annotations on the model
- Add proper validation restrictions
- Make the column headers (or future mappings) part of the model
  by adding a proper enum, representing the (TSV column) order and
  the key values
@poikilotherm poikilotherm mentioned this issue Sep 3, 2021
Closed
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 3, 2021
@poikilotherm
feat(schema): make MetadataBlock data-binding possible IQSS#8085
82b87f6 
- Use the Univocity annotations on the model
- Add proper validation restrictions
- Make the column headers (or future mappings) part of the model
  by adding a proper enum, representing the (TSV column) order and
  the key values
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 6, 2021
@poikilotherm
test: make MetadataBlockParsingTest use header extraction IQSS#8085
71faf25
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 6, 2021
@poikilotherm
refactor: make MetadataBlock name regex a public constant IQSS#8085
7289ff6
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 6, 2021
@poikilotherm
feat(schema): make DatasetFieldType data-binding possible IQSS#8085
4d9c044 
- Make column headers part of the model class in a reusable fashion
  by using an enum plus constants. The enum also represent order and
  key values for the parser.
- Add Univocity parsing annotations to setter methods of the model class
- Add validation annotations where necessary
- Add tests for everything
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 6, 2021
@poikilotherm
feat(schema): make DatasetFieldType data-binding possible IQSS#8085
fcb0dbd 
- Make column headers part of the model class in a reusable fashion
  by using an enum plus constants. The enum also represent order and
  key values for the parser.
- Add Univocity parsing annotations to setter methods of the model class
- Add validation annotations where necessary
- Add tests for everything
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 7, 2021
@poikilotherm
feat(schema): make DatasetFieldType displayOnCreate mapping backward …
ae620d6 
…compatible IQSS#8085
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 7, 2021
@poikilotherm
fix(schema): add missing validation annotations in DatasetFieldType I…
16bf03c 
…QSS#8085
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 7, 2021
@poikilotherm
feat(schema): enable ControlledVocabularyValue data-binding from TSV I…
336825d 
…QSS#8085

- Add headers enum like with MetadataBlock and DatasetFieldType
- Add a placeholder for alternate values that need references to
  the dataset field this CVV is a part of
- Make the alternatives come from a column with a header
  (This is undocumented behaviour in the docs currently!)
- Proper validation as with the other data bindings
- Includes lot's of tests to make sure we notice when it breaks.
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 9, 2021
@poikilotherm
docs(schema): refactor and complete controlled vocabulary TSV section I…
2004f8c 
…QSS#8085
poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Sep 9, 2021
@poikilotherm
refactor(schema): change metadata TSVs to contain altValue column hea…
cd6f1e7 
…der IQSS#8085
@pdurbin
Copy link
Member

pdurbin commented Oct 1, 2022

Isn't @JR-1991 doing some nice parsing of the TSVs in https://github.com/gdcc/easyDataverse ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@poikilotherm poikilotherm

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

8085 decoupling tsv poikilotherm/dataverse
2 participants
@pdurbin @poikilotherm