Exports v2 #1054
Merged
Exports v2 #1054
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * use vectorized timestamp conversion * minor formatting --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com>
* preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * rewrite riskofbias exports * update naming for domain and metric --------- Co-authored-by: Daniel Rabstejnek <rabstejnek@gmail.com>
* preliminary epi rewrite * changes * changes * changes * fix * added back something accidentally deleted * moved code * fix test * remove old stuff * cleanup: * epimeta export rewrite * remove obsolete code * changes * merge fix * update admin site to browse data pivot by evidence type --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com>
* Convert invitro exports * Fix exporter datetime converter when datetime is None * Fix exports where there's no data * DTXSID in export should be None instead of useless dict if missing * add endpoint categories * Remove bloated dsstox dict from exports in favor of dtxsid --------- Co-authored-by: Andy Shapiro <shapiromatron@gmail.com>
* Made animal model exports, began configuring exporters * first export largely done, second export mostly done * Some cleanups, finished fourth animal exporter * Changes after more testing * Fix tests * Remove remaining flat_complete_* methods * Cleanup * Update sql_display to accept dict * Add TODO to comment to easily find it * Rename exporters to match the flat file exporter class names
b4ddf3e to
f7fdc1e
Compare
* remove duplicate rename calls * add ci calculation * update treatment period calculation * add back space; fix caused too many false positives in comparison * properly handle a 500 response from the server in a data pivot * fix epi logic from plot review * fix invitro export when category id is null; remove category id from export * add unique rob columns * use same method for categories * refactor into a reusable method * remove pytest warning "Marks applied to fixtures have no effect" * refactor; write tests * add tests * rewrite animal export test using same pattern * add udf to openapi * remove extra whitespace (will change visuals; known change)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Convert all our tabular exports to build from ORM exports cast into pandas data frames instead of lists of nested JSON objects. This should improve export performance significantly in many cases, and should make the code simpler and easier to maintain in the future.
This work was originally started October 12, 2023 (260 days ago). Nice job @rabstejnek!!!
This original work was more of a 1:1 conversion; future work can be added to dramatically improve performance if we find any performance bottlenecks in the future.