Releases: shapiromatron/hawc
2022-Q2
July 12, 2022
Updates were light this quarter; we were focused on other projects. One major change is rewriting how rasterized visuals are created; we're using native tooling in the browser now instead of generating server side; this should result in increased performance and accuracy to the rendered visual. #630
The complete changelog is described below:
- Visualizations:
- Client side rasterization of SVG images #630
- Literature screening:
- Study evaluations:
- Client:
- Admin:
- Developer updates:
- Current in development (behind feature flags):
2022-Q1
April 12, 2022
Our major feature this quarter is the ability to create Study Evaluation Tables, or SETs. These tables are designed to present study evaluations (or risk of bias) findings for multiple studies, but are more customizable than the existing heatmap tables we build. Users can select which studies and metrics to present, can create custom columns, and can even show the same metric across multiple health outcomes. For animal bioassay data, you can include additional metadata for a study such as study design, doses and dose units, or endpoints extracted.
In addition to the SET tables, we've made a few other notable changes:
- After hearing from users, we've added new views to 1) bulk-publish content when making assessments public, 2) rapidly convert references into studies where data can be extracted or study evaluations can be completed, and 3) create studies for extraction/risk of bias given a PubMed or HERO ID without the intermediate step of making a reference
- Added custom sort orders for visuals as well as textures to help distinguish data across studies
- Expansion of the high level assessment metadata, making the objective required and adding a publicly available authors field to describe what users or groups created the assessment
This release also includes a first pass of a revised epidemiology data extraction module. This hasn't been yet publicly released and is only available by a feature flag; we hope to complete next quarter after further evaluation from epidemiologists.
The complete changelog is described below:
New features:
-
Assessment
- Improved publicly available metadata about the purpose of assessments and authors #574 #600
- Added page for bulk publishing/unpublishing content #582
- Paginated assessment list views #611
- Assessment attachment changes now happen in an inline form #564
- Removed inactive users from team-members list #601
-
Literature
-
Study
-
Risk of Bias/Study evaluation
-
Summary tables and visualizations
-
Animal
- Converted dose tables into a more compact list format #612
-
Epidemiology
-
General user-interface improvements
-
Administrative
- Created an admin dashboard framework and new charts/tables #580
-
Client
Software upgrades/security/stability:
- Updated python packages #579 #628
- Refreshed test cassettes #578
- Replaced Django JSON views with DRF implementation #560
- Moved admin components into a new hawc_admin app #580
- Expanded reachable APIs in the OpenAPI schema #588
- Added a class to hold current and future feature flags #622
- Added additional logging to APIs #610
- Updated DSSTox source #572
- Fixed vulnerability on endpoint copy page #616
Bug/regressions fixed:
- Fixed generational form display choices #560
- Fixed dose activation on endpoints #560
- Fixed layout of untagged references #560
- Fixed overlap on donut visuals #584
- Fixed the rendering of BMD values on data pivots #585
- Fixed rendering of 500 error page #587
- Fixed errors on API calls of empty tagtrees #591
- Fixed some errors raised by POST requests with partial data #592
- Fixed some regressions caused by updated d3 #597
- Fixed error on visual create page #606
- Handle deletion of score overrides when study changes warrant deletion #625
- Fixed pages that show a user's assigned tasks #629
- Fixed several miscellaneous bugs identified during new site launch #596
- Removed bulk update task assignment page #624
- Removed unused API endpoints #589
- Removed the EXTRA_BRANDING setting #614
2021-Q4
January 14, 2022
Big changes this quarter include integrating external authentication systems into HAWC, instead of always requiring to create a custom user account (#525). We also added a much requested bulk-literature tagging application (#519), so you can apply (or remove) tags to dozens of references at once from an Excel spreadsheet, and improved and expanded handling of DOIs throughout the literature application. Finally, we've improved searching and filtering pages for HAWC for greater flexibility (#547), as well more flexible data filtering in creating custom visuals (#520).
The complete changelog is described below:
New features:
-
Authentication
-
Literature:
- Added new bulk-tag literature feature from an Excel spreadsheet #514 #519
- Improved DOI validation and capture from literature searches and imports #533
- Add DOI to literature exports #526 #530 #558
- Search and import descriptions now use rich text #537
- Added "download" actions for all tag views including in visualization popups #549
-
Data extraction
- Added internal communications fields for assessments #528 #539
- Added internal communications fields for studies #528 #539
- Added improved searching/filtering for studies #547
- Added improved searching/filtering for visuals #547
- Added improved searching/filtering for assessments #547
- Enabled copying of data visualizations (not just data pivots) #554
- Enabled copying of summary tables (not just data pivots) #554
-
Benchmark dose modeling
- Allow multiple selected models for an endpoint, one per dose-unit #521
-
Data visualization
-
API:
- Added change audit-logging to API writeable views #524
- Tie API tokens to user-login sessions #523 #559
- Add swagger user interface to browse/test the API (admin-only) #545
- Split study API for the final review and individual risk of bias evaluations #534
- Add literature tag tree fetch/clone/update methods to API and client #513
-
Development environment
- Reorganize model choices to move to a constants file #543
- Improved developer environment with new syncing command #542
- Code inside the scripts/private path is no longer formatted #540
- Change default setting for
HAWC_LOAD_TEST_DBand add new option 'delete' #536 - Deployed a new test-environment for ecological data extraction #482
Software upgrades/security/stability
- Add change audit logs to logging and database for redundancy #546
- Standardize javascript applications startup in templates #507 #567
- Convert client module from single file to a python package #562
- Convert test files into a python package #563
- Add a flag to conditionally deploy the HAWC admin #522
- Add sendgrid as a new email provider #548
- Improved application monitoring by adding container healthchecks #516
Bug/regressions fixed:
- Fix use of native javascript Set/Map #515
- Prevent error if unauthenticated user attempts to logout #529
- Improve data validation for requesting plot downloads #529
- Fix API bugs #535
- Fix autocomplete for admin user assignment #538
- Fix various bugs related to Heatmaps #527
- Remove
evalmethod in JavaScript #544 - Minor improvements to test fixture and API #551
- Fix literature downloads to include child tags #570
2021-Q3
October 1, 2021
Our key feature this quarter is a new view of assessment-level change logs for team members. Team members can browse and search the logs to find which objects changed, including the ability to investigate the logs for particular objects. At this point we're not presenting the content which changed, but we may find a way to present it in the future.
In addition, a few additional major updates include:
- Rewrite the study evaluation/risk of bias user assignment page for speed and flexibility, including the ability to toggle active/inactive status
- Improve the literature tagging and browsing interface including new search features
- Documentation on how to build interactive HAWC visuals with an uploaded dataset
The complete changelog is described below:
New features:
- Overall:
- Literature
- Study evaluation
- Summary and visualizations
- Administrative
Software upgrades/security/stability:
- Update python 3.6 to python 3.9; package updates #478 #481
- Update postgres 9.6 to 12 #478
- Update javascript build tools and packages #485
- Refactor build to use standard python wheels instead of a custom approach #477
- Add javascript unit-testing and integration with CI/CD #501
- Refactor javascript code to standardize startup/code-splitting, improve editor introspection #477 #491 #504 #505
- Worker healthcheck for automated checking and plotting #476
- Revise/update development and documentation #480
Bug/regressions fixed:
- Add randomness for generating long slugs #502
- Standardize error messages for assessment uniqueness #490
- Fix Actions dropdown display on endpoint detail page #500
- Fix risk of bias score creation signals and defaults #494
- Make relative URLs absolute to HAWC in a table docx export #483
- Properly handle SvgRenderer in API #488
- Heatmap tooltip positioning fix #489
- Add async task expirations for recurring tasks #493
- Assorted bugfixes regarding BMD modeling and dose-response confidence intervals #501
- Fix bugs regarding state where dose-groups do not align with endpoint-groups #505
2021-Q2
July 2, 2021
Our key feature this quarter is expanded flexibility and interoperability of the risk of bias / study evaluation module. This module continues to be one of the most frequently used in HAWC, with over 8,500 studies where study evaluation was conducted and over 200,000 individual metric responses. We've added the ability to load an existing approach (instead of a default approach), and added flexibility in which judgment (previously score) options can be used. We've also overhauled the user interface to make it a snappier and more reliable experience.
A few additional notable updates:
- Expanded flexibility in evidence profile tables and exploratory heatmaps
- Performance updates (faster study evaluation views, literature pagination, permission checks)
- Redesign for improved accessibility and consistency with EPA style guides (EPA flavor)
The complete changelog is described below:
New features:
- Literature screening
- Risk of bias/study evaluation
- Default evaluation approaches are no longer performed and can be selected for each assessment #451
- Risk of bias domains and study evaluations are now sortable #440
- Metrics can have customizable judgment response instead of one list for all #441
- Appropriately show correct study evaluation for override cases on endpoints and outcomes in heatmaps and data pivots #409 #413 #456
- User interface improvement
- Make the score/judgment field optional for entry and presentation #463
- Add timestamps to exports for QA/QC #422
- Vocabulary
- Exploratory heatmaps
- Evidence profile table
- Styling
- Assorted changes
Software upgrades/security/stability:
- Enable HSTS security settings #430
- Add Referer checking on all safe methods #415
- Add admin auditing view for browsing media #443
- Add epi models to admin #448
- Vendor all js/css instead of external css requests #416
- Change all URLs from updating to /update/ for consistency #429
- Refactor and improve code quality for risk of bias application #457 #461 #464 #465 #468 #470 #469 #466
- Update logging configuration in development #454
- Make keys-stable in test comparisons to prevent false diffs #472
Bug/regressions fixed:
- Fix layout recursion on risk of bias heatmaps #438
- Fix regression where risk of bias table cells when shown on a modal caused a display error #414
- Fix bug where exploratory heatmap input text prevented adding spaces #444
- Fix error with updating dosing-regime invalidating endpoint-groups #424
- Fix validation errors on APIs #421 #427
- Fix URL validation in evidence profile tables #419
- Fix test failure requiring remote requests #425
- Show 404 error for missing dataset revisions #433
2021-Q1
April 1, 2021.
HAWC has long had the ability to build result summary figures in a variety of graphical formats, but tabular options have been limited. This releases introduces HAWC summary tables, which enable users to create summary tables in their assessments. At this point, we've created two different table formats - a generic table, which allows a user to define their own table structure and content, and an evidence profile table, which is a predefined table structure for expert-summarization by evidence type. HAWC tables natively have a web view and an export to Microsoft Word. The current two table formats require new data entry into HAWC; future HAWC table updates will include query-based tables from extracted data in HAWC.
We've also overhauled the literature tagging and browsing view, including a quicksearch feature, making tags hyperlinked, and adding a wordcloud and filterable histogram of results by year. On the backend, we made a few changes to improve performance (aka speed), with a few more in the queue for future updates.
A few additional notable updates:
- Custom literature tagtrees can be resized for larger visuals with more nodes
- Exploratory heatmap tables can now be filtered to only include a subset of the data, including risk of bias summaries
- Epidemiology data in HAWC can now be created/updated via API, allowing data migration from external tools
The full changelog is below:
New features:
- New table creation features #407
- Overhaul literature tags, searching, and displays #393
- Make tag badges interactive and clickable so you can view related References
- Added a quicksearch to enable searching by title, author, or year
- Added a filterable histogram by year
- Added a wordcloud
- Improved query performance for faster pageviews
- Visualization updates
- Literature tagtree height/width are now customizable to prevent tag node edge conflicts #405
- Evidence heatmap updates #404
- For custom heatmaps, add the ability to filter the dataset to display only a subset of the data
- Add new calculated summary fields for bioassay data, consistent with fields in data pivot
- Add overall study evaluation/risk of bias fields to epi and bioassay data
- Changed default URLs for visuals to be slugs instead of IDs, consistent with other other pages in HAWC. ID links now redirect to the slug view automatically #401
- Literature tagtree height/width are now customizable to prevent tag node edge conflicts #405
- By default, links to public HERO are shown. However a user can modify their user profile to request to see private HERO links. #398
- Update header footer layout, customizable contact us #389
- Audit-logging added for delete views #391
- Add new guardrails for literature reference deletion to prevent unintentional data loss #392
- Make new editable, cacheable frontmatter and about page #394 #395
- Added draft ecological data extraction schemas for admin testing #355
New APIs for bulk-modifying content:
- Epi content can now be created via API #411
- Simplified reference -> study creation API, making
short_citationandfull_citationoptional, and calculated if not provided, consistent with how studies are created in the form view #396
Software upgrades/security/stability:
- Improve container layout and deployment documentation #385
- Add new admin diagnostic action for checking email service #402
- Added new healthcheck for the bmds2 webservice to periodically check for proper functioning; as well as a diagnostic test in the admin #406
- Add new optional load of test database for dev/test environments from CI #402
- Move healthcheck status url outside admin #388
- Migrated from google analytics to google tag manager; make optional specified by environment variables #399
- Improve performance (speed) for animal endpoint list, animal groups, and literature tagging #400 #393
- Support Python 3.9 with pubmed XML parser #412
Bug/regressions fixed:
- Always show default, non-override scores in data pivot exports #387
- Fix epi country filter after migration to many to many field #384
- Restored entry of null-booleans after regression (eg., dosing regime, positive control) #383
- Fix endpoint list sorting regression #408
List of changes:
https://github.com/shapiromatron/hawc/pulls?q=is%3Apr+is%3Amerged+sort%3Aupdated-desc+closed%3A%3E2021-01-15+created%3A%3C2021-04-01+
2020-Q4
January 14, 2021.
A large amount of our time this quarter was spent updating and redesigning the HAWC frontend. The project has grown organically over around eight years, patching new features and functionality as it grew. Unfortunately, we hit a point where the site was looking dated, and there were user-interface inconsistencies, making it difficult to add new features. We've therefore updated from a 2012 framework to a modern one, and added the ability to more easily skin an application with branding.
In addition to the website redesign, a few critical new features were added:
- References in HAWC can now be manually updated to associate with a Pubmed and/or HERO ID
- Restore copy buttons for risk of bias/study evaluation text
- Unaccented author search feature (eg., a search for "fred" returns "Frédéric")
- Added new programmatic APIs to copy risk of bias across assessments, and associate existing bioassay endpoints with controlled vocabulary terms
Finally, we added the ability to (optionally) restrict assessment creation to a subgroup of users. In some cases organizations may want to limit the content that is added to HAWC, and therefore, if enabled, HAWC admins can assign users to a group that has the ability to create new assessments. Once an assessment is created, members of the assessment can manage just as before.
The full changelog is below:
New features:
- overhaul look and feel of site, migrating from Bootstrap 2 (2012) to Bootstrap 4 (2018+)
#334 #335 #336 #337 #338 #338 #343 #345 #346 #347 #348 #354 #356 #358 #359 #360 #361 #362 - enable users option to specify controlled vocabulary for assessment #326
- for risk of bias/study evaluation, rename
scores -> judgements#331 - restyle header, footer, and navigation bar #342 #367
- enable updating any reference to manually add PubmedID or HERO ID after import #364
- Restore copy buttons for risk of bias/study evaluation text #366
- (optionally) restrict assessment creation to subgroup of users #369
- Enable unaccented search for author names #373
- Improve risk of bias score admin view and identify orphaned override content objects #374
New APIs for bulk-modifying content:
- add new API to bulk update endpoint term IDs #323
- allow copying risk of bias/study evaluations across assessments #332
- add new API for fetching required bioassay metadata #371
Software upgrades/security/stability:
- upgrade to django 3.1 #324
- add server-side browser check to possible client incompatibility #339
- move breadcrumb logic to view instead of template #341
- re-enable and expand integration tests #350
- sanitize referer and fix XSS vulnerability #349 #363
- remove nginx configuration from docker container deploys #352
Bug/regressions fixed:
- cache EHV vocabulary page: #322
- restore model ordering defaults #325
- fix model ordering distinct issues #327
- fix cog button in visualizations #328
- update pending deprecation warnings #329
- remove non-final reviews from rob heatmaps #330
- fix NA scores on data pivots #333
- fix tabular export when literature tag names are not unique #368
- fix url slugs not properly escaping special characters #370
2020-Q3
October 11, 2020.
This quarter's update has three major new features/updates:
- Allowing use of controlled vocabulary for in vivo bioassay endpoint data extraction in HAWC. To take advantage of this new feature, we've also released the Environmental Health Vocabulary (EHV), designed by health scientists at EPA/ORD/CHPEA, this is designed to be a living vocabulary which will be updated and curated frequently as new health assessments extract content in new biological domains (https://hawcproject.org/vocab/ehv/; login required). Controlled vocabularies in HAWC allow mapping of terms to external ontologies, including potentially mapping to adverse outcome pathways (AOP) or other ways of organizing biological causality and critical effect. Use of the vocabulary in HAWC is optional; you can still enter content without using the vocabulary as before.
- Integration with the US EPA Chemistry dashboard (https://comptox.epa.gov/dashboard) by using DTXSID as a preferred chemical identifier instead of name/CASRN, which allows for QA/QC of data entry by using this highly curated scientific database. Assessments, bioassay endpoints, invitro endpoints, and epidemiological exposures can now be mapped to DTXSIDs.
- Improving software stability by full-stack integration tests in our CI/CD pipeline, which allows for testing visualizations and rich frontend client applications
A full list of changes are available below:
-
Vocabulary
- Create new application for controlled vocabulary and mapping to related external entities
- Load v1 of enviro health vocabulary EHV for EPA HAWC assessments, developed by EPA/ORD/CPHEA colleagues
- Add a display view,
/vocab/ehv/to browse the vocabulary
-
Assessment
- Add DTXSID integration and modify dashboard to query Chemistry dashboard API
- Migrate existing CASRN to DTXSID
- Add assessment-level (and overall) logs
- Add blog model for reporting HAWC updates
- Add job-model for asynchronous reports
- Datasets can now be published or unpublished, just like studies or visuals
- Rewrite data-extraction cleanup using mobX instead of redux, ignoring controlled vocabulary
-
Literature
- Added new API endpoint for replacing HERO identifiers
- Added new API endpoint for updating all reference metadata from HERO
- Bugfix: update API to check for unique title/slug
- Update reference page to show all tags currently applied in all views
- Update tagtree visualization to show/hide nodes with no references
- Catch common RIS formatting upload errors and report to user as 400s instead of 500s
-
Risk of bias/study evaluation
- Modify "default" score label to "overall" and only present if a label was not entered
- Add ability to selectively filter which scores to present in an RoB heatmap
-
Animal bioassay
- Add DTXSID integration and modify dashboard to query Chemistry Dashboard API
- Migrate existing CASRN to DTXSID
- Update bioassay endpoint forms for use of vocabulary, including a quick insert by ID method
- Add mechanism (admin only) to migrate terms to use the new vocabulary
- Update API and client to add endpoints using controlled vocabulary
-
Study
- fixed bug for project managers locking/unlocking studies
- Show ask_authors text details, preserving whitespace
-
Epi
- Add DTXSID field to exposure objects in HAWC
-
Management
- Add ability to unset users in tasks
-
Client
- Added new client calls for new literature APIs
- Update bioassay endpoint to allow use of controlled vocabulary
- Updated python HAWC client for October release https://pypi.org/project/hawc-client/2020.10/
-
Testing
- Add new integration tests which use a browser to test interactive javascript functionality
- Bring in external dependencies into hawc (
litter_getteranddocx_utils) - Ensure IE11 error is still displayed on older browsers
- Fix bugfix on /about/ page and add tests for reliability
-
Development
- Remove eslint warnings to further standardize javascript frontend application code
- Update d3 from v3 to v5
- Generate lines of code report
- Force ordered querysets in all assessments for deterministic reporting of data
-
Deployment
- refactor to allow for kubernetes deployment in addition to docker-compose
- added new
/admin/:uuid/healthcheck/url
2020-Q2
Released July 15 2020.
This quarter's updates focused on developing new visualizations for displaying data extracted in HAWC or in other tools. Evidence maps created in Tableau can now be embedded into HAWC, or if data has been extracted in HAWC, exploratory heatmaps can be created to view the data. In addition, standardized, commonly used views of data in an assessment are available whenever data has been extracted into HAWC (eg., see the dashboards in PFBS). These new views are available on all new and existing assessments.
Updates were also made in expanding data-capture and display of study evaluation/risk of bias data. In addition, a new assessment-level Dataset data type was added to allow users to upload datasets created in external systems (such as DistillerSR, SWIFT, or Excel) and use in visualizations in HAWC. We anticipate other features will be available with the new Dataset type in future updates.
We've also expanded the API and HAWC client so that studies, study evaluation/risk of bias responses, and animal bioassay data can be created via API in addition to our standard exports of these data. This should allow for easier interoperability with external datasets, databases, and applications.
A full list of changes are available below:
-
Assessment
- Add new model,
Dataset, where tabular datasets can be uploaded and used in HAWC assessments. Datasets are versioned and only the most recent version is available to the public if an assessment is made publicly available. Datasets can currently be used in creating custom exploratory heatmaps, but it is expected that they will be used in the future for other purposes. - Added a new assessment option to include data extracted from assessment in an anonymized machine learning (ML) dataset export
- beta-testing: Added new ML exports using HAWC data
- Add new model,
-
Literature
- Previously, only references tagged with "Included" or child-tag could be extracted in HAWC; this is now a configurable option to allow for more dynamic tag-trees
- Integration of literature tags with new exploratory heatmap visualizations
- beta-testing: topic modeling for literature using title and abstract and visualization of topics. Preliminary approach using TFIDF + NMF + TSNE with the scikit-learn library.
-
Study evaluation/Risk of bias
- Add new (optional) field to all risk of bias/study evaluation - direction of effect
- Update visualizations to include the direction of effect when entered
-
Visualization
- Add new custom visualization type - exploratory heatmap. Exploratory heatmaps can be created with data currently extracted in HAWC (literature tags, bioassay extraction, or epi extraction) or with a tabular dataset upload
- In addition to custom exploratory heatmaps, standard generic heatmap visuals are available for extracted HAWC data. These are preloaded using commonly-used displays of bioassay and epi data (eg., PFBS), and are available for all assessments that have data extracted.
- Add new visualization type - embedding external visuals. This allows embedding of visuals from external systems; it currently supports the public Tableau website.
- Data pivot updates:
- Update user interface (UI) for conditional formatting options in data pivots
- Fix bug regarding column text alignment with maximum text-size
- Fix bug, remove dose-groups where data are not extracted for a single Endpoint, even if it was captured in the Dosing Regime
- Crossview updates:
- Add
endpoint nameas a new filter option
- Add
- Study evaluation/risk of bias heatmap
- Update legend to include footnotes for additional extraction indications
- Update heatmap to display direction of effect if evaluated
- beta-testing: new visualization for exploring critical effect levels (LOEL, NOEL, BMD, etc) across all endpoints in an assessment, grouped by system
-
Client
- Release python client on pypi to allow for easier install (eg.,
pip install hawc_client) - Update python client to create study, risk of bias response, or bioassay data
- Add a status-bar indicator for large dataset downloads
- Added a test-suite to Python client for continuous integration testing
- Release python client on pypi to allow for easier install (eg.,
-
API
- Add new endpoints to create risk of bias response
- Add new endpoints to create study
- Add new endpoints to create animal bioassay extraction
- Add new tabular exports, commonly used in exploratory heatmap visuals
- Add new assessment dataset API views
- Add OpenAPI API schema definition file (admin-only)
-
Testing
- Add vcrpy to test suite for faster tests and to remove dependency on external websites like PubMed, HERO, etc.
- Add new
make coveragecommand to create testing coverage reports - Add test-suite for
hawc_clientand integrate with CI - Expanded tests for tabular exports to improve stability, commonly used in visualizations
-
Other
- Refactor tabular exports to use REST API instead of custom solution
- Remove
printfin javascript in favor of ES6 string literals - Large tabular datasets are now cached and responses are reused over a period of hours. If data changes overtime, users now have the option to delete assessment caches and build new.
hawc-client 2020.5
First pypi HAWC client library release.