Skip to content

Commit

Permalink
new: waf import completed
Browse files Browse the repository at this point in the history
  • Loading branch information
nickumia-reisys committed Sep 25, 2023
1 parent 084881a commit f1a2172
Show file tree
Hide file tree
Showing 2 changed files with 52 additions and 4 deletions.
54 changes: 51 additions & 3 deletions docs/waf_xml.mmd
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,63 @@ flowchart LR
is([IMPORT STARTED])
ie([IMPORT ENDED])
is ==> is_object_empty
is_object_empty == No ==> is_force_import
is_force_import == Yes ==> change_object_2
is_force_import == No ==> check_status_from_gather
change_object_2 ==> get_existing_object_2
check_status_from_gather ==> get_existing_object_2
get_existing_object_2 ==> is_delete_2
is_delete_2 == Yes ==> delete
is_delete_2 == No ==> is_iso_2
is_iso_2 == No ==> transform_to_iso
transform_to_iso == Success ==> save_content_2
save_content_2 == tranform ==> parse_iso
is_iso_2 == Yes ==> is_object_content_empty
is_object_content_empty == No ==> _validate_document
_validate_document == valid ==> parse_iso
_validate_document == not valid ==> continue_on_validation_errors
continue_on_validation_errors == Yes ==> parse_iso
continue_on_validation_errors == No ==> ie
parse_iso ==> update_object_reference
update_object_reference ==> is_guid_current
is_guid_current == No ==> update_guid
update_guid ==> is_guid_present
is_guid_present == No ==> generate_guid
is_guid_present == Yes ==> get_modified_date
generate_guid ==> get_modified_date
get_modified_date ==> spatial_package_create
spatial_package_create ==> is_source_private
is_source_private == Yes ==> mark_object_private
%% BUG: if source marked as private --> harvest --> changed to public --> harvest --> datasets remain private
is_source_private == No ==> is_source_part_of_topic
mark_object_private ==> is_source_part_of_topic
is_source_part_of_topic == Yes ==> mark_object_part_of_topic
is_source_part_of_topic == No ==> mark_as_geospatial
mark_object_part_of_topic ==> mark_as_geospatial
mark_as_geospatial ==> update_object_reference_2
update_object_reference_2 ==> is_status_new
is_status_new == Yes ==> create
is_status_new == No ==> is_modified_newer
is_modified_newer == No ==> transfer_job_history
transfer_job_history ==> delete_old_object
delete_old_object ==> reindex_package
is_modified_newer == Yes ==> update
create ==> ie
update ==> ie
is_object_content_empty-. Yes .-> error_3
parse_iso-. exception .-> error_3
is_guid_current-. Yes .-> error_3
get_modified_date-. exception .-> error_3
is_object_empty-. Yes .-> skip_2
reindex_package -.-> skip_2
end

%% Data
error[\Error/]
error_2[\Error/]
error_3[\Error/]
skip[/Skip\]
skip_2[/Skip\]
new_datasets[(New Datasets)]
deleted_datasets[(Deleted Datasets)]
changed_datasets[(Changed Datasets)]
Expand All @@ -93,9 +143,7 @@ flowchart LR
guess_standard[[Guess Metadata Standard]]
save_content[["Save Content (ISO)"]]
save_original_document[["Save Original Content (non-ISO)"]]

update[[Update Dataset]]
do_nothing[[Nothing to do]]
create[[Create New Package]]
update_object_reference[[Make this package the Current Harvest Object]]
update_object_reference_2[[Make this package the Current Harvest Object]]
Expand All @@ -119,7 +167,6 @@ flowchart LR
%% Conditional Checks
is_date_present{Does the source object have a date?}
is_ids{Is there data to process?}
is_existing_object{Does the object exist?}
is_iso{Is document ISO?}
is_iso_2{Is document ISO?}
is_object_empty{Is Object Empty?}
Expand All @@ -129,6 +176,7 @@ flowchart LR
is_guid_current{Is the GUID current?}
is_guid_present{Does the GUID exist?}
is_delete{Should the dataset be deleted?}
is_delete_2{Should the dataset be deleted?}
is_source_private{Is the Harvest Source Private?}
is_source_part_of_topic{Is the Harvest Source part of a Topic?}
is_status_new{Is the Status new?}
2 changes: 1 addition & 1 deletion docs/waf_xml.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

1 comment on commit f1a2172

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage

Coverage Report
FileStmtsMissCoverMissing
harvester
   __init__.py30100% 
harvester/db/models
   __init__.py50100% 
   models.py530100% 
harvester/extract
   __init__.py1922 89%
   dcatus.py1122 82%
harvester/utils
   __init__.py00100% 
   json.py2266 73%
   pg.py3544 89%
   s3.py2466 75%
harvester/validate
   __init__.py00100% 
   dcat_us.py240100% 
TOTAL1962090% 

Tests Skipped Failures Errors Time
29 0 💤 0 ❌ 0 🔥 21.359s ⏱️

Please sign in to comment.