Skip to content

Commit

Permalink
docs: add code reference to complex functions
Browse files Browse the repository at this point in the history
unify import stage for single xml and waf
  • Loading branch information
nickumia-reisys committed Sep 25, 2023
1 parent f1a2172 commit 75491d4
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 4 deletions.
3 changes: 3 additions & 0 deletions docs/single_xml.mmd
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,10 @@ flowchart LR
skip[/Skip\]

%% Functons

%% Code: https://github.com/ckan/ckanext-spatial/blob/e59a295431247fcd605fe55bb4fd9a2ecfc28d2b/ckanext/spatial/harvesters/base.py#L835-L860
_get_content_as_unicode[[Download XML File]]

get_existing_object[[Get Existing Object]]
get_existing_object_2[[Get Existing Object]]
create_object[[Create New Object]]
Expand Down
20 changes: 16 additions & 4 deletions docs/waf_xml.mmd
Original file line number Diff line number Diff line change
Expand Up @@ -46,16 +46,18 @@ flowchart LR
is_delete-. Yes .-> skip
is_delete == No ==> get_location
get_location-. No URL .-> error_2
get_location ==> get_data
get_data-. Exception .-> error_2
get_data ==> guess_standard
get_location ==> _get_content_as_unicode
_get_content_as_unicode-. Exception .-> error_2
_get_content_as_unicode ==> guess_standard
guess_standard ==> is_iso
is_iso == Yes ==> save_content
is_iso == No ==> save_original_document
save_content ==> fe
save_original_document ==> fe
end
subgraph import_stage [Import Stage]
%% WAFHarvester does not implement this code, it inherits from
%% SpatialHarvester, which is shared with DocHarvester
direction TB
is([IMPORT STARTED])
ie([IMPORT ENDED])
Expand Down Expand Up @@ -124,9 +126,15 @@ flowchart LR

%% Functons
get_source_url[[Get Source Root URL]]

%% Code: https://github.com/ckan/ckanext-spatial/blob/e59a295431247fcd605fe55bb4fd9a2ecfc28d2b/ckanext/spatial/harvesters/waf.py#L269-L277
_get_scraper[[Determine the WAF Server Type]]

get_current_objects[[Load Current Objects]]

%% Code: https://github.com/ckan/ckanext-spatial/blob/e59a295431247fcd605fe55bb4fd9a2ecfc28d2b/ckanext/spatial/harvesters/waf.py#L279-L334
_extract_waf[[Extract Source Objects]]

compare[[Compare Source and Current Objects]]
for_each_new[[For Each New Object]]
hash_new[[Create GUID based on hash of new object]]
Expand All @@ -139,7 +147,10 @@ flowchart LR
return_ids[[Return IDS]]
return_none[[Return NONE]]
get_location[[Get URL from Object]]
get_data[[Extract Source Object Data]]

%% Code: https://github.com/ckan/ckanext-spatial/blob/e59a295431247fcd605fe55bb4fd9a2ecfc28d2b/ckanext/spatial/harvesters/base.py#L835-L860
_get_content_as_unicode[[Download XML File]]

guess_standard[[Guess Metadata Standard]]
save_content[["Save Content (ISO)"]]
save_original_document[["Save Original Content (non-ISO)"]]
Expand All @@ -160,6 +171,7 @@ flowchart LR
transfer_job_history[[Transfer old object job history to new object]]
delete_old_object[[Delete old object]]
reindex_package[[Reindex package to reflect new date]]

%% Spatial Package Create: https://github.com/ckan/ckanext-spatial/blob/e59a295431247fcd605fe55bb4fd9a2ecfc28d2b/ckanext/spatial/harvesters/base.py#L233-L492
spatial_package_create[[Create Package Data ..see reference in code..]]

Expand Down

1 comment on commit 75491d4

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coverage

Coverage Report
FileStmtsMissCoverMissing
harvester
   __init__.py30100% 
harvester/db/models
   __init__.py50100% 
   models.py530100% 
harvester/extract
   __init__.py1922 89%
   dcatus.py1122 82%
harvester/utils
   __init__.py00100% 
   json.py2266 73%
   pg.py3544 89%
   s3.py2466 75%
harvester/validate
   __init__.py00100% 
   dcat_us.py240100% 
TOTAL1962090% 

Tests Skipped Failures Errors Time
29 0 💤 0 ❌ 0 🔥 14.862s ⏱️

Please sign in to comment.