Enhancement/Bugfix: Downstream qld-gov-au fix's #232

duttonw · 2024-10-25T00:50:40Z

feat: @JVickery-TBS work on validation integration (await successful validation prior to doing datastore work via 'IPipeValidation'
fix: handle gracefully if tabulator load fails by trying 'direct load'
fix: Excel blank header row bug
fix: Datastore truncate, restart identity so numbering restarts from 0 again (when imported data has same columns and types
fix: parital fix on DB deadlock by adding timeouts on DDL events
test: test_simple_large_file, test_with_blanks, test_with_empty_lines, test_with_extra_blank_cells
test: test_require_validation, test_enforce_validation_schema
chore: min version requirements for cve's,
requests>=2.32.0
urllib3>=2.2.2
zipp>=3.19.1

Upgrade to GitHub-native Dependabot

Develop to master

Develop to master - merge from upstream

Develop to master - make datastore more robust

fix workflow

- If all types have been rejected, ensure that the fallback flag is correctly set

- replace empty strings with None if they have types that will choke on empty string

…into QOLDEV-347-fix-ckan-2.10

- Column that has some rows with free text and others with numeric data - Column that has some rows with timestamp and others with empty string

- Remove unused imports, or tag those that serve a purpose (testing what can be imported) - Remove obsolete exclusions from Flake8 config

QOLDEV-347 Prepare for CKAN 2.10

- Show the first 100 and last 100 logs, with a message between to say how many were skipped

- Start with the first and last 50 rows, provide a link to double it

- Add a notice at the top of the page when logs have been hidden. - Clarify that we are hiding logs, not skipping actual processing of data. - Add 'Show All' link to show all logs if the user is confident they can handle it.

[QOLSVC-1863] truncate on-page XLoader logs if there are too many

Develop to master - merge and fix upstream changes

…otes

…otes using tabulator

…gle and double quotes

…aviour

- This is more efficient than setting it on each call, and applies even to code that just reads the config without accepting an override.

QOLDEV-424 fix parsing of DESBT data

…/validation-support Validation Extension Support (Minified)

Develop -> Master: feat(dev): implement IPipeValidation + updates

…GovAuToCkan) Conflicts Fixed: ckanext/xloader/jobs.py ckanext/xloader/loader.py ckanext/xloader/plugin.py ckanext/xloader/templates/package/resource_read.html ckanext/xloader/tests/samples/mixed_numeric_string_sample.csv ckanext/xloader/tests/samples/sample_with_blanks.csv ckanext/xloader/tests/test_loader.py ckanext/xloader/tests/test_plugin.py ckanext/xloader/utils.py requirements.txt Changes to be committed: modified: ckanext/xloader/config_declaration.yaml modified: ckanext/xloader/jobs.py modified: ckanext/xloader/loader.py modified: ckanext/xloader/plugin.py modified: ckanext/xloader/templates/package/resource_read.html modified: ckanext/xloader/tests/samples/sample_with_blanks.csv new file: ckanext/xloader/tests/samples/sample_with_extra_blank_cells.csv modified: ckanext/xloader/tests/test_loader.py modified: ckanext/xloader/tests/test_plugin.py modified: ckanext/xloader/utils.py modified: dev-requirements.txt modified: requirements.txt

… into snyk-fix-92fccbfc3cb4ea7e0386dab1141f96ac

…dab1141f96ac [Snyk] Fix for 4 vulnerabilities

- Added status badges. Author: Jesse Vickery <jesse.vickery@tbs-sct.gc.ca>

- Reverse debug badge logic.

Frontend Status Badges

…, remove display bug from badge display

fix: badges

…outer that is not rendered

fix: asset inclusion must be in the block being rendered,

… develop

… to Datastore

ThrawnCA · 2024-10-25T00:56:54Z

ckanext/xloader/jobs.py

@@ -124,6 +124,7 @@ def xloader_data_into_datastore(input):
            if tries < MAX_RETRIES:
                tries = tries + 1
                log.info("Job %s failed due to temporary error [%s], retrying", job_id, e)
+                logger.info("Job failed due to temporary error [%s], retrying", e)


This seems to be a duplicate. Should use one logger or the other, not both, I think?

we have root file
log = logging.getLogger(__name__)

and then we have per job logging

level = logging.DEBUG handler.setLevel(level) logger = logging.getLogger(job_id) handler.setFormatter(logging.Formatter('%(message)s')) logger.addHandler(handler) # also show logs on stderr logger.addHandler(logging.StreamHandler()) logger.setLevel(logging.DEBUG)

… develop

…lity

chore: use more unique tag, include test sumary in github actions

dependabot-preview bot and others added 30 commits April 28, 2021 22:14

Upgrade to GitHub-native Dependabot

d59f4ea

Merge pull request #29 from qld-gov-au/dependabot/add-v2-config-file

8ecfa62

Upgrade to GitHub-native Dependabot

Merge pull request #31 from qld-gov-au/develop

79d6ea0

Develop to master

Merge pull request #33 from qld-gov-au/develop

0b1b9c5

Develop to master

Merge pull request #35 from qld-gov-au/develop

450c8ab

Develop to master

Merge pull request #37 from qld-gov-au/develop

a4e350e

Develop to master

Merge pull request #39 from qld-gov-au/develop

e336c83

Develop to master - merge from upstream

Merge pull request #42 from qld-gov-au/develop

031de33

Develop to master - make datastore more robust

fix workflow

963ada1

Merge pull request #43 from qld-gov-au/develop

a938b26

fix workflow

[QOLDEV-347] apply 'str' fallback type correctly, ckan#182

9f96e16

- If all types have been rejected, ensure that the fallback flag is correctly set

[QOLDEV-347] fix validation errors on empty strings, ckan#182

cf04a5c

- replace empty strings with None if they have types that will choke on empty string

Merge branch 'master' of https://github.com/qld-gov-au/ckanext-xloader …

f4f3665

…into QOLDEV-347-fix-ckan-2.10

[QOLDEV-347] add tests for edge cases we're fixing

b8b9914

- Column that has some rows with free text and others with numeric data - Column that has some rows with timestamp and others with empty string

[QOLDEV-347] tighten Flake8 rules

b62aa6c

- Remove unused imports, or tag those that serve a purpose (testing what can be imported) - Remove obsolete exclusions from Flake8 config

Merge pull request #44 from qld-gov-au/QOLDEV-347-fix-ckan-2.10

93aaf0c

QOLDEV-347 Prepare for CKAN 2.10

[QOLSVC-1863] truncate on-page XLoader logs if there are too many

939ff6b

- Show the first 100 and last 100 logs, with a message between to say how many were skipped

[QOLSVC-1863] make XLoader log truncation configurable

05b2b88

- Start with the first and last 50 rows, provide a link to double it

[QOLSVC-1863] adjust XLoader log truncation appearance

d937dea

- Add a notice at the top of the page when logs have been hidden. - Clarify that we are hiding logs, not skipping actual processing of data. - Add 'Show All' link to show all logs if the user is confident they can handle it.

[QOLSVC-1863] ignore negative numbers of logs to display

288b0ab

Merge pull request #46 from qld-gov-au/QOLSVC-1863-xloader-page-timeout

5481f8b

[QOLSVC-1863] truncate on-page XLoader logs if there are too many

Merge pull request #45 from qld-gov-au/develop

99d538c

Develop to master - merge and fix upstream changes

[QOLDEV-424] add unit test for parsing CSV file with commas inside qu…

b351e95

…otes

[QOLDEV-424] add unit test for parsing CSV file with commas inside qu…

83e1b86

…otes using tabulator

[QOLDEV-424] add unit test for parsing CSV file with a mixture of sin…

9ae1b26

…gle and double quotes

[QOLDEV-424] reuse sample size constant for both loading methods

0829801

[QOLDEV-424] increase CSV sample size to better match Messytables beh…

b444c6c

…aviour

[QOLDEV-424] set default CSV sample size in config

37a2a54

- This is more efficient than setting it on each call, and applies even to code that just reads the config without accepting an override.

[QOLDEV-424] alter sample mixed-quotes file to use generic data

51fffad

Merge pull request #47 from qld-gov-au/QOLDEV-424-fix-desbt-parsing

a7ead01

QOLDEV-424 fix parsing of DESBT data

duttonw and others added 18 commits July 24, 2024 07:19

Merge pull request #104 from JVickery-TBS/feature/qld-gov-au/minified…

8186398

…/validation-support Validation Extension Support (Minified)

Merge pull request #105 from qld-gov-au/develop

16eb59b

Develop -> Master: feat(dev): implement IPipeValidation + updates

Pin setuptools only for 2.9 to allow CICD to pass

ded839c

Merge pull request #106 from qld-gov-au/qldGovAuToCkan

9325784

Merge branch 'develop' of https://github.com/qld-gov-au/ckanext-xloader…

e9fe6ba

… into snyk-fix-92fccbfc3cb4ea7e0386dab1141f96ac

Merge pull request #101 from qld-gov-au/snyk-fix-92fccbfc3cb4ea7e0386…

a3ef7db

…dab1141f96ac [Snyk] Fix for 4 vulnerabilities

feat(templates): status badges;

8329512

- Added status badges. Author: Jesse Vickery <jesse.vickery@tbs-sct.gc.ca>

fix(logic): reverse debug logic;

29b1c7b

- Reverse debug badge logic.

doc: Add additional ckan config options

f29e179

Merge pull request #108 from qld-gov-au/JVickery-TBS/feature/badges

e042527

Frontend Status Badges

fix: Move from static image to css badge which allows i18n more easily

2a4f965

Merge pull request #110 from qld-gov-au/JVickery-TBS/feature/badges

73ff70a

fix: webassets not loaded on snippets, correct display vs hover title…

5a527ef

…, remove display bug from badge display

Merge pull request #111 from qld-gov-au/JVickery-TBS/feature/badges

d7dfc83

fix: badges

fix: asset inclusion must be in the block being rendered, not on the …

a713803

…outer that is not rendered

Merge pull request #112 from qld-gov-au/JVickery-TBS/feature/badges

b2f82ba

fix: asset inclusion must be in the block being rendered,

Merge branch 'master' of https://github.com/ckan/ckanext-xloader into…

04f1d09

… develop

duttonw added the enhancement label Oct 25, 2024

duttonw requested a review from ThrawnCA October 25, 2024 00:50

duttonw self-assigned this Oct 25, 2024

duttonw changed the base branch from develop to master October 25, 2024 04:55

fix: action_manage instead of action_manage_inner for standart access…

5fa19cb

… to Datastore

ThrawnCA approved these changes Oct 29, 2024

View reviewed changes

ThrawnCA and others added 5 commits October 30, 2024 10:35

feat: limit PyPI publish to upstream repo

6cc48b7

Merge branch 'master' of https://github.com/ckan/ckanext-xloader into…

b270fb1

… develop

fix: run docker container as root user not as ckan for cicd compatabi…

d36cace

…lity

chore: use more unique tag, include test sumary in github actions

2ef1ccb

Merge pull request #116 from qld-gov-au/chore_container_uplift

402f1cc

chore: use more unique tag, include test sumary in github actions

duttonw merged commit 9f398b0 into ckan:master Nov 21, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhancement/Bugfix: Downstream qld-gov-au fix's #232

Enhancement/Bugfix: Downstream qld-gov-au fix's #232

duttonw commented Oct 25, 2024

ThrawnCA Oct 25, 2024

duttonw Oct 30, 2024 •

edited

Loading

Enhancement/Bugfix: Downstream qld-gov-au fix's #232

Enhancement/Bugfix: Downstream qld-gov-au fix's #232

Conversation

duttonw commented Oct 25, 2024

ThrawnCA Oct 25, 2024

Choose a reason for hiding this comment

duttonw Oct 30, 2024 • edited Loading

Choose a reason for hiding this comment

duttonw Oct 30, 2024 •

edited

Loading