Refactor to class

[rshewitt]

Pull Request

Related to #4577

About

• refactoring harvester functionality and tests as classes
• i'm using the @property decorator to make some fields read-only. it serves a similar benefit to const in node.
  • i'm not using the decorator in some cases where the dtype is a dictionary because we'd need to create a custom dict class and overload the __setitem__ method. is it better to be consistent?
  • the non-dict fields which will be assigned after initialization have setter methods configured to them with basic type checking
• a majority of tests have been removed. this is for a couple reasons
  • the number of tests aren't needed ( e.g. validation and extract )
  • some tests weren't added back into this refactor in an effort to get this codebase available to @btylerburton more quickly so he can perform his load test.
  • furthermore, I simply wanted to get the coverage reasonably high while also minimizing our codebase as a sort-of fresh start.
• nothing is being done with the transform ( i.e no tests or functionality ). we're not transforming anything in our load test because our transformation service isn't ready although progress is actively being made so maybe we include it?
• RemoteCKAN is created outside of any dataclass to avoid any potential serialization issue when/if we decide to store an intermediate HarvestSource instance in somewhere like s3.
• Record instantiation occurs primarily in harvest_to_id_hash but only for records that need to be created or updated NOT deleted. for the sake of record-keeping ( no pun-intended ) I instantiated Record for deleted records immediately before deleting them on ckan.
• there's purposefully no try/excepts ( excluding the validation function because i copy/pasted that ) because we don't have a concrete idea of our exception handling.

PR TASKS

• The actual code changes.
• Tests written and passed.
• Any changes to docs?
• Bumped version number in pyproject.toml (also checked on PyPi).

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
harvester
__init__.py	0	0	100%
harvest.py	348	54	54	84%
utils.py	10	0	100%
TOTAL	358	54	85%

Tests	Skipped	Failures	Errors	Time
7	0 💤	0 ❌	0 🔥	0.963s ⏱️

[jbrown-xentity]

As someone who's been tracking this progress all week, my suggestion for reviewing this is to look at the code state at the latest commit, not the changes. GitHub doesn't know how to show this many changes, it's messy. But the code is much cleaner.

[jbrown-xentity]

This all looks good to me, reviewed in person.