Outline Routes to Flask app

[krishnasanaka]

User Story

In order to facilitate the creation, reading, updating, and deletion of records in our harvest source database, datagovteam wants to design the routes of a flask app.

Links:

• Datagov Harvest Orchestrator repo: https://github.com/GSA/datagov-harvest-orchestrator
• Tutorial for adding a basic form to your Flask app: https://www.digitalocean.com/community/tutorials/how-to-use-flask-sqlalchemy-to-interact-with-databases-in-a-flask-application
• Previous work done on Flask / DB integration: Create Harvest DB and Table Structure #4612 (comment)

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I want to create, update, delete, and read records from our harvest database
  AND a list of potential routes
  THEN a group discussion will occur to determine what routes will be implemented.

Background

In continuation of flask app ticket(#4619 ) where we created GET and POST methods.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Create a list of routes to use
• Discuss the list and determine what routes should be implemented.
• Document decision(s) on why a route is included and the business purpose it accomplishes
• Document decision on why a route isn't included
• Document business logic on the implementation of the functionality bound to the route

[gujral-rei]

• Do we need an interface?
• Add different routes (delete, sync, edit etc)
• Outline success criteria for a fully functional solution
• Define databases we need
• Outline approach/requirements

[btylerburton]

WIP for Harvster UI routes here: https://docs.google.com/document/d/1XzfTrPxu-asJ_55GoeZ2UOJsie9CuCegStS28BAL_40/edit#heading=h.njcvzj6q25gg

[rshewitt]

this isn't a set list. just brainstorming.

notes:

• "/harvest_record/field_name?={field_name}"
  • field_name could be name, owner_org, identifier, author, author_email, maintainer, etc.. . there's a lot to pick from.

signature	HTTP method	description	Authed
/organization	GET	returns a list of all organizations	No
/organization	POST	add an organization	Yes
/organization/{org_id}	GET	return a single organization	No
/organization/{org_id}	PUT	update an organization	Yes
/organization/{org_id}	DELETE	delete an organization	Yes
/harvest_source	GET	returns a list of all harvest sources	No
/harvest_source	POST	add a harvest sources	Yes
/harvest_source/{source_id}	GET	return a single harvest source	No
/harvest_source/{source_id}	PUT	update a harvest source	Yes
/harvest_source/{source_id}	DELETE	delete a harvest source	Yes
/harvest_source/?organization_id={org_id}	GET	get all harvest sources of an organization	No
/harvest_record	GET	get all records	No
/harvest_record	POST	add a record	Yes
/harvest_record/{record_id}	GET	get a record	Yes
/harvest_record/{record_id}	PUT	update a record	Yes
/harvest_record/{record_id}	DELETE	delete a record	Yes
/harvest_record/field_name?={field_name}	GET	retrieve records based on general query	No
/harvest_job	GET	get all harvest jobs	No
/harvest_job	POST	add a job	Yes
/harvest_job/{job_id}	GET	return a job	No
/harvest_job/{job_id}	DELETE	delete a job	Yes
/harvest_job/{job_id}/?include_source={boolean}	GET	get a job and associated source info	No
/harvest_job/?date_start={date}&date_end={date}	GET	get jobs within a date/time range	No
/harvest_job/?harvest_source_id={source_id}	GET	get all jobs of a harvest source	No
/harvest_error	GET	get all harvest errors	No
/harvest_error	POST	add a harvest error	Yes
/harvest_error/{error_id}	GET	get a single harvest error	No
/harvest_error/?harvest_source_id={source_id}	GET	get all harvest errors of a harvest source	No
/harvest_error/?job_id={job_id}	GET	get all errors of a job	No
/harvest_error/?date_start={date}&date_end={date}	GET	get all errors within a date range	No
/harvest_error/?type={error_type}	GET	get all errors of a particular type ( i.e. job or record )	No

[rshewitt]

adding an access pattern section above our table definitions in the harvester 2.0 doc

[rshewitt]

no routes should return a massive amount of data ( i.e. getting all records from a table ). how many organization/sources/records/errors do we have now?

• organizations 132
• sources 917
• records 292,027
• errors 2,211,189

[rshewitt]

What are our access patterns?

• What is write-heavy?
  • Harvest error table
  • Harvest record table
• What is read-heavy?
  • Harvest error table?
  • Harvest record table
  • Harvest source table?

This is more of what's expected from the db but since the api talks to the db it could be valuable to include this?

[rshewitt]

going to break this out into 2 tickets since the content here involves designing our routes ( 1 ticket ) and implementing the design ( 1 ticket ). we have an existing implementation ( i.e @Jin-Sun-tts work ) but it seems there's still need for clarity on the design. we should talk about this as a stay-after or pairing tomorrow ( 03/11 ) because if the group feels that our current flask app covers everything that we need then maybe we don't go back to design

[rshewitt]

talking with @FuhuXia yesterday about a simple manual harvest workflow which looks like...

• button on the harvest source page which when clicked will post to something like /harvest_job/{source}, adding a record to the harvest job table
• meanwhile, a cron GHA running every 15 minutes will query the harvest job table looking for records/jobs with a status of new
• grab a job to perform and run it via cf run-task {app_name} -c python harvest.py -i { job_id }
• harvest.py will parse the incoming argument and grab the job/source information to perform the job ( this means we need a sqlachemy command for grabbing a job and joining it to its harvest source record ).

considering just this workflow, the sole purpose of the flask app at this moment is to facilitate CRUD operations in a database. these operations are reflected by the routes we choose.

[rshewitt]

one outcome of our route design discussion is finalizing the anatomy of the route. I like the pattern of using only nouns in the route and letting the http method provide the action/verb but if adding a verb like edit to the route makes sense to the team i'm cool with that too. i just want to be consistent with our decision.

[rshewitt]

discussion doc