BCDA-3710 - Resolve issues with stale db connections

[mtrang1263]

Fixes BCDA-3710

Previously, our driver dependencies (pg [v1.0.0] and pgx [v3.2.0]) had issues with re-using stale connections. For example, if our database instances were restarted, we would receive invalid/dead connection errors.

Issues:

1. pg issue: db operation fails with "broken pipe" instead of reconnecting transparently after server restart lib/pq#870
2. pgx issue: reduce unecessary memory allocation jackc/pgx#49

We cannot upgrade our pgx driver to v4 because of an incompatibility with our que-go dependency. que-go relies on pgx v3 with no plans to move up to pgx v4.

The pg driver issue was resolved in this PR.

Change Details

1. Upgrade pg driver from v1.0.0 to v1.9.0
2. Add connection pooling goroutine that checks and verifies the connections in our pools. Implementation was based off of the pgxv4 fix.

No changes are needed to the pgx driver used in bcdaworker. Since we instantiate a quego worker, there's goroutine(s) that run that verify and refresh the connection pool. See.

Security Implications

• new software dependencies
  Upgraded our pg client from v1.0.0 to v1.9.0

• security controls or supporting software altered

• new data stored or transmitted

• security checklist is completed for this change

• requires more information or team discussion to evaluate security implications

• no PHI/PII is affected by this change

Acceptance Validation

1. Verified the stale connection issue was resolved locally by running the following process.

make load-fixtures
docker-compose restart db queue
make smoke-test

When running this, we would encounter failures because of the invalid connections:

{"file":"/go/src/github.com/CMSgov/bcda-app/bcda/api/requests.go:293","func":"github.com/CMSgov/bcda-app/bcda/api.(*Handler).bulkRequest","level":"error","msg":"conn is dead","time":"2020-12-11T13:27:49-05:00"}

2. Ran multiple smoke tests jobs after restarting the dev database to verify that connections are automatically refreshed.

[mtrang1263]

Fixes issue related to invalid connections when using the pg driver.

[mtrang1263]

Still needed to keep this restart here because of issues uncovered in our CI process. It's caused by the database being dropped while the app is initializing.

Removing the TODO since the restart is not related to the restarting of the db instance.

[SMLuthi]

This LGTM! Not super comfortable with go routines but this all makes sense to me.

[PatKelsh]

looks good to me

[PatKelsh]

Good idea to link to resources explaining the implementation here

[SMLuthi]

Reapproving!