refactor CMAQ ingest to be more efficient / faster

[mjstealey]

• Existing implementation works well for small sets, but takes weeks (or longer) to ingest data formatted at the 299 x 459 grid size

Example:

Processes started to ingest for 2011 CMAQ data:

25124 Sat Dec 23 20:24:15 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25268 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25269 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25270 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25271 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25272 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25273 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25274 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25275 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25276 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25277 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_
25278 Sat Dec 23 20:40:27 2017 ./venv/bin/python ./ingest-cmaq-file.py /projects/datatrans/CMAQ/2011/raw/CCTM_CMAQ_v51_Release_Oct23_NoDust_

As of 2018-01-16, the most populated set is January, 2011 with 90 of 459 columns completed. The rest of the months are between 21 and 24 of 459 completed.

psql (9.6.6)
Type "help" for help.

cmaq=# select utc_date_time::date, max(row), max(col) from exposure_data where utc_date_time::date >= '2011-01-01' group by utc_date_time::date order by utc_date_time::date;
 utc_date_time | max | max
---------------+-----+-----
 2011-01-01    | 299 |  90
 2011-01-02    | 299 |  90
 2011-01-03    | 299 |  90
 2011-01-04    | 299 |  90
 2011-01-05    | 299 |  90
 2011-01-06    | 299 |  90
 2011-01-07    | 299 |  90
 2011-01-08    | 299 |  90
 2011-01-09    | 299 |  90
 2011-01-10    | 299 |  90
 2011-01-11    | 299 |  90
 2011-01-12    | 299 |  90
 2011-01-13    | 299 |  90
 2011-01-14    | 299 |  90
 2011-01-15    | 299 |  90
 2011-01-16    | 299 |  90
 2011-01-17    | 299 |  90
 2011-01-18    | 299 |  90
 2011-01-19    | 299 |  90
 2011-01-20    | 299 |  90
 2011-01-21    | 299 |  90
 2011-01-22    | 299 |  90
 2011-01-23    | 299 |  90
 2011-01-24    | 299 |  90
 2011-01-25    | 299 |  90
 2011-01-26    | 299 |  90
 2011-01-27    | 299 |  90
 2011-01-28    | 299 |  90
 2011-01-29    | 299 |  90
 2011-01-30    | 299 |  90
 2011-01-31    | 299 |  90
 2011-02-01    | 299 |  90
 2011-02-02    | 299 |  24
 2011-02-03    | 299 |  24
 2011-02-04    | 299 |  24
 2011-02-05    | 299 |  24
 2011-02-06    | 299 |  24
 2011-02-07    | 299 |  24
...
 2011-12-22    | 299 |  21
 2011-12-23    | 299 |  21
 2011-12-24    | 299 |  21
 2011-12-25    | 299 |  21
 2011-12-26    | 299 |  21
 2011-12-27    | 299 |  21
 2011-12-28    | 299 |  21
 2011-12-29    | 299 |  21
 2011-12-30    | 299 |  21
 2011-12-31    | 299 |  21
 2012-01-01    | 299 |  21
(366 rows)