DYD Data

Overview

What heating/cooling setpoints are appropriate for the reference building in a high-performance residential standard?

The Task

Extract the data from the BigQuery dataset (7.6TB) and get in a format that can be analyzed by staff at Mathis Consulting. The team determined that it wanted the data by state/province.

Data

There are two tables in the dataset, “dyd” and “meta_data” which exist in the Google Cloud Platform. The “dyd” table is quite large and has a lot of fields that are not needed. The “meta_data” table is smaller. The “Identifier” field exists in both tables and will be the field in which data will be merged.

Process

In Big Query, clean extract the wanted columns from the tables and clean all the data by removing duplicates and NULLS.
Merge the two datasets on "Identifier."
Select the wanted time and dates (by day hours and night hours, month).
Establish final queries based state, year, day/night.
Export tables to CSV files. These tables are very large.
Bring large tables exported from Big Query in to Jupyter Notebook and aggregate the data for temperature control, heating, and cooling based on each identifier.
Export CSV files for each state, month, and day/night hours.
Added a checker to the notebook to ensure the data was correct per state.

https://klharp.github.io/DYD_Data/

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
Scraper_Output/State_Month_Day		Scraper_Output/State_Month_Day
data		data
forTableau		forTableau
static		static
.gitignore		.gitignore
DYD_AK-day.ipynb		DYD_AK-day.ipynb
DYD_AL-day.ipynb		DYD_AL-day.ipynb
DYD_AR-day.ipynb		DYD_AR-day.ipynb
DYD_AZ-day.ipynb		DYD_AZ-day.ipynb
DYD_CA-day.ipynb		DYD_CA-day.ipynb
DYD_CO-day.ipynb		DYD_CO-day.ipynb
DYD_CT-day.ipynb		DYD_CT-day.ipynb
DYD_DC-day.ipynb		DYD_DC-day.ipynb
DYD_DE-day.ipynb		DYD_DE-day.ipynb
DYD_FL-day.ipynb		DYD_FL-day.ipynb
DYD_GA-day.ipynb		DYD_GA-day.ipynb
DYD_HI-day.ipynb		DYD_HI-day.ipynb
DYD_IA-day.ipynb		DYD_IA-day.ipynb
DYD_ID-day.ipynb		DYD_ID-day.ipynb
DYD_IL-day.ipynb		DYD_IL-day.ipynb
DYD_IN-day.ipynb		DYD_IN-day.ipynb
DYD_KS-day.ipynb		DYD_KS-day.ipynb
DYD_KY-day.ipynb		DYD_KY-day.ipynb
DYD_LA-day.ipynb		DYD_LA-day.ipynb
DYD_MA-day.ipynb		DYD_MA-day.ipynb
DYD_MD-day.ipynb		DYD_MD-day.ipynb
DYD_ME-day.ipynb		DYD_ME-day.ipynb
DYD_MI-day.ipynb		DYD_MI-day.ipynb
DYD_MN-day.ipynb		DYD_MN-day.ipynb
DYD_MO-day.ipynb		DYD_MO-day.ipynb
DYD_MS-day.ipynb		DYD_MS-day.ipynb
DYD_MT-day.ipynb		DYD_MT-day.ipynb
DYD_NC-day.ipynb		DYD_NC-day.ipynb
DYD_ND-day.ipynb		DYD_ND-day.ipynb
DYD_NE-day.ipynb		DYD_NE-day.ipynb
DYD_NH-day.ipynb		DYD_NH-day.ipynb
DYD_NJ-day.ipynb		DYD_NJ-day.ipynb
DYD_NM-day.ipynb		DYD_NM-day.ipynb
DYD_NV-day.ipynb		DYD_NV-day.ipynb
DYD_NY-day.ipynb		DYD_NY-day.ipynb
DYD_OH-day.ipynb		DYD_OH-day.ipynb
DYD_OK-day.ipynb		DYD_OK-day.ipynb
DYD_OR-day.ipynb		DYD_OR-day.ipynb
DYD_PA-day.ipynb		DYD_PA-day.ipynb
DYD_RI-day.ipynb		DYD_RI-day.ipynb
DYD_SC-day.ipynb		DYD_SC-day.ipynb
DYD_SD-day.ipynb		DYD_SD-day.ipynb
DYD_TN-day.ipynb		DYD_TN-day.ipynb
DYD_TX-day.ipynb		DYD_TX-day.ipynb
DYD_UT-day.ipynb		DYD_UT-day.ipynb
DYD_VA-day.ipynb		DYD_VA-day.ipynb
DYD_VT-day.ipynb		DYD_VT-day.ipynb
DYD_WA-day.ipynb		DYD_WA-day.ipynb
DYD_WI-day.ipynb		DYD_WI-day.ipynb
DYD_WV-day.ipynb		DYD_WV-day.ipynb
DYD_WY-day.ipynb		DYD_WY-day.ipynb
Project 2 ETL notebook.ipynb		Project 2 ETL notebook.ipynb
README.md		README.md
cityZipMerge.ipynb		cityZipMerge.ipynb
index.html		index.html

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

DYD Data

Overview

The Task

Data

Process

About

Releases

Packages

Languages

klharp/DYD_Data

Folders and files

Latest commit

History

Repository files navigation

DYD Data

Overview

The Task

Data

Process

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages