Skip to content

Commit

Permalink
Removed notebook outputs from commit. Updated documentation.
Browse files Browse the repository at this point in the history
  • Loading branch information
@GeorgeRought-NOAA
GeorgeRought-NOAA committed Jan 16, 2025
1 parent 4918d42 commit 15f7766
Showing 1 changed file with 14 additions and 220 deletions.
234 changes: 14 additions & 220 deletions Core/Manual_Workflows/static_services/Critical_Infrastructure_Service.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## This script ingests the Critical Infrastructure Service into HydroVIS into the egis database. ##"
"## This script ingests the Critical Infrastructure Service into the HydroVIS EGIS Database. ##"
]
},
{
Expand Down Expand Up @@ -51,30 +51,18 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The Critical Infrastructure point layer is hosted as a feature service. To retreive it within this script, you'll have to login to your TI ArcGIS Online account. You must setup environment variables \"EKS_EGIS_TI_USERNAME\" and \"EKS_EGIS_TI_PASSWORD\" in the windows environment variables. Once done, you'll need to close/reopen your IDE for it to successfully find your environment variables. "
"The Critical Infrastructure point layer is hosted as a feature service for storage and accessibility purposes. To retreive it within this script, you'll have to login to your TI ArcGIS Online account. You must setup environment variables \"EKS_EGIS_TI_USERNAME\" and \"EKS_EGIS_TI_PASSWORD\" in the windows environment variables. Once done, you'll need to close/reopen your IDE for it to successfully find your environment variables. \n",
"\n",
"This data is from https://services2.arcgis.com/C8EMgrsFcRFL6LrL/arcgis/rest/services/All_Infrastructure_Merge/FeatureServer/0. This data can still be accessed by logging in to https://noaa.maps.arcgis.com/home/ and finding the \"All_Infrastructure_Merge\" feature service. This data is derived from FEMA infrastructure datasaets. \n",
"\n",
"To get an editable copy of this data, the services2.arcgis.com link was used to connect to a new server in ArcGIS Pro. Feature service -> feature service in the geoprocessing toolbox was used to save it as a new local geodatabase. Then, for future accessbility, the data was published as a feature service within the EKS EGIS TI portal with organization wide access. This feature service is then connected to below for ingestion to the egis database."
]
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Setting `verify_cert` to False is a security risk, use at your own risk.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Attempting to login...\n",
"Succesfully logged in as george.rought@noaa.gov\n"
]
}
],
"outputs": [],
"source": [
"#Defining constants\n",
"FEATURE_SERVICE_URL = \"https://eks-maps-testing.water.noaa.gov/server/rest/services/Hosted/static_public_critical_infrastructure_points_fema/FeatureServer/0/\"\n",
Expand Down Expand Up @@ -201,173 +189,9 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Fetching data from: https://eks-maps-testing.water.noaa.gov/server/rest/services/Hosted/static_public_critical_infrastructure_points_fema/FeatureServer/0/\n",
"Fetched 2000 features. Total so far: 2000\n",
"Fetched 2000 features. Total so far: 4000\n",
"Fetched 2000 features. Total so far: 6000\n",
"Fetched 2000 features. Total so far: 8000\n",
"Fetched 2000 features. Total so far: 10000\n",
"Fetched 2000 features. Total so far: 12000\n",
"Fetched 2000 features. Total so far: 14000\n",
"Fetched 2000 features. Total so far: 16000\n",
"Fetched 2000 features. Total so far: 18000\n",
"Fetched 2000 features. Total so far: 20000\n",
"Fetched 2000 features. Total so far: 22000\n",
"Fetched 2000 features. Total so far: 24000\n",
"Fetched 2000 features. Total so far: 26000\n",
"Fetched 2000 features. Total so far: 28000\n",
"Fetched 2000 features. Total so far: 30000\n",
"Fetched 2000 features. Total so far: 32000\n",
"Fetched 2000 features. Total so far: 34000\n",
"Fetched 2000 features. Total so far: 36000\n",
"Fetched 2000 features. Total so far: 38000\n",
"Fetched 2000 features. Total so far: 40000\n",
"Fetched 2000 features. Total so far: 42000\n",
"Fetched 2000 features. Total so far: 44000\n",
"Fetched 2000 features. Total so far: 46000\n",
"Fetched 2000 features. Total so far: 48000\n",
"Fetched 2000 features. Total so far: 50000\n",
"Fetched 2000 features. Total so far: 52000\n",
"Fetched 2000 features. Total so far: 54000\n",
"Fetched 2000 features. Total so far: 56000\n",
"Fetched 2000 features. Total so far: 58000\n",
"Fetched 2000 features. Total so far: 60000\n",
"Fetched 2000 features. Total so far: 62000\n",
"Fetched 2000 features. Total so far: 64000\n",
"Fetched 2000 features. Total so far: 66000\n",
"Fetched 2000 features. Total so far: 68000\n",
"Fetched 2000 features. Total so far: 70000\n",
"Fetched 2000 features. Total so far: 72000\n",
"Fetched 2000 features. Total so far: 74000\n",
"Fetched 2000 features. Total so far: 76000\n",
"Fetched 2000 features. Total so far: 78000\n",
"Fetched 2000 features. Total so far: 80000\n",
"Fetched 2000 features. Total so far: 82000\n",
"Fetched 2000 features. Total so far: 84000\n",
"Fetched 2000 features. Total so far: 86000\n",
"Fetched 2000 features. Total so far: 88000\n",
"Fetched 2000 features. Total so far: 90000\n",
"Fetched 2000 features. Total so far: 92000\n",
"Fetched 2000 features. Total so far: 94000\n",
"Fetched 2000 features. Total so far: 96000\n",
"Fetched 2000 features. Total so far: 98000\n",
"Fetched 2000 features. Total so far: 100000\n",
"Fetched 2000 features. Total so far: 102000\n",
"Fetched 2000 features. Total so far: 104000\n",
"Fetched 2000 features. Total so far: 106000\n",
"Fetched 2000 features. Total so far: 108000\n",
"Fetched 2000 features. Total so far: 110000\n",
"Fetched 2000 features. Total so far: 112000\n",
"Fetched 2000 features. Total so far: 114000\n",
"Fetched 2000 features. Total so far: 116000\n",
"Fetched 2000 features. Total so far: 118000\n",
"Fetched 2000 features. Total so far: 120000\n",
"Fetched 2000 features. Total so far: 122000\n",
"Fetched 2000 features. Total so far: 124000\n",
"Fetched 2000 features. Total so far: 126000\n",
"Fetched 2000 features. Total so far: 128000\n",
"Fetched 2000 features. Total so far: 130000\n",
"Fetched 2000 features. Total so far: 132000\n",
"Fetched 2000 features. Total so far: 134000\n",
"Fetched 2000 features. Total so far: 136000\n",
"Fetched 2000 features. Total so far: 138000\n",
"Fetched 2000 features. Total so far: 140000\n",
"Fetched 2000 features. Total so far: 142000\n",
"Fetched 2000 features. Total so far: 144000\n",
"Fetched 2000 features. Total so far: 146000\n",
"Fetched 2000 features. Total so far: 148000\n",
"Fetched 2000 features. Total so far: 150000\n",
"Fetched 2000 features. Total so far: 152000\n",
"Fetched 2000 features. Total so far: 154000\n",
"Fetched 2000 features. Total so far: 156000\n",
"Fetched 2000 features. Total so far: 158000\n",
"Fetched 2000 features. Total so far: 160000\n",
"Fetched 2000 features. Total so far: 162000\n",
"Fetched 2000 features. Total so far: 164000\n",
"Fetched 2000 features. Total so far: 166000\n",
"Fetched 2000 features. Total so far: 168000\n",
"Fetched 2000 features. Total so far: 170000\n",
"Fetched 2000 features. Total so far: 172000\n",
"Fetched 2000 features. Total so far: 174000\n",
"Fetched 2000 features. Total so far: 176000\n",
"Fetched 2000 features. Total so far: 178000\n",
"Fetched 2000 features. Total so far: 180000\n",
"Fetched 2000 features. Total so far: 182000\n",
"Fetched 2000 features. Total so far: 184000\n",
"Fetched 2000 features. Total so far: 186000\n",
"Fetched 2000 features. Total so far: 188000\n",
"Fetched 2000 features. Total so far: 190000\n",
"Fetched 2000 features. Total so far: 192000\n",
"Fetched 2000 features. Total so far: 194000\n",
"Fetched 2000 features. Total so far: 196000\n",
"Fetched 2000 features. Total so far: 198000\n",
"Fetched 2000 features. Total so far: 200000\n",
"Fetched 2000 features. Total so far: 202000\n",
"Fetched 2000 features. Total so far: 204000\n",
"Fetched 2000 features. Total so far: 206000\n",
"Fetched 2000 features. Total so far: 208000\n",
"Fetched 2000 features. Total so far: 210000\n",
"Fetched 2000 features. Total so far: 212000\n",
"Fetched 2000 features. Total so far: 214000\n",
"Fetched 2000 features. Total so far: 216000\n",
"Fetched 2000 features. Total so far: 218000\n",
"Fetched 2000 features. Total so far: 220000\n",
"Fetched 2000 features. Total so far: 222000\n",
"Fetched 2000 features. Total so far: 224000\n",
"Fetched 2000 features. Total so far: 226000\n",
"Fetched 2000 features. Total so far: 228000\n",
"Fetched 2000 features. Total so far: 230000\n",
"Fetched 2000 features. Total so far: 232000\n",
"Fetched 2000 features. Total so far: 234000\n",
"Fetched 2000 features. Total so far: 236000\n",
"Fetched 2000 features. Total so far: 238000\n",
"Fetched 2000 features. Total so far: 240000\n",
"Fetched 2000 features. Total so far: 242000\n",
"Fetched 2000 features. Total so far: 244000\n",
"Fetched 2000 features. Total so far: 246000\n",
"Fetched 2000 features. Total so far: 248000\n",
"Fetched 2000 features. Total so far: 250000\n",
"Fetched 2000 features. Total so far: 252000\n",
"Fetched 2000 features. Total so far: 254000\n",
"Fetched 2000 features. Total so far: 256000\n",
"Fetched 2000 features. Total so far: 258000\n",
"Fetched 2000 features. Total so far: 260000\n",
"Fetched 2000 features. Total so far: 262000\n",
"Fetched 2000 features. Total so far: 264000\n",
"Fetched 2000 features. Total so far: 266000\n",
"Fetched 2000 features. Total so far: 268000\n",
"Fetched 2000 features. Total so far: 270000\n",
"Fetched 2000 features. Total so far: 272000\n",
"Fetched 2000 features. Total so far: 274000\n",
"Fetched 2000 features. Total so far: 276000\n",
"Fetched 2000 features. Total so far: 278000\n",
"Fetched 2000 features. Total so far: 280000\n",
"Fetched 2000 features. Total so far: 282000\n",
"Fetched 2000 features. Total so far: 284000\n",
"Fetched 2000 features. Total so far: 286000\n",
"Fetched 2000 features. Total so far: 288000\n",
"Fetched 2000 features. Total so far: 290000\n",
"Fetched 2000 features. Total so far: 292000\n",
"Fetched 2000 features. Total so far: 294000\n",
"Fetched 2000 features. Total so far: 296000\n",
"Fetched 2000 features. Total so far: 298000\n",
"Fetched 2000 features. Total so far: 300000\n",
"Fetched 2000 features. Total so far: 302000\n",
"Fetched 2000 features. Total so far: 304000\n",
"Fetched 2000 features. Total so far: 306000\n",
"Fetched 923 features. Total so far: 306923\n",
"No more features found.\n",
"Total features fetched: 306923\n"
]
}
],
"outputs": [],
"source": [
"#Fetch the data\n",
"print(f\"Fetching data from: {FEATURE_SERVICE_URL}\")\n",
Expand All @@ -376,31 +200,9 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"No ObjectID column found. Making it from scratch.\n",
"\n",
"Sample of the GeoDataFrame with the added 'oid' column:\n",
"country naics_code specialty excess_bed val_method tot_enroll city latitude county level_ source type st_grade sourcedate st_grade_1 alias state id helipad build_type end_grade longitude zip trauma naics_desc website address naicscode telephone addressbui tot_emp population enrollment val_date zipcode alt_name naicsdescr ft_teacher name beds objectid status geometry oid\n",
" USA 0 0 HEBRON 42.465566 MCHENRY HIGH 09 NaN IL SCHOOLS 12 -88.431010 60034 ELEMENTARY AND SECONDARY SCHOOLS NOT AVAILABLE 9604 ILLINOIS ST (815) 648-2442 0 123 112 NaN 11 ALDEN-HEBRON HIGH SCHOOL 0 1 POINT (-9844095.045 5230976.133) 1\n",
" USA 0 0 CHARLOTTE 35.231909 MECKLENBURG ELEMENTARY KG NaN NC SCHOOLS 08 -80.911501 28208 ELEMENTARY AND SECONDARY SCHOOLS http://schools.cms.k12.nc.us/westerlyhillsES 4420 DENVER AVE (980) 343-6021 0 540 495 NaN 45 WESTERLY HILLS ACADEMY 0 2 POINT (-9007027.123 4195441.532) 2\n",
" USA 0 0 FRANKLIN 39.489753 JOHNSON ELEMENTARY PK NaN IN SCHOOLS 04 -86.062430 46131 ELEMENTARY AND SECONDARY SCHOOLS NOT AVAILABLE 965 GRIZZLY CUB DR (317) 346-8900 0 503 478 NaN 25 NORTHWOOD ELEMENTARY SCHOOL 0 3 POINT (-9580425.87 4792069.206) 3\n",
" USA 0 0 QUEEN CREEK 33.242888 MARICOPA ELEMENTARY PK NaN AZ SCHOOLS 06 -111.687306 85298 ELEMENTARY AND SECONDARY SCHOOLS NOT AVAILABLE 22700 S. POWER RD. (480) 812-7000 0 1009 958 NaN 51 DR. GARY AND ANNETTE AUXIER ELEMENTARY SCHOOL 0 4 POINT (-12432974.033 3927587.87) 4\n",
" USA 0 0 QUEEN CREEK 33.222170 MARICOPA HIGH 07 NaN AZ SCHOOLS 12 -111.683768 85142 ELEMENTARY AND SECONDARY SCHOOLS NOT AVAILABLE 24901 S. POWER RD. (480) 424-8100 0 2409 2294 NaN 115 DR. CAMILLE CASTEEL HIGH SCHOOL 0 5 POINT (-12432580.229 3924830.599) 5\n",
"\n",
"Columns in the GeoDataFrame\n",
"['country', 'naics_code', 'specialty', 'excess_bed', 'val_method', 'tot_enroll', 'city', 'latitude', 'county', 'level_', 'source', 'type', 'st_grade', 'sourcedate', 'st_grade_1', 'alias', 'state', 'id', 'helipad', 'build_type', 'end_grade', 'longitude', 'zip', 'trauma', 'naics_desc', 'website', 'address', 'naicscode', 'telephone', 'addressbui', 'tot_emp', 'population', 'enrollment', 'val_date', 'zipcode', 'alt_name', 'naicsdescr', 'ft_teacher', 'name', 'beds', 'objectid', 'status', 'geometry', 'oid']\n",
"\n",
"Number of rows in datraframe: \n",
"306923\n"
]
}
],
"outputs": [],
"source": [
"#Create GeoDataFrame from the downloaded features\n",
"df = create_dataframe_from_features(features)"
Expand Down Expand Up @@ -434,17 +236,9 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Successfully copied dev.static_public_critical_infrastructure_points_fema from the VIZ DB to reference.static_public_critical_infrastructure_points_fema in the EGIS DB!\n"
]
}
],
"outputs": [],
"source": [
"move_data_from_viz_to_egis('dev.static_public_critical_infrastructure_points_fema', 'reference.static_public_critical_infrastructure_points_fema')"
]
Expand Down

0 comments on commit 15f7766

Please sign in to comment.