Skip to content

home

Jump to bottom
David Hoese edited this page Oct 16, 2015 · 37 revisions

CSPOV? huh?

It's a working title. Community Satellite Persistence of Vision? Presenter of Variables?

purpose

This is a moon-shot standalone rapid prototype for fluidly presenting meteorological imagery - specifically large-scale datasets like AHI/ABI/AMI - in combination with other sources. It should be immediately approachable to the novice, sufficient in accuracy for real analysis exercises, and free for use and modification by the community.

Ultimately, this should make use of the capability of modern multiplatform OpenGL (including WebGL at some point) to provide full-resolution imagery manipulation, animation, as well as scientific combination and enhancement capability in a classroom setting.

what we hope to learn

A lot of capability of modern computers resides in the video subsystem. Efforts such as vispy show that a lot of heavy lifting for time-critical or data-intensive rendering can be done using well-standardized OpenGL capabilities across languages, operating systems, and even in the web browser using WebGL now.

By smartly buffering into OpenGL the right subset tiles of a huge (up to 22000x22000 x16bands x12frames) dataset, we hope to achieve a fluid and resource-efficient user experience for data analysis.

Later experiments will include using GLSL (shading language) to allow real-time adjustments, enhancements and combinations of bands to be rendered in realtime.

Once we have a stable and useful system of applications/scripts/plugins, we can consider porting or bridging the engine and tools to other target systems, e.g. Web/mobile/tablet.

methodology

The initial prototyping is done in Anaconda Python 3 using PyQt4 and PyOpenGL. This is a stable, science-oriented, modular and modern development environment available for Windows, Mac and Linux.

We will lean on other high-quality open source systems in the geoscience and satellite communities to provide subsystems, plugins or know-how. GDAL for large-scale data warping, PROJ.4 for projection math, Shapely for geographic area representation, grib_api for ground station data, et cetera. Much code will be borrowed from Polar2Grid, CSPP, and so on.

Establishing simple-to-approach extensibility and cooperation with other programs is a priority. Building a monolithic system incapable of shedding past missteps is a sure route to failure. A simple internal data representation ("workspace") should allow others to bring their data into the system in minutes, not days. Authoring plugins should not require deep knowledge of the internals of the visualization application.

In fact, multiple complementary or alternative applications in multiple languages may have to cooperate in one workspace to get the user what s/he wants. For instance, a probe process written in MATLAB should be able to take shape selection information from the map GUI, and provide an HTML control updated with a D3.js javascript visualization.

dev environment setup and startup

Download Anaconda 3 v2.3 or newer for your platform. You don't need admin to install it.

conda create -n cspov python=3.4 anaconda
source activate cspov
conda install netCDF4=1.1.9 h5py=2.5.0 pyopengl=3.1.1a1 gdal=2.0.0 pyproj=1.9.4 numpy=1.9.3
pip install git+https://github.com/vispy/vispy.git@65edc3245aaf1e2e55d60d9c3c4abb68564b5972
pip install pyshp
export PYTHONPATH=/path/to/CSPOV/py
python -m cspov --glob-pattern /path/to/files/????/HS*B02*.tif

Developer info is here

Viper01 Development Setup

  1. Download and install Anaconda3 64-bit for Windows
  2. Download and install git 64-bit for Windows
  3. Open "Git CMD", "mkdir Repos", "cd Repos", "mkdir Git", "cd Git"
  4. Follow development environment instructions above except use "Git CMD" to download the vispy source (anaconda pip doesn't support git prefix). Then install vispy from anaconda terminal by changing directories and running "python setup.py install". Also instead of gdal 2.0 do "conda install gdal=1.11.2".
  5. Run "git clone https://gitlab.ssec.wisc.edu/rayg/CSPOV.git"
  6. Run "git checkout -b develop origin/develop"
  7. Use "Git Bash" to SSH in to ash and port forward test data from /delta to /c/Users/guamadmin/test_data
  8. In the anaconda terminal change to the CSPOV git checkout, then to the "py" directory, and run "python setup.py develop".
  9. Run "python -m cspov -vvv --glob-pattern C:\Users\guamadmin\test_data\0??0\HSB02merc.tif".

Viper02 Production Setup

  1. Download and install Anaconda 64-bit for Windows with Python 3.4

    a. Register Python 3.4 as the system python

  2. Open Anaconda Command Prompt and create an Anaconda subenvironment and activate it:

    conda create -n cspov python=3.4 anaconda activate cspov

  3. Install CSPOV and its dependencies:

    conda install -c http://larch.ssec.wisc.edu/channels/cspov cspov

  4. Create easy update and run scripts on the desktop...TODO.

Mac Production Installation

  1. Install Anaconda for 64-bit Mac with Python 3.4 (Google "Download Anaconda" to find download and installation instructions). Make sure that after installation you can do "which python" in the terminal and it returns the Anaconda path. This may require temporary or permanent PATH manipulation.

  2. Run the following to install CSPOV and its dependencies (mid-CSPOV/SIFT rename):

    conda create -n sift python=3.4 anaconda source activate sift conda install -c http://larch.ssec.wisc.edu/channels/cspov cspov

  3. Run the program by running:

    python -m cspov

There are other command line options for loading geotiffs automatically or changing other defaults, use -h to see them. NOTE: You must run source activate sift every time before running python -m cspov.

Clone this wiki locally