From b8d0612881d3a1053e134f29c43fadc0725eb92c Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 20 Feb 2024 13:53:17 -0700 Subject: [PATCH 01/56] start to reorganize Overview page --- README.md | 40 ++++++++++++++++++++++++++++++++++------ mkdocs.yml | 4 ++-- 2 files changed, 36 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 657ca4c7..10f11fc9 100644 --- a/README.md +++ b/README.md @@ -30,32 +30,60 @@

-## **Overview** +## **Welcome to `earthaccess`** -*earthaccess* is a **python library to search, download or stream NASA Earth science data** with just a few lines of code. +`earthaccess` is a python library to **search for**, and **download** or **stream** NASA Earth science data with just a few lines of code. In the age of cloud computing, the power of open science only reaches its full potential if we have easy-to-use workflows that facilitate research in an inclusive, efficient and reproducible way. Unfortunately —as it stands today— scientists and students alike face a steep learning curve adapting to systems that have grown too complex and end up spending more time on the technicalities of the tools, cloud and NASA APIs than focusing on their important science. During several workshops organized by [NASA Openscapes](https://nasa-openscapes.github.io/events.html), the need to provide easy-to-use tools to our users became evident. Open science is a collaborative effort; it involves people from different technical backgrounds, and the data analysis to solve the pressing problems we face cannot be limited by the complexity of the underlying systems. Therefore, providing easy access to NASA Earthdata regardless of the data storage location (hosted within or outside of the cloud) is the main motivation behind this Python library. -## **Installing earthaccess** +Try it in your browser without installing anything! [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/nsidc/earthaccess/main) + + +## **Getting Started** + +### **Installing earthaccess** + +The latest release of `earthaccess` can be installed with `mamba`, `conda` or `pip`. We recommend using `mamba` because it is faster. You will need Python 3.8 or higher installed. -Install the latest release using conda +#### Using `mamba` + +```bash +mamba install -c conda-forge earthaccess +``` + +#### Using `conda` ```bash conda install -c conda-forge earthaccess ``` -Using Pip +#### Using `pip` ```bash pip install earthaccess ``` -Try it in your browser without installing anything! [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/nsidc/earthaccess/main) + +### Check `earthaccess` is installed + +This should run seamlessly (fingers-crossed). To check `earthaccess` is correctly installed you can start a python interpreter (either python or ipython) and run the following code. + +``` +$ python +Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 08:03:24) [GCC 12.3.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import earthaccess +>>> earthaccess.__version__ +'0.8.2' +``` + +> Note: +> Your `python` and `earthaccess` versions may be different. ## **Usage** diff --git a/mkdocs.yml b/mkdocs.yml index c1905ba3..0911390b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -46,8 +46,8 @@ plugins: - "**/.ipynb_checkpoints/*" nav: - - OVERVIEW: - - "Readme": "index.md" + - Getting Started: + - "index.md" - "Getting started": "tutorials/getting-started.ipynb" - "Resources": "resources.md" - HOW-TO: From 02cf2ec022a8a52397257af6629534ba37778689 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 19 Mar 2024 12:58:10 -0600 Subject: [PATCH 02/56] reorganized docs --- README.md | 90 +++++++---------------------------- docs/howto/access-data.md | 45 ++++++++++++++++++ docs/howto/authenticate.md | 31 ++++++++++++ docs/howto/search-granules.md | 21 ++++++++ 4 files changed, 115 insertions(+), 72 deletions(-) create mode 100644 docs/howto/access-data.md diff --git a/README.md b/README.md index 10f11fc9..bf53b7c6 100644 --- a/README.md +++ b/README.md @@ -34,11 +34,12 @@ `earthaccess` is a python library to **search for**, and **download** or **stream** NASA Earth science data with just a few lines of code. - -In the age of cloud computing, the power of open science only reaches its full potential if we have easy-to-use workflows that facilitate research in an inclusive, efficient and reproducible way. Unfortunately —as it stands today— scientists and students alike face a steep learning curve adapting to systems that have grown too complex and end up spending more time on the technicalities of the tools, cloud and NASA APIs than focusing on their important science. +Open science only reaches its full potential if we have easy-to-use workflows that facilitate research in an inclusive, efficient and reproducible way. Unfortunately —as it stands today— scientists and students alike face a steep learning curve adapting to systems that have grown too complex and end up spending more time on the technicalities of the tools, cloud and NASA APIs than focusing on their important science. During several workshops organized by [NASA Openscapes](https://nasa-openscapes.github.io/events.html), the need to provide easy-to-use tools to our users became evident. Open science is a collaborative effort; it involves people from different technical backgrounds, and the data analysis to solve the pressing problems we face cannot be limited by the complexity of the underlying systems. Therefore, providing easy access to NASA Earthdata regardless of the data storage location (hosted within or outside of the cloud) is the main motivation behind this Python library. +***earthaccess*** handles authentication with [NASA's Earthdata Login (EDL)](https://urs.earthdata.nasa.gov), search using NASA's [CMR](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) and access through [`fsspec`](https://github.com/fsspec/filesystem_spec). + Try it in your browser without installing anything! [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/nsidc/earthaccess/main) @@ -86,105 +87,50 @@ Type "help", "copyright", "credits" or "license" for more information. > Your `python` and `earthaccess` versions may be different. -## **Usage** +## **Get Data in 3 Steps** - -With *earthaccess* we can login, search and download data with a few lines of code and even more relevant, our code will work the same way if we are running it in the cloud or from our laptop. ***earthaccess*** handles authentication with [NASA's Earthdata Login (EDL)](https://urs.earthdata.nasa.gov), search using NASA's [CMR](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) and access through [`fsspec`](https://github.com/fsspec/filesystem_spec). +`earthaccess` allows you to search for and access data in as little as three steps. We give a very quick example below. These three steps allow you to get data whether you are working in the cloud or on your local laptop or workstation. Read the [User Guide](user_guide.qmd) for more information. If you want to quickly find how to perform some common searches and data access, +take a look at our [How-to](how_to.qmd) guide. The only requirement to use this library is to open a free account with NASA [EDL](https://urs.earthdata.nasa.gov). +### Step 1: Login -### **Authentication** - -By default, `earthaccess` with automatically look for your EDL account credentials in two locations: +To access NASA data, you have to login using your Earth Data Login credentials. You can register for a free Earth Data Login account [here](https://urs.earthdata.nasa.gov/). -1. A `~/.netrc` file -2. `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD` environment variables +By default, `earthaccess` will look for your Earth Data Login credentials in a `.netrc` file, or in environment variables `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD`. If you don't +have either of these set up, you can login manually. See [Authenticating](authenticate.qmd) to learn how to create a `.netrc` file or environment variables. -If neither of these options are configured, you can authenticate by calling the `earthaccess.login()` method -and manually entering your EDL account credentials. - -```python +``` import earthaccess earthaccess.login() ``` -Note you can pass `persist=True` to `earthaccess.login()` to have the EDL account credentials you enter -automatically saved to a `~/.netrc` file for future use. - - -Once you are authenticated with NASA EDL you can: -* Get a file from a DAAC using a `fsspec` session. -* Request temporary S3 credentials from a particular DAAC (needed to download or stream data from an S3 bucket in the cloud). -* Use the library to download or stream data directly from S3. -* Regenerate CMR tokens (used for restricted datasets) +### Step 2: Search for data +As an example, we'll search for data from the NASA ICESat-2 mission. ATL06 -### **Searching for data** - -Once we have selected our dataset we can search for the data granules using *doi*, *short_name* or *concept_id*. -If we are not sure or we don't know how to search for a particular dataset, we can start with the ["Introducing NASA earthaccess"](https://nsidc.github.io/earthaccess/tutorials/demo/#querying-for-datasets) tutorial or through the [NASA Earthdata Search portal](https://search.earthdata.nasa.gov/). For a complete list of search parameters we can use visit the extended [API documentation](https://earthaccess.readthedocs.io/en/latest/user-reference/api/api/). - -```python +``` results = earthaccess.search_data( - short_name='SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205', - cloud_hosted=True, + short_name='ATL06' bounding_box=(-10, 20, 10, 50), temporal=("1999-02", "2019-03"), count=10 ) - - ``` -Now that we have our results we can do multiple things: We can iterate over them to get HTTP (or S3) links, we can download the files to a local folder, or we can open these files and stream their content directly to other libraries e.g. xarray. - -### **Accessing the data** - -**Option 1: Using the data links** - -If we already have a workflow in place for downloading our data, we can use *earthaccess* as a search-only library and get HTTP links from our query results. This could be the case if our current workflow uses a different language and we only need the links as input. +### Step 3. Download the files -```python - -# if the data set is cloud hosted there will be S3 links available. The access parameter accepts "direct" or "external", direct access is only possible if you are in the us-west-2 region in the cloud. -data_links = [granule.data_links(access="direct") for granule in results] - -# or if the data is an on-prem dataset -data_links = [granule.data_links(access="external") for granule in results] +Once you have found the files you want, you can download them to your local machine. ``` - -> Note: *earthaccess* can get S3 credentials for us, or auhenticated HTTP sessions in case we want to use them with a different library. - -**Option 2: Download data to a local folder** - -This option is practical if you have the necessary space available on disk. The *earthaccess* library will print out the approximate size of the download and its progress. -```python files = earthaccess.download(results, "./local_folder") - ``` -**Option 3: Direct S3 Access - Stream data directly to xarray** - -This method works best if you are in the same Amazon Web Services (AWS) region as the data (us-west-2) and you are working with gridded datasets (processing level 3 and above). - -```python -import xarray as xr - -files = earthaccess.open(results) - -ds = xr.open_mfdataset(files) - -``` - -And that's it! Just one line of code, and this same piece of code will also work for data that are not hosted in the cloud, i.e. located at NASA storage centers. - - -> More examples coming soon! +If you are working in the cloud and the data files are hosted in the cloud, you can stream the data directly, without having to download data. See [Direct S3 Access]() ### Compatibility diff --git a/docs/howto/access-data.md b/docs/howto/access-data.md new file mode 100644 index 00000000..23fbf0d4 --- /dev/null +++ b/docs/howto/access-data.md @@ -0,0 +1,45 @@ +### **Accessing the data** + +**Option 1: Using the data links** + +If we already have a workflow in place for downloading our data, we can use *earthaccess* as a search-only library and get HTTP links from our query results. This could be the case if our current workflow uses a different language and we only need the links as input. + +```python + +# if the data set is cloud hosted there will be S3 links available. The access parameter accepts "direct" or "external", direct access is only possible if you are in the us-west-2 region in the cloud. +data_links = [granule.data_links(access="direct") for granule in results] + +# or if the data is an on-prem dataset +data_links = [granule.data_links(access="external") for granule in results] + +``` + +> Note: *earthaccess* can get S3 credentials for us, or auhenticated HTTP sessions in case we want to use them with a different library. + +**Option 2: Download data to a local folder** + +This option is practical if you have the necessary space available on disk. The *earthaccess* library will print out the approximate size of the download and its progress. +```python +files = earthaccess.download(results, "./local_folder") + +``` + +**Option 3: Direct S3 Access - Stream data directly to xarray** + +This method works best if you are in the same Amazon Web Services (AWS) region as the data (us-west-2) and you are working with gridded datasets (processing level 3 and above). + +```python +import xarray as xr + +files = earthaccess.open(results) + +ds = xr.open_mfdataset(files) + +``` + +And that's it! Just one line of code, and this same piece of code will also work for data that are not hosted in the cloud, i.e. located at NASA storage centers. + + +> More examples coming soon! + + diff --git a/docs/howto/authenticate.md b/docs/howto/authenticate.md index 26525ec6..5b3edd74 100644 --- a/docs/howto/authenticate.md +++ b/docs/howto/authenticate.md @@ -29,3 +29,34 @@ If you wish to enter your Earthdata Login credentials when prompted with optiona ```py auth = earthaccess.login(strategy="interactive", persist=True) ``` + + + +### **Authentication** + +By default, `earthaccess` with automatically look for your EDL account credentials in two locations: + +1. A `~/.netrc` file +2. `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD` environment variables + +If neither of these options are configured, you can authenticate by calling the `earthaccess.login()` method +and manually entering your EDL account credentials. + +```python +import earthaccess + +earthaccess.login() +``` + +Note you can pass `persist=True` to `earthaccess.login()` to have the EDL account credentials you enter +automatically saved to a `~/.netrc` file for future use. + + +Once you are authenticated with NASA EDL you can: + +* Get a file from a DAAC using a `fsspec` session. +* Request temporary S3 credentials from a particular DAAC (needed to download or stream data from an S3 bucket in the cloud). +* Use the library to download or stream data directly from S3. +* Regenerate CMR tokens (used for restricted datasets) + + diff --git a/docs/howto/search-granules.md b/docs/howto/search-granules.md index 9d106a82..06286eb9 100644 --- a/docs/howto/search-granules.md +++ b/docs/howto/search-granules.md @@ -13,3 +13,24 @@ results = earthaccess.search_data( ) ``` +### **Searching for data** + +Once we have selected our dataset we can search for the data granules using *doi*, *short_name* or *concept_id*. +If we are not sure or we don't know how to search for a particular dataset, we can start with the ["Introducing NASA earthaccess"](https://nsidc.github.io/earthaccess/tutorials/demo/#querying-for-datasets) tutorial or through the [NASA Earthdata Search portal](https://search.earthdata.nasa.gov/). For a complete list of search parameters we can use visit the extended [API documentation](https://earthaccess.readthedocs.io/en/latest/user-reference/api/api/). + +```python + +results = earthaccess.search_data( + short_name='SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205', + cloud_hosted=True, + bounding_box=(-10, 20, 10, 50), + temporal=("1999-02", "2019-03"), + count=10 +) + + +``` + +Now that we have our results we can do multiple things: We can iterate over them to get HTTP (or S3) links, we can download the files to a local folder, or we can open these files and stream their content directly to other libraries e.g. xarray. + + From 998ea02da2a7e262a3d7535d6feb2a4b1875416a Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 2 Apr 2024 11:59:02 -0600 Subject: [PATCH 03/56] Make "Welcome" page single level --- mkdocs.yml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 0911390b..3d5be946 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -46,10 +46,9 @@ plugins: - "**/.ipynb_checkpoints/*" nav: - - Getting Started: - - "index.md" - - "Getting started": "tutorials/getting-started.ipynb" - - "Resources": "resources.md" + - Welcome: "index.md" +# - "Getting started": "tutorials/getting-started.ipynb" +# - "Resources": "resources.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" - "Search NASA datasets using filters": "howto/search-collections.md" From bd0bf694c08b8159cc9909c6b61d3d919b5582fd Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 2 Apr 2024 14:27:28 -0600 Subject: [PATCH 04/56] add user_guide section --- mkdocs.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 3d5be946..d36c46d2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -49,6 +49,11 @@ nav: - Welcome: "index.md" # - "Getting started": "tutorials/getting-started.ipynb" # - "Resources": "resources.md" + - User Guide: + - "Overview": "user_guide/overview.md" + - "Authentication": "user_guide/authenticate.md" + - "Search": "user_guide/search.md" + - "Access": "user_guide/access.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" - "Search NASA datasets using filters": "howto/search-collections.md" From b5648d79356030b93fb925f1f47683554f7d46d0 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 2 Apr 2024 14:27:47 -0600 Subject: [PATCH 05/56] add user guide page stubs --- docs/user_guide/access.md | 9 +++++++++ docs/user_guide/authenticate.md | 3 +++ docs/user_guide/overview.md | 13 +++++++++++++ docs/user_guide/search.md | 5 +++++ 4 files changed, 30 insertions(+) create mode 100644 docs/user_guide/access.md create mode 100644 docs/user_guide/authenticate.md create mode 100644 docs/user_guide/overview.md create mode 100644 docs/user_guide/search.md diff --git a/docs/user_guide/access.md b/docs/user_guide/access.md new file mode 100644 index 00000000..8f369db8 --- /dev/null +++ b/docs/user_guide/access.md @@ -0,0 +1,9 @@ +# Access + +**Access** includes downloading files from a NASA DAAC or Earthdata Cloud to a local computer, a laptop, workstation or the storage space for a high-performace-computer (HPC); and also streaming data. + + +## Downloading data + + +## Streaming data \ No newline at end of file diff --git a/docs/user_guide/authenticate.md b/docs/user_guide/authenticate.md new file mode 100644 index 00000000..fd69f543 --- /dev/null +++ b/docs/user_guide/authenticate.md @@ -0,0 +1,3 @@ +# Authentication + +Introduces the `earthaccess.login` method for managing Earthdata Login and cloud credentials. \ No newline at end of file diff --git a/docs/user_guide/overview.md b/docs/user_guide/overview.md new file mode 100644 index 00000000..aed94d5e --- /dev/null +++ b/docs/user_guide/overview.md @@ -0,0 +1,13 @@ +# Overview + +`earthaccess` enables authentication, search and access for NASA Earth science data. The package enables users to find, and download or stream data archived both at NASA DAACs and in the AWS-hosted Earthdata Cloud. The User Guide introduces these three functions. + +The [Authentication](user_guide/authentication.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. + +The [Search](user_guide/search.md) section introduces how to search for NASA datasets (collections) and specific data granules. + +The [Access](user_guide/access.md) section introduces how to use `earthaccess` to download data to a local machine or stream data by opening it directly in memory. + +Additional usage information can be found in the User Reference section. The How-To sections contains short code examples for specific workflows. Longer tutorials for working with particular datasets are in the Tutorials section. + +The Glossary section contains definitions for NASA terminology and Cloud computing. \ No newline at end of file diff --git a/docs/user_guide/search.md b/docs/user_guide/search.md new file mode 100644 index 00000000..bb5043d0 --- /dev/null +++ b/docs/user_guide/search.md @@ -0,0 +1,5 @@ +# Search + +## `search_datasets` + +## `search_data` From 34f6b1eb2b920d4594b77d6764287dc3bf0bd271 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 16 Apr 2024 12:04:39 -0600 Subject: [PATCH 06/56] remove glossary link from welcome --- README.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/README.md b/README.md index 90ab83a1..2b2de687 100644 --- a/README.md +++ b/README.md @@ -152,10 +152,6 @@ Welcome! 😊👋 ### [Project Board](https://github.com/nsidc/earthdata/discussions). -### Glossary - -NASA Earth Science Glossary - ## License earthaccess is licensed under the MIT license. See [LICENSE](LICENSE.txt). From 60ab0ac3b13a9a30626cca271def70a5e226cfb2 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 16 Apr 2024 12:24:53 -0600 Subject: [PATCH 07/56] Move license to welcome and remove level of support sections. --- README.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 2b2de687..47b9fdbf 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,8 @@ Open science only reaches its full potential if we have easy-to-use workflows th During several workshops organized by [NASA Openscapes](https://nasa-openscapes.github.io/events.html), the need to provide easy-to-use tools to our users became evident. Open science is a collaborative effort; it involves people from different technical backgrounds, and the data analysis to solve the pressing problems we face cannot be limited by the complexity of the underlying systems. Therefore, providing easy access to NASA Earthdata regardless of the data storage location (hosted within or outside of the cloud) is the main motivation behind this Python library. +The library is an open source community effort under an [MIT license](LICENSE.txt). We welcome contributions to improve `earthaccess`. Please see the [Contributing Guide](CONTRIBUTING.md) to learn how to get involved. + ***earthaccess*** handles authentication with [NASA's Earthdata Login (EDL)](https://urs.earthdata.nasa.gov), search using NASA's [CMR](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) and access through [`fsspec`](https://github.com/fsspec/filesystem_spec). Try it in your browser without installing anything! [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/nsidc/earthaccess/main) @@ -151,15 +153,3 @@ Welcome! 😊👋 > Please see the [Contributing Guide](CONTRIBUTING.md). ### [Project Board](https://github.com/nsidc/earthdata/discussions). - -## License - -earthaccess is licensed under the MIT license. See [LICENSE](LICENSE.txt). - -## Level of Support - -
-
- This repository is supported by a joint effort of NSIDC, NASA DAACs, and the Earth science community, and we welcome any contribution in the form of issue submissions, pull requests, or discussions. Issues labeled as https://github.com/nsidc/earthaccess/labels/good%20first%20issue are a great place to get started. -
- From bf82ad74de5a9c1d469bd3adb4aa4c710855b05c Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 16 Apr 2024 12:50:20 -0600 Subject: [PATCH 08/56] move getting started to quick_start.md --- README.md | 90 +----------------------------------- docs/welcome/quick_start.md | 90 ++++++++++++++++++++++++++++++++++++ docs/welcome/work_with_us.md | 1 + mkdocs.yml | 4 +- 4 files changed, 95 insertions(+), 90 deletions(-) create mode 100644 docs/welcome/quick_start.md create mode 100644 docs/welcome/work_with_us.md diff --git a/README.md b/README.md index 47b9fdbf..5a5a2f72 100644 --- a/README.md +++ b/README.md @@ -45,94 +45,10 @@ The library is an open source community effort under an [MIT license](LICENSE.tx Try it in your browser without installing anything! [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/nsidc/earthaccess/main) -## **Getting Started** -### **Installing earthaccess** +## Getting Started -The latest release of `earthaccess` can be installed with `mamba`, `conda` or `pip`. We recommend using `mamba` because it is faster. - -You will need Python 3.8 or higher installed. - -#### Using `mamba` - -```bash -mamba install -c conda-forge earthaccess -``` - -#### Using `conda` - -```bash -conda install -c conda-forge earthaccess -``` - -#### Using `pip` - -```bash -pip install earthaccess -``` - - -### Check `earthaccess` is installed - -This should run seamlessly (fingers-crossed). To check `earthaccess` is correctly installed you can start a python interpreter (either python or ipython) and run the following code. - -``` -$ python -Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 08:03:24) [GCC 12.3.0] on linux -Type "help", "copyright", "credits" or "license" for more information. ->>> import earthaccess ->>> earthaccess.__version__ -'0.8.2' -``` - -> Note: -> Your `python` and `earthaccess` versions may be different. - - -## **Get Data in 3 Steps** - -`earthaccess` allows you to search for and access data in as little as three steps. We give a very quick example below. These three steps allow you to get data whether you are working in the cloud or on your local laptop or workstation. Read the [User Guide](user_guide.qmd) for more information. If you want to quickly find how to perform some common searches and data access, -take a look at our [How-to](how_to.qmd) guide. - -The only requirement to use this library is to open a free account with NASA [EDL](https://urs.earthdata.nasa.gov). - -### Step 1: Login - -To access NASA data, you have to login using your Earth Data Login credentials. You can register for a free Earth Data Login account [here](https://urs.earthdata.nasa.gov/). - -By default, `earthaccess` will look for your Earth Data Login credentials in a `.netrc` file, or in environment variables `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD`. If you don't -have either of these set up, you can login manually. See [Authenticating](authenticate.qmd) to learn how to create a `.netrc` file or environment variables. - -``` -import earthaccess - -earthaccess.login() -``` - - -### Step 2: Search for data - -As an example, we'll search for data from the NASA ICESat-2 mission. ATL06 - - -``` -results = earthaccess.search_data( - short_name='ATL06' - bounding_box=(-10, 20, 10, 50), - temporal=("1999-02", "2019-03"), - count=10 -) -``` - -### Step 3. Download the files - -Once you have found the files you want, you can download them to your local machine. - -``` -files = earthaccess.download(results, "./local_folder") -``` - -If you are working in the cloud and the data files are hosted in the cloud, you can stream the data directly, without having to download data. See [Direct S3 Access]() +Go to the [Quick Start](welcome/quick_start.md) to learn how to install and see a simple example of using `earthaccess`. ### Compatibility @@ -140,8 +56,6 @@ If you are working in the cloud and the data files are hosted in the cloud, you Only **Python 3.8+** is supported. - - ## Contributors [![Contributors](https://contrib.rocks/image?repo=nsidc/earthaccess)](https://github.com/nsidc/earthaccess/graphs/contributors) diff --git a/docs/welcome/quick_start.md b/docs/welcome/quick_start.md new file mode 100644 index 00000000..27d51e5d --- /dev/null +++ b/docs/welcome/quick_start.md @@ -0,0 +1,90 @@ +# Quick Start + +## **Getting Started** + +### **Installing earthaccess** + +The latest release of `earthaccess` can be installed with `mamba`, `conda` or `pip`. We recommend using `mamba` because it is faster. + +You will need Python 3.8 or higher installed. + +#### Using `mamba` + +```bash +mamba install -c conda-forge earthaccess +``` + +#### Using `conda` + +```bash +conda install -c conda-forge earthaccess +``` + +#### Using `pip` + +```bash +pip install earthaccess +``` + + +### Check `earthaccess` is installed + +This should run seamlessly (fingers-crossed). To check `earthaccess` is correctly installed you can start a python interpreter (either python or ipython) and run the following code. + +``` +$ python +Python 3.12.1 | packaged by conda-forge | (main, Dec 23 2023, 08:03:24) [GCC 12.3.0] on linux +Type "help", "copyright", "credits" or "license" for more information. +>>> import earthaccess +>>> earthaccess.__version__ +'0.8.2' +``` + +> Note: +> Your `python` and `earthaccess` versions may be different. + + +## **Get Data in 3 Steps** + +`earthaccess` allows you to search for and access data in as little as three steps. We give a very quick example below. These three steps allow you to get data whether you are working in the cloud or on your local laptop or workstation. Read the [User Guide](user_guide.qmd) for more information. If you want to quickly find how to perform some common searches and data access, +take a look at our [How-to](how_to.qmd) guide. + +The only requirement to use this library is to open a free account with NASA [EDL](https://urs.earthdata.nasa.gov). + +### Step 1: Login + +To access NASA data, you have to login using your Earth Data Login credentials. You can register for a free Earth Data Login account [here](https://urs.earthdata.nasa.gov/). + +By default, `earthaccess` will look for your Earth Data Login credentials in a `.netrc` file, or in environment variables `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD`. If you don't +have either of these set up, you can login manually. See [Authenticating](authenticate.qmd) to learn how to create a `.netrc` file or environment variables. + +``` +import earthaccess + +earthaccess.login() +``` + + +### Step 2: Search for data + +As an example, we'll search for data from the NASA ICESat-2 mission. ATL06 + + +``` +results = earthaccess.search_data( + short_name='ATL06' + bounding_box=(-10, 20, 10, 50), + temporal=("1999-02", "2019-03"), + count=10 +) +``` + +### Step 3. Download the files + +Once you have found the files you want, you can download them to your local machine. + +``` +files = earthaccess.download(results, "./local_folder") +``` + +If you are working in the cloud and the data files are hosted in the cloud, you can stream the data directly, without having to download data. See [Direct S3 Access]() diff --git a/docs/welcome/work_with_us.md b/docs/welcome/work_with_us.md new file mode 100644 index 00000000..895dabe7 --- /dev/null +++ b/docs/welcome/work_with_us.md @@ -0,0 +1 @@ +# Work With Us \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index d36c46d2..358ffa45 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -47,8 +47,8 @@ plugins: nav: - Welcome: "index.md" -# - "Getting started": "tutorials/getting-started.ipynb" -# - "Resources": "resources.md" + - "Quick Start": "welcome/quick_start.md" + - "Work With Us": "welcome/work_with_us.md" - User Guide: - "Overview": "user_guide/overview.md" - "Authentication": "user_guide/authenticate.md" From 5d5c9e69585efde39ea0764ec70eb3890a2b4f9a Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 16 Apr 2024 15:09:38 -0600 Subject: [PATCH 09/56] update headings --- docs/welcome/quick_start.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/welcome/quick_start.md b/docs/welcome/quick_start.md index 27d51e5d..6dc2cce4 100644 --- a/docs/welcome/quick_start.md +++ b/docs/welcome/quick_start.md @@ -1,8 +1,6 @@ # Quick Start -## **Getting Started** - -### **Installing earthaccess** +## **Installing earthaccess** The latest release of `earthaccess` can be installed with `mamba`, `conda` or `pip`. We recommend using `mamba` because it is faster. From 774afce9b18452cd415eb0678e3d4e106cc1fcf2 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 16 Apr 2024 15:16:12 -0600 Subject: [PATCH 10/56] Update headings and fix text --- README.md | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 5a5a2f72..c2267df3 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ +# Welcome +

earthaccess, a python library to search, download or stream NASA Earth science data with just a few lines of code

@@ -30,7 +32,8 @@

-## **Welcome to `earthaccess`** + +## What is `earthaccess`? `earthaccess` is a python library to **search for**, and **download** or **stream** NASA Earth science data with just a few lines of code. @@ -46,24 +49,24 @@ Try it in your browser without installing anything! [![Binder](https://mybinder. -## Getting Started +## How to Get Started with `earthaccess` Go to the [Quick Start](welcome/quick_start.md) to learn how to install and see a simple example of using `earthaccess`. -### Compatibility +## Compatibility Only **Python 3.8+** is supported. -## Contributors +## How to Contribute to `earthaccess` -[![Contributors](https://contrib.rocks/image?repo=nsidc/earthaccess)](https://github.com/nsidc/earthaccess/graphs/contributors) +If you want to find out how to contribute to `earthaccess` checkout the [Contributing Guide](CONTRIBUTING.md). -## Contributing Guide -Welcome! 😊👋 +## Contributors + +[![Contributors](https://contrib.rocks/image?repo=nsidc/earthaccess)](https://github.com/nsidc/earthaccess/graphs/contributors) -> Please see the [Contributing Guide](CONTRIBUTING.md). ### [Project Board](https://github.com/nsidc/earthdata/discussions). From 0431c6c22911975d0a34e40184aa76412fe0bbb3 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 16 Apr 2024 15:16:51 -0600 Subject: [PATCH 11/56] Fix block mapping --- mkdocs.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 358ffa45..db4c4b45 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -46,7 +46,8 @@ plugins: - "**/.ipynb_checkpoints/*" nav: - - Welcome: "index.md" + - Welcome: + - "What is earthaccess?": "index.md" - "Quick Start": "welcome/quick_start.md" - "Work With Us": "welcome/work_with_us.md" - User Guide: From 50bc4342ae3afbd582b243e0748727a8bf66cdf1 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 16:14:13 -0600 Subject: [PATCH 12/56] Add content about hack days to our docs --- docs/welcome/work_with_us.md | 1 - docs/work-with-us.md | 21 +++++++++++++++++++++ mkdocs.yml | 6 +++++- 3 files changed, 26 insertions(+), 2 deletions(-) delete mode 100644 docs/welcome/work_with_us.md create mode 100644 docs/work-with-us.md diff --git a/docs/welcome/work_with_us.md b/docs/welcome/work_with_us.md deleted file mode 100644 index 895dabe7..00000000 --- a/docs/welcome/work_with_us.md +++ /dev/null @@ -1 +0,0 @@ -# Work With Us \ No newline at end of file diff --git a/docs/work-with-us.md b/docs/work-with-us.md new file mode 100644 index 00000000..3f1eac18 --- /dev/null +++ b/docs/work-with-us.md @@ -0,0 +1,21 @@ +# How to collaborate with the _earthaccess_ team + +## Bi-weekly (alternating weeks) _earthaccess_ hack days + +???+ info + + For an invitation to our recurring hack day meeting, please visit our + [announcement thread on GitHub Discussions](https://github.com/nsidc/earthaccess/discussions/440#) + and make a comment to request a calendar invitation and Zoom link. + + +Hack days... + +* Occur on alternating Tuesdays at 11AM - 1PM Mountain Time. +* Are self-determining; you can work on what sounds fun to you! +* Will support you; _earthaccess_ developers, maintainers, and community managers will + be present on the call. +* Include live demos on request! + +For a glimpse in to the work we do on a typical hack day, please visit our +[hack day share-out space in GitHub Discussions](https://github.com/nsidc/earthaccess/discussions/categories/hack-days)! diff --git a/mkdocs.yml b/mkdocs.yml index db4c4b45..1b59b52c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -49,12 +49,16 @@ nav: - Welcome: - "What is earthaccess?": "index.md" - "Quick Start": "welcome/quick_start.md" - - "Work With Us": "welcome/work_with_us.md" + - "Work With Us": "work-with-us.md" - User Guide: - "Overview": "user_guide/overview.md" - "Authentication": "user_guide/authenticate.md" - "Search": "user_guide/search.md" - "Access": "user_guide/access.md" + - OVERVIEW: + - "Readme": "index.md" + - "Getting started": "tutorials/getting-started.ipynb" + - "Resources": "resources.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" - "Search NASA datasets using filters": "howto/search-collections.md" From 88951d198f4051fbd41421fc3a976feedce1e0f5 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 10 Apr 2024 11:23:47 -0600 Subject: [PATCH 13/56] Fix up broken callout and add title --- docs/work-with-us.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/work-with-us.md b/docs/work-with-us.md index 3f1eac18..535f13f5 100644 --- a/docs/work-with-us.md +++ b/docs/work-with-us.md @@ -2,11 +2,11 @@ ## Bi-weekly (alternating weeks) _earthaccess_ hack days -???+ info +???+ info "How to get invited" - For an invitation to our recurring hack day meeting, please visit our - [announcement thread on GitHub Discussions](https://github.com/nsidc/earthaccess/discussions/440#) - and make a comment to request a calendar invitation and Zoom link. + For an invitation to our recurring hack day meeting, please visit our + [announcement thread on GitHub Discussions](https://github.com/nsidc/earthaccess/discussions/440#) + and make a comment to request a calendar invitation and Zoom link. Hack days... From 07c6c83c535c1b1208061b204f723fcadd38aacf Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 16:18:25 -0600 Subject: [PATCH 14/56] Remove accidentally-re-added config --- mkdocs.yml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 1b59b52c..537ba655 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -50,15 +50,12 @@ nav: - "What is earthaccess?": "index.md" - "Quick Start": "welcome/quick_start.md" - "Work With Us": "work-with-us.md" + - "Resources": "resources.md" - User Guide: - "Overview": "user_guide/overview.md" - "Authentication": "user_guide/authenticate.md" - "Search": "user_guide/search.md" - "Access": "user_guide/access.md" - - OVERVIEW: - - "Readme": "index.md" - - "Getting started": "tutorials/getting-started.ipynb" - - "Resources": "resources.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" - "Search NASA datasets using filters": "howto/search-collections.md" From 76e8789b0fdb43576083b6a190647eb0cff06c4f Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Wed, 24 Apr 2024 15:09:20 -0600 Subject: [PATCH 15/56] Modified "voice" for consistency --- docs/work-with-us.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/work-with-us.md b/docs/work-with-us.md index 535f13f5..a2183b1d 100644 --- a/docs/work-with-us.md +++ b/docs/work-with-us.md @@ -13,7 +13,7 @@ Hack days... * Occur on alternating Tuesdays at 11AM - 1PM Mountain Time. * Are self-determining; you can work on what sounds fun to you! -* Will support you; _earthaccess_ developers, maintainers, and community managers will +* Are supportive; _earthaccess_ developers, maintainers, and community managers will be present on the call. * Include live demos on request! From a8e5c64a016d45e3ca6cfb9923f4f13241c1f33e Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 17:16:40 -0600 Subject: [PATCH 16/56] Move meet-up page under contributing --- docs/{work-with-us.md => contributing/our-meet-ups.md} | 0 mkdocs.yml | 6 ++++-- 2 files changed, 4 insertions(+), 2 deletions(-) rename docs/{work-with-us.md => contributing/our-meet-ups.md} (100%) diff --git a/docs/work-with-us.md b/docs/contributing/our-meet-ups.md similarity index 100% rename from docs/work-with-us.md rename to docs/contributing/our-meet-ups.md diff --git a/mkdocs.yml b/mkdocs.yml index 3dce0d64..9494e6bc 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -48,8 +48,10 @@ plugins: nav: - Welcome: - "What is earthaccess?": "index.md" - - "Quick Start": "welcome/quick_start.md" - - "Work With Us": "work-with-us.md" + - "Quick start": "welcome/quick_start.md" + - "Work with us": + - "contributing/index.md" + - "Our meet-ups": "contributing/our-meet-ups.md" - "Resources": "resources.md" - User Guide: - "Overview": "user_guide/overview.md" From 71803e093dfc5ef166881ad8235713addc0d9774 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 17:34:10 -0600 Subject: [PATCH 17/56] Split out development environment and releasing docs --- docs/contributing/development.md | 62 +++++++++++++++++++ docs/contributing/index.md | 102 ++----------------------------- docs/contributing/releasing.md | 28 +++++++++ 3 files changed, 95 insertions(+), 97 deletions(-) create mode 100644 docs/contributing/development.md create mode 100644 docs/contributing/releasing.md diff --git a/docs/contributing/development.md b/docs/contributing/development.md new file mode 100644 index 00000000..2cea2e8e --- /dev/null +++ b/docs/contributing/development.md @@ -0,0 +1,62 @@ +# Development environment setup + +1. Fork [nsidc/earthaccess](https://github.com/nsidc/earthaccess) +1. Clone your fork (`git clone git@github.com:{my-username}/earthaccess`) + +`earthaccess` uses Poetry to build and publish the package to PyPI, the defacto Python repository. In order to develop new features or fix bugs etc. we need to set up a virtual environment and install the library locally. We can accomplish this with Poetry and/or Conda. + +### Using Conda + +If we have `mamba` (or `conda`) installed, we can use the environment file included in the `ci` folder. This will install all the libraries we need (including Poetry) to start developing `earthaccess`: + +```bash +mamba env update -f ci/environment-dev.yml +mamba activate earthaccess-dev +poetry install +``` + +After activating our environment and installing the library with Poetry we can run Jupyter lab and start testing the local distribution or we can use `make` to run the tests and lint the code. +Now we can create a feature branch and push those changes to our fork! + +### Using Poetry + +If we want to use Poetry, first we need to [install it](https://python-poetry.org/docs/#installation). After installing Poetry we can use the same workflow we used for Conda, first we install the library locally: + +```bash +poetry install +``` + +and now we can run the local Jupyter Lab and run the scripts etc. using Poetry: + +```bash +poetry run jupyter lab +``` + +!!! note + + You may need to use `poetry run make ...` to run commands in the environment. + +### Managing Dependencies + +If you need to add a dependency, you should do the following: + +- Run `poetry add ` for a required (non-development) dependency +- Run `poetry add --group=dev ` for a development dependency, such + as a testing or code analysis dependency + +Both commands will add an entry to `pyproject.toml` with a version that is +compatible with the rest of the dependencies. However, `poetry` pins versions +with a caret (`^`), which is not what we want. Therefore, you must locate the +new entry in `pyproject.toml` and change the `^` to `>=`. (See +[poetry-relax](https://github.com/zanieb/poetry-relax) for the reasoning behind +this.) + +In addition, you must also add a corresponding entry to +`ci/environment-mindeps.yaml`. You'll notice in that file that required +dependencies should be pinned exactly to the versions specified in +`pyproject.toml` (after changing `^` to `>=` there), and that development +dependencies should be left unpinned. + +Finally, for _development dependencies only_, you must add an entry to +`ci/environment-dev.yaml` with the same version constraint as in +`pyproject.toml`. diff --git a/docs/contributing/index.md b/docs/contributing/index.md index e94ecec4..92babfbd 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -1,78 +1,15 @@ # Contributing When contributing to this repository, please first discuss the change you wish to make via issue, -email, or any other method with the owners of this repository before making a change. +email, or any other method with the maintainers before making a change. -Please note that we have a [code of conduct](./CODE_OF_CONDUCT.md). Please follow it in all of your interactions with the project. - -## Development environment - -1. Fork [nsidc/earthaccess](https://github.com/nsidc/earthaccess) -1. Clone your fork (`git clone git@github.com:{my-username}/earthaccess`) - -`earthaccess` uses Poetry to build and publish the package to PyPI, the defacto Python repository. In order to develop new features or fix bugs etc. we need to set up a virtual environment and install the library locally. We can accomplish this with Poetry and/or Conda. - -### Using Conda - -If we have `mamba` (or `conda`) installed, we can use the environment file included in the `ci` folder. This will install all the libraries we need (including Poetry) to start developing `earthaccess`: - -```bash -mamba env update -f ci/environment-dev.yml -mamba activate earthaccess-dev -poetry install -``` - -After activating our environment and installing the library with Poetry we can run Jupyter lab and start testing the local distribution or we can use `make` to run the tests and lint the code. -Now we can create a feature branch and push those changes to our fork! - -### Using Poetry - -If we want to use Poetry, first we need to [install it](https://python-poetry.org/docs/#installation). After installing Poetry we can use the same workflow we used for Conda, first we install the library locally: - -```bash -poetry install -``` - -and now we can run the local Jupyter Lab and run the scripts etc. using Poetry: - -```bash -poetry run jupyter lab -``` - -!!! note - - You may need to use `poetry run make ...` to run commands in the environment. - -### Managing Dependencies - -If you need to add a dependency, you should do the following: - -- Run `poetry add ` for a required (non-development) dependency -- Run `poetry add --group=dev ` for a development dependency, such - as a testing or code analysis dependency - -Both commands will add an entry to `pyproject.toml` with a version that is -compatible with the rest of the dependencies. However, `poetry` pins versions -with a caret (`^`), which is not what we want. Therefore, you must locate the -new entry in `pyproject.toml` and change the `^` to `>=`. (See -[poetry-relax](https://github.com/zanieb/poetry-relax) for the reasoning behind -this.) - -In addition, you must also add a corresponding entry to -`ci/environment-mindeps.yaml`. You'll notice in that file that required -dependencies should be pinned exactly to the versions specified in -`pyproject.toml` (after changing `^` to `>=` there), and that development -dependencies should be left unpinned. - -Finally, for _development dependencies only_, you must add an entry to -`ci/environment-dev.yaml` with the same version constraint as in -`pyproject.toml`. +Please note that we have a [code of conduct](/CODE_OF_CONDUCT.md). Please follow it in all of your interactions with the project. ## First Steps to contribute -- Read the documentation -- Fork this repo (see "Development environment" section above for more) -- Install environment (see "Development environment" section above for more) +- Read the documentation! +- Fork this repo and set up development environment (see + [development environment documentation](./development.md) for details) - Run the unit tests successfully in `main` branch: - `make test` @@ -144,32 +81,3 @@ the stubs appear under `stubs/cmr`. 1. You may merge the Pull Request once you have the sign-off of another developer, or if you do not have permission to do that, you may request the reviewer to merge it for you. - -## Release process - -> :memo: The versioning scheme we use is [SemVer](http://semver.org/). Note that until -> we agree we're ready for v1.0.0, we will not increment the major version. - -1. Ensure all desired features are merged to `main` branch and `CHANGELOG.md` is updated. -1. Use `bump-my-version` to increase the version number in all needed places, e.g. to - increase the minor version (`1.2.3` to `1.3.0`): - - ```plain - bump-my-version bump minor - ``` - -1. Push a tag on the new commit containing the version number, prefixed with `v`, e.g. - `v1.3.0`. -1. [Create a new GitHub Release](https://github.com/nsidc/earthaccess/releases/new). We - hand-curate our release notes to be valuable to humans. Please do not auto-generate - release notes and aim for consistency with the GitHub Release descriptions from other - releases. - -> :gear: After the GitHub release is published, multiple automations will trigger: -> -> - Zenodo will create a new DOI. -> - GitHub Actions will publish a PyPI release. - -> :memo: `earthaccess` is published to conda-forge through the -> [earthdata-feedstock](https://github.com/conda-forge/earthdata-feedstock), as this -> project was renamed early in its life. The conda package is named `earthaccess`. diff --git a/docs/contributing/releasing.md b/docs/contributing/releasing.md new file mode 100644 index 00000000..fe62a000 --- /dev/null +++ b/docs/contributing/releasing.md @@ -0,0 +1,28 @@ +# Release process + +> :memo: The versioning scheme we use is [SemVer](http://semver.org/). Note that until +> we agree we're ready for v1.0.0, we will not increment the major version. + +1. Ensure all desired features are merged to `main` branch and `CHANGELOG.md` is updated. +1. Use `bump-my-version` to increase the version number in all needed places, e.g. to + increase the minor version (`1.2.3` to `1.3.0`): + + ```plain + bump-my-version bump minor + ``` + +1. Push a tag on the new commit containing the version number, prefixed with `v`, e.g. + `v1.3.0`. +1. [Create a new GitHub Release](https://github.com/nsidc/earthaccess/releases/new). We + hand-curate our release notes to be valuable to humans. Please do not auto-generate + release notes and aim for consistency with the GitHub Release descriptions from other + releases. + +> :gear: After the GitHub release is published, multiple automations will trigger: +> +> - Zenodo will create a new DOI. +> - GitHub Actions will publish a PyPI release. + +> :memo: `earthaccess` is published to conda-forge through the +> [earthdata-feedstock](https://github.com/conda-forge/earthdata-feedstock), as this +> project was renamed early in its life. The conda package is named `earthaccess`. From e88695e14ce6292b5c7feb0bf2df3a74977b4dc7 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 17:36:53 -0600 Subject: [PATCH 18/56] Enable navigation index pages --- mkdocs.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 9494e6bc..a734f298 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -16,6 +16,10 @@ theme: toggle: icon: material/toggle-switch name: Switch to light mode + features: + # Allow navigation sections to link to index/overview pages. + # See: https://squidfunk.github.io/mkdocs-material/setup/setting-up-navigation/#section-index-pages + - navigation.indexes repo_name: nsidc/earthaccess repo_url: https://github.com/nsidc/earthaccess From f7ab38bb6cc94b27d707dbe78493dbf9981ed756 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 17:49:06 -0600 Subject: [PATCH 19/56] Un-nest pages from "Welcome" section --- README.md | 2 +- docs/{welcome/quick_start.md => quick-start.md} | 0 mkdocs.yml | 13 ++++++------- 3 files changed, 7 insertions(+), 8 deletions(-) rename docs/{welcome/quick_start.md => quick-start.md} (100%) diff --git a/README.md b/README.md index c2267df3..f0888a39 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Try it in your browser without installing anything! [![Binder](https://mybinder. ## How to Get Started with `earthaccess` -Go to the [Quick Start](welcome/quick_start.md) to learn how to install and see a simple example of using `earthaccess`. +Go to the [Quick Start](quick-start.md) to learn how to install and see a simple example of using `earthaccess`. ## Compatibility diff --git a/docs/welcome/quick_start.md b/docs/quick-start.md similarity index 100% rename from docs/welcome/quick_start.md rename to docs/quick-start.md diff --git a/mkdocs.yml b/mkdocs.yml index a734f298..1e231d0b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -50,13 +50,12 @@ plugins: - "**/.ipynb_checkpoints/*" nav: - - Welcome: - - "What is earthaccess?": "index.md" - - "Quick start": "welcome/quick_start.md" - - "Work with us": - - "contributing/index.md" - - "Our meet-ups": "contributing/our-meet-ups.md" - - "Resources": "resources.md" + - "What is earthaccess?": "index.md" + - "Quick start": "quick-start.md" + - "Work with us": + - "contributing/index.md" + - "Our meet-ups": "contributing/our-meet-ups.md" + - "Resources": "resources.md" - User Guide: - "Overview": "user_guide/overview.md" - "Authentication": "user_guide/authenticate.md" From 632700de55f8ed78cdabe094494f61a72c5dc9ce Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 18:17:54 -0600 Subject: [PATCH 20/56] Use navigation index functionality to remove a sub-nav from user guide --- docs/user_guide/{overview.md => index.md} | 0 mkdocs.yml | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) rename docs/user_guide/{overview.md => index.md} (100%) diff --git a/docs/user_guide/overview.md b/docs/user_guide/index.md similarity index 100% rename from docs/user_guide/overview.md rename to docs/user_guide/index.md diff --git a/mkdocs.yml b/mkdocs.yml index 1e231d0b..fd9c3331 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -57,7 +57,7 @@ nav: - "Our meet-ups": "contributing/our-meet-ups.md" - "Resources": "resources.md" - User Guide: - - "Overview": "user_guide/overview.md" + - "user_guide/index.md" - "Authentication": "user_guide/authenticate.md" - "Search": "user_guide/search.md" - "Access": "user_guide/access.md" From 975602c17165fd0d26c1c8163740e0a6e00a5b4f Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 24 Apr 2024 18:52:45 -0600 Subject: [PATCH 21/56] Clarify conda/poetry choice Co-authored-by: Andy Barrett --- docs/contributing/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 2cea2e8e..7425fcba 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -3,7 +3,7 @@ 1. Fork [nsidc/earthaccess](https://github.com/nsidc/earthaccess) 1. Clone your fork (`git clone git@github.com:{my-username}/earthaccess`) -`earthaccess` uses Poetry to build and publish the package to PyPI, the defacto Python repository. In order to develop new features or fix bugs etc. we need to set up a virtual environment and install the library locally. We can accomplish this with Poetry and/or Conda. +`earthaccess` uses Poetry to build and publish the package to PyPI, the defacto Python repository. In order to develop new features or fix bugs etc. we need to set up a virtual environment and install the library locally. We can accomplish this with Conda and Poetry, or just with Poetry. Both workflows achieve the same result. ### Using Conda From 0e21d1d9a2f249d374b3be86469452b8e810b3ec Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 24 Apr 2024 18:53:02 -0600 Subject: [PATCH 22/56] Apply suggested wording change Co-authored-by: Andy Barrett --- docs/contributing/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 7425fcba..954be7ae 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -38,7 +38,7 @@ poetry run jupyter lab ### Managing Dependencies -If you need to add a dependency, you should do the following: +If you need to add a new dependency, you should do the following: - Run `poetry add ` for a required (non-development) dependency - Run `poetry add --group=dev ` for a development dependency, such From 3b4e0017d8720d1dc678d3cf10d16374abf0e47e Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 24 Apr 2024 18:53:18 -0600 Subject: [PATCH 23/56] More concise wording Co-authored-by: Andy Barrett --- docs/contributing/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index 954be7ae..c4ec122b 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -44,7 +44,7 @@ If you need to add a new dependency, you should do the following: - Run `poetry add --group=dev ` for a development dependency, such as a testing or code analysis dependency -Both commands will add an entry to `pyproject.toml` with a version that is +Both commands add an entry to `pyproject.toml` with a version that is compatible with the rest of the dependencies. However, `poetry` pins versions with a caret (`^`), which is not what we want. Therefore, you must locate the new entry in `pyproject.toml` and change the `^` to `>=`. (See From e76456c9300080fb4ed55659ec48c3e4ebc3dfb7 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 24 Apr 2024 18:53:30 -0600 Subject: [PATCH 24/56] Clarify wording Co-authored-by: Andy Barrett --- docs/contributing/development.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/development.md b/docs/contributing/development.md index c4ec122b..0350329f 100644 --- a/docs/contributing/development.md +++ b/docs/contributing/development.md @@ -52,7 +52,7 @@ new entry in `pyproject.toml` and change the `^` to `>=`. (See this.) In addition, you must also add a corresponding entry to -`ci/environment-mindeps.yaml`. You'll notice in that file that required +`ci/environment-mindeps.yaml`. You'll notice in this file that required dependencies should be pinned exactly to the versions specified in `pyproject.toml` (after changing `^` to `>=` there), and that development dependencies should be left unpinned. From 8a4e2a0ceee3d5826ecfc3262b43cb9d5eb285d2 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 24 Apr 2024 18:58:14 -0600 Subject: [PATCH 25/56] Improve intro text to contributing docs --- docs/contributing/index.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/contributing/index.md b/docs/contributing/index.md index 92babfbd..50a606b9 100644 --- a/docs/contributing/index.md +++ b/docs/contributing/index.md @@ -1,7 +1,10 @@ # Contributing -When contributing to this repository, please first discuss the change you wish to make via issue, -email, or any other method with the maintainers before making a change. +When contributing to this repository, please first discuss the change you wish to make +with the community and maintainers via +[a GitHub issue](https://github.com/nsidc/earthaccess/issues), +[a GitHub Discussion](https://github.com/nsidc/earthaccess/discussions), +or [any other method](our-meet-ups.md). Please note that we have a [code of conduct](/CODE_OF_CONDUCT.md). Please follow it in all of your interactions with the project. From 3b3858024f409b9db0f472b6a09167a341458b66 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 17:58:19 -0600 Subject: [PATCH 26/56] Split the README into a GitHub README and doc page Previously, the README file was symlinked into our docs. This led to a problem where we could only write links that worked within one of the two systems. Docs links need to be relative to the symlink to render correctly, and GitHub README links need to be relative to the original file to work correctly. --- README.md | 19 +++++-------------- docs/index.md | 12 +++++++++++- 2 files changed, 16 insertions(+), 15 deletions(-) mode change 120000 => 100644 docs/index.md diff --git a/README.md b/README.md index f0888a39..8eea0480 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Welcome +# _earthaccess_

earthaccess, a python library to search, download or stream NASA Earth science data with just a few lines of code @@ -33,25 +33,16 @@

-## What is `earthaccess`? - `earthaccess` is a python library to **search for**, and **download** or **stream** NASA Earth science data with just a few lines of code. -Open science only reaches its full potential if we have easy-to-use workflows that facilitate research in an inclusive, efficient and reproducible way. Unfortunately —as it stands today— scientists and students alike face a steep learning curve adapting to systems that have grown too complex and end up spending more time on the technicalities of the tools, cloud and NASA APIs than focusing on their important science. - -During several workshops organized by [NASA Openscapes](https://nasa-openscapes.github.io/events.html), the need to provide easy-to-use tools to our users became evident. Open science is a collaborative effort; it involves people from different technical backgrounds, and the data analysis to solve the pressing problems we face cannot be limited by the complexity of the underlying systems. Therefore, providing easy access to NASA Earthdata regardless of the data storage location (hosted within or outside of the cloud) is the main motivation behind this Python library. - -The library is an open source community effort under an [MIT license](LICENSE.txt). We welcome contributions to improve `earthaccess`. Please see the [Contributing Guide](CONTRIBUTING.md) to learn how to get involved. - -***earthaccess*** handles authentication with [NASA's Earthdata Login (EDL)](https://urs.earthdata.nasa.gov), search using NASA's [CMR](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) and access through [`fsspec`](https://github.com/fsspec/filesystem_spec). +Visit [our documentation](https://earthaccess.readthedocs.io/en/latest) to learn more! Try it in your browser without installing anything! [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/nsidc/earthaccess/main) - ## How to Get Started with `earthaccess` -Go to the [Quick Start](quick-start.md) to learn how to install and see a simple example of using `earthaccess`. +Visit [our quick start guide](https://earthaccess.readthedocs.io/en/latest/quick-start.html) to learn how to install and see a simple example of using `earthaccess`. ## Compatibility @@ -61,10 +52,10 @@ Only **Python 3.8+** is supported. ## How to Contribute to `earthaccess` -If you want to find out how to contribute to `earthaccess` checkout the [Contributing Guide](CONTRIBUTING.md). +If you want to find out how to contribute to `earthaccess` checkout the [Contributing Guide](https://earthaccess.readthedocs.io/en/latest/contributing/). -## Contributors +### Contributors [![Contributors](https://contrib.rocks/image?repo=nsidc/earthaccess)](https://github.com/nsidc/earthaccess/graphs/contributors) diff --git a/docs/index.md b/docs/index.md deleted file mode 120000 index 32d46ee8..00000000 --- a/docs/index.md +++ /dev/null @@ -1 +0,0 @@ -../README.md \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 00000000..9e116b31 --- /dev/null +++ b/docs/index.md @@ -0,0 +1,11 @@ +# What is `earthaccess`? + +`earthaccess` is a python library to **search for**, and **download** or **stream** NASA Earth science data with just a few lines of code. + +Open science only reaches its full potential if we have easy-to-use workflows that facilitate research in an inclusive, efficient and reproducible way. Unfortunately —as it stands today— scientists and students alike face a steep learning curve adapting to systems that have grown too complex and end up spending more time on the technicalities of the tools, cloud and NASA APIs than focusing on their important science. + +During several workshops organized by [NASA Openscapes](https://nasa-openscapes.github.io/events.html), the need to provide easy-to-use tools to our users became evident. Open science is a collaborative effort; it involves people from different technical backgrounds, and the data analysis to solve the pressing problems we face cannot be limited by the complexity of the underlying systems. Therefore, providing easy access to NASA Earthdata regardless of the data storage location (hosted within or outside of the cloud) is the main motivation behind this Python library. + +The library is an open source community effort under an [MIT license](LICENSE.txt). We welcome contributions to improve `earthaccess`. Please see the [Contributing Guide](contributing/index.md) to learn how to get involved. + +***earthaccess*** handles authentication with [NASA's Earthdata Login (EDL)](https://urs.earthdata.nasa.gov), search using NASA's [CMR](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html) and access through [`fsspec`](https://github.com/fsspec/filesystem_spec). From 3c213c03962bfb32c6cad7778b4afd2ce6a90df0 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 19 Apr 2024 18:04:47 -0600 Subject: [PATCH 27/56] Enable mkdocs strict mode to catch broken links --- mkdocs.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index fd9c3331..ca5877bd 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -2,6 +2,9 @@ site_name: earthaccess site_description: Client library for NASA Earthdata APIs site_url: https://github.com/nsidc/earthaccess +# Fail on broken links +strict: true + theme: name: "material" logo: earth.png From c06908ae8f622a7c4a12772b37b110050e99e226 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Wed, 24 Apr 2024 20:35:13 -0600 Subject: [PATCH 28/56] Fixup mkdocs broken links --- README.md | 2 +- docs/quick-start.md | 10 +++++++--- docs/user_guide/index.md | 8 ++++---- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 8eea0480..d9bfb9a4 100644 --- a/README.md +++ b/README.md @@ -52,7 +52,7 @@ Only **Python 3.8+** is supported. ## How to Contribute to `earthaccess` -If you want to find out how to contribute to `earthaccess` checkout the [Contributing Guide](https://earthaccess.readthedocs.io/en/latest/contributing/). +If you want to contribute to `earthaccess` checkout the [Contributing Guide](https://earthaccess.readthedocs.io/en/latest/contributing/). ### Contributors diff --git a/docs/quick-start.md b/docs/quick-start.md index 6dc2cce4..38545302 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -44,8 +44,12 @@ Type "help", "copyright", "credits" or "license" for more information. ## **Get Data in 3 Steps** -`earthaccess` allows you to search for and access data in as little as three steps. We give a very quick example below. These three steps allow you to get data whether you are working in the cloud or on your local laptop or workstation. Read the [User Guide](user_guide.qmd) for more information. If you want to quickly find how to perform some common searches and data access, -take a look at our [How-to](how_to.qmd) guide. +`earthaccess` allows you to search for and access data in as little as three steps. We +give a very quick example below. These three steps allow you to get data whether you +are working in the cloud or on your local laptop or workstation. Read the +[User Guide](user_guide/index.md) for more information. If you want to quickly find how +to perform some common searches and data access, take a look at our how-to guides in the +sidebar. The only requirement to use this library is to open a free account with NASA [EDL](https://urs.earthdata.nasa.gov). @@ -54,7 +58,7 @@ The only requirement to use this library is to open a free account with NASA [ED To access NASA data, you have to login using your Earth Data Login credentials. You can register for a free Earth Data Login account [here](https://urs.earthdata.nasa.gov/). By default, `earthaccess` will look for your Earth Data Login credentials in a `.netrc` file, or in environment variables `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD`. If you don't -have either of these set up, you can login manually. See [Authenticating](authenticate.qmd) to learn how to create a `.netrc` file or environment variables. +have either of these set up, you can login manually. See [Authenticating](howto/authenticate.md) to learn how to create a `.netrc` file or environment variables. ``` import earthaccess diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index aed94d5e..4688bdeb 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -2,12 +2,12 @@ `earthaccess` enables authentication, search and access for NASA Earth science data. The package enables users to find, and download or stream data archived both at NASA DAACs and in the AWS-hosted Earthdata Cloud. The User Guide introduces these three functions. -The [Authentication](user_guide/authentication.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. +The [Authentication](/user_guide/authentication.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. -The [Search](user_guide/search.md) section introduces how to search for NASA datasets (collections) and specific data granules. +The [Search](/user_guide/search.md) section introduces how to search for NASA datasets (collections) and specific data granules. -The [Access](user_guide/access.md) section introduces how to use `earthaccess` to download data to a local machine or stream data by opening it directly in memory. +The [Access](/user_guide/access.md) section introduces how to use `earthaccess` to download data to a local machine or stream data by opening it directly in memory. Additional usage information can be found in the User Reference section. The How-To sections contains short code examples for specific workflows. Longer tutorials for working with particular datasets are in the Tutorials section. -The Glossary section contains definitions for NASA terminology and Cloud computing. \ No newline at end of file +The Glossary section contains definitions for NASA terminology and Cloud computing. From 4ffe156d47fb8e738782cd39ca59c33584096e10 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 26 Apr 2024 12:33:50 -0600 Subject: [PATCH 29/56] Update nav for new contributing refactor --- mkdocs.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index ca5877bd..4d4266f6 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -57,6 +57,8 @@ nav: - "Quick start": "quick-start.md" - "Work with us": - "contributing/index.md" + - "Development": "contributing/development.md" + - "Releasing": "contributing/releasing.md" - "Our meet-ups": "contributing/our-meet-ups.md" - "Resources": "resources.md" - User Guide: @@ -64,7 +66,6 @@ nav: - "Authentication": "user_guide/authenticate.md" - "Search": "user_guide/search.md" - "Access": "user_guide/access.md" - - "Contributing": "contributing/index.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" - "Search NASA datasets using filters": "howto/search-collections.md" From ca108208b362eecfe31c1c5944d22c3cb153e691 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Fri, 26 Apr 2024 12:44:13 -0600 Subject: [PATCH 30/56] Display user guide in nav as all-caps Making this change because the user guide is the 4th diataxis quadrant (discussion/explanation/concepts) --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 4d4266f6..c8db9eb5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -61,7 +61,7 @@ nav: - "Releasing": "contributing/releasing.md" - "Our meet-ups": "contributing/our-meet-ups.md" - "Resources": "resources.md" - - User Guide: + - USER GUIDE: - "user_guide/index.md" - "Authentication": "user_guide/authenticate.md" - "Search": "user_guide/search.md" From a2dad0aaf93f516e42802fb88f79d99f46ac9d48 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Fri, 26 Apr 2024 12:54:27 -0600 Subject: [PATCH 31/56] Fix link to authenticate.md --- docs/user_guide/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index 4688bdeb..89f848ae 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -2,7 +2,7 @@ `earthaccess` enables authentication, search and access for NASA Earth science data. The package enables users to find, and download or stream data archived both at NASA DAACs and in the AWS-hosted Earthdata Cloud. The User Guide introduces these three functions. -The [Authentication](/user_guide/authentication.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. +The [Authentication](/user_guide/authenticate.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. The [Search](/user_guide/search.md) section introduces how to search for NASA datasets (collections) and specific data granules. From a1f2e3ddc8f565495f0a9e121d1932a2cb44c495 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Fri, 26 Apr 2024 13:06:23 -0600 Subject: [PATCH 32/56] Add admonition to redirect users to howto While the documentation is under construction and content is not complete, we are adding admonitions to redirect users from User Guide pages to existing How-Tos and Tutorials. --- docs/user_guide/access.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/user_guide/access.md b/docs/user_guide/access.md index 8f369db8..beafa3c2 100644 --- a/docs/user_guide/access.md +++ b/docs/user_guide/access.md @@ -2,8 +2,14 @@ **Access** includes downloading files from a NASA DAAC or Earthdata Cloud to a local computer, a laptop, workstation or the storage space for a high-performace-computer (HPC); and also streaming data. +!!! note "This Page is a Work in Progress" + We are reorganizing and updating the documentation, so not all pages are complete. If you are looking for information about accessing data using earthaccess see the + HOW-TO pages below. + [How-to download data](/howto/onprem.md) + [Getting Started Tutorial](/tutorials/getting-started.ipynb) + ## Downloading data -## Streaming data \ No newline at end of file +## Streaming data From a5a7c159f71b844d2725de7bcbcdc9dd4703f6c5 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Fri, 26 Apr 2024 13:11:55 -0600 Subject: [PATCH 33/56] Add admonition with redirect to how-to and tutorials Added admonition to indicate docs are under construction with redirect to how-to and tutorials for authentication. --- docs/user_guide/authenticate.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/user_guide/authenticate.md b/docs/user_guide/authenticate.md index fd69f543..aa40e2eb 100644 --- a/docs/user_guide/authenticate.md +++ b/docs/user_guide/authenticate.md @@ -1,3 +1,10 @@ # Authentication -Introduces the `earthaccess.login` method for managing Earthdata Login and cloud credentials. \ No newline at end of file +Introduces the `earthaccess.login` method for managing Earthdata Login and cloud credentials. + +!!! note "This Page is a Work in Progress" + + We are reorganizing and updating the documentation, so not all pages are complete. If you are looking for information about authenticating using earthaccess see the + How-Tos and Tutorials in links below. + [How-To Authenticate with earthaccess](/howto/authenticate.md) + [Getting Started Tutorial](/tutorials/getting-started.ipynb) From fad6cf86a5ff931cb062262b68f32c7d00cd3cad Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Fri, 26 Apr 2024 13:13:50 -0600 Subject: [PATCH 34/56] Add admonition to redirect to access how-to and tutorial --- docs/user_guide/search.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/user_guide/search.md b/docs/user_guide/search.md index bb5043d0..cea2ffe2 100644 --- a/docs/user_guide/search.md +++ b/docs/user_guide/search.md @@ -1,5 +1,12 @@ # Search +!!! note "This Page is a Work in Progress" + + We are reorganizing and updating the documentation, so not all pages are complete. If you are looking for information about authenticating using earthaccess see the + How-Tos and Tutorials in links below. + [How-To Access Data](/howto/access-data.md) + [Getting Started Tutorial](/tutorials/getting-started.ipynb) + ## `search_datasets` ## `search_data` From 97a47ca8373a8a52cdc20768b88e3a2024346ac8 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 11:57:01 -0600 Subject: [PATCH 35/56] Disable strict mode when previewing docs --- scripts/docs-live.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/docs-live.sh b/scripts/docs-live.sh index 6ed6a4be..f9f1b9de 100755 --- a/scripts/docs-live.sh +++ b/scripts/docs-live.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash set -ex -mkdocs serve --dev-addr 0.0.0.0:8008 --dirtyreload +# HACK: --no-strict is on because --dirtyreload ALWAYS throws errors. Better solution? +mkdocs serve --dev-addr 0.0.0.0:8008 --dirtyreload --no-strict From 1f5a7b5f90d8185b2ecc5240af9f724f89090b90 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:04:18 -0600 Subject: [PATCH 36/56] Remove outdated tutorial Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/tutorials/getting-started.ipynb | 1254 -------------------------- 1 file changed, 1254 deletions(-) delete mode 100644 docs/tutorials/getting-started.ipynb diff --git a/docs/tutorials/getting-started.ipynb b/docs/tutorials/getting-started.ipynb deleted file mode 100644 index d069e8af..00000000 --- a/docs/tutorials/getting-started.ipynb +++ /dev/null @@ -1,1254 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "56ba2387-54e8-4aa5-aedb-c7d90644536f", - "metadata": { - "tags": [] - }, - "source": [ - "\n", - "# Getting started with earthaccess\n", - "\n", - "\n", - "#### TL;DR: [**earthaccess**](https://github.com/nsidc/earthaccess) is a Python package to search, preview and access NASA datasets (on-prem or in the cloud) with a few lines of code.\n", - "\n", - "\n", - "## Why? \n", - "\n", - "**Programmatic, Easy, Reproducible.**\n", - "\n", - "There are many ways to access NASA datasets, we can use the [Earthdata search portal](https://search.earthaccess.nasa.gov/). We can use DAAC specific portals or tools.\n", - "We could even use [data.gov](https://data.gov)! Web portals are great but they are not designed for programmatic access and reproducible workflows. This is extremely important in the age of the cloud and reproducible open science. \n", - "\n", - "The good news is that NASA also exposes APIs that allows us to search, transform and access data in a programmatic way. Many of these libraries contain amazing features and some similarities. In this context, **earthaccess** aims to be a simple library that can deal with the important parts of the metadata so we can access or download data without having to worry if a given dataset is on-prem or in the cloud.\n", - "\n", - "\n", - "## How?\n", - "\n", - "> Note: There are a lot of acronyms that we need to get familiar with before any of this makes sense, here is a brief glossary for NASA Earthdata terms: [NASA glossary](glossary.md)\n", - "\n", - "\n", - "### Authentication: Before we can use `earthaccess` we need an account with **[NASA EDL](https://urs.earthaccess.nasa.gov/)**\n", - "\n", - "\n", - "Earthdata Login provides free and immediate access to thousands of EOSDIS data products covering all Earth science disciplines and topic areas for researchers, applied science users, application developers, and the general public.\n", - "\n", - "Once we have our NASA EDL login credentials we can start accessing NASA data in a programmatic way.\n", - "\n", - "\n", - "## Querying CMR using earthaccess\n", - "\n", - "This short tutorial uses the `collection_query()` and `granule_query()` methods, these methods return a lower level Query Builder instance that can be used to query NASA's CMR.\n", - "For convenience the top level API also offers the `dataset_search(**kwargs)` and `data_search(**kwargs)` methods that map what these query builders do. \n", - "\n", - "For instance \n", - "\n", - "```python\n", - "query = earthaccess.granule_query().doi(\"some_doi\").temporal(\"1990-01-01\", \"2020-12-31\").cloud_hosted(True)\n", - "granules = query.get(10)\n", - "\n", - "```\n", - "\n", - "is equivalent to\n", - "\n", - "```python\n", - "granules = earthaccess.search_data(\n", - " doi=\"some_doi\",\n", - " temporal = (\"1990-01-01\",\"2020-12-31\"),\n", - " cloud_hosted=True,\n", - " limit=10\n", - ")\n", - "```" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "10f6c9ed-fe58-4e03-b29b-c6c447061f84", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/plain": [ - "'0.9.0'" - ] - }, - "execution_count": 1, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import earthaccess\n", - "earthaccess.__version__" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "496c1e3e-5b1a-44f8-ae13-84c42ea814af", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "auth = earthaccess.login()" - ] - }, - { - "cell_type": "markdown", - "id": "95121ff7-5222-4778-a4de-25625e23884b", - "metadata": {}, - "source": [ - "#### Auth()\n", - "\n", - "`earthaccess`'s **Auth** class provides 3 different strategies to authenticate ourselves with NASA EDL.\n", - "\n", - "* **netrc**: Do we have a `.netrc` file with our EDL credentials? if so, we can use it with `earthaccess`.\n", - "If we don't have it and want to create one we can, earthaccess allows users to type their credentials and persist them into a `.netrc` file.\n", - "\n", - "```python\n", - "auth = earthaccess.login(strategy=\"netrc\")\n", - "```\n", - "* **environment**: If we have our EDL credentials as environment variables \n", - " * EARTHDATA_USERNAME\n", - " * EARTHDATA_PASSWORD\n", - "```python\n", - "auth = earthaccess.login(strategy=\"environment\")\n", - "```\n", - "* **interactive**: We will be asked for our EDL credentials with optinal persistance to `.netrc`\n", - "\n", - "```python\n", - "auth = earthaccess.login(strategy=\"interactive\", persist=True)\n", - "```\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "id": "fcc8b461-8c68-4719-94e5-34057159dac7", - "metadata": {}, - "source": [ - "## Querying for datasets\n", - "\n", - "The `DataCollections` class can query CMR for any collection (dataset) using all of CMR's Query parameters and has built-in functions to extract useful information from the response.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "8d5bf4c9-571b-4c93-af94-e66bd51cb584", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Datasets found: 25\n" - ] - } - ], - "source": [ - "# The first step is to create a DataCollections query \n", - "\n", - "datasets = earthaccess.search_datasets(\n", - " keyword = \"fire\",\n", - " cloud_hosted= True,\n", - " bounding_box = (-180, -90, 180, 90),\n", - " temporal = (\"2019\", \"2023\")\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "26962664-cbe8-453f-b617-80d473df9c75", - "metadata": {}, - "source": [ - "The results from a DataCollections and DataGranules query are enhanced python dictionaries, this means\n", - "that we can access all the keys and values like we usually do with Python dictionaries." - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "8cb5154c-f131-44ad-a68f-cf0fa21ce18f", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "C2271754179-LPCLOUD The Moderate Resolution Imaging Spectroradiometer (MODIS) Thermal Anomalies and Fire MOD14 Version 6.1 product is produced daily in 5-minute temporal satellite increments (swaths) at 1 kilometer (km) spatial resolution. The MOD14 product is used to generate all of the higher level fire products, but can also be used to identify fires and other thermal anomalies, such as volcanoes. Each swath of data is approximately 2,030 kilometers along track (long), and 2,300 kilometers across track (wide). \n", - "\n", - "Validation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for all MODIS Thermal Anomalies and Fire products. Further details regarding MODIS land product validation for the MOD14 data product is available from the MODIS land team validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD14).\n", - "\n", - "Improvements/Changes from Previous Versions\n", - "\n", - "* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n", - "* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB).\n", - "\n", - "----------------------------------------------------------------------------------------------------------------------------------------------------------------\n", - "C2278858993-LPCLOUD The Aqua Moderate Resolution Imaging Spectroradiometer (MODIS) Thermal Anomalies and Fire MYD14 Version 6.1 product is produced daily in 5-minute temporal satellite increments (swaths) at a 1 kilometer (km) spatial resolution. The MYD14 product is used to generate all of the higher level fire products, but can also be used to identify fires and other thermal anomalies, such as volcanoes. Each swath of data is approximately 2,030 kilometers along track (long), and 2,300 kilometers across track (wide). \n", - "\n", - "Validation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for all MODIS Thermal Anomalies and Fire products. Further details regarding MODIS land product validation for the MOD14 data product is available from the MODIS land team validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD14).\n", - "\n", - "Improvements/Changes from Previous Versions\n", - "\n", - "* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n", - "* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB).\n", - "----------------------------------------------------------------------------------------------------------------------------------------------------------------\n", - "C2565791013-LPCLOUD The Terra Moderate Resolution Imaging Spectroradiometer (MODIS) Thermal Anomalies and Fire Daily (MOD14A1) Version 6.1 data are generated every eight days at 1 kilometer (km) spatial resolution as a Level 3 product. MOD14A1 contains eight consecutive days of fire data conveniently packaged into a single file.\n", - "\n", - "The Science Dataset (SDS) layers include the fire mask, pixel quality indicators, maximum fire radiative power (MaxFRP), and the position of the fire pixel within the scan. Each layer consists of daily per pixel information for each of the eight days of data acquisition. \n", - "\n", - "Validation at stage 3 (https://modis-land.gsfc.nasa.gov/MODLAND_val.html) has been achieved for all MODIS Thermal Anomalies and Fire products. Further details regarding MODIS land product validation for the MOD14 data product is available from the MODIS land team validation site (https://modis-land.gsfc.nasa.gov/ValStatus.php?ProductID=MOD14).\n", - "\n", - " Improvements/Changes from Previous Versions\n", - "\n", - "* The Version 6.1 Level-1B (L1B) products have been improved by undergoing various calibration changes that include: changes to the response-versus-scan angle (RVS) approach that affects reflectance bands for Aqua and Terra MODIS, corrections to adjust for the optical crosstalk in Terra MODIS infrared (IR) bands, and corrections to the Terra MODIS forward look-up table (LUT) update for the period 2012 - 2017.\n", - "* A polarization correction has been applied to the L1B Reflective Solar Bands (RSB).\n", - "\n", - "----------------------------------------------------------------------------------------------------------------------------------------------------------------\n" - ] - } - ], - "source": [ - "for dataset in datasets[0:3]:\n", - " print(dataset.concept_id(), dataset.abstract())\n", - " print(\"--\"*80)" - ] - }, - { - "cell_type": "markdown", - "id": "28d45a6f-ac37-4744-bcfe-88ac3dd6ac07", - "metadata": {}, - "source": [ - "The DataCollections class returns python dictionaries with some handy methods.\n", - "\n", - "```python \n", - "collection.concept_id() # returns the concept-id, used to search for data granules\n", - "collection.abstract() # returns the abstract\n", - "collection.landing_page() # returns the landing page if present in the UMM fields\n", - "collection.get_data() # returns the portal where data can be accessed.\n", - "```\n", - "\n", - "The same results can be obtained using the `dict` syntax:\n", - "\n", - "```python\n", - "collection[\"meta\"][\"concept-id\"] # concept-id\n", - "collection[\"umm\"][\"RelatedUrls\"] # URLs, with GET DATA, LANDING PAGE etc\n", - "```\n" - ] - }, - { - "cell_type": "markdown", - "id": "4bb9c3bb-ac8b-48e8-8233-8c44da8fb7bc", - "metadata": {}, - "source": [ - "## Querying for data files (granules)\n", - "\n", - "The DataGranules class provides similar functionality as the collection class. To query for granules in a more reliable way concept-id would be the main key.\n", - "You can search data granules using a short name but that could (more likely will) return different versions of the same data granules. \n", - "\n", - "In this example we're querying for 10 data grnaules from ICESat-2 [ATL06](https://nsidc.org/data/ATL06/versions/) version `005` dataset. \n", - "\n", - "\n", - "> **Note**: Generally speaking we won't need authenticated queries unless they are restricted datasets for early adopters." - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "9364d737-5a79-4089-853f-76d2ad1c85a7", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Granules found: 87\n" - ] - } - ], - "source": [ - "from pprint import pprint\n", - "\n", - "# We query for our data\n", - "\n", - "results = earthaccess.search_data(\n", - " short_name = 'HLSL30',\n", - " cloud_cover = (0,30),\n", - " bounding_box = (-134.7, 58.9,-133.9, 59.2),\n", - " count=5\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "40aa8035-f4a8-4592-b19c-49d5c06331fb", - "metadata": {}, - "source": [ - "## Pretty printing data granules\n", - "\n", - "Since we are in a notebook we can take advantage of it to see a more user friendly version of the granules with the built-in function `display`\n", - "This will render browse image for the granule if available and eventually will have a similar representation as the one from the Earthdata search portal" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "0b56b119-ec9b-4922-911a-f37501597451", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/html": [ - "\n", - "
\n", - " \n", - "
\n", - "
\n", - "
\n", - "
\n", - "
\n", - "

Data: HLS.L30.T08VNL.2013147T201428.v2.0.B11.tifHLS.L30.T08VNL.2013147T201428.v2.0.B06.tifHLS.L30.T08VNL.2013147T201428.v2.0.B09.tifHLS.L30.T08VNL.2013147T201428.v2.0.B01.tifHLS.L30.T08VNL.2013147T201428.v2.0.VAA.tifHLS.L30.T08VNL.2013147T201428.v2.0.VZA.tifHLS.L30.T08VNL.2013147T201428.v2.0.Fmask.tifHLS.L30.T08VNL.2013147T201428.v2.0.B02.tifHLS.L30.T08VNL.2013147T201428.v2.0.SZA.tifHLS.L30.T08VNL.2013147T201428.v2.0.B04.tifHLS.L30.T08VNL.2013147T201428.v2.0.B05.tifHLS.L30.T08VNL.2013147T201428.v2.0.B07.tifHLS.L30.T08VNL.2013147T201428.v2.0.B03.tifHLS.L30.T08VNL.2013147T201428.v2.0.SAA.tifHLS.L30.T08VNL.2013147T201428.v2.0.B10.tif

\n", - "

Size: 45.62 MB

\n", - "

Cloud Hosted: True

\n", - "
\n", - "
\n", - " \"Data\n", - "
\n", - "
\n", - "
\n", - "
\n", - " " - ], - "text/plain": [ - "Collection: {'EntryTitle': 'HLS Landsat Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0'}\n", - "Spatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'GPolygons': [{'Boundary': {'Points': [{'Longitude': -134.90188398, 'Latitude': 58.55224361}, {'Longitude': -134.23067278, 'Latitude': 59.53608808}, {'Longitude': -135.00035365, 'Latitude': 59.53834913}, {'Longitude': -135.00034366, 'Latitude': 58.55228107}, {'Longitude': -134.90188398, 'Latitude': 58.55224361}]}}]}}}\n", - "Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2013-05-27T20:14:28.010Z', 'EndingDateTime': '2013-05-27T20:14:51.892Z'}}\n", - "Size(MB): 45.62166404724121\n", - "Data: ['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B11.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B06.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B09.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B01.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.VAA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.VZA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.Fmask.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B02.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.SZA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B04.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B05.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B07.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B03.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.SAA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013147T201428.v2.0/HLS.L30.T08VNL.2013147T201428.v2.0.B10.tif']" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "text/html": [ - "\n", - "
\n", - " \n", - "
\n", - "
\n", - "
\n", - "
\n", - "
\n", - "

Data: HLS.L30.T08VNL.2013163T201424.v2.0.SZA.tifHLS.L30.T08VNL.2013163T201424.v2.0.B04.tifHLS.L30.T08VNL.2013163T201424.v2.0.B02.tifHLS.L30.T08VNL.2013163T201424.v2.0.B03.tifHLS.L30.T08VNL.2013163T201424.v2.0.B10.tifHLS.L30.T08VNL.2013163T201424.v2.0.B11.tifHLS.L30.T08VNL.2013163T201424.v2.0.VAA.tifHLS.L30.T08VNL.2013163T201424.v2.0.SAA.tifHLS.L30.T08VNL.2013163T201424.v2.0.VZA.tifHLS.L30.T08VNL.2013163T201424.v2.0.B09.tifHLS.L30.T08VNL.2013163T201424.v2.0.Fmask.tifHLS.L30.T08VNL.2013163T201424.v2.0.B01.tifHLS.L30.T08VNL.2013163T201424.v2.0.B05.tifHLS.L30.T08VNL.2013163T201424.v2.0.B07.tifHLS.L30.T08VNL.2013163T201424.v2.0.B06.tif

\n", - "

Size: 48.39 MB

\n", - "

Cloud Hosted: True

\n", - "
\n", - "
\n", - " \"Data\n", - "
\n", - "
\n", - "
\n", - "
\n", - " " - ], - "text/plain": [ - "Collection: {'EntryTitle': 'HLS Landsat Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0'}\n", - "Spatial coverage: {'HorizontalSpatialDomain': {'Geometry': {'GPolygons': [{'Boundary': {'Points': [{'Longitude': -134.8807487, 'Latitude': 58.55222573}, {'Longitude': -134.20998754, 'Latitude': 59.53596485}, {'Longitude': -135.00035365, 'Latitude': 59.53834913}, {'Longitude': -135.00034366, 'Latitude': 58.55228107}, {'Longitude': -134.8807487, 'Latitude': 58.55222573}]}}]}}}\n", - "Temporal coverage: {'RangeDateTime': {'BeginningDateTime': '2013-06-12T20:14:24.508Z', 'EndingDateTime': '2013-06-12T20:14:48.386Z'}}\n", - "Size(MB): 48.38529682159424\n", - "Data: ['https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.SZA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B04.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B02.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B03.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B10.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B11.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.VAA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.SAA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.VZA.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B09.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.Fmask.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B01.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B05.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B07.tif', 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T08VNL.2013163T201424.v2.0/HLS.L30.T08VNL.2013163T201424.v2.0.B06.tif']" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "quicklooks = [display(g) for g in results[0:2]]" - ] - }, - { - "cell_type": "markdown", - "id": "ec8f32a4-026d-4a5f-af66-69026cabe966", - "metadata": {}, - "source": [ - "### Spatiotemporal queries\n", - "\n", - "Our granules and collection classes accept the same spatial and temporal arguments as CMR so we can search for granules that match spatiotemporal criteria.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": 35, - "id": "00aa39ec-e2fb-49d1-bc54-8d8a2f0655aa", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Granules found: 2\n" - ] - } - ], - "source": [ - "granules = earthaccess.search_data(\n", - " short_name=\"ATL08\",\n", - " temporal = (\"2020-03-01\", \"2020-03-30\"),\n", - " bounding_box =(-134.7,58.9,-133.9,59.2),\n", - " cloud_hosted=True\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 37, - "id": "8c493585-0d48-41bb-8815-6c83ad20ae80", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[{'links': ['https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL08/006/2020/03/10/ATL08_20200310121504_11420606_006_01.h5'],\n", - " 'size (MB):': 1.7647275924682617},\n", - " {'links': ['https://data.nsidc.earthdatacloud.nasa.gov/nsidc-cumulus-prod-protected/ATLAS/ATL08/006/2020/03/12/ATL08_20200312233336_11800602_006_01.h5'],\n", - " 'size (MB):': 57.75613307952881}]" - ] - }, - "execution_count": 37, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# Now we can print some info about these granules using the built-in methods\n", - "data_links = [{'links': g.data_links(access=\"on_prem\"), 'size (MB):': g.size()} for g in granules]\n", - "data_links" - ] - }, - { - "cell_type": "markdown", - "id": "f7c90c43-6e17-42f5-8bf5-95fdd3cb0dce", - "metadata": {}, - "source": [ - "## **Accessing the data**\n", - "\n", - "With `earthaccess` a researcher can get the files regardless if they are on-prem or cloud based with the same API call, although an important consideration is that if we want to access data in the cloud (direct access) we must run the code in the cloud. This is because some S3 buckets are configured to only allow direct access (s3:// links) if the requester is in the same zone, `us-west-2`.\n", - "\n", - "## On-prem access: DAAC hosted data 📡\n" - ] - }, - { - "cell_type": "code", - "execution_count": 38, - "id": "e7b80520-5cae-45c5-9397-f990a1ba0f26", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Granules found: 6\n", - "Granules found: 6\n", - "Granules found: 6\n", - "Granules found: 6\n", - "Granules found: 6\n", - "Granules found: 6\n", - "Granules found: 6\n", - "Granules found: 7\n", - "Granules found: 7\n", - "Granules found: 7\n" - ] - } - ], - "source": [ - "granules = []\n", - "\n", - "# we just grab 1 granule from May for each year of the dataset\n", - "for year in range(2009, 2019):\n", - " results = earthaccess.search_data(\n", - " doi = \"10.5067/SLREF-CDRV3\",\n", - " temporal=(f\"{year}-05\", f\"{year}-06\"),\n", - " count=1\n", - " )\n", - " if len(results)>0:\n", - " granules.append(results[0])" - ] - }, - { - "cell_type": "markdown", - "id": "4239e041-db87-40d1-b81a-12c26e9e0a47", - "metadata": {}, - "source": [ - "For this example we are going to use a PODAAC dataset `SMAP_JPL_L3_SSS_CAP_8DAY-RUNNINGMEAN_V5` which we previously queried (see querying for datasets) and got the concept id: `C1972955240-PODAAC`" - ] - }, - { - "cell_type": "markdown", - "id": "436e8a72-64c5-4e6b-950b-ac801d7b926e", - "metadata": {}, - "source": [ - "### Finally! let's get the data\n", - "\n", - "The Store class accepts the results from a `DataGranules()` query or it can also accept a list of URLs for the data files. In the second case we'll have to specify the DAAC since it cannot infer which credentials to use solely on the URL." - ] - }, - { - "cell_type": "code", - "execution_count": 39, - "id": "434466a3-602b-4dff-a260-f7db6901514a", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - " Getting 1 granules, approx download size: 0.01 GB\n" - ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "49c62c6d7a334cf6b739dc0fdb5f56a3", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "QUEUEING TASKS | : 0%| | 0/1 [00:00\n", - "\n", - "\n", - "\n", - "With `earthaccess` a researcher can get the files regardless if they are on-prem or cloud based with the same API call, although an important consideration is that if we want to access data in the cloud we must run the code in the cloud. This is because some S3 buckets are configured to only allow direct access (s3:// links) if the requester is in the same zone, `us-west-2`.\n", - "\n", - "Same API, just a different place, in this case the `concept-id` for the same dataset is `C2208422957-POCLOUD`\n", - "> Note: The `concept-id` changed even though is the same dataset." - ] - }, - { - "cell_type": "code", - "execution_count": 42, - "id": "b2a294f1-b1f9-4cd4-8751-dfc32feacec1", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Opening 10 granules, approx size: 0.09 GB\n" - ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "9c38a684f4f341bfbf8a60834f5b1c0a", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "QUEUEING TASKS | : 0%| | 0/10 [00:00 **Recommended read: [Skip the download! Stream NASA data directly into Python objects](https://medium.com/pangeo/intake-stac-nasa-4cd78d6246b7)**" - ] - }, - { - "cell_type": "code", - "execution_count": 44, - "id": "aecdb529-5961-4fa6-b7e0-70bbd0d85041", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "\n", - "
<xarray.Dataset>\n",
-       "Dimensions:      (Longitude: 2160, nv: 2, Latitude: 960, Time: 1)\n",
-       "Coordinates:\n",
-       "  * Longitude    (Longitude) float32 0.08333 0.25 0.4167 ... 359.6 359.8 359.9\n",
-       "  * Latitude     (Latitude) float32 -79.92 -79.75 -79.58 ... 79.58 79.75 79.92\n",
-       "  * Time         (Time) datetime64[ns] 2009-05-04T12:00:00\n",
-       "Dimensions without coordinates: nv\n",
-       "Data variables:\n",
-       "    Lon_bounds   (Longitude, nv) float32 ...\n",
-       "    Lat_bounds   (Latitude, nv) float32 ...\n",
-       "    Time_bounds  (Time, nv) datetime64[ns] ...\n",
-       "    SLA          (Time, Latitude, Longitude) float32 ...\n",
-       "    SLA_ERR      (Time, Latitude, Longitude) float32 ...\n",
-       "Attributes: (12/21)\n",
-       "    Conventions:            CF-1.6\n",
-       "    ncei_template_version:  NCEI_NetCDF_Grid_Template_v2.0\n",
-       "    Institution:            Jet Propulsion Laboratory\n",
-       "    geospatial_lat_min:     -79.916664\n",
-       "    geospatial_lat_max:     79.916664\n",
-       "    geospatial_lon_min:     0.083333336\n",
-       "    ...                     ...\n",
-       "    version_number:         2205\n",
-       "    Data_Pnts_Each_Sat:     {"16": 777555, "1003": 564903}\n",
-       "    source_version:         commit 7ba4d2404171bab4068e03a666a335c7a44c5f09\n",
-       "    SLA_Global_MEAN:        0.017279774082311437\n",
-       "    SLA_Global_STD:         0.08781475750200256\n",
-       "    latency:                final
" - ], - "text/plain": [ - "\n", - "Dimensions: (Longitude: 2160, nv: 2, Latitude: 960, Time: 1)\n", - "Coordinates:\n", - " * Longitude (Longitude) float32 0.08333 0.25 0.4167 ... 359.6 359.8 359.9\n", - " * Latitude (Latitude) float32 -79.92 -79.75 -79.58 ... 79.58 79.75 79.92\n", - " * Time (Time) datetime64[ns] 2009-05-04T12:00:00\n", - "Dimensions without coordinates: nv\n", - "Data variables:\n", - " Lon_bounds (Longitude, nv) float32 ...\n", - " Lat_bounds (Latitude, nv) float32 ...\n", - " Time_bounds (Time, nv) datetime64[ns] ...\n", - " SLA (Time, Latitude, Longitude) float32 ...\n", - " SLA_ERR (Time, Latitude, Longitude) float32 ...\n", - "Attributes: (12/21)\n", - " Conventions: CF-1.6\n", - " ncei_template_version: NCEI_NetCDF_Grid_Template_v2.0\n", - " Institution: Jet Propulsion Laboratory\n", - " geospatial_lat_min: -79.916664\n", - " geospatial_lat_max: 79.916664\n", - " geospatial_lon_min: 0.083333336\n", - " ... ...\n", - " version_number: 2205\n", - " Data_Pnts_Each_Sat: {\"16\": 777555, \"1003\": 564903}\n", - " source_version: commit 7ba4d2404171bab4068e03a666a335c7a44c5f09\n", - " SLA_Global_MEAN: 0.017279774082311437\n", - " SLA_Global_STD: 0.08781475750200256\n", - " latency: final" - ] - }, - "execution_count": 44, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "import warnings\n", - "warnings.filterwarnings('ignore')\n", - "warnings.simplefilter('ignore')\n", - "import xarray as xr\n", - "\n", - "ds = xr.open_dataset(fileset[0], engine=\"h5netcdf\")\n", - "ds" - ] - }, - { - "cell_type": "markdown", - "id": "d276eca3-ca27-4970-9367-7e65e5f2302f", - "metadata": {}, - "source": [ - "## Now to the important science! 🚀 \n", - "\n", - "### Recap\n", - "\n", - "```python\n", - "\n", - "import earthaccess\n", - "\n", - "# first we authenticate with NASA EDL\n", - "auth = earthaccess.login()\n", - "\n", - "# We query for our metadata\n", - "\n", - "granules = earthaccess.search_data(\n", - " short_name=\"SEA_SURFACE_HEIGHT_ALT_GRIDS_L4_2SATS_5DAY_6THDEG_V_JPL2205\" # Same as doi = \"10.5067/SLREF-CDRV3\",\n", - " bounding_box = (-180, -90, 180, 90),\n", - " temporal=(\"2019-11\", \"2019-12\"), \n", - " count=-1 # we can specify max results here\n", - ")\n", - "\n", - "# Now it{s time to download (or open) our data granules list with get()\n", - "files = earthaccess.download(granules, local_path='./data')\n", - "\n", - "# Now to the important science!\n", - "```" - ] - }, - { - "cell_type": "markdown", - "id": "d779e877-2f0c-4da2-92d5-6cc299204956", - "metadata": {}, - "source": [ - "### Related links\n", - "\n", - "**Github repository**: https://github.com/nsidc/earthaccess\n", - "\n", - "**CMR** API documentation: https://cmr.earthaccess.nasa.gov/search/site/docs/search/api.html\n", - "\n", - "**EDL** API documentation: https://urs.earthaccess.nasa.gov/\n", - "\n", - "NASA OpenScapes: https://nasa-openscapes.github.io/earthaccess-cloud-cookbook/\n", - "\n", - "NSIDC: https://nsidc.org\n", - "\n", - "\n", - "Contact: luis.lopez@nsidc.org" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.6" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} From 531d27a99194203e8d28396e35b60477ca4ff4bb Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:07:00 -0600 Subject: [PATCH 37/56] Add access data how-to to the nav Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index c8db9eb5..5b281cbb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -68,6 +68,7 @@ nav: - "Access": "user_guide/access.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" + - "Access data": "howto/access-data.md" - "Search NASA datasets using filters": "howto/search-collections.md" - "Search for data using filters": "howto/search-granules.md" - "Determine if a dataset is available in the cloud": "howto/storage-location.md" From e84852425696c8d109b7558e4d83887d4b87d0e4 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:09:22 -0600 Subject: [PATCH 38/56] Fix broken links We don't know why these links were not working; the paths seemed to be correct. Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/user_guide/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index 89f848ae..5812687e 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -2,11 +2,11 @@ `earthaccess` enables authentication, search and access for NASA Earth science data. The package enables users to find, and download or stream data archived both at NASA DAACs and in the AWS-hosted Earthdata Cloud. The User Guide introduces these three functions. -The [Authentication](/user_guide/authenticate.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. +The [Authentication](./authenticate.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. -The [Search](/user_guide/search.md) section introduces how to search for NASA datasets (collections) and specific data granules. +The [Search](./search.md) section introduces how to search for NASA datasets (collections) and specific data granules. -The [Access](/user_guide/access.md) section introduces how to use `earthaccess` to download data to a local machine or stream data by opening it directly in memory. +The [Access](./access.md) section introduces how to use `earthaccess` to download data to a local machine or stream data by opening it directly in memory. Additional usage information can be found in the User Reference section. The How-To sections contains short code examples for specific workflows. Longer tutorials for working with particular datasets are in the Tutorials section. From ec1261520ba30124267c1dc4f077d6b02e2543f3 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:11:35 -0600 Subject: [PATCH 39/56] Highlight python code in quick start Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/quick-start.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index 38545302..3824be21 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -60,7 +60,7 @@ To access NASA data, you have to login using your Earth Data Login credentials. By default, `earthaccess` will look for your Earth Data Login credentials in a `.netrc` file, or in environment variables `EARTHDATA_USERNAME` and `EARTHDATA_PASSWORD`. If you don't have either of these set up, you can login manually. See [Authenticating](howto/authenticate.md) to learn how to create a `.netrc` file or environment variables. -``` +```python import earthaccess earthaccess.login() @@ -72,7 +72,7 @@ earthaccess.login() As an example, we'll search for data from the NASA ICESat-2 mission. ATL06 -``` +```python results = earthaccess.search_data( short_name='ATL06' bounding_box=(-10, 20, 10, 50), @@ -85,7 +85,7 @@ results = earthaccess.search_data( Once you have found the files you want, you can download them to your local machine. -``` +```python files = earthaccess.download(results, "./local_folder") ``` From a4ce5cbe96a6f77338dad49ee6f6ef45227dc49c Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:11:52 -0600 Subject: [PATCH 40/56] Fix missing comma in quick start example code :bell: Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/quick-start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index 3824be21..eae211b1 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -74,7 +74,7 @@ As an example, we'll search for data from the NASA ICESat-2 mission. ATL06 ```python results = earthaccess.search_data( - short_name='ATL06' + short_name='ATL06', bounding_box=(-10, 20, 10, 50), temporal=("1999-02", "2019-03"), count=10 From d65ade988862e7b300d802d249e38dcf5aba400b Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:30:50 -0600 Subject: [PATCH 41/56] Fix last sentence of quick start Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/quick-start.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index eae211b1..31e7cb1b 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -89,4 +89,5 @@ Once you have found the files you want, you can download them to your local mach files = earthaccess.download(results, "./local_folder") ``` -If you are working in the cloud and the data files are hosted in the cloud, you can stream the data directly, without having to download data. See [Direct S3 Access]() +Data can also be opened in-memory with `earthacces.open()`. See [our API +docs](user-reference/api/api.md) for more. From ff76a70ddb348223db4e5ab7e130496618515ac3 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:32:01 -0600 Subject: [PATCH 42/56] Fix whitespace Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/user_guide/access.md | 2 +- docs/user_guide/search.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user_guide/access.md b/docs/user_guide/access.md index beafa3c2..873635fa 100644 --- a/docs/user_guide/access.md +++ b/docs/user_guide/access.md @@ -8,7 +8,7 @@ HOW-TO pages below. [How-to download data](/howto/onprem.md) [Getting Started Tutorial](/tutorials/getting-started.ipynb) - + ## Downloading data diff --git a/docs/user_guide/search.md b/docs/user_guide/search.md index cea2ffe2..2dd050f3 100644 --- a/docs/user_guide/search.md +++ b/docs/user_guide/search.md @@ -6,7 +6,7 @@ How-Tos and Tutorials in links below. [How-To Access Data](/howto/access-data.md) [Getting Started Tutorial](/tutorials/getting-started.ipynb) - + ## `search_datasets` ## `search_data` From bc4c1f2c48e1f0ea972fa29535681d9bedd2f594 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:40:04 -0600 Subject: [PATCH 43/56] Add context about how to run quick start code Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/quick-start.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index 31e7cb1b..dfe33185 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -51,7 +51,11 @@ are working in the cloud or on your local laptop or workstation. Read the to perform some common searches and data access, take a look at our how-to guides in the sidebar. -The only requirement to use this library is to open a free account with NASA [EDL](https://urs.earthdata.nasa.gov). +The only requirement to use this library is to open a free account with NASA +[EDL](https://urs.earthdata.nasa.gov). + +The following steps can be executed in a Python interpreter, a Python file, or a +[Jupyter notebook](https://docs.jupyter.org/en/latest/start/index.html). ### Step 1: Login From aac2f052892efbde8bbc7fdd154d7e6ef69922b8 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 12:48:08 -0600 Subject: [PATCH 44/56] Add callout about destination directory for download step Co-Authored-By: Mikala Beig Co-Authored-By: Arie Knoester Co-Authored-By: Andy Barrett Co-Authored-By: Jennifer Tran --- docs/quick-start.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/quick-start.md b/docs/quick-start.md index dfe33185..f7742555 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -93,5 +93,11 @@ Once you have found the files you want, you can download them to your local mach files = earthaccess.download(results, "./local_folder") ``` +!!! note + + This will download the data to a directory named `local_folder` in your current + working directory (the directory from which you are running this code, also known as + `.`). If that directory doesn't exist, it will be created automatically. + Data can also be opened in-memory with `earthacces.open()`. See [our API docs](user-reference/api/api.md) for more. From 6d295c0abe5e34b3be6ca99242246abcf3623365 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 30 Apr 2024 19:03:55 -0600 Subject: [PATCH 45/56] End bullet with period for consistency Co-authored-by: Jennifer Tran <12633533+botanical@users.noreply.github.com> --- docs/howto/authenticate.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/authenticate.md b/docs/howto/authenticate.md index 5b3edd74..68c16ff6 100644 --- a/docs/howto/authenticate.md +++ b/docs/howto/authenticate.md @@ -57,6 +57,6 @@ Once you are authenticated with NASA EDL you can: * Get a file from a DAAC using a `fsspec` session. * Request temporary S3 credentials from a particular DAAC (needed to download or stream data from an S3 bucket in the cloud). * Use the library to download or stream data directly from S3. -* Regenerate CMR tokens (used for restricted datasets) +* Regenerate CMR tokens (used for restricted datasets). From e1bbbf63d17061244aa3708e544d08b12b29dbea Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:40:26 -0600 Subject: [PATCH 46/56] Add statement about welcoming environment Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/contributing/our-meet-ups.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/our-meet-ups.md b/docs/contributing/our-meet-ups.md index a2183b1d..78181747 100644 --- a/docs/contributing/our-meet-ups.md +++ b/docs/contributing/our-meet-ups.md @@ -14,7 +14,7 @@ Hack days... * Occur on alternating Tuesdays at 11AM - 1PM Mountain Time. * Are self-determining; you can work on what sounds fun to you! * Are supportive; _earthaccess_ developers, maintainers, and community managers will - be present on the call. + be present on the call. We welcome and aim to foster new contributions and community members. * Include live demos on request! For a glimpse in to the work we do on a typical hack day, please visit our From 928fa695060aa03abd09d7f3290406408248d213 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:41:40 -0600 Subject: [PATCH 47/56] Punctuation adjustment Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/howto/access-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/access-data.md b/docs/howto/access-data.md index 23fbf0d4..b3b9951b 100644 --- a/docs/howto/access-data.md +++ b/docs/howto/access-data.md @@ -6,7 +6,7 @@ If we already have a workflow in place for downloading our data, we can use *ear ```python -# if the data set is cloud hosted there will be S3 links available. The access parameter accepts "direct" or "external", direct access is only possible if you are in the us-west-2 region in the cloud. +# if the data set is cloud-hosted there will be S3 links available. The access parameter accepts "direct" or "external". Direct access is only possible if you are in the us-west-2 region in the cloud. data_links = [granule.data_links(access="direct") for granule in results] # or if the data is an on-prem dataset From f0c71fbcc156fb7490b24128bda08afd418597b2 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:42:43 -0600 Subject: [PATCH 48/56] Fix typo Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/howto/access-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/access-data.md b/docs/howto/access-data.md index b3b9951b..e4d4c8e8 100644 --- a/docs/howto/access-data.md +++ b/docs/howto/access-data.md @@ -14,7 +14,7 @@ data_links = [granule.data_links(access="external") for granule in results] ``` -> Note: *earthaccess* can get S3 credentials for us, or auhenticated HTTP sessions in case we want to use them with a different library. +> Note: *earthaccess* can get S3 credentials for us, or authenticated HTTP sessions in case we want to use them with a different library. **Option 2: Download data to a local folder** From 65375704c4ed6e0be72aad22db3e5cf55ee1a219 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:43:50 -0600 Subject: [PATCH 49/56] Clarification Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/howto/access-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/access-data.md b/docs/howto/access-data.md index e4d4c8e8..5e3a66f2 100644 --- a/docs/howto/access-data.md +++ b/docs/howto/access-data.md @@ -26,7 +26,7 @@ files = earthaccess.download(results, "./local_folder") **Option 3: Direct S3 Access - Stream data directly to xarray** -This method works best if you are in the same Amazon Web Services (AWS) region as the data (us-west-2) and you are working with gridded datasets (processing level 3 and above). +This method works best if you are in the same Amazon Web Services (AWS) region as the data (us-west-2) and you are working with gridded (processing level 3 and above) datasets without nested or hierarchical data variables. ```python import xarray as xr From d50db6171bfc8b3d46b1debe4a4d85e1032c6b29 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:44:31 -0600 Subject: [PATCH 50/56] Better sentence flow Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/howto/access-data.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/howto/access-data.md b/docs/howto/access-data.md index 5e3a66f2..22f19521 100644 --- a/docs/howto/access-data.md +++ b/docs/howto/access-data.md @@ -37,7 +37,7 @@ ds = xr.open_mfdataset(files) ``` -And that's it! Just one line of code, and this same piece of code will also work for data that are not hosted in the cloud, i.e. located at NASA storage centers. +And that's it in just one line of code! This same piece of code will also work for data that are not hosted in the cloud, i.e. located at NASA storage centers. > More examples coming soon! From 05545a7d5e8905af9e6f19dd5b2347df1868fd70 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:45:01 -0600 Subject: [PATCH 51/56] Spell it out for clarity Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/quick-start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index f7742555..f0f1de93 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -52,7 +52,7 @@ to perform some common searches and data access, take a look at our how-to guide sidebar. The only requirement to use this library is to open a free account with NASA -[EDL](https://urs.earthdata.nasa.gov). +[Earthdata Login](https://urs.earthdata.nasa.gov). The following steps can be executed in a Python interpreter, a Python file, or a [Jupyter notebook](https://docs.jupyter.org/en/latest/start/index.html). From 4db0387db07a845471752b406fccb0024cf2a4c9 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:46:14 -0600 Subject: [PATCH 52/56] Clarify quick start prose Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/quick-start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index f0f1de93..de8789e4 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -73,7 +73,7 @@ earthaccess.login() ### Step 2: Search for data -As an example, we'll search for data from the NASA ICESat-2 mission. ATL06 +As an example, we'll search for the "ATLAS/ICESat-2 L3A Land Ice Height", or ATL06, dataset from the NASA [ICESat-2](https://nsidc.org/data/icesat-2) mission. ```python From 507ff368f563b0c74febf25b4c6084f433ef5ea3 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:46:45 -0600 Subject: [PATCH 53/56] Fix typo Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/quick-start.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/quick-start.md b/docs/quick-start.md index de8789e4..46f17b0a 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -99,5 +99,5 @@ files = earthaccess.download(results, "./local_folder") working directory (the directory from which you are running this code, also known as `.`). If that directory doesn't exist, it will be created automatically. -Data can also be opened in-memory with `earthacces.open()`. See [our API +Data can also be opened in-memory with `earthaccess.open()`. See [our API docs](user-reference/api/api.md) for more. From 07eee04f7b9afef8721d843af3c02f9b2ea2d4d4 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Mon, 6 May 2024 22:47:03 -0600 Subject: [PATCH 54/56] Spelling Co-authored-by: Amy Steiker <47193922+asteiker@users.noreply.github.com> --- docs/user_guide/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user_guide/index.md b/docs/user_guide/index.md index 5812687e..d3bddfbd 100644 --- a/docs/user_guide/index.md +++ b/docs/user_guide/index.md @@ -2,7 +2,7 @@ `earthaccess` enables authentication, search and access for NASA Earth science data. The package enables users to find, and download or stream data archived both at NASA DAACs and in the AWS-hosted Earthdata Cloud. The User Guide introduces these three functions. -The [Authentication](./authenticate.md) section shows how `earthaccess` can be used to manage Earth Data Login and also Earthdata Cloud credentials. +The [Authentication](./authenticate.md) section shows how `earthaccess` can be used to manage Earthdata Login and also Earthdata Cloud credentials. The [Search](./search.md) section introduces how to search for NASA datasets (collections) and specific data granules. From d5b59ef89a940d74711385398807d6809580feb4 Mon Sep 17 00:00:00 2001 From: Andy Barrett Date: Tue, 7 May 2024 13:58:50 -0600 Subject: [PATCH 55/56] Update mkdocs.yml --- mkdocs.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 5b281cbb..c99162a7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -68,10 +68,10 @@ nav: - "Access": "user_guide/access.md" - HOW-TO: - "Authenticate with Earthdata Login": "howto/authenticate.md" - - "Access data": "howto/access-data.md" - "Search NASA datasets using filters": "howto/search-collections.md" - "Search for data using filters": "howto/search-granules.md" - "Determine if a dataset is available in the cloud": "howto/storage-location.md" + - "Access data": "howto/access-data.md" - "Using authenticated sessions to access data": "howto/edl.ipynb" - "Download data from on-prem location": "howto/onprem.md" - "Direct S3 access - Open/stream files in the cloud": "howto/cloud.md" From 52e70c083f20c7c8ab53d9a6208cdff889bd5eb3 Mon Sep 17 00:00:00 2001 From: Matt Fisher Date: Tue, 7 May 2024 18:56:35 -0600 Subject: [PATCH 56/56] Hack: Restore RTD build by pinning poetry --- readthedocs.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/readthedocs.yml b/readthedocs.yml index acea8084..86d9a434 100644 --- a/readthedocs.yml +++ b/readthedocs.yml @@ -14,8 +14,12 @@ build: post_create_environment: # Install poetry - asdf plugin add poetry - - asdf install poetry latest - - asdf global poetry latest + # HACK: As of ~2024-05-07, poetry "latest" is not installable with + # asdf ("Version 1.8.3 does not exist") because the latest tagged version + # of poetry is not yet released. Should we avoid using "latest" from here + # on, or change it back once 1.8.3 is released? + - asdf install poetry 1.8.2 + - asdf global poetry 1.8.2 # Tell poetry to not use a virtual environment - poetry config virtualenvs.create false post_install: