How to access elements of data variable in xarray when dask clusters are created?

[aftabhunzai]

Whenever I create Dask Clusters on PC then I can't access elements of data variables of xarray? what is the solution to this issue? Please need help.

[delgadom]

the point of dask is the data is processed on separate threads, processes, or machines, to enable efficient and easy parallel and out of core (not held in RAM) computation. so you can't access the data directly by design. you need to use special methods such as compute in order to see the data - see the docs on xarray and dask. There is also an xarray with dask tutorial on the dask site.

If you have a more specific question and demonstration of your issue feel free to ask! It's hard to answer something so broad without just pointing you to the docs.

[aftabhunzai]

Thank you for response. I got the answer when I present a specific example here: microsoft/PlanetaryComputerExamples#150
However, based on (this linked) discussion I still have a specific question which is as follows if you would like to answer:
I used SAS token to read a modis file from Azure Blob directly to use Gateway Clusters. However, I got an error which is given below followed by program example code. Please guide, what is the issue in code, as the modis dataset is public so I am using access_key as empty. Does rasterio or Azure server not support empty_key to access public dataset?

import rioxarray as rxr
from azure.storage.blob import BlobServiceClient
from azure.storage.blob import ResourceTypes, AccountSasPermissions, generate_account_sas
from azure.storage.blob import BlobSasPermissions
from datetime import datetime, timedelta
from azure.storage.blob import baseblobservice

def get_sas_url(modis_account_name,account_key,modis_container_name,blob_name,url):
    service = baseblobservice.BaseBlobService(account_name=modis_account_name, account_key=account_key)
    token = service.generate_blob_shared_access_signature(modis_container_name, blob_name, 
    permission=BlobSasPermissions(read=True), expiry=datetime.utcnow() + timedelta(hours=1),)
    url_with_sas = f"{url}?{token}"
    return url_with_sas

account_key=""
modis_account_name = 'modissa'
modis_container_name = 'modis-006'
blob_name="MYD10A1/23/05/2018001/MYD10A1.A2018001.h23v05.006.2018003025822.hdf"
modis_account_url = 'https://' + modis_account_name + '.blob.core.windows.net/'
url=modis_account_url+modis_container_name+'/'+blob_name
print("URL is :",url)
sas_url=get_sas_url(modis_account_name,account_key,modis_container_name,blob_name,url)
print("this is sas url:  ",sas_url)
aa=rxr.open_rasterio(sas_url, chunks=(4, 2097152, 2097152,2097152,2097152), lock=False)
print(aa)`

Error is as follows: 
`
URL is : https://modissa.blob.core.windows.net/modis-006/MYD10A1/23/05/2018001/MYD10A1.A2018001.h23v05.006.2018003025822.hdf
https://modissa.blob.core.windows.net/modis-006/MYD10A1/23/05/2018001/MYD10A1.A2018001.h23v05.006.2018003025822.hdf?se=2022-04-21T05%3A11%3A41Z&sp=rt&sv=2018-03-28&sr=b&sig=0DknLoYQURphxsoqND6ZqKqSpKLYosrobUoVYEqN3N8%3D
this is sasurl:   https://modissa.blob.core.windows.net/modis-006/MYD10A1/23/05/2018001/MYD10A1.A2018001.h23v05.006.2018003025822.hdf?se=2022-04-21T05%3A11%3A41Z&sp=rt&sv=2018-03-28&sr=b&sig=0DknLoYQURphxsoqND6ZqKqSpKLYosrobUoVYEqN3N8%3D
---------------------------------------------------------------------------
CPLE_HttpResponseError                    Traceback (most recent call last)
File rasterio/_base.pyx:261, in rasterio._base.DatasetBase.__init__()

File rasterio/_shim.pyx:78, in rasterio._shim.open_dataset()

File rasterio/_err.pyx:216, in rasterio._err.exc_wrap_pointer()

CPLE_HttpResponseError: HTTP response code: 403

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
Input In [16], in <module>
     25 sas_url=get_sas_url(modis_account_name,account_key,modis_container_name,blob_name,url)
     27 print("this is sasurl:  ",sas_url)
---> 28 aa=rxr.open_rasterio(sas_url, chunks=(4, 2097152, 2097152,2097152,2097152), lock=False)
     29 print(aa)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rioxarray/_io.py:833, in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, decode_times, decode_timedelta, **open_kwargs)
    831     else:
    832         manager = URIManager(rasterio.open, filename, mode="r", kwargs=open_kwargs)
--> 833     riods = manager.acquire()
    834     captured_warnings = rio_warnings.copy()
    836 # raise the NotGeoreferencedWarning if applicable

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rioxarray/_io.py:117, in URIManager.acquire(self, needs_lock)
    113 if self._local.thread_manager is None:
    114     self._local.thread_manager = ThreadURIManager(
    115         self._opener, *self._args, mode=self._mode, kwargs=self._kwargs
    116     )
--> 117 return self._local.thread_manager.file_handle

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rioxarray/_io.py:73, in ThreadURIManager.file_handle(self)
     71 if self._file_handle is not None:
     72     return self._file_handle
---> 73 self._file_handle = self._opener(*self._args, mode=self._mode, **self._kwargs)
     74 return self._file_handle

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rasterio/env.py:437, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    434     session = DummySession()
    436 with env_ctor(session=session):
--> 437     return f(*args, **kwds)

File /srv/conda/envs/notebook/lib/python3.8/site-packages/rasterio/__init__.py:220, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    216 # Create dataset instances and pass the given env, which will
    217 # be taken over by the dataset's context manager if it is not
    218 # None.
    219 if mode == 'r':
--> 220     s = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    221 elif mode == "r+":
    222     s = get_writer_for_path(path, driver=driver)(
    223         path, mode, driver=driver, sharing=sharing, **kwargs
    224     )

File rasterio/_base.pyx:263, in rasterio._base.DatasetBase.__init__()

RasterioIOError: HTTP response code: 403

[delgadom]

Thanks for the update. This is pretty hard to read though! Check out the guide on formatting code blocks with github-flavored markdown. I see you have some single backticks in there... the code should be formatted with tripple backticks (make sure to get the open and close) for all of your code and tracebacks. For example, in your question you could write...

here's my function
```python
def my_example_func():
    print 'hello'
```
and this raises the following error:
```python
  Input In [1]
    print 'hello'
    ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?
```

And whenever editing a question, you can click the "Preview" tab within the editor to see what the post will look like. Thanks!

[delgadom]

just a hint on this error though - a 403 error means the request has been acknowledged by the server, but it has denied the request, often because you don't have sufficient permissions or haven't been authenticated. This seems like an issue with the way you're trying to access the azure storage, and seems like a question for the Planetary Computer team, not xarray.