Implementation of other sensors

[yuvraajnarula]

Pull Request

Description

Added implementation for other tensor datasets and their unit tests for sensor data processing functionality.

This PR adds comprehensive unit tests for the sensor data processing module, specifically testing the SensorDataset class and collate_fn function. The tests cover various sensor types including AMSU-A, ATMS, MHS, IASI, and CrIS.

Key changes:

• Added test_sensor_dataset to verify basic dataset functionality
• Added test_collate_function to ensure proper data batching
• Added test_sensor_datasets to test multiple sensor types
• Implemented mock DataCatalog fixture for consistent testing

Fixes #128

Test Results

tests/test_nnjai.py::test_sensor_dataset PASSED                [ 33%]
tests/test_nnjai.py::test_collate_function PASSED             [ 66%]
tests/test_nnjai.py::test_sensor_datasets PASSED              [100%]
================ 3 passed in 7.78s ================

Test output

Testing sensor: amsu-1bamua-NC021023
Dataset length for amsu-1bamua-NC021023: 100
First item from amsu-1bamua-NC021023:
{'timestamp': tensor(1.7383e+09), 'latitude': tensor(45.), 'longitude': tensor(-120.), 'metadata': tensor([250., 260.])}
Metadata for amsu-1bamua-NC021023: tensor([250., 260.])


Testing sensor: atms-atms-NC021203
Dataset length for atms-atms-NC021203: 100
First item from atms-atms-NC021203:
{'timestamp': tensor(1.7383e+09), 'latitude': tensor(45.), 'longitude': tensor(-120.), 'metadata': tensor([250., 260.])}
Metadata for atms-atms-NC021203: tensor([250., 260.])


Testing sensor: mhs-1bmhs-NC021027
Dataset length for mhs-1bmhs-NC021027: 100
First item from mhs-1bmhs-NC021027:
{'timestamp': tensor(1.7383e+09), 'latitude': tensor(45.), 'longitude': tensor(-120.), 'metadata': tensor([250., 260.])}
Metadata for mhs-1bmhs-NC021027: tensor([250., 260.])


Testing sensor: iasi-mtiasi-NC021241
Dataset length for iasi-mtiasi-NC021241: 100
First item from iasi-mtiasi-NC021241:
{'timestamp': tensor(1.7383e+09), 'latitude': tensor(45.), 'longitude': tensor(-120.), 'metadata': tensor([250., 260.])}
Metadata for iasi-mtiasi-NC021241: tensor([250., 260.])


Testing sensor: cris-crisf4-NC021206
Dataset length for cris-crisf4-NC021206: 100
First item from cris-crisf4-NC021206:
{'timestamp': tensor(1.7383e+09), 'latitude': tensor(45.), 'longitude': tensor(-120.), 'metadata': tensor([250., 260.])}
Metadata for cris-crisf4-NC021206: tensor([250., 260.])

Testing Details

• Created mock data fixtures to avoid external data dependencies
• Verified tensor shapes, data types, and structures
• Tested data loading and processing for multiple sensor types
• Validated collate function for batch processing
• All tests pass successfully with pytest

Checklist:

• Code follows OCF's coding style guidelines
• Performed self-review of code
• No documentation changes needed (test-only changes)
• Added comprehensive test suite
• Checked code for misspellings

The PR focuses on improving test coverage for the sensor data processing pipeline, ensuring reliable data handling across different sensor types while maintaining code quality standards.

[jacobbieker]

Thanks for this! A good improvement over the original one, just a few changes I think, then we can merge this in.

[jacobbieker]

I don't think we should actually import this here, since we have nnja as an optional dependency. I think to import this dataset, like in the tests, we should just do from graph_weather.data.nnja_ai import SensorDataset for example. I also renamed this on the main branch, but could you update the nnjai_wrapp.py to be named nnja_ai as I think that is maybe more consistent naming.

[jacobbieker]

Suggested change

	class SensorDataset(Dataset):
	class NNJADataset(Dataset):

We might (probably will, as I am looking into adding other, non NNJA sensors) have other sensors that won't fit in this dataset. I think naming this just NNJADataset then is more descriptive and easier to parse where this data is coming from.

[yuvraajnarula]

Could you please tell me the status of any changes for this PR?

[jacobbieker]

A few changes needed I think for updating the tests and removing an old file. But then I think it would be good to merge. If you can also resolve the merge conflicts, that would be great.

[jacobbieker]

Could you remove this file as its not used now?

[jacobbieker]

Suggested change

	Tests for the nnjai_wrapp module in the graph_weather package.
	Tests for the nnja_ai module in the graph_weather package.

[jacobbieker]

Do these tests still pass? I think they need to be updated for the new names. Ideally, there should be tests for all the different sensor types as well in NNJADataset. You should be able to do that with pytest.parameterize I think.