A syrupy replacement (#2473)

* restructure of geometry

* add radial1d boundary logic

* black format

* several fixes

* fix epsilon

* add testing of boundaries

* change the r_inner_active

* first integration with `from_config` working

* hunting down density indexing bug

* all model tests (without csvy) pass

* more fixes

* fix of model to simulation_state

* fix inner boundary packet error

* fix some leftovers

* final fix for csvy

* blackify

* restructure to readers and remove some leftover code

* further cleanup

* first start of the restructure

* add comment about removing quantitiness

* add velocity check

* add new abundance functions

* remove default units

* add new matter module

* several restructures. move decay from io to tardis/model/matter

* mid restructure

* slow progress on abundance refactor

* include effective_element_masses

* further updates

* fix of csvy readers

* last fixes to custom abundance widget

* restructuring the radiation field

* some cleanup

* cleanup

* further cleanup

* clearnup

* changes

* remove matter base.py

* removing matter

* fixed last tests

* first commit for regression_data

* add several parsers

* add composition

* remove isotopemassfractions

* some small fixes

* fixed w, t_rad to dilution_factor, t_radiative

* add setters for w, t_rad

* several fixes for instantiating the right t_rads.
also ensuring that they are positive and len(t_rad) == dilution factor

* add the from_config atom_data

* fixing model.ipynb

* fix some more tests

* fix tests

* fix some of the issues wdigets

* assert mass fractions positive.
Change IsotopeAbundances to Massfractions

* add radiation_field_state

* Fix isotope mass fraction typo and t_radiatve duplicate definition

* Document remove functionality for effective element masses

* Fix variable name in
convert_to_nuclide_mass_fraction function

* Refactor test names and variable names in
test_base.py

* Refactor gamma ray simulation setup and test
functions

* fix gamma ray tests

* Add model_isotope_time_0 property to abundances
schema

* Add test for montecarlo main loop with vpacket
tracking

* Refactor config_reader and config_validator
modules

* Fix model_isotope_time_0 property in
model_definitions.yml

* Refactor simulation module to use pathlib

* remove bad comment and refactor with ruff

* Fix variable name in test_simulation_state_mass()
function

* Fix unit conversion in calculate_cell_masses
function

* Fix logger debug message in CSV reader and update
method names in Composition class

* black formatting

* Refactor code to use composition variable in
test_gamma_ray_transport.py

* Refactor imports in atom_web_download.py and
update return statement in config_internal.py

* fix for documentation not builiding

* add fix for model_isotope_time_0

* fix grid test

* restructure the grid

* final change

* add stuff

* add vpacket_log

* Refactor model_reader.py and shell_info.py
changing w to dilution_factor

* Update damping constants and variable names
in doc notebook

* fix w in several places in tardis

* slowly fixing radiation_field changes

* last fix

* change hdf from model to simulation_state

* fixing model->simulation-state

* Fix model_reader.py imports and formatting issues

* Refactor variable names for clarity and
consistency and being correct

* Update variable names in model.ipynb

* Update variable name for average temperature
calculation

* Implement regression data configurations

* Refactor snapshot fixtures and add regression data
handling

* Update to new testing framework

* Refactor imports and fixtures in conftest.py and
test_nlte_solver.py

* Make regression-data optional

* Fix regression data synchronization in plasma
tests

* Refactor variable names in model reader and
simulation base

* Refactor store_model_to_hdf to
store_simulation_state_to_hdf

* Refactor HDF file handling and model reader
imports

* add docstr

* Remove syrupy_extensions.py

* setting up regression with github actions

* Update LFS cache IDs for refdata and regression
data

* Fix test_montecarlo_radial1d function

* Update sync_hdf_store method in RegressionData class

* Refactor test_base.py for improved accuracy and precision

* update docstrings

* Fix warnings and add HDF output in sdec_plot.ipynb and cleanup regression_data.py

* Update demo.hdf file

Delete demo.hdf file

Update file paths in code

* Update URL for atomic data repository

* Refactor code to remove unnecessary conversion to numpy array in test_hdf_scalars

.github/workflows/tests.yml

@@ -15,7 +15,7 @@ on:
 
 env:
   CACHE_NUMBER: 0  # increase to reset cache manually
-  PYTEST_FLAGS: --tardis-refdata=${{ github.workspace }}/tardis-refdata --tardis-snapshot-data=${{ github.workspace }}/tardis-regressions
+  PYTEST_FLAGS: --tardis-refdata=${{ github.workspace }}/tardis-refdata --tardis-regression-data=${{ github.workspace }}/tardis-regression-data
                 --cov=tardis --cov-report=xml --cov-report=html 
   CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
 
@@ -62,36 +62,68 @@ jobs:
 
       - name: Restore LFS cache
         uses: actions/cache/restore@v3
-        id: lfs-cache
+        id: lfs-cache-refdata
         with:
           path: tardis-refdata/.git/lfs
           key: ${{ runner.os }}-lfs-${{ hashFiles('tardis-refdata/.lfs-assets-id') }}-v1
 
       - name: Git LFS Pull
         run: git lfs pull
         working-directory: tardis-refdata
-        if: steps.lfs-cache.outputs.cache-hit != 'true'
+        if: steps.lfs-cache-refdata.outputs.cache-hit != 'true'
 
       - name: Git LFS Checkout
         run: git lfs checkout
         working-directory: tardis-refdata
-        if: steps.lfs-cache.outputs.cache-hit == 'true'
+        if: steps.lfs-cache-refdata.outputs.cache-hit == 'true'
 
       - name: Save LFS cache if not found
         # uses fake ternary 
         # for reference: https://github.com/orgs/community/discussions/26738#discussioncomment-3253176
         if: ${{ steps.lfs-cache.outputs.cache-hit != 'true'  && always() || false }}
         uses: actions/cache/save@v3
-        id: lfs-cache-save
+        id: lfs-cache-refdata-save
         with:
           path: tardis-refdata/.git/lfs
           key: ${{ runner.os }}-lfs-${{ hashFiles('tardis-refdata/.lfs-assets-id') }}-v1
 
-      - name: Clone tardis-sn/tardis-regressions
+      - name: Clone tardis-sn/tardis-regression-data
         uses: actions/checkout@v4
         with:
-          repository: tardis-sn/tardis-regressions
-          path: tardis-regressions
+          repository: tardis-sn/tardis-regression-data
+          path: tardis-regression-data
+
+      - name: Create LFS file list
+        run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
+        working-directory: tardis-refdata
+
+      - name: Restore LFS cache
+        uses: actions/cache/restore@v3
+        id: lfs-cache-regression-data
+        with:
+          path: tardis-regression-data/.git/lfs
+          key: ${{ runner.os }}-lfs-${{ hashFiles('tardis-regression-data/.lfs-assets-id') }}-v1
+
+      - name: Git LFS Pull
+        run: git lfs pull
+        working-directory: tardis-regression-data
+        if: steps.lfs-cache-regression-data.outputs.cache-hit != 'true'
+
+      - name: Git LFS Checkout
+        run: git lfs checkout
+        working-directory: tardis-regression-data
+        if: steps.lfs-cache-regression-data.outputs.cache-hit == 'true'
+
+      - name: Save LFS cache if not found
+        # uses fake ternary 
+        # for reference: https://github.com/orgs/community/discussions/26738#discussioncomment-3253176
+        if: ${{ steps.lfs-cache.outputs.cache-hit != 'true'  && always() || false }}
+        uses: actions/cache/save@v3
+        id: lfs-cache-regression-data-save
+        with:
+          path: tardis-regression-data/.git/lfs
+          key: ${{ runner.os }}-lfs-${{ hashFiles('tardis-regression-data/.lfs-assets-id') }}-v1
+
 
       - name: Setup environment
         uses: conda-incubator/setup-miniconda@v2

docs/io/visualization/abundance_widget.ipynb

@@ -113,7 +113,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# widget = CustomAbundanceWidget.from_hdf(\"demo.hdf\")"
+    "# widget = CustomAbundanceWidget.from_hdf(\"demo.h5\")"
    ]
   },
   {

docs/io/visualization/generating_widgets.ipynb

@@ -127,7 +127,7 @@
    },
    "outputs": [],
    "source": [
-    "# shell_info_widget = shell_info_from_hdf('demo.hdf')\n",
+    "# shell_info_widget = shell_info_from_hdf('demo.h5')\n",
     "# shell_info_widget.display()"
    ]
   },

docs/io/visualization/sdec_plot.ipynb

@@ -22,7 +22,7 @@
    "source": [
     "# We filter out warnings throughout this notebook\n",
     "import warnings\n",
-    "warnings.filterwarnings('ignore');\n",
+    "warnings.filterwarnings('ignore')\n",
     "\n",
     "# Due to the large size of the SDEC plots in SVG format, we request output as a\n",
     "# high-resolution PNG\n",
@@ -54,7 +54,7 @@
     "# We download the atomic data needed to run the simulation\n",
     "download_atom_data('kurucz_cd23_chianti_H_He')\n",
     "\n",
-    "sim = run_tardis(\"tardis_example.yml\", virtual_packet_logging=True)"
+    "sim = run_tardis(\"tardis_example.yml\", virtual_packet_logging=True)\n"
    ]
   },
   {
@@ -139,7 +139,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl();"
+    "plotter.generate_plot_mpl()"
    ]
   },
   {
@@ -162,7 +162,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(\"real\");"
+    "plotter.generate_plot_mpl(\"real\")"
    ]
   },
   {
@@ -198,7 +198,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(packet_wvl_range=[3000, 9000] * u.AA);"
+    "plotter.generate_plot_mpl(packet_wvl_range=[3000, 9000] * u.AA)"
    ]
   },
   {
@@ -226,7 +226,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(packet_wvl_range=[2000, 8000] * u.AA, nelements = 3);"
+    "plotter.generate_plot_mpl(packet_wvl_range=[2000, 8000] * u.AA, nelements = 3)"
    ]
   },
   {
@@ -254,7 +254,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(packet_wvl_range=[2000, 8000] * u.AA, species_list = ['Si II', 'S I-V', 'Ca']);"
+    "plotter.generate_plot_mpl(packet_wvl_range=[2000, 8000] * u.AA, species_list = ['Si II', 'S I-V', 'Ca'])"
    ]
   },
   {
@@ -275,7 +275,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(nelements = 3, species_list = ['Si II', 'S I-V', 'Ca']);"
+    "plotter.generate_plot_mpl(nelements = 3, species_list = ['Si II', 'S I-V', 'Ca'])"
    ]
   },
   {
@@ -297,7 +297,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(distance=100 * u.Mpc);"
+    "plotter.generate_plot_mpl(distance=100 * u.Mpc)"
    ]
   },
   {
@@ -338,7 +338,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(observed_spectrum = (observed_spectrum_wavelength, observed_spectrum_flux), distance = 6 * u.Mpc);"
+    "plotter.generate_plot_mpl(observed_spectrum = (observed_spectrum_wavelength, observed_spectrum_flux), distance = 6 * u.Mpc)"
    ]
   },
   {
@@ -360,7 +360,7 @@
    },
    "outputs": [],
    "source": [
-    "plotter.generate_plot_mpl(show_modeled_spectrum=False);"
+    "plotter.generate_plot_mpl(show_modeled_spectrum=False)"
    ]
   },
   {
@@ -382,7 +382,7 @@
    },
    "outputs": [],
    "source": [
-    "# To list all avaialble options (or parameters) with their description\n",
+    "# To list all available options (or parameters) with their description\n",
     "help(plotter.generate_plot_mpl)"
    ]
   },
@@ -490,7 +490,7 @@
    },
    "outputs": [],
    "source": [
-    "# To list all avaialble options (or parameters) with their description\n",
+    "# To list all available options (or parameters) with their description\n",
     "help(plotter.generate_plot_ply)"
    ]
   },
@@ -513,7 +513,7 @@
    },
    "outputs": [],
    "source": [
-    "hdf_plotter = SDECPlotter.from_hdf(\"demo.hdf\")"
+    "# hdf_plotter = SDECPlotter.from_hdf(\"demo.h5\") ## Files is too large - just as an example"
    ]
   },
   {
@@ -535,7 +535,7 @@
    "outputs": [],
    "source": [
     "# Static plot with virtual packets mode\n",
-    "hdf_plotter.generate_plot_mpl();"
+    "# hdf_plotter.generate_plot_mpl()"
    ]
   },
   {
@@ -550,7 +550,7 @@
    "outputs": [],
    "source": [
     "# Static plot with real packets mode\n",
-    "hdf_plotter.generate_plot_mpl(\"real\");"
+    "# hdf_plotter.generate_plot_mpl(\"real\")"
    ]
   },
   {
@@ -565,7 +565,7 @@
    "outputs": [],
    "source": [
     "# Interactive plot with virtual packets mode\n",
-    "hdf_plotter.generate_plot_ply()"
+    "# hdf_plotter.generate_plot_ply()"
    ]
   }
  ],

tardis/conftest.py

@@ -1,3 +1,19 @@
+import os
+from pathlib import Path
+
+import pandas as pd
+import pytest
+from astropy.version import version as astropy_version
+
+from tardis.io.configuration.config_reader import Configuration
+from tardis.io.util import YAMLLoader, yaml_load_file
+from tardis.simulation import Simulation
+from tardis.tests.fixtures.atom_data import *
+from tardis.tests.fixtures.regression_data import regression_data
+
+# ensuring that regression_data is not removed by ruff
+assert regression_data is not None
+
 """Configure Test Suite.
 
 This file is used to configure the behavior of pytest when using the Astropy
@@ -7,10 +23,6 @@
 
 """
 
-import os
-from pathlib import Path
-
-from astropy.version import version as astropy_version
 
 # For Astropy 3.0 and later, we can use the standalone pytest plugin
 if astropy_version < "3.0":
@@ -70,32 +82,24 @@ def pytest_configure(config):
 # To ignore some specific deprecation warning messages for Python version
 # MAJOR.MINOR or later, add:
 #     warnings_to_ignore_by_pyver={(MAJOR, MINOR): ['Message to ignore']}
-# from astropy.tests.helper import enable_deprecations_as_exceptions  # noqa
+# from astropy.tests.helper import enable_deprecations_as_exceptions
 # enable_deprecations_as_exceptions()
 
 # -------------------------------------------------------------------------
 # Here the TARDIS testing stuff begins
 # -------------------------------------------------------------------------
 
-import re
-import pytest
-import pandas as pd
-from tardis.io.util import yaml_load_file, YAMLLoader
-from tardis.io.configuration.config_reader import Configuration
-from tardis.simulation import Simulation
-from tardis.util.syrupy_extensions import (
-    SingleFileSanitizedNames,
-    NumpySnapshotExtenstion,
-    PandasSnapshotExtenstion,
-)
-
-pytest_plugins = "syrupy"
-
 
 def pytest_addoption(parser):
     parser.addoption(
         "--tardis-refdata", default=None, help="Path to Tardis Reference Folder"
     )
+    parser.addoption(
+        "--tardis-regression-data",
+        default=None,
+        help="Path to the TARDIS regression data directory",
+    )
+
     parser.addoption(
         "--integration-tests",
         dest="integration-tests",
@@ -108,13 +112,6 @@ def pytest_addoption(parser):
         default=False,
         help="generate reference data instead of testing",
     )
-
-    parser.addoption(
-        "--tardis-snapshot-data",
-        default=None,
-        help="Path to Tardis Snapshot Folder",
-    )
-
     parser.addoption(
         "--less-packets",
         action="store_true",
@@ -168,9 +165,6 @@ def tardis_snapshot_path(request):
         )
 
 
-from tardis.tests.fixtures.atom_data import *
-
-
 @pytest.yield_fixture(scope="session")
 def tardis_ref_data(tardis_ref_path, generate_reference):
     if generate_reference:
@@ -238,23 +232,3 @@ def simulation_verysimple(config_verysimple, atomic_dataset):
     sim = Simulation.from_config(config_verysimple, atom_data=atomic_data)
     sim.iterate(4000)
     return sim
-
-
-# -------------------------------------------------------------------------
-# fixtures and plugins for syrupy/regression data testing
-# -------------------------------------------------------------------------
-
-
-@pytest.fixture
-def pandas_snapshot_extention():
-    return PandasSnapshotExtenstion
-
-
-@pytest.fixture
-def numpy_snapshot_extension():
-    return NumpySnapshotExtenstion
-
-
-@pytest.fixture
-def singlefilesanitized():
-    return SingleFileSanitizedNames

tardis/data/atomic_data_repo.yml

@@ -1,7 +1,7 @@
 default: kurucz_cd23_chianti_H_He
 
 kurucz_cd23_chianti_H_He:
-  url: https://dev.azure.com/tardis-sn/TARDIS/_apis/git/repositories/tardis-refdata/items?path=atom_data/kurucz_cd23_chianti_H_He.h5&resolveLfs=true
+  url: https://media.githubusercontent.com/media/tardis-sn/tardis-refdata/master/atom_data/kurucz_cd23_chianti_H_He.h5
   mirrors:
     - https://dev.azure.com/tardis-sn/TARDIS/_apis/git/repositories/tardis-refdata/items?path=atom_data/kurucz_cd23_chianti_H_He.h5&resolveLfs=true
     - https://media.githubusercontent.com/media/tardis-sn/tardis-refdata/master/atom_data/kurucz_cd23_chianti_H_He.h5