Add uc 08 derive simulation production configuration parameters #1098

tobiaskleiner · 2024-08-01T13:25:14Z

This PR adds the functionality defined in UC8 with the following four applications:

simtools-production-calculate-resource-estimates
-- calculates compute and storage resources --
simtools-production-generate-grid
-- generates a grid of simulation points --
simtools-production-generate-simulation-config
-- generates simulation parameters for a specific grid point --
simtools-production-scale-events
-- metric evaluation and statistical error calculations --

Modules are stored in production_configuration:

calculate_statistical_errors_grid_point.py
derive_computing_resources.py
generate_production_grid.py
generate_simulation_config.py
interpolation_handler.py

GernotMaier · 2024-09-10T09:37:50Z

Would like start the discussions how to best organize the code. At this point most of it is in the applications, but we want to move it into modules.

Suggest to start a module to put those classes in. What would be a good name?

simtools.production_tools (don't like tools, seems to cover everything)
simtools.production_configuration
simtools.simulation_production
...something shorter...?

This module would include all classes currently defined in

simtools/applications/derive_resources.py (which will be renamed to simtools/applications/derive_computing_resources.py
simtools/applications/generate_grid.py (which will be renamed to simtools/applications/generate_production_grid.py)
simtools/applications/generate_simulation_config.py
`simtools/utils/calculate_statistical_errors_grid_point.py (maybe we can find a shorter name)

@orelgueta , @tobiaskleiner : please comment.

orelgueta · 2024-09-10T09:52:24Z

Not much for me to comment. This is the first time I see that those modules are in applications and I definitely agree they should go into simtools instead. In terms of which name to put them under, I would vote for simtools.production_configuration.
However, I would also check if we can integrate these classes with the current modules we have (at least partly).

tobiaskleiner · 2024-09-10T13:40:24Z

I agree with the simtools.production_configuration suggestion and to move some parts into visualization or in other parts.

GernotMaier · 2024-10-08T07:41:47Z

simtools/production_configuration/calculate_statistical_errors_grid_point.py

+
+        # Determine the effective area threshold (50% of max effective area)
+        max_efficiency = np.max(efficiencies)
+        threshold_efficiency = 0.1 * max_efficiency


The docstring says "exceeds 50%". Do you use actually 10%?

tobiaskleiner · 2024-11-12T08:27:54Z

@GernotMaier thanks for the review. I went through the comments and adressed most of them. Few more points need discussion/implementation see #1233, #1227, #1219. Will let you know when I have fixed the unit tests for another review.

tobiaskleiner · 2024-11-15T16:20:36Z

@GernotMaier thanks again for the review of the first part of the PR. Went through your comments and fixed the unit/integration tests. I factored out the event scaling logic and added a file for helper functions in the production configuration folder. In there we could also move the dl2 reading part in a later stage.
Let me know if the changes make sense and if so you could start reviewing the other parts of the PR.

GernotMaier

A couple of more comments on the production_generate_simulation_config.

I will talk to you directly about a couple of points.

GernotMaier · 2024-11-19T06:41:17Z

simtools/applications/production_generate_simulation_config.py

+    The data level for the simulation (e.g., 'A', 'B', 'C').
+science_case (str, required)
+    The science case for the simulation.
+file_path (str, required)


Suggest to change the file_path doc string to

Path to file with MC events at CTAO DL2 data level. Used for statistical uncertainty evaluation.

GernotMaier · 2024-11-19T06:41:32Z

simtools/applications/production_generate_simulation_config.py

+elevation (float, required)
+    Elevation angle in degrees.
+nsb (float, required)
+    Night sky background value.


I think the units for the NSB is "Hz" (but please check)

Not 1/(srnscm**2) ?

please find it out. I know that in some places we use Hz (which requires the knowledge of the pixel fov)

GernotMaier · 2024-11-19T06:46:30Z

simtools/applications/production_generate_simulation_config.py

+    config.parser.add_argument(
+        "--data_level", type=str, required=True, help="Data level (e.g., 'A', 'B', 'C')."
+    )
+    config.parser.add_argument(


Agree, added this to the list of discussions.

GernotMaier · 2024-11-19T06:46:56Z

simtools/applications/production_generate_simulation_config.py

+        "--science_case", type=str, required=True, help="Science case for the simulation."
+    )
+    config.parser.add_argument(
+        "--file_path", type=str, required=True, help="Path to the dl2_mc_events_file FITS file."


See comment above.

Path to MC event file in DL2 format

Adding some comments here, but the changes are done in the separate part1 PR.
Changed the comment to your suggestion.

GernotMaier · 2024-11-19T06:48:20Z

simtools/applications/production_generate_simulation_config.py

+        "--metrics_file",
+        required=True,
+        type=str,
+        help="Path to YAML file containing metrics and required precision as values (required).",


I think you can remove the (required) from the comment (it is the only parameter with this added, although others are required).

GernotMaier · 2024-11-19T08:11:40Z

simtools/production_configuration/calculate_statistical_errors_grid_point.py

+        max_error : float
+            Maximum relative error.
+        """
+        if "relative_errors" in self.metric_results["error_eff_area"]:


Trying to understand a case where relative_errors is not in metric_results by error_eff_area is filled. If I understand it correctly, both variables are always filled in calculate_metrics?

No currently this depends on the production_simulation_config_metrics file and what metrics are given there (i.e. which metric computation is required).

GernotMaier · 2024-11-19T08:25:42Z

simtools/production_configuration/calculate_statistical_errors_grid_point.py

+            )
+        valid = (simulated_event_counts > 0 * u.ct) & (triggered_event_counts > 0 * u.ct)
+
+        uncertainties = np.zeros_like(triggered_event_counts) * u.ct**-0.5


Can you explain the '-0.5'?

good point, this was wrongly implemented and the errors should be dimensionless. Previously when keeping the units the rel error turns out with this dimension.

GernotMaier · 2024-11-19T08:29:19Z

simtools/production_configuration/calculate_statistical_errors_grid_point.py

+
+        return efficiencies, relative_errors
+
+    def calculate_energy_threshold(self):


Suggest to replace the hardwired 10% by a variable (which default is 10%)

GernotMaier · 2024-11-19T08:31:22Z

simtools/production_configuration/calculate_statistical_errors_grid_point.py

+        bin_edges = np.concatenate([bin_edges_low, [bin_edges_high[-1]]])
+        return np.unique(bin_edges)
+
+    def compute_histogram(self, event_energies_reco, bin_edges):


Suggest 'compute_triggered_event_histogram' to make the purpose of this function clearer.

GernotMaier · 2024-11-19T08:32:22Z

simtools/production_configuration/calculate_statistical_errors_grid_point.py

+        Parameters
+        ----------
+        event_energies_reco : array
+            Array of energies of the observed events.


Array of reconstructed energy per event

…UC-08-derive-simulation-production-configuration-parameters

…masim/simtools into add-UC-08-derive-simulation-production-configuration-parameters

…rameters' of https://github.com/gammasim/simtools into add-UC-08-derive-simulation-production-configuration-parameters

This reverts commit dd2cc81.

…UC-08-derive-simulation-production-configuration-parameters

ctao-dpps-sonarqube · 2024-11-26T13:55:10Z

Analysis Details

0 Issues

0 Bugs
0 Vulnerabilities
0 Code Smells

Coverage and Duplications

82.20% Coverage (92.80% Estimated after merge)
0.00% Duplicated Code (0.00% Estimated after merge)

Project ID: gammasim_simtools_AY_ssha9WiFxsX-2oy_w

View in SonarQube

GernotMaier · 2025-02-20T09:41:09Z

@tobiaskleiner - I am not sure on the status of this pull request and branch. Is this still relevant given all the other updates you have done? Please have a look and remove it if necessary.

tobiaskleiner · 2025-02-20T17:59:01Z

@GernotMaier the only part that is not yet included in the code or in the other PR is generate_production_grid. But maybe its best if we close this one and I open another PR with just this part included?

GernotMaier · 2025-02-21T07:47:01Z

Agree. Please close the PR and put a comment on where the updated work is taking place.

tobiaskleiner · 2025-02-21T15:03:32Z

Closing this PR as its parts are now either implemented (threshold/limits calculation for production, event scaling for production) or moved into other PRs:#1254 #1388.