[Feature branch] Add Neural Stats API #1208

q-andy · 2025-03-04T22:25:04Z

Description

Implementing Neural Stats API framework design proposed in #1196. This initial PR sets up the foundation for the framework to track event and state stats throughout the neural search plugin and exposed their values via API.

Event-based stats
- Event stats are recorded in code at a node level (processor executions, documents ingested, etc)
- When an API call is made, all node-level maps are fetched via transport action and returned in the response.
State stats
- State stats are defined by helper functions that populate state stat values
- When an API call is made, the functions are invoked and the information is added to the response on demand

See RFC for more details.

Initial implementation includes 3 stats:

Text embedding processor executions
Text embedding processors in pipelines
Cluster version

Related Issues

Resolves #1196 #1104
Related: #1146

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

Example requests

GET /_plugins/_neural/stats
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"cluster_version": "3.0.0",
	"processors": {
		"ingest": {
			"text_embedding_processors_in_pipelines": 0
		}
	},
	"all_nodes": {
		"processors": {
			"ingest": {
				"text_embedding_executions": 0
			}
		}
	},
	"nodes": {
		"r-hVPa7-Ra6FBbdpfVrHjg": {
			"processors": {
				"ingest": {
					"text_embedding_executions": 0
				}
			}
		}
	}
}

GET /_plugins/_neural/stats?include_metadata=true
{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"cluster_version": {
		"value": "3.0.0",
		"stat_type": "settable"
	},
	"processors": {
		"ingest": {
			"text_embedding_processors_in_pipelines": {
				"value": 0,
				"stat_type": "countable"
			}
		}
	},
	"all_nodes": {
		"processors": {
			"ingest": {
				"text_embedding_executions": {
					"value": 0,
					"stat_type": "timestamped_counter",
					"trailing_interval_value": 0,
					"minutes_since_last_event": 29018783
				}
			}
		}
	},
	"nodes": {
		"r-hVPa7-Ra6FBbdpfVrHjg": {
			"processors": {
				"ingest": {
					"text_embedding_executions": {
						"value": 0,
						"stat_type": "timestamped_counter",
						"trailing_interval_value": 0,
						"minutes_since_last_event": 29018783
					}
				}
			}
		}
	}
}

GET _plugins/_neural/stats/text_embedding_executions?include_metadata=true&flat_keys=true

{
	"_nodes": {
		"total": 1,
		"successful": 1,
		"failed": 0
	},
	"cluster_name": "integTest",
	"all_nodes.processors.ingest.text_embedding_executions": {
		"value": 0,
		"stat_type": "timestamped_counter",
		"trailing_interval_value": 0,
		"minutes_since_last_event": 29018784
	},
	"nodes": {
		"r-hVPa7-Ra6FBbdpfVrHjg": {
			"processors.ingest.text_embedding_executions": {
				"value": 0,
				"stat_type": "timestamped_counter",
				"trailing_interval_value": 0,
				"minutes_since_last_event": 29018784
			}
		}
	}
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Andy Qin <qinandy@amazon.com>

heemin32 · 2025-03-05T00:55:24Z

Shouldn't it be a full flat? Similar to cluster setting. https://opensearch.org/docs/latest/api-reference/cluster-api/cluster-settings/#path-parameters

GET _plugins/_neural/stats/text_embedding_executions?include_metadata=true&flat_keys=true

{
	"_nodes.total": 1,
        "_nodes.successful": 1,
        "_nodes.failed": 0
        "all_nodes.processors.ingest.text_embedding_executions.value": 0,
        ...
        "nodes.r-hVPa7-Ra6FBbdpfVrHjg.processors.ingest.text_embedding_executions.minutes_since_last_event": 29018784
}

q-andy · 2025-03-05T17:55:42Z

It can go either way, but I think preserving some basic structure is helpful for navigating the high level categories. For example, cluster setting still divides the flat keys between persistent, transient, and defaults fields.

The way I see it, flatten is an option to simplify navigating the nested fields, not a hard rule to flatten everything. Keeping some info grouped together still makes it easier to parse and navigate than if it was completely flat, e.g. stat metadata is easier to recursively parse since all the info is "close by".

Example cluster settings call with flatten enabled:

GET _cluster/settings?include_defaults=true&flat_settings=true

{
	"persistent": {
		"plugins.ml_commons.model_access_control_enabled": "true",
		"plugins.ml_commons.native_memory_threshold": "99",
		"plugins.ml_commons.only_run_on_ml_node": "false",
		"plugins.neural_search.stats_enabled": "true"
	},
	"transient": {},
	"defaults": {
		"_plugin_neural_search_hybrid_query_executor.queue_size": "1000",
		"_plugin_neural_search_hybrid_query_executor.size": "20",
		"action.auto_create_index": "true",
         . . .
       }
}

Compare to non-flat

GET _cluster/settings?include_defaults=true
{
	"persistent": {
		"plugins": {
			"neural_search": {
				"stats_enabled": "true"
			},
			"ml_commons": {
				"only_run_on_ml_node": "false",
				"model_access_control_enabled": "true",
				"native_memory_threshold": "99"
			}
		}
	},
	"transient": {},
	"defaults": {
		"task_resource_tracking": {
			"enabled": "true"
		},
	. . .
        },
        . . . 
}

heemin32 · 2025-03-06T00:21:38Z

That make sense. We only need to flatten the key for easier search. Then, shouldn't it be like this?

"all_nodes" {
      "processors.ingest.text_embedding_executions": {
		"value": 0,
		"stat_type": "timestamped_counter",
		"trailing_interval_value": 0,
		"minutes_since_last_event": 29018784
	},
}

Also wondering if this could be better.

"all_nodes" {
      "processors.ingest.text_embedding_executions.value": 0,
      "processors.ingest.text_embedding_executions.stat_type": "timestamped_counter",
      "processors.ingest.text_embedding_executions.trailing_interval_value": 0,
      "processors.ingest.text_embedding_executions.minutes_since_last_event": 29018784
}

heemin32 · 2025-03-06T00:24:33Z

There are two components: Stats API and Metric Management. For the Metric Management code, it might be better to have a separate package to make it more portable for use in other plugins.

q-andy · 2025-03-06T17:54:25Z

Then, shouldn't it be like this? (all_nodes unflattened)

Yes, that makes more sense, good catch

Also wondering if this could be better. (metadata flattened)

I considered that as well, what I'm thinking is the "stat metadata" object replaces the raw value so the flat key name is the same whether or not metadata is included or not. Like the format will always be at "path.to.stat: <data>", sometimes the data is the raw value, sometimes its a the metadata, but the stat key name is same.

It also keeps things organized when you parse recursively. For example. if a cluster manager is periodically calling the stats API and parsing the response without hardcoding stat names, if we keep metadata non-flat, for every stat you can foreach on the categories with flat_keys and parse each stat and all its metadata one by one in a single pass. Whereas if it was completely flat metadata, running a foreach would need additional logic to split, sort, and group the metadata keys by the stat name again. Also you would probably never be interested in the value of path.to.stat.stat_type independently of the other stat metadata.

Just some thoughts. I don't see the usecase right now but I could still include an option to flatten the stat metadata as well, it's a pretty small change.

q-andy · 2025-03-06T17:59:31Z

There are two components: Stats API and Metric Management. For the Metric Management code, it might be better to have a separate package to make it more portable for use in other plugins.

Sure, although it'd take a little more intricate design to decouple some stuff like state stat calculation from plugin-specific details and refactoring to make it all extensible. Maybe that'd be a separate RFC?

vibrantvarun · 2025-03-06T23:56:45Z

Please add the change in changelog.md

Signed-off-by: Andy Qin <qinandy@amazon.com>

vibrantvarun

Completed 1st round of review.

vibrantvarun · 2025-03-06T23:57:10Z

src/main/java/org/opensearch/neuralsearch/plugin/NeuralSearch.java

+
+        PipelineServiceUtil.instance().initialize(clusterService);
+        NeuralSearchSettingsAccessor.instance().initialize(clusterService, environment.settings());
+


Remove extra line

vibrantvarun · 2025-03-06T23:57:20Z

src/main/java/org/opensearch/neuralsearch/plugin/NeuralSearch.java

@@ -105,7 +123,11 @@ public Collection<Object> createComponents(
        NeuralSparseQueryBuilder.initialize(clientAccessor);
        HybridQueryExecutor.initialize(threadPool);
        normalizationProcessorWorkflow = new NormalizationProcessorWorkflow(new ScoreNormalizer(), new ScoreCombiner());
-        return List.of(clientAccessor);
+


Remove extra line

vibrantvarun · 2025-03-06T23:59:03Z

src/main/java/org/opensearch/neuralsearch/processor/util/RestActionUtils.java

+
+import java.util.Optional;
+
+public class RestActionUtils {


Add javadoc of class.

Take reference from ProcessorUtils

vibrantvarun · 2025-03-07T00:00:10Z

src/main/java/org/opensearch/neuralsearch/processor/util/RestActionUtils.java

+import java.util.Optional;
+
+public class RestActionUtils {
+    public static Optional<String[]> splitCommaSeparatedParam(RestRequest request, String paramName) {


Add javadoc

vibrantvarun · 2025-03-07T00:00:21Z

src/main/java/org/opensearch/neuralsearch/processor/util/RestActionUtils.java

+        return Optional.ofNullable(request.param(paramName)).map(s -> s.split(","));
+    }
+
+    public static Optional<String> getStringParam(RestRequest request, String paramName) {


Add javadoc

vibrantvarun · 2025-03-07T00:38:58Z

src/main/java/org/opensearch/neuralsearch/settings/NeuralSearchSettingsAccessor.java

+import org.opensearch.common.settings.Settings;
+import org.opensearch.neuralsearch.stats.events.EventStatsManager;
+
+public class NeuralSearchSettingsAccessor {


Add javadoc

vibrantvarun · 2025-03-07T00:39:48Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+        .map(String::toLowerCase)
+        .collect(Collectors.toSet());
+
+    private NeuralSearchSettingsAccessor settingsAccessor;


Move this declaration to top

vibrantvarun · 2025-03-07T00:40:26Z

src/main/java/org/opensearch/neuralsearch/settings/NeuralSearchSettingsAccessor.java

+    private boolean initialized;
+
+    @Getter
+    private volatile Boolean isStatsEnabled;


Suggested change

private volatile Boolean isStatsEnabled;

private volatile boolean isStatsEnabled;

vibrantvarun · 2025-03-07T00:41:47Z

src/main/java/org/opensearch/neuralsearch/stats/NeuralStatsInput.java

+ * Holds user requester input parameters for retrieving neural stats via rest handler API.
+ * It allows filtering statistics by node IDs, event statistic types, and state statistic types.


Suggested change

* Holds user requester input parameters for retrieving neural stats via rest handler API.

* It allows filtering statistics by node IDs, event statistic types, and state statistic types.

* Entity class that holds input parameters for retrieving neural stats.

* It is responsible for filtering statistics by node IDs, event statistic types, and state statistic types.

vibrantvarun · 2025-03-07T00:45:02Z

src/main/java/org/opensearch/neuralsearch/stats/events/EventStatsManager.java

+import java.util.Map;
+
+/**
+ * Manager class for event counter stats, used to track monotonically increasing counts plus some possible metadata


Suggested change

* Manager class for event counter stats, used to track monotonically increasing counts plus some possible metadata

* Manager class for event counter stats, used to track monotonically increasing counts plus some possible metadata

Please rewrite to make it more reader friendly. It should provide high level information of this class.

heemin32

Overall, I have two comments:

The most frequently called method (currently the increment method) should always operate in O(1) time complexity, regardless of the number of metrics defined in the future. Hash.get() does not always guarantee O(1) performance when a large number of items are stored.
We should avoid using the public static instance() pattern. This approach should only be considered as a last resort when no other alternatives exist, as it can make unit testing more difficult.

heemin32 · 2025-03-06T00:36:58Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+        }
+
+        // If no stats are specified, add all stats to retrieve all by default
+        if (retrieveAllStats) {


Even if stats is specified, if there is not matching stats, do we want to return all stats instead of empty?

heemin32 · 2025-03-06T00:45:00Z

src/main/java/org/opensearch/neuralsearch/settings/NeuralSearchSettingsAccessor.java

+     * Return instance of the cluster context, must be initialized first for proper usage
+     * @return instance of cluster context
+     */
+    public static synchronized NeuralSearchSettingsAccessor instance() {


I understand that this is a commonly used pattern, but can we avoid this approach as much as possible? Implementing a singleton this way can create long-term testing challenges. If one test modifies the settings, it affects all other tests.

It shouldn't be globally accessible through static method. Instead, it was created once and passed around by injection through constructor.

Sure, I can switch most of the singletons to injection. PipelineServiceUtil, NeuralSearchSettingsAccessor, and StateStatsManager should be trivial.

For the EventStatsManager, though, incrementing stats via static method call seems cleaner otherwise we'd have to add the instance to every single constructor of every object that wants to call it which is pretty invasive. I see OS core has an injector interface and uses @Inject for things like Transport Action, but it doesn't seem utilized in many other places, maybe for event stats its easier to keep the singleton instance

Yes. For EventStatsManager, it might not be suitable to inject.

heemin32 · 2025-03-06T01:09:30Z

src/main/java/org/opensearch/neuralsearch/stats/events/TimestampedEventStat.java

+
+        // Synchronized thread call shouldn't have performance hit, this will be called infrequently
+        synchronized (this) {
+            buckets = newBuckets;


I think resetting the value might be a better approach than creating a new object since this object has been alive for a long time. As a result, it may have been moved to the old generation, making it less likely to be collected during young generation cleanup by JVM.

bucket.timestamp.set(0); bucket.count.reset();

heemin32 · 2025-03-07T17:02:55Z

src/main/java/org/opensearch/neuralsearch/stats/NeuralStatsInput.java

+     * @throws IOException if there's an error reading from the stream
+     */
+    public NeuralStatsInput(StreamInput input) throws IOException {
+        nodeIds = input.readBoolean() ? new HashSet<>(input.readStringList()) : new HashSet<>();


readOptionalStringList(); might be better in terms of readability.
Wonder if we can make nodeIds as List so that we can avoid this list to set conversion.

heemin32 · 2025-03-07T17:03:34Z

src/main/java/org/opensearch/neuralsearch/stats/NeuralStatsInput.java

+     * @param set the EnumSet to write
+     * @throws IOException If there's an error writing to the stream
+     */
+    private void writeOptionalEnumSet(StreamOutput out, EnumSet<?> set) throws IOException {


Might be good to have corresponding readOptionalEnumSet()
Better if we can add this method inside core :)

heemin32 · 2025-03-07T17:11:48Z

src/main/java/org/opensearch/neuralsearch/stats/events/EventStatsManager.java

+     */
+    public void inc(EventStatName eventStatName) {
+        boolean enabled = settingsAccessor.getIsStatsEnabled();
+        if (enabled && stats.containsKey(eventStatName)) {


This could become an issue when there are many eventStatName values. While HashMap.get is generally constant time, its performance may degrade as the number of items grows. Given that this method is called frequently, we may need to explore an alternative approach to access the instance directly for better efficiency.

Sure, I can have the EventStat be a member of the EventStatName enum and be instantiated in the enum constructor. That way we can access it directly without using a map at all.

If we have performance issues with hash map get, though, we do have other options, my understanding is the O(n) worst case happens when there are key object hash code collisions leading to putting objects in the same bucket and requiring a search on the bucket. There are ways we can mitigate this like implementing our hash code methods to ensure uniqueness, and Java has its own optimizations to address this like storing collisions in a tree rather than linked list.

I'll update again once I finish running performance benchmarks

Yes. Storing it on a tree is also not O(1). Also, even if the hashcode are distinct, I don't think they will all have their own bucket. The number of bucket can be smaller then the number of hashcode.

heemin32 · 2025-03-07T17:50:57Z

Then, shouldn't it be like this? (all_nodes unflattened)

Yes, that makes more sense, good catch

Also wondering if this could be better. (metadata flattened)

I considered that as well, what I'm thinking is the "stat metadata" object replaces the raw value so the flat key name is the same whether or not metadata is included or not. Like the format will always be at "path.to.stat: <data>", sometimes the data is the raw value, sometimes its a the metadata, but the stat key name is same.

It also keeps things organized when you parse recursively. For example. if a cluster manager is periodically calling the stats API and parsing the response without hardcoding stat names, if we keep metadata non-flat, for every stat you can foreach on the categories with flat_keys and parse each stat and all its metadata one by one in a single pass. Whereas if it was completely flat metadata, running a foreach would need additional logic to split, sort, and group the metadata keys by the stat name again. Also you would probably never be interested in the value of path.to.stat.stat_type independently of the other stat metadata.

Just some thoughts. I don't see the usecase right now but I could still include an option to flatten the stat metadata as well, it's a pretty small change.

Yes. One use case that I can think of is that, when operator want to grep stats containing processor, this whole flat option can be handy.

_plugin/neural/stats?flat=true | grep processor

martin-gaievski · 2025-03-08T18:29:27Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+
+    @Override
+    public List<Route> routes() {
+        return ImmutableList.of(


this list can be a class constant, no need to construct it on every call as it's immutable object

martin-gaievski · 2025-03-08T18:35:01Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+            splitCommaSeparatedParam(request, "nodeId");
+            splitCommaSeparatedParam(request, "stat");
+            request.paramAsBoolean(FLATTEN_PARAM, false);
+            request.paramAsBoolean(INCLUDE_METADATA_PARAM, false);


why we need to process parameters and have those 4 lines if we are a going to return 403 or whatever response in case user did opted in?

martin-gaievski · 2025-03-08T18:42:05Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+     */
+    private NeuralStatsRequest getRequest(RestRequest request) {
+        // parse the nodes the user wants to query
+        String[] nodeIdsArr = null;


nodeIdsArr array can be null, add comment about how the null value will be handled: will that trigger an exception or we treat it as "all nodes" etc.

martin-gaievski · 2025-03-08T18:43:49Z

src/main/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandler.java

+                } else if (STATE_STAT_NAMES.contains(stat)) {
+                    retrieveAllStats = false;
+                    neuralStatsInput.getStateStatNames().add(StateStatName.from(stat));
+                }


add a log statement in case of neither if nor else are not true

martin-gaievski · 2025-03-08T18:46:01Z

src/main/java/org/opensearch/neuralsearch/settings/NeuralSearchSettingsAccessor.java

+    }
+
+    public void initialize(ClusterService clusterService, Settings settings) {
+        if (initialized) return;


please add curly braces for better readability

martin-gaievski · 2025-03-08T18:57:12Z

src/test/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandlerIT.java

+        updateClusterSettings();
+
+        // Only enable stats for this IT to prevent collisions
+        updateClusterSettings("plugins.neural_search.stats_enabled", true);


you should be able to get this name from that constant NEURAL_STATS_ENABLED

martin-gaievski · 2025-03-08T19:02:02Z

src/test/java/org/opensearch/neuralsearch/rest/RestNeuralStatsHandlerIT.java

+import java.util.Map;
+
+@Log4j2
+public class RestNeuralStatsHandlerIT extends BaseNeuralSearchIT {


we also need a BWC test. It's fine if there is no 2.x version that supports stats, it's needed for future 3.x releases

q-andy added 19 commits February 26, 2025 09:32

Add neural stats

19480ae

Signed-off-by: Andy Qin <qinandy@amazon.com>

Refactor event/state stats

a1f24f0

Signed-off-by: Andy Qin <qinandy@amazon.com>

Extend state stats and refactor stat combination

064465b

Signed-off-by: Andy Qin <qinandy@amazon.com>

Fix rest integ test

e4abeca

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add stats and nodes filtering

9274a71

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add flatten and include metadata query parameter

e1cb163

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add EventStatData to snapshot EventStat info

282db07

Signed-off-by: Andy Qin <qinandy@amazon.com>

Refactor stat filtering

7d08433

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add stat aggregations, metadata

df81f11

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add trailing interval and minutes since last event metadata

4677199

Signed-off-by: Andy Qin <qinandy@amazon.com>

Reorganize response processing logic

8cc6698

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add enable stats setting and reset

24760a5

Signed-off-by: Andy Qin <qinandy@amazon.com>

Adding documentation for state and event stats

c15bcff

Signed-off-by: Andy Qin <qinandy@amazon.com>

More documentation

a8c3d0a

Signed-off-by: Andy Qin <qinandy@amazon.com>

Clean up unused state stats manager methods

9425021

Signed-off-by: Andy Qin <qinandy@amazon.com>

Add unit tests and update ITs

7ba9dcb

Signed-off-by: Andy Qin <qinandy@amazon.com>

Merge branch 'opensearch-project:main' into neural-stats

fe8a616

Add transport tests, refactor toXContent tests

56c2d19

Signed-off-by: Andy Qin <qinandy@amazon.com>

Clean up files

d540f28

Signed-off-by: Andy Qin <qinandy@amazon.com>

q-andy requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, martin-gaievski, sean-zheng-amazon, model-collapse and zane-neo as code owners March 4, 2025 22:25

q-andy requested review from vibrantvarun, zhichao-aws, yuye-aws and minalsha as code owners March 4, 2025 22:25

github-actions bot added v3.0.0 v3.0.0 labels Mar 4, 2025

Merge branch 'opensearch-project:main' into neural-stats

3da5765

vibrantvarun changed the base branch from main to feature/neural-stats-api March 6, 2025 23:52

q-andy changed the title ~~Add Neural Stats API~~ [Feature branch] Add Neural Stats API Mar 6, 2025

Update changelog

64032f3

Signed-off-by: Andy Qin <qinandy@amazon.com>

vibrantvarun reviewed Mar 7, 2025

View reviewed changes

heemin32 reviewed Mar 7, 2025

View reviewed changes

martin-gaievski reviewed Mar 8, 2025

View reviewed changes


		PipelineServiceUtil.instance().initialize(clusterService);
		NeuralSearchSettingsAccessor.instance().initialize(clusterService, environment.settings());

	private volatile Boolean isStatsEnabled;
	private volatile boolean isStatsEnabled;

		* Holds user requester input parameters for retrieving neural stats via rest handler API.
		* It allows filtering statistics by node IDs, event statistic types, and state statistic types.

	* Manager class for event counter stats, used to track monotonically increasing counts plus some possible metadata
	* Manager class for event counter stats, used to track monotonically increasing counts plus some possible metadata

[Feature branch] Add Neural Stats API #1208

Are you sure you want to change the base?

[Feature branch] Add Neural Stats API #1208

Conversation

q-andy commented Mar 4, 2025 • edited Loading

Description

Related Issues

Check List

Example requests

heemin32 commented Mar 5, 2025 • edited Loading

q-andy commented Mar 5, 2025 • edited Loading

heemin32 commented Mar 6, 2025

heemin32 commented Mar 6, 2025

q-andy commented Mar 6, 2025

q-andy commented Mar 6, 2025

vibrantvarun commented Mar 6, 2025

vibrantvarun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heemin32 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

q-andy Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

heemin32 Mar 7, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heemin32 commented Mar 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

q-andy commented Mar 4, 2025 •

edited

Loading

heemin32 commented Mar 5, 2025 •

edited

Loading

q-andy commented Mar 5, 2025 •

edited

Loading

q-andy Mar 7, 2025 •

edited

Loading

heemin32 Mar 7, 2025 •

edited

Loading