Automatic http failover base on Srv Dns #442

RichardChen820 · 2023-08-01T07:19:49Z

Automatic http failover base on Srv Dns

For supporting automatic Http failover, there are SRV DNS record set being added for each replica by RP.

Each replica has _origin._tcp.* SRV record been created to indicate the origin store of the replica
_origin._tcp. contoso-replica1.azconfig.io --> SRV 1 100 443 contoso.azconfig.io
Each replica has _alt*._tcp.* SRV record created in _alt{i}._tcp record set for querying auto failover hosts.
_alt0._tcp. contoso.azconfig.io --> SRV 1 100 443 contoso-replica1.azconfig.io
_alt0._tcp. contoso.azconfig.io --> SRV 1 100 443 contoso-replica1.azconfig.io

"i" is a self-increment index since each record set only supports to accommodate up to 20 records, if there're 30 replicas, it will dispersed into two _alt* record sets.

Design:

auto failover should be enabled by default, CX can explicitly disable it.
auto failover happens when all clients that passed by CX failed because of any failoverable issue.
If multiple auto failover client are detected, .net provider pick up one from them randomly.

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Constants/LoggingConstants.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/DnsClient/DnsProcessor.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationOptions.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConnectionStringUtils.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

avanigupta · 2023-08-30T01:31:42Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+            }
+            else
+            {
+                throw new ArgumentNullException(nameof(_endpoints));


I dont think we should throw for ArgumentNullException for _endpoints here. Null/empty _connectionStrings could also be the cause for ArgumentNullException here.

How about throw new ArgumentNullException(nameof(_clients));

It looks more like an InvalidOperationException if both connectrings and endpoints are not provided.

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/DnsClient/SrvRecord.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/DnsClient/DnsProcessor.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

avanigupta · 2023-09-11T03:31:16Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+        // Only remove the client if it is not in the user passed connection string or endpoints, as well as not in returned SRV records.
+        private bool IsEligibleToRemove(IEnumerable<string> srvEndpointHosts, ConfigurationClientWrapper client)
+        {
+            if (_connectionStrings != null && _connectionStrings.Any(c => GetHostFromConnectionString(c).Equals(client.Endpoint.Host, StringComparison.OrdinalIgnoreCase)))


This would parse the endpoint from each connection string everytime we invoke IsEligibleToRemove.
How about storing the list of endpoints instead of connection strings? Looks like we only use the first connection string anyway. Maybe we can store that one connection string separately and extract endpoint from all other connection strings before storing in _endpoint list.
That also makes it simpler to check isUsingConnectionString. We can avoid the check for _connectionStrings.Any() and just check if the single connection string is set.

I have updated this, could you please tale a look again?

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationOptions.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Extensions/EnumerableExtensions.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs

avanigupta · 2023-09-21T03:19:55Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+                endpoint = _endpoints.First();
+            }
+
+            var lookup = new SrvLookupClient(logger);


Is it necessary to create a new SrvLookupClient every time we get auto failover clients? Can we create once and reuse it?

Good point, this is also what I want to optimize, the udpClient and tcpClient in it should be reusable. Besides this, I use cache in the SrvLookupClient, if new SrvLookupClient every time, using cache is meanless.

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs

avanigupta · 2023-09-21T03:51:29Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs

+
+                return resultRecords;
+            }
+            catch (Exception e)


We shouldn't catch base exception in any case. Only known exceptions should be caught and handled.
Also, InternalQueryAsync is the only method that can throw, so it would be better to wrap only that line of code in its own try/catch block so that its easy to understand where the exception is coming from.

We don't need to catch any exceptions here. Dns lookup error is nothing different from other errors, just throw is fine

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs

...ration.AzureAppConfiguration/Microsoft.Extensions.Configuration.AzureAppConfiguration.csproj

jimmyca15 · 2023-09-22T16:28:38Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationOptions.cs

+        /// <summary>
+        /// Flag to indicate whether discover the replicas automatically.
+        /// </summary>
+        public bool AutoDiscoverReplica { get; set; } = true;


Suggested change

public bool AutoDiscoverReplica { get; set; } = true;

public bool EnableReplicaDiscovery { get; set; } = true;

See this comment

My suggestion should be ReplicaDiscoveryEnabled in order to avoid a verb as a property name.

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationProvider.cs

jimmyca15 · 2023-09-22T20:09:55Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+            }
+            else
+            {
+                throw new ArgumentNullException(nameof(_clients));


_clients is not an argument to this constructor.

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

jimmyca15 · 2023-09-22T20:19:33Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+            IEnumerable<SrvRecord> results = await _srvLookupClient.Value.QueryAsync(endpoint.DnsSafeHost, cancellationToken).ConfigureAwait(false);
+
+            // shuffle the results to ensure hosts can be picked randomly.
+            IEnumerable<string> srvTargetHosts = results.Select(r => $"{r.Target.Value.Trim('.')}").Shuffle().ToList();


Do you mind shuffling on the non-transformed enumerable.

What's that trim doing, just removing a preceding '.' ? If so, may be clearer to be explicit about that.

@RichardChen820

Why was this resolved? Did other comments get resolved without being considered?

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs

NOTICE

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationProvider.cs

jimmyca15 · 2023-12-07T18:07:25Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+            {
+                await RefreshFallbackClients(cts.Token).ConfigureAwait(false);
+            }
+            catch (OperationCanceledException e)


Shouldn't log a warning on cancellation due to dispose.

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Extensions/TaskExtensions.cs

drago-draganov · 2023-12-08T01:15:33Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs

+                results = results.Concat(records);
+
+                // If we get no record from _alt{i} SRV, we have reached the end of _alt* list
+                if (records.Count() == 0)


We can concact empty. The check can be above to avoid it.

drago-draganov · 2023-12-08T01:25:29Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+                _ = DiscoverFallbackClients();
+            }
+
+            List<ConfigurationClient> clients = new List<ConfigurationClient>(_clients.Where(c => c.BackoffEndTime <= now).Select(c => c.Client));


To ensure thread safety, the _dynamicClients could be updated asynchronously

Actually, we don't have to worry about this, updated.

drago-draganov · 2023-12-08T01:25:59Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs

+                _ = DiscoverFallbackClients();
+            }
+
+            List<ConfigurationClient> clients = new List<ConfigurationClient>(_clients.Select(c => c.Client));


that creates copy

drago-draganov · 2023-12-08T20:01:49Z

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/IConfigurationClientManager.cs


 namespace Microsoft.Extensions.Configuration.AzureAppConfiguration
 {
    internal interface IConfigurationClientManager
    {
-        IEnumerable<ConfigurationClient> GetAvailableClients(DateTimeOffset time);
+        IEnumerable<ConfigurationClient> GetAvailableClients();

        IEnumerable<ConfigurationClient> GetAllClients();


These two are odd. Separate method for a subset of data is not great for an interface.
In this case we can always return all and have a property in ConfigurationClient that the caller can filter on instead.

The ConfigurationClient type is not defined in this project, it's from .net SDK, we can't add a property to it, we can only extend it or wrap it.

Then it seems would be a little bit overkill since the callers only care about the clients, they don't have to worry about how the clients are provided.

From my perspective , changing this seems doesn't make it much better.
https://github.com/Azure/AppConfiguration-DotnetProvider/pull/502/files

What's the reason we need two different subsets of ConfigurationClients (Available vs All)?

GetAllClients is a new method since 7.0.0 for time-based retry on start-up. When the app starts, there is no longer a fixed number of retries for each client, but as long as the time allows (default is 100s), we will continue to try all clients until a certain client can work or the time runs out. Therefore, it is necessary to obtain all Clients. Of course, we will not keep trying all Clients without limit. Whenever a round of attempts is made and all Clients are invalid, it will back off for a certain period of time and then start trying the next round.

GetAvailableClients is for getting the available clients for refreshing scenario, each client has unique backoff time which is determined by its failure attempts. Hence, we would always get the clients whose back off time is due.

I think we should have uniform strategy to use client instances in all cases (startup and refresh). Beside on startup, backoff isn't specified (no one has failed yet). I would assume that at startup GetAvailableClients always return all clients. Am I missing something?

cc @jimmyca15

Sounds like we should move the backoff tracking out of ConfigurationClientManager and use ConfigurationClientManager just as factory.
This can be done in a separate PR.

Good idea, I would merge this one first, go out in 7.1.0-preview first.

* Automatic http failover base on Srv Dns * update failover condition * Logging messages for auto failover * Not use generic method * Remove useless constructor * Failover to TCP when response is truncated * Error handling of TCP failover * Resolve comments and add thrid party notice * Fix the tests * Fix the tess * Fix the test * Update to use DnsClient.NET * Remove unecessary code * Resolve conflict * Stop once it has been auto failovered * Update GetAvaliableClient method * Resolve comments * Resolve comments * Internalize getting dynamic client in ConfigurationClientManager * Fix format * Fix format * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Not await RefreshFallbackClients when startup * Revert non-await RefreshFallbackClients * Validate domain of Srv endpoints (#489) * Validate domain of Srv endpoints * Resolve comments * Resolve comments * Create object in using * Add the origin host to the srv lookup result * Validate endpoint by domain name (#499) * Validate endpoint by domain name * Resolve comments * Remove *-test * Fix formatResolve conflict * Use await using for IAsyncDisposible * Replace IAsyncEumerable with ValueTask<IEnumerable> * Resolve comments * Remove cancallation token * Resolve comments

* Automatic http failover base on Srv Dns (#442) * Automatic http failover base on Srv Dns * update failover condition * Logging messages for auto failover * Not use generic method * Remove useless constructor * Failover to TCP when response is truncated * Error handling of TCP failover * Resolve comments and add thrid party notice * Fix the tests * Fix the tess * Fix the test * Update to use DnsClient.NET * Remove unecessary code * Resolve conflict * Stop once it has been auto failovered * Update GetAvaliableClient method * Resolve comments * Resolve comments * Internalize getting dynamic client in ConfigurationClientManager * Fix format * Fix format * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Resolve comments * Not await RefreshFallbackClients when startup * Revert non-await RefreshFallbackClients * Validate domain of Srv endpoints (#489) * Validate domain of Srv endpoints * Resolve comments * Resolve comments * Create object in using * Add the origin host to the srv lookup result * Validate endpoint by domain name (#499) * Validate endpoint by domain name * Resolve comments * Remove *-test * Fix formatResolve conflict * Use await using for IAsyncDisposible * Replace IAsyncEumerable with ValueTask<IEnumerable> * Resolve comments * Remove cancallation token * Resolve comments * Refactor and move backoff logic to AzureAppConfigurationProvider (#508) * move backoff logic to provider, can't access backoffendtime from manager for dynamic clients * add condition before calling getclients in refreshasync, add refreshclients method * fix logic to check if backoff ended for client * fix indents * check client backoff before refresh, fix tests in progress * remove tolist * comment changes, fix refresh tests * reduce minimum backoff when testing * fix tests, fix typo with failover logic * update comments in tests * resolve PR comments * resolve small comments * fix tests backoff duration * fix tests backoff duration * fix error from merging main branch in * add back fix for attempts backoff calculation --------- Co-authored-by: Richard chen <99175581+RichardChen820@users.noreply.github.com>

RichardChen820 requested a review from amerjusupovic August 1, 2023 07:19

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from 9087339 to fff39fb Compare August 1, 2023 07:26

amerjusupovic requested a review from avanigupta August 4, 2023 20:14

amerjusupovic reviewed Aug 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Constants/LoggingConstants.cs Outdated Show resolved Hide resolved

amerjusupovic reviewed Aug 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Constants/LoggingConstants.cs Outdated Show resolved Hide resolved

amerjusupovic reviewed Aug 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

amerjusupovic reviewed Aug 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

amerjusupovic reviewed Aug 7, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/DnsClient/DnsProcessor.cs Outdated Show resolved Hide resolved

avanigupta reviewed Aug 30, 2023

View reviewed changes

avanigupta reviewed Sep 11, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

avanigupta reviewed Sep 11, 2023

View reviewed changes

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from ee5ce94 to 68ed6e9 Compare September 15, 2023 13:25

RichardChen820 requested review from jimmyca15 and drago-draganov September 18, 2023 01:46

avanigupta reviewed Sep 19, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationOptions.cs Outdated Show resolved Hide resolved

avanigupta reviewed Sep 21, 2023

View reviewed changes

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

...ration.AzureAppConfiguration/Microsoft.Extensions.Configuration.AzureAppConfiguration.csproj Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationProvider.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Sep 22, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/SrvLookupClient.cs Outdated Show resolved Hide resolved

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch 2 times, most recently from f92fa5a to b6a344d Compare December 4, 2023 09:25

Replace IAsyncEumerable with ValueTask<IEnumerable>

bc94a95

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from b6a344d to bc94a95 Compare December 4, 2023 09:26

jimmyca15 reviewed Dec 4, 2023

View reviewed changes

NOTICE Show resolved Hide resolved

jimmyca15 reviewed Dec 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Dec 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Dec 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Dec 4, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

Resolve comments

cec2f6a

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from e209865 to cec2f6a Compare December 6, 2023 15:12

jimmyca15 reviewed Dec 6, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/ConfigurationClientManager.cs Outdated Show resolved Hide resolved

jimmyca15 reviewed Dec 6, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/AzureAppConfigurationProvider.cs Outdated Show resolved Hide resolved

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from 0f8a5d2 to 684a374 Compare December 7, 2023 09:32

Remove cancallation token

fdaa51f

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from 684a374 to fdaa51f Compare December 7, 2023 09:37

jimmyca15 reviewed Dec 7, 2023

View reviewed changes

src/Microsoft.Extensions.Configuration.AzureAppConfiguration/Extensions/TaskExtensions.cs Outdated Show resolved Hide resolved

jimmyca15 approved these changes Dec 7, 2023

View reviewed changes

drago-draganov reviewed Dec 8, 2023

View reviewed changes

Resolve comments

312ad19

RichardChen820 force-pushed the user/junbchen/srvDnsClient branch from a141380 to 312ad19 Compare December 8, 2023 09:21

drago-draganov reviewed Dec 8, 2023

View reviewed changes

drago-draganov approved these changes Dec 12, 2023

View reviewed changes

RichardChen820 merged commit 06f237f into preview Dec 12, 2023
2 checks passed

avanigupta mentioned this pull request Oct 3, 2024

Add release notes for .NET Provider 8.0.0 stable release Azure/AppConfiguration#975

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automatic http failover base on Srv Dns #442

Automatic http failover base on Srv Dns #442

RichardChen820 commented Aug 1, 2023 •

edited

Loading

avanigupta Aug 30, 2023

RichardChen820 Aug 30, 2023

avanigupta Sep 11, 2023

avanigupta Sep 11, 2023

RichardChen820 Sep 15, 2023

avanigupta Sep 21, 2023

RichardChen1122 Sep 22, 2023 •

edited

Loading

avanigupta Sep 21, 2023

RichardChen1122 Sep 22, 2023

jimmyca15 Sep 22, 2023

jimmyca15 Oct 2, 2023

jimmyca15 Sep 22, 2023

jimmyca15 Sep 22, 2023

jimmyca15 Sep 22, 2023

jimmyca15 Oct 10, 2023

jimmyca15 Dec 7, 2023

RichardChen820 Dec 8, 2023

drago-draganov Dec 8, 2023

drago-draganov Dec 8, 2023

RichardChen820 Dec 8, 2023 •

edited

Loading

RichardChen820 Dec 8, 2023

drago-draganov Dec 8, 2023

drago-draganov Dec 8, 2023

RichardChen820 Dec 10, 2023

RichardChen820 Dec 10, 2023

drago-draganov Dec 10, 2023

RichardChen820 Dec 11, 2023

drago-draganov Dec 11, 2023

drago-draganov Dec 12, 2023

RichardChen820 Dec 12, 2023

	public bool AutoDiscoverReplica { get; set; } = true;
	public bool EnableReplicaDiscovery { get; set; } = true;

Automatic http failover base on Srv Dns #442

Automatic http failover base on Srv Dns #442

Conversation

RichardChen820 commented Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardChen1122 Sep 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardChen820 Dec 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardChen820 commented Aug 1, 2023 •

edited

Loading

RichardChen1122 Sep 22, 2023 •

edited

Loading

RichardChen820 Dec 8, 2023 •

edited

Loading