[SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect #49107

HyukjinKwon · 2024-12-09T04:53:44Z

What changes were proposed in this pull request?

This PR proposes to add a new configuration called spark.api.mode (default classic) that allows existing Spark Classic applications to easily use Spark Connect.

Why are the changes needed?

In order users to easily try Spark Connect with their existing Spark Classic applications.

Does this PR introduce any user-facing change?

It adds a new configuration spark.api.mode without changing the default behaviour.

How was this patch tested?

For PySpark applications, added unittests for Spark Submissions in Yarn and Kubernates, and manual tests.
For Scala applications, it is difficult to add a test because SBT picks up the complied jars into the classpathes automatically (whereas we can easily control it for production environment).

For this case, I manually tested as below:

git clone https://github.com/HyukjinKwon/spark-connect-example
cd spark-connect-example
build/sbt package
cd ..
git clone https://github.com/apache/spark.git
cd spark
build/sbt package
# sbin/start-connect-server.sh
bin/spark-submit --name "testApp" --master "local[*]" --conf spark.api.mode=connect --class com.hyukjinkwon.SparkConnectExample ../spark-connect-example/target/scala-2.13/spark-connect-example_2.13-0.0.1.jar

Was this patch authored or co-authored using generative AI tooling?

No.

HyukjinKwon · 2024-12-10T01:25:30Z

sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkConnectClient.scala

@@ -620,8 +620,18 @@ object SparkConnectClient {
     * Configure the builder using the env SPARK_REMOTE environment variable.
     */
    def loadFromEnvironment(): Builder = {
+      lazy val isAPIModeConnect = Option(System.getProperty("spark.api.mode"))


This is needed because we directly execute ConnectRepl (which is same as Python or R shells).

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile

HyukjinKwon · 2025-01-03T01:14:12Z

docs/app-dev-spark-connect.md

+This is similar to using `--conf spark.api.mode=connect` with `--master ...`. However, note that
+`spark.remote` and `--remote` are limited to `local*` values, while `--conf spark.api.mode=connect`
+with `--master ...` supports additional cluster URLs, such as spark://, for broader compatibility with
+Spark Classic.


@hvanhovell this part is slightly different from what we discussed offline. My logic here is to restrict it first instead of allowing it: --remote and spark.remote will support only local* as it was.

cloud-fan · 2025-01-27T04:57:12Z

how is it going?

HyukjinKwon · 2025-01-27T07:18:53Z

Needs some review from @hvanhovell ..

HyukjinKwon · 2025-01-27T07:28:09Z

rebased ..

core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala

docs/app-dev-spark-connect.md

…park Connect ### What changes were proposed in this pull request? This PR proposes to add a new configuration called `spark.api.mode` (default `classic`) that allows existing Spark Classic applications to easily use Spark Connect. ### Why are the changes needed? In order users to easily try Spark Connect with their existing Spark Classic applications. ### Does this PR introduce _any_ user-facing change? It adds a new configuration `spark.api.mode` without changing the default behaviour. ### How was this patch tested? For PySpark applications, added unittests for Spark Submissions in Yarn and Kubernates, and manual tests. For Scala applications, it is difficult to add a test because SBT picks up the complied jars into the classpathes automatically (whereas we can easily control it for production environment). For this case, I manually tested as below: ```bash git clone https://github.com/HyukjinKwon/spark-connect-example cd spark-connect-example build/sbt package cd .. git clone https://github.com/apache/spark.git cd spark build/sbt package # sbin/start-connect-server.sh bin/spark-submit --name "testApp" --master "local[*]" --conf spark.api.mode=connect --class com.hyukjinkwon.SparkConnectExample ../spark-connect-example/target/scala-2.13/spark-connect-example_2.13-0.0.1.jar ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49107 from HyukjinKwon/api-mode-draft-3. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 1bc4d15) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala

core/src/main/scala/org/apache/spark/internal/config/package.scala

docs/app-dev-spark-connect.md

Kimahriman · 2025-02-05T22:25:55Z

Questions based on questions on the mailing list thread

Is this supposed to work with cluster deploy mode?
If so, how would this work on YARN? It looks like it will only use an ephemeral port when running tests, I think YARN would also need to use an ephemeral port?
And if so, how is the connect server being secured? If this is running in a secured but shared environment, the connect server may be accessible by anyone else using the cluster.

HyukjinKwon · 2025-02-06T00:18:49Z

Is this supposed to work with cluster deploy mode?

Yes. For --master ..., It will run a local Spark Connect server where Spark Driver is running.

If so, how would this work on YARN? It looks like it will only use an ephemeral port when running tests, I think YARN would also need to use an ephemeral port?

Yes, it will launch a Spark Connect server locally.

And if so, how is the connect server being secured? If this is running in a secured but shared environment, the connect server may be accessible by anyone else using the cluster.

This is the same as Py4J server. It does not enable the auth feature by default. We could have the implementation in Spark Connect to but I would prefer to run this separately.

HyukjinKwon · 2025-02-06T00:38:21Z

I am going to merge this if that sounds fine to you @Kimahriman

Kimahriman · 2025-02-06T00:41:55Z

If so, how would this work on YARN? It looks like it will only use an ephemeral port when running tests, I think YARN would also need to use an ephemeral port?

Yes, it will launch a Spark Connect server locally.

But if it uses the default 15002 port, it will hit issues when multiple drivers are running on the same node, unless it uses an ephemeral port.

This is the same as Py4J server. It does not enable the auth feature by default. We could have the implementation in Spark Connect to but I would prefer to run this separately.

Where is auth not enabled for Py4J? It looks like it always generates a secret that the client uses to connect to it

spark/core/src/main/scala/org/apache/spark/api/python/Py4JServer.scala

Line 32 in e2ef5a4

private[spark] val secret: String = Utils.createSecret(sparkConf)

HyukjinKwon · 2025-02-06T00:46:57Z

Oops, sorry you're correct. We enabled it in Spark specifically yes.

But if it uses the default 15002 port, it will hit issues when multiple drivers are running on the same node, unless it uses an ephemeral port.

Yes, we should also use ephemeral port. We're doing it when we running it locally in some cases, e.g., pyspark shell with --remote local (see DefaultChannelBuilder.default_port)

HyukjinKwon · 2025-02-06T00:48:57Z

I think we can address them all anyway because we already have Py4J that exactly works like Spark Connect server.

Kimahriman · 2025-02-06T00:49:15Z

Oops, sorry you're correct. We enabled it in Spark specifically yes.

WIthout a default mechanism to authenticate to the connect server, this seems like a massive security vulnerability then, right?

Yes, we should also use ephemeral port. We're doing it when we running it locally in some cases, e.g., pyspark shell with --remote local (see DefaultChannelBuilder.default_port)

Yeah saw that, should an ephemeral port just always be used for the auto launching of the local server?

HyukjinKwon · 2025-02-06T01:28:09Z

@Kimahriman mind I if I run it separately? We already have the logic of running it in any event.

Kimahriman · 2025-02-06T01:41:19Z

@Kimahriman mind I if I run it separately? We already have the logic of running it in any event.

Fine either way, brought this up on the mailing list to bring a little more awareness

HyukjinKwon · 2025-02-06T01:51:09Z

Yup, thanks for chiming in. Let me followup.

Merged to master and branch-4.0.

…park Connect ### What changes were proposed in this pull request? This PR proposes to add a new configuration called `spark.api.mode` (default `classic`) that allows existing Spark Classic applications to easily use Spark Connect. ### Why are the changes needed? In order users to easily try Spark Connect with their existing Spark Classic applications. ### Does this PR introduce _any_ user-facing change? It adds a new configuration `spark.api.mode` without changing the default behaviour. ### How was this patch tested? For PySpark applications, added unittests for Spark Submissions in Yarn and Kubernates, and manual tests. For Scala applications, it is difficult to add a test because SBT picks up the complied jars into the classpathes automatically (whereas we can easily control it for production environment). For this case, I manually tested as below: ```bash git clone https://github.com/HyukjinKwon/spark-connect-example cd spark-connect-example build/sbt package cd .. git clone https://github.com/apache/spark.git cd spark build/sbt package # sbin/start-connect-server.sh bin/spark-submit --name "testApp" --master "local[*]" --conf spark.api.mode=connect --class com.hyukjinkwon.SparkConnectExample ../spark-connect-example/target/scala-2.13/spark-connect-example_2.13-0.0.1.jar ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #49107 from HyukjinKwon/api-mode-draft-3. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 96bd8f1) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

HyukjinKwon · 2025-02-06T01:56:18Z

am working on it now

…park Connect ### What changes were proposed in this pull request? This PR proposes to add a new configuration called `spark.api.mode` (default `classic`) that allows existing Spark Classic applications to easily use Spark Connect. ### Why are the changes needed? In order users to easily try Spark Connect with their existing Spark Classic applications. ### Does this PR introduce _any_ user-facing change? It adds a new configuration `spark.api.mode` without changing the default behaviour. ### How was this patch tested? For PySpark applications, added unittests for Spark Submissions in Yarn and Kubernates, and manual tests. For Scala applications, it is difficult to add a test because SBT picks up the complied jars into the classpathes automatically (whereas we can easily control it for production environment). For this case, I manually tested as below: ```bash git clone https://github.com/HyukjinKwon/spark-connect-example cd spark-connect-example build/sbt package cd .. git clone https://github.com/apache/spark.git cd spark build/sbt package # sbin/start-connect-server.sh bin/spark-submit --name "testApp" --master "local[*]" --conf spark.api.mode=connect --class com.hyukjinkwon.SparkConnectExample ../spark-connect-example/target/scala-2.13/spark-connect-example_2.13-0.0.1.jar ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#49107 from HyukjinKwon/api-mode-draft-3. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

LuciferYang · 2025-02-06T17:09:24Z

resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnClusterSuite.scala

@@ -237,6 +265,16 @@ class YarnClusterSuite extends BaseYarnClusterSuite {
    testPySpark(false)
  }

+  test("run Python application with Spark Connect in yarn-client mode") {


@HyukjinKwon These two new test cases should assume sql.IntegratedUDFTestUtils.isPandasAvailable, otherwise they may lead to test failures, such as: https://github.com/apache/spark/actions/runs/13179439122/job/36786327176.

However, the yarn module currently does not have a dependency on the sql test module. Should we add this dependency to the yarn module, or is there other better alternative solution?

hmm... It appears that we need to check the entire Connect runtime environment, not just pandas and pyarrow.

let me skip the test for now.

Actually the reason seems to be:

Traceback (most recent call last): File "/home/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/utils.py", line 28, in require_minimum_pandas_version ModuleNotFoundError: No module named 'pandas' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/runner/work/spark/spark/resource-managers/yarn/target/tmp/spark-ba2c7cc1-250b-4e3d-89aa-a6c729012dcf/test.py", line 13, in <module> "spark.api.mode", "connect").master("yarn").getOrCreate() ^^^^^^^^^^^^^ File "/home/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 492, in getOrCreate File "/home/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/connect/session.py", line 19, in <module> File "/home/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/connect/utils.py", line 35, in check_dependencies File "/home/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/utils.py", line 43, in require_minimum_pandas_version pyspark.errors.exceptions.base.PySparkImportError: [PACKAGE_NOT_INSTALLED] Pandas >= 2.0.0 must be installed; however, it was not found. 14:03:30.599 INFO org.apache.spark.util.ShutdownHookManager: Shutdown hook called 14:03:30.604 INFO org.apache.spark.util.ShutdownHookManager: Deleting directory /tmp/spark-f973a6e2-72c5-4759-8709-b18b15afc3d2 14:03:30.608 INFO org.apache.spark.util.ShutdownHookManager: Deleting directory /tmp/localPyFiles-ce5279c9-5a6a-4547-84e9-3d01302054d0 (BaseYarnClusterSuite.scala:242) - run Python application with Spark Connect in yarn-cluster mode *** FAILED *** FAILED did not equal FINISHED WARNING: Using incubator modules: jdk.incubator.vector Exception in thread "main" org.apache.spark.SparkException: Application application_1738850370406_0018 finished with failed status at org.apache.spark.deploy.yarn.Client.run(Client.scala:1393) at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1827) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1032) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:204) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:227) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:96) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1137) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1146) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) (BaseYarnClusterSuite.scala:242) - run Python application in yarn-cluster mode using spark.yarn.appMasterEnv to override local envvar - ```

### What changes were proposed in this pull request? This PR proposes to disallow Spark Connect strings in `--master` when Spark API mode is `connect`. This is Python specific issue. ### Why are the changes needed? Should work as documented in #49107 ### Does this PR introduce _any_ user-facing change? Not yet because the main change has not been released (#49107) ### How was this patch tested? Manually tested: ``` ./bin/pyspark --master "sc://localhost:15002" --conf spark.api.mode=connect ``` ``` Python 3.11.9 (main, Apr 19 2024, 11:44:45) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information. /.../spark/python/pyspark/shell.py:77: UserWarning: Failed to initialize Spark session. warnings.warn("Failed to initialize Spark session.") Traceback (most recent call last): File "/.../spark/python/pyspark/shell.py", line 52, in <module> spark = SparkSession.builder.getOrCreate() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.../spark/python/pyspark/sql/session.py", line 512, in getOrCreate raise PySparkRuntimeError( pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_INVALID] Master must either be yarn or start with spark, k8s, or local. ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50000 from HyukjinKwon/SPARK-51254. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

### What changes were proposed in this pull request? This PR proposes to disallow Spark Connect strings in `--master` when Spark API mode is `connect`. This is Python specific issue. ### Why are the changes needed? Should work as documented in #49107 ### Does this PR introduce _any_ user-facing change? Not yet because the main change has not been released (#49107) ### How was this patch tested? Manually tested: ``` ./bin/pyspark --master "sc://localhost:15002" --conf spark.api.mode=connect ``` ``` Python 3.11.9 (main, Apr 19 2024, 11:44:45) [Clang 14.0.6 ] on darwin Type "help", "copyright", "credits" or "license" for more information. /.../spark/python/pyspark/shell.py:77: UserWarning: Failed to initialize Spark session. warnings.warn("Failed to initialize Spark session.") Traceback (most recent call last): File "/.../spark/python/pyspark/shell.py", line 52, in <module> spark = SparkSession.builder.getOrCreate() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/.../spark/python/pyspark/sql/session.py", line 512, in getOrCreate raise PySparkRuntimeError( pyspark.errors.exceptions.base.PySparkRuntimeError: [MASTER_URL_INVALID] Master must either be yarn or start with spark, k8s, or local. ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #50000 from HyukjinKwon/SPARK-51254. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 6603a4e) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

HyukjinKwon marked this pull request as draft December 9, 2024 04:53

github-actions bot added SQL CORE PYTHON CONNECT labels Dec 9, 2024

HyukjinKwon force-pushed the api-mode-draft-3 branch 2 times, most recently from 9dd1ec7 to 6bdbebc Compare December 9, 2024 05:54

HyukjinKwon commented Dec 10, 2024

View reviewed changes

HyukjinKwon force-pushed the api-mode-draft-3 branch from f6df8f4 to 0fabc3a Compare December 10, 2024 03:31

github-actions bot added the YARN label Dec 10, 2024

HyukjinKwon force-pushed the api-mode-draft-3 branch 2 times, most recently from bc35f57 to 51186fb Compare December 11, 2024 05:15

github-actions bot added KUBERNETES INFRA labels Dec 11, 2024

HyukjinKwon commented Dec 12, 2024

View reviewed changes

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/bindings/python/Dockerfile Outdated Show resolved Hide resolved

HyukjinKwon force-pushed the api-mode-draft-3 branch from 48d7163 to 041feed Compare December 18, 2024 04:03

github-actions bot added the DOCS label Dec 18, 2024

HyukjinKwon force-pushed the api-mode-draft-3 branch from 041feed to be264da Compare December 18, 2024 04:09

HyukjinKwon changed the title ~~[DO-NOT-MERGE] Support SQL API mode, prototype 3~~ [SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect Dec 18, 2024

HyukjinKwon force-pushed the api-mode-draft-3 branch from be264da to 8907020 Compare December 18, 2024 04:15

HyukjinKwon marked this pull request as ready for review December 18, 2024 04:18

HyukjinKwon commented Jan 3, 2025

View reviewed changes

HyukjinKwon force-pushed the api-mode-draft-3 branch from e4cf517 to 137e68c Compare January 27, 2025 07:27

hvanhovell reviewed Jan 28, 2025

View reviewed changes

core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala Outdated Show resolved Hide resolved

hvanhovell reviewed Jan 28, 2025

View reviewed changes

docs/app-dev-spark-connect.md Outdated Show resolved Hide resolved

HyukjinKwon closed this in 1bc4d15 Feb 4, 2025

HyukjinKwon added 2 commits February 5, 2025 10:41

fixup

454b501

address comments

da90697

HyukjinKwon force-pushed the api-mode-draft-3 branch from f019164 to da90697 Compare February 5, 2025 01:41

fixup

fab02b4

cloud-fan reviewed Feb 5, 2025

View reviewed changes

core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Feb 5, 2025

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/package.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Feb 5, 2025

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/package.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Feb 5, 2025

View reviewed changes

docs/app-dev-spark-connect.md Show resolved Hide resolved

cloud-fan approved these changes Feb 5, 2025

View reviewed changes

fixup

717a74a

HyukjinKwon closed this in 96bd8f1 Feb 6, 2025

LuciferYang reviewed Feb 6, 2025

View reviewed changes

HyukjinKwon mentioned this pull request Feb 7, 2025

[SPARK-50605][CONNECT][FOLLOW-UP] Install dependencies for Yarn tests in Maven build #49845

Closed

HyukjinKwon mentioned this pull request Feb 19, 2025

[SPARK-51254][PYTHON][CONNECT] Disallow --master with Spark Connect URL #50000

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect #49107

[SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect #49107

HyukjinKwon commented Dec 9, 2024 •

edited

Loading

HyukjinKwon Dec 10, 2024

HyukjinKwon Jan 3, 2025

cloud-fan commented Jan 27, 2025

HyukjinKwon commented Jan 27, 2025

HyukjinKwon commented Jan 27, 2025

Kimahriman commented Feb 5, 2025 •

edited

Loading

HyukjinKwon commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

Kimahriman commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025 •

edited

Loading

HyukjinKwon commented Feb 6, 2025

Kimahriman commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

Kimahriman commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

LuciferYang Feb 6, 2025 •

edited

Loading

LuciferYang Feb 6, 2025 •

edited

Loading

HyukjinKwon Feb 7, 2025

HyukjinKwon Feb 7, 2025

HyukjinKwon Feb 7, 2025

HyukjinKwon Feb 7, 2025

[SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect #49107

[SPARK-50605][CONNECT] Support SQL API mode for easier migration to Spark Connect #49107

Conversation

HyukjinKwon commented Dec 9, 2024 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

HyukjinKwon Dec 10, 2024

Choose a reason for hiding this comment

HyukjinKwon Jan 3, 2025

Choose a reason for hiding this comment

cloud-fan commented Jan 27, 2025

HyukjinKwon commented Jan 27, 2025

HyukjinKwon commented Jan 27, 2025

Kimahriman commented Feb 5, 2025 • edited Loading

HyukjinKwon commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

Kimahriman commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025 • edited Loading

HyukjinKwon commented Feb 6, 2025

Kimahriman commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

Kimahriman commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

HyukjinKwon commented Feb 6, 2025

LuciferYang Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

LuciferYang Feb 6, 2025 • edited Loading

Choose a reason for hiding this comment

HyukjinKwon Feb 7, 2025

Choose a reason for hiding this comment

HyukjinKwon Feb 7, 2025

Choose a reason for hiding this comment

HyukjinKwon Feb 7, 2025

Choose a reason for hiding this comment

HyukjinKwon Feb 7, 2025

Choose a reason for hiding this comment

HyukjinKwon commented Dec 9, 2024 •

edited

Loading

Kimahriman commented Feb 5, 2025 •

edited

Loading

HyukjinKwon commented Feb 6, 2025 •

edited

Loading

LuciferYang Feb 6, 2025 •

edited

Loading

LuciferYang Feb 6, 2025 •

edited

Loading