Code

Spark 49297#6

Code

Merged

attilapiros merged 203 commits intomasterattilapiros/spark:masterfrom SPARK-49297attilapiros/spark:SPARK-49297Copy head branch name to clipboardOct 24, 2024

+23,883-7,162

Commits

Commits on Oct 5, 2024

• [SPARK-49683][SQL] Block trim collation

  Show description for 3e69b40
  

  jovanpavl-db

  and

  HyukjinKwon

  committed
  3e69b40

  Copy full SHA for 3e69b40

• [SPARK-49866][SQL] Improve the error message for describe table with partition columns

  Show description for 37f2966
  

  mihailoale-db

  authored and

  MaxGekk

  committed
  37f2966

  Copy full SHA for 37f2966

Commits on Oct 6, 2024

• [MINOR][SQL] Move iterator.hasNext into try block in executeTask

  Show description for 06c70ba
  

  viirya

  authored and

  huaxingao

  committed
  06c70ba

  Copy full SHA for 06c70ba

• [SPARK-49744][SS][PYTHON] Implement TTL support for ListState in TransformWithStateInPandas

  Show description for 06a8f6b
  

  bogao007

  authored and

  HeartSaVioR

  committed
  06a8f6b

  Copy full SHA for 06a8f6b

Commits on Oct 7, 2024

• [SPARK-49773][SQL] Uncaught Java exception from make_timestamp() with bad timezone

  Show description for 9ce0564
  itholic

  committed
  9ce0564

  Copy full SHA for 9ce0564

• Revert "[SPARK-49808][SQL] Fix a deadlock in subquery execution due o lazy vals"

  Show description for a177ed1
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  a177ed1

  Copy full SHA for a177ed1

• [SPARK-49890][PYTHON] Extract the preparation of df.sample to parent class

  Show description for be546ff
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  be546ff

  Copy full SHA for be546ff

• [SPARK-49877][SQL] Change classifyException function signature: add isRuntime argument

  Show description for 5132ab1
  

  3 peoplecommitted
  5132ab1

  Copy full SHA for 5132ab1

• [SPARK-49896][K8S][DOCS] Drop K8s v1.28 Support

  Show description for 724e1bf
  dongjoon-hyun

  committed
  724e1bf

  Copy full SHA for 724e1bf

Commits on Oct 8, 2024

• [SPARK-49852][SQL] Fix deadlock caused by explain string generation

  Show description for d8aca18
  

  liuzqt

  authored and

  cloud-fan

  committed
  d8aca18

  Copy full SHA for d8aca18

• [SPARK-49894][PYTHON][CONNECT] Refine the string representation of column field operations

  Show description for c6b09c0
  

  zhengruifeng

  and

  HyukjinKwon

  committed
  c6b09c0

  Copy full SHA for c6b09c0

• [SPARK-49564][SQL] Add SQL pipe syntax for the JOIN operator

  Show description for 51af177
  

  dtenedor

  authored and

  cloud-fan

  committed
  51af177

  Copy full SHA for 51af177

• [SPARK-49889][PYTHON] Add argument trim for functionstrim/ltrim/rtrim

  Show description for 78135dc
  zhengruifeng

  committed
  78135dc

  Copy full SHA for 78135dc

• [SPARK-49898][CORE] Fix documentation and default for event log task metrics accumulator logging flag from SPARK-42204

  Show description for ef142c4
  

  JoshRosen

  authored and

  HyukjinKwon

  committed
  ef142c4

  Copy full SHA for ef142c4

• [SPARK-49903][BUILD] Upgrade Parquet to 1.14.3

  Show description for 83f732c
  

  panbingkun

  authored and

  dongjoon-hyun

  committed
  83f732c

  Copy full SHA for 83f732c

• [SPARK-49901][BUILD] Upgrade dropwizard metrics to 4.2.28

  Show description for 92e79e3
  

  panbingkun

  authored and

  dongjoon-hyun

  committed
  92e79e3

  Copy full SHA for 92e79e3

Commits on Oct 9, 2024

• [SPARK-49906][SQL] Introduce and use CONFLICTING_DIRECTORY_STRUCTURES error for PartitioningUtils

  Show description for 345a2be
  

  vladimirg-db

  authored and

  HyukjinKwon

  committed
  345a2be

  Copy full SHA for 345a2be

• [SPARK-49569][CONNECT][SQL] Add shims to support SparkContext and RDD

  Show description for 80ae411
  hvanhovell

  committed
  80ae411

  Copy full SHA for 80ae411

• [SPARK-49895][SQL] Improve error when encountering trailing comma in SELECT clause

  Show description for 5f64e80
  

  stefankandic

  and

  HyukjinKwon

  committed
  5f64e80

  Copy full SHA for 5f64e80

• [SPARK-49022][CONNECT][SQL][FOLLOW-UP] Parse unresolved identifier to keep the behavior same

  Show description for c1f18a0
  HyukjinKwon

  committed
  c1f18a0

  Copy full SHA for c1f18a0

• [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs

  Show description for 5e27eec
  

  nikhilsheoran-db

  authored and

  cloud-fan

  committed
  5e27eec

  Copy full SHA for 5e27eec

• [SPARK-49564][SQL] Add SQL pipe syntax for set operations

  Show description for 135cbc6
  

  dtenedor

  authored and

  cloud-fan

  committed
  135cbc6

  Copy full SHA for 135cbc6

• [SPARK-49909][SQL] Fix the pretty name of some expressions

  Show description for 52538f0
  

  panbingkun

  authored and

  MaxGekk

  committed
  52538f0

  Copy full SHA for 52538f0

• [SPARK-49418][CONNECT][SQL] Shared Session Thread Locals

  Show description for b565a8d
  hvanhovell

  committed
  b565a8d

  Copy full SHA for b565a8d

• [SPARK-49857][SQL] Add storageLevel to Dataset localCheckpoint API

  Show description for b1ff767
  

  Julek Sompolski

  authored and

  cloud-fan

  committed
  b1ff767

  Copy full SHA for b1ff767

• [SPARK-49873][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1325

  Show description for 97a5aa6
  

  itholic

  authored and

  MaxGekk

  committed
  97a5aa6

  Copy full SHA for 97a5aa6

• [SPARK-49569][BUILD][FOLLOWUP] Adds scala-library maven dependency to the spark-connect-shims module to fix Maven build errors

  Show description for fed9a8d
  LuciferYang

  committed
  fed9a8d

  Copy full SHA for fed9a8d

• [SPARK-43838][SQL][FOLLOWUP] Replace HashSet with HashMap to improve performance of DeduplicateRelations

  Show description for f69d03e
  

  mihailotim-db

  authored and

  HyukjinKwon

  committed
  f69d03e

  Copy full SHA for f69d03e

• [MINOR][PYTHON][TESTS] Reduce the python worker error log of test_toDF_with_schema_string

  Show description for 6ed4bdf
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  6ed4bdf

  Copy full SHA for 6ed4bdf

Commits on Oct 10, 2024

• [SPARK-49905][SQL][SS] Use different ShuffleOrigin for the shuffle required from stateful operators

  Show description for 7e82e29
  HeartSaVioR

  committed
  7e82e29

  Copy full SHA for 7e82e29

• [SPARK-49918][CORE] Use read-only access to conf in SparkContext where appropriate

  Show description for ea60e93
  

  pmenon

  authored and

  cloud-fan

  committed
  ea60e93

  Copy full SHA for ea60e93

• [SPARK-48714][PYTHON][FOLLOW-UP] Skip tests if test class is not available

  Show description for 0912217
  HyukjinKwon

  committed
  0912217

  Copy full SHA for 0912217

• [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output

  Show description for d7772f2
  

  LantaoJin

  authored and

  cloud-fan

  committed
  d7772f2

  Copy full SHA for d7772f2

• [SPARK-49549][SQL] Assign a name to the error conditions _LEGACY_ERROR_TEMP_3055, 3146

  Show description for f0498f0
  

  mrk-andreev

  authored and

  MaxGekk

  committed
  f0498f0

  Copy full SHA for f0498f0

• [SPARK-49542][SQL] Partition transform exception evaluate error

  Show description for f003638
  

  dusantism-db

  authored and

  MaxGekk

  committed
  f003638

  Copy full SHA for f003638

• [SPARK-49908][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_0044

  Show description for 38d66fd
  

  itholic

  authored and

  MaxGekk

  committed
  38d66fd

  Copy full SHA for 38d66fd

• [SPARK-49748][CORE] Add getCondition and deprecate getErrorClass in SparkThrowable

  Show description for e693af0
  MaxGekk

  committed
  e693af0

  Copy full SHA for e693af0

• [SPARK-49756][SQL] Postgres dialect supports pushdown datetime functions

  Show description for ab1315b
  

  beliefer

  authored and

  cloud-fan

  committed
  ab1315b

  Copy full SHA for ab1315b

• [SPARK-49920][INFRA] Install R for ubuntu 24.04 when GA run k8s-integration-tests

  Show description for e589ccd
  

  panbingkun

  authored and

  LuciferYang

  committed
  e589ccd

  Copy full SHA for e589ccd

• [SPARK-49569][BUILD][FOLLOWUP] Exclude spark-connect-shims from sql/core module

  Show description for b056e0b
  LuciferYang

  committed
  b056e0b

  Copy full SHA for b056e0b

Commits on Oct 11, 2024

• [SPARK-49927][SS] pyspark.sql.tests.streaming.test_streaming_listener to wait longer

  Show description for 2af6536
  

  siying

  and

  HyukjinKwon

  committed
  2af6536

  Copy full SHA for 2af6536

• [SPARK-48567][PYTHON][TESTS][FOLLOW-UP] Make the query scope higher so finally can access to it

  Show description for 4666972
  HyukjinKwon

  committed
  4666972

  Copy full SHA for 4666972

• [SPARK-49927][SS][PYTHON][TESTS][FOLLOW-UP] Fixes q.lastProgress.batchId to q.lastProgress.progress.batchId

  Show description for cacd261
  HyukjinKwon

  committed
  cacd261

  Copy full SHA for cacd261

• Revert "[SPARK-49927][SS][PYTHON][TESTS][FOLLOW-UP] Fixes q.lastProgress.batchId to q.lastProgress.progress.batchId"

  Show description for c0d396f
  HyukjinKwon

  committed
  c0d396f

  Copy full SHA for c0d396f

• Revert "[SPARK-48567][PYTHON][TESTS][FOLLOW-UP] Make the query scope higher so finally can access to it"

  Show description for b93281e
  HyukjinKwon

  committed
  b93281e

  Copy full SHA for b93281e

• Revert "[SPARK-49927][SS] pyspark.sql.tests.streaming.test_streaming_listener to wait longer"

  Show description for ed0a63f
  HyukjinKwon

  committed
  ed0a63f

  Copy full SHA for ed0a63f

• [SPARK-49915][SQL] Handle zeros and ones in ReorderAssociativeOperator

  Show description for 5104d1d
  yaooqinn

  committed
  5104d1d

  Copy full SHA for 5104d1d

• [SPARK-49615] Bugfix: Make ML column schema validation conforms with spark config spark.sql.caseSensitive

  Show description for 8e1d317
  WeichenXu123

  committed
  8e1d317

  Copy full SHA for 8e1d317

• [SPARK-49748][CORE][FOLLOWUP] Add getCondition and deprecate getErrorClass in QueryCompilationErrorsSuite

  Show description for 6d0b838
  

  panbingkun

  authored and

  MaxGekk

  committed
  6d0b838

  Copy full SHA for 6d0b838

• [SPARK-49925][SQL] Add tests for order by with collated strings

  Show description for c79e2d6
  

  ilicmarkodb

  authored and

  cloud-fan

  committed
  c79e2d6

  Copy full SHA for c79e2d6

• [MINOR][PYTHON] Minor refine LiteralExpression

  Show description for 04ec55e
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  04ec55e

  Copy full SHA for 04ec55e

Commits on Oct 12, 2024

• [SPARK-49930][SS] Ensure that socket updates are flushed on exception from the python worker

  Show description for 3ecfe8e
  

  anishshri-db

  authored and

  HyukjinKwon

  committed
  3ecfe8e

  Copy full SHA for 3ecfe8e

• [SPARK-49927][SS][PYTHON][TESTS] pyspark.sql.tests.streaming.test_streaming_listener to wait for longer

  Show description for cf657e5
  

  siying

  authored and

  HyukjinKwon

  committed
  cf657e5

  Copy full SHA for cf657e5

• [SPARK-49935][BUILD] Exclude spark-connect-shims from assembly module

  Show description for 1fb3d57
  

  LuciferYang

  authored and

  HyukjinKwon

  committed
  1fb3d57

  Copy full SHA for 1fb3d57

• [SPARK-49932][CORE] Use tryWithResource release JsonUtils#toJsonString resources to avoid memory leaks

  Show description for 6734d48
  

  panbingkun

  authored and

  LuciferYang

  committed
  6734d48

  Copy full SHA for 6734d48

• [SPARK-49766][SQL] Codegen Support for json_array_length (by Invoke & RuntimeReplaceable)

  Show description for 1244c5a
  

  panbingkun

  authored and

  MaxGekk

  committed
  1244c5a

  Copy full SHA for 1244c5a

• [SPARK-49937][INFRA] Ban call the method SparkThrowable#getErrorClass

  Show description for ed4847f
  

  panbingkun

  authored and

  HyukjinKwon

  committed
  ed4847f

  Copy full SHA for ed4847f

• [SPARK-49924][SQL] Keep containsNull after ArrayCompact replacement

  Show description for 62ade5f
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  62ade5f

  Copy full SHA for 62ade5f

Commits on Oct 13, 2024

• [MINOR][SQL] Improved broadcast timeout message prompt

  Show description for 083f44d
  

  huangxiaopingRD

  authored and

  MaxGekk

  committed
  083f44d

  Copy full SHA for 083f44d

• [SPARK-49939][SQL] Codegen Support for json_object_keys (by Invoke & RuntimeReplaceable)

  Show description for 54fd408
  

  panbingkun

  authored and

  MaxGekk

  committed
  54fd408

  Copy full SHA for 54fd408

• [SPARK-49943][PS] Remove timestamp_ntz_to_long from PythonSQLUtils

  Show description for 1abfd49
  zhengruifeng

  committed
  1abfd49

  Copy full SHA for 1abfd49

Commits on Oct 14, 2024

• [SPARK-49928][PYTHON][TESTS] Refactor plot-related unit tests

  Show description for 1aae160
  xinrong-meng

  committed
  1aae160

  Copy full SHA for 1aae160

• [SPARK-49891][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2271

  Show description for 36b2a4e
  

  itholic

  authored and

  MaxGekk

  committed
  36b2a4e

  Copy full SHA for 36b2a4e

• [SPARK-49904][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2140

  Show description for 5b9b8da
  

  itholic

  authored and

  MaxGekk

  committed
  5b9b8da

  Copy full SHA for 5b9b8da

• [MINOR][CORE] Fix the regenerate command in SparkThrowableSuite

  Show description for a2ad4d4
  MaxGekk

  committed
  a2ad4d4

  Copy full SHA for a2ad4d4

• [SPARK-49892][SQL] Assign proper error class for _LEGACY_ERROR_TEMP_1136

  Show description for 560748c
  

  itholic

  authored and

  MaxGekk

  committed
  560748c

  Copy full SHA for 560748c

• [SPARK-49949][PS] Avoid unnecessary analyze task in attach_sequence_column

  Show description for eeb044e
  zhengruifeng

  committed
  eeb044e

  Copy full SHA for eeb044e

• [SPARK-49952][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1142

  Show description for d53fa23
  

  itholic

  authored and

  MaxGekk

  committed
  d53fa23

  Copy full SHA for d53fa23

• [SPARK-49864][SQL][FOLLOW-UP] Fix default suggestion for binary arithmetic overflow

  Show description for 0606512
  

  mihailom-db

  authored and

  MaxGekk

  committed
  0606512

  Copy full SHA for 0606512

• [SPARK-49955][SQL] null value does not mean corrupted file when parsing JSON string RDD

  Show description for 96c4953
  

  cloud-fan

  authored and

  MaxGekk

  committed
  96c4953

  Copy full SHA for 96c4953

• [SPARK-49661][SQL] Implement trim collation hashing and comparison

  Show description for 74aed77
  

  jovanpavl-db

  authored and

  MaxGekk

  committed
  74aed77

  Copy full SHA for 74aed77

Commits on Oct 15, 2024

• [SPARK-49929][PYTHON][CONNECT] Support box plots

  Show description for 488f680
  xinrong-meng

  committed
  488f680

  Copy full SHA for 488f680

• [SPARK-49965][BUILD] Upgrade ASM to 9.7.1

  Show description for 217e0da
  

  panbingkun

  authored and

  LuciferYang

  committed
  217e0da

  Copy full SHA for 217e0da

• [SPARK-49451][SQL][FOLLOW-UP] Improve duplicate key exception test

  Show description for c3176a7
  

  harshmotw-db

  authored and

  MaxGekk

  committed
  c3176a7

  Copy full SHA for c3176a7

• [SPARK-49956] Disabled collations with collect_set expression

  Show description for 2ccdaba
  

  vladanvasi-db

  authored and

  cloud-fan

  committed
  2ccdaba

  Copy full SHA for 2ccdaba

• [SPARK-49974][SQL] Move resolveRelations(...) out of the Analyzer.scala

  Show description for 14c01eb
  

  vladimirg-db

  authored and

  MaxGekk

  committed
  14c01eb

  Copy full SHA for 14c01eb

• [SPARK-49916][SQL] Throw appropriate Exception for type mismatch between ColumnType and data type in some rows

  Show description for f2bd314
  

  RaleSapic

  authored and

  MaxGekk

  committed
  f2bd314

  Copy full SHA for f2bd314

• [SPARK-49959][SQL] Fix ColumnarArray.copy() to read nulls from the correct offset

  Show description for 5f2bd5c
  

  chenhao-db

  authored and

  MaxGekk

  committed
  5f2bd5c

  Copy full SHA for 5f2bd5c

• [SPARK-49954][SQL] Codegen Support for SchemaOfJson (by Invoke & RuntimeReplaceable)

  Show description for 1269b35
  

  panbingkun

  authored and

  MaxGekk

  committed
  1269b35

  Copy full SHA for 1269b35

• [SPARK-49911][SQL] Fix semantic of support binary equality

  Show description for 0b48828
  

  jovanpavl-db

  authored and

  MaxGekk

  committed
  0b48828

  Copy full SHA for 0b48828

• [SPARK-49969][BUILD] Simplify dependency management in YARN module

  Show description for 856cfe7
  

  pan3793

  authored and

  dongjoon-hyun

  committed
  856cfe7

  Copy full SHA for 856cfe7

• [SPARK-49973][INFRA] Upgrade python to 3.11 for non-python tests

  Show description for 0f6bc3b
  

  zhengruifeng

  authored and

  dongjoon-hyun

  committed
  0f6bc3b

  Copy full SHA for 0f6bc3b

• [SPARK-49964][BUILD] Remove ws-rs-api package

  Show description for 111473d
  

  hiufung-kwok

  authored and

  dongjoon-hyun

  committed
  111473d

  Copy full SHA for 111473d

• [SPARK-49922][BUILD] Upgrade sbt-assembly to 2.3.0

  Show description for add4a9c
  

  panbingkun

  authored and

  dongjoon-hyun

  committed
  add4a9c

  Copy full SHA for add4a9c

• [SPARK-49792][PYTHON][BUILD] Upgrade to numpy 2 for building and testing Spark branches

  Show description for 0e75d19
  

  xinrong-meng

  authored and

  dongjoon-hyun

  committed
  0e75d19

  Copy full SHA for 0e75d19

Commits on Oct 16, 2024

• [SPARK-49948][PS][CONNECT] Add parameter "precision" to pandas on Spark box plot

  Show description for 861b5e9
  xinrong-meng

  committed
  861b5e9

  Copy full SHA for 861b5e9

• [SPARK-49970][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_2069

  Show description for f3b2535
  

  itholic

  authored and

  MaxGekk

  committed
  f3b2535

  Copy full SHA for f3b2535

• [SPARK-49966][SQL] Codegen Support for JsonToStructs(from_json)

  Show description for 2a13011
  

  panbingkun

  authored and

  MaxGekk

  committed
  2a13011

  Copy full SHA for 2a13011

• [SPARK-49946][CORE] Require an error class in SparkOutOfMemoryError

  Show description for 39112e4
  MaxGekk

  committed
  39112e4

  Copy full SHA for 39112e4

• [SPARK-49987][SQL] Fix the error prompt when seedExpression is non-foldable in randstr

  Show description for 8d1cb76
  

  panbingkun

  authored and

  MaxGekk

  committed
  8d1cb76

  Copy full SHA for 8d1cb76

• [SPARK-49981][CORE][TESTS] Fix AsyncRDDActionsSuite.FutureAction result, timeout test case to be robust

  Show description for a3b9124
  dongjoon-hyun

  committed
  a3b9124

  Copy full SHA for a3b9124

• [SPARK-49983][CORE][TESTS] Fix BarrierTaskContextSuite.successively sync with allGather and barrier test case to be robust

  Show description for bcfe62b
  dongjoon-hyun

  committed
  bcfe62b

  Copy full SHA for bcfe62b

• [SPARK-49957][SQL] Scala API for string validation functions

  Show description for 60200ae
  

  uros-db

  authored and

  MaxGekk

  committed
  60200ae

  Copy full SHA for 60200ae

• [SPARK-48155][FOLLOWUP][SQL] AQEPropagateEmptyRelation for left anti join should check if remain child is just BroadcastQueryStageExec

  Show description for f860af6
  

  zml1206

  authored and

  cloud-fan

  committed
  f860af6

  Copy full SHA for f860af6

• [SPARK-49057][SQL][TESTS][FOLLOWUP] Handle _LEGACY_ERROR_TEMP_2235 error case

  Show description for 31a4117
  dongjoon-hyun

  committed
  31a4117

  Copy full SHA for 31a4117

• [SPARK-49643][SQL] Merge _LEGACY_ERROR_TEMP_2042 into ARITHMETIC_OVERFLOW

  Show description for f5e6b05
  

  mihailom-db

  authored and

  MaxGekk

  committed
  f5e6b05

  Copy full SHA for f5e6b05

• [SPARK-49978][R] Move sparkR deprecation warning to package attach time

  Show description for e92bf37
  

  tinglongliao-db

  authored and

  HyukjinKwon

  committed
  e92bf37

  Copy full SHA for e92bf37

Commits on Oct 17, 2024

• [SPARK-49986][INFRA] Restore scipy installation in dockerfile

  Show description for 224d3ba
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  224d3ba

  Copy full SHA for 224d3ba

• [SPARK-49945][PS][CONNECT] Add alias for distributed_id

  Show description for baa5f40
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  baa5f40

  Copy full SHA for baa5f40

• [SPARK-49951][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_(1099|3085)

  Show description for 9af705d
  itholic

  committed
  9af705d

  Copy full SHA for 9af705d

• [SPARK-49947][SQL][TESTS] Upgrade MsSql docker image version

  Show description for 948aeba
  

  panbingkun

  authored and

  HyukjinKwon

  committed
  948aeba

  Copy full SHA for 948aeba

• [SPARK-49876][CONNECT] Get rid of global locks from Spark Connect Service

  Show description for 070f2bd
  

  changgyoopark-db

  authored and

  HyukjinKwon

  committed
  070f2bd

  Copy full SHA for 070f2bd

• [SPARK-49979][SQL] Fix AQE hanging issue when collecting twice on a failed plan

  Show description for e374b94
  

  liuzqt

  authored and

  cloud-fan

  committed
  e374b94

  Copy full SHA for e374b94

• [SPARK-49977][SQL] Use stack-based iterative computation to avoid creating many Scala List objects for deep expression trees

  Show description for 175d563
  

  utkarsh39

  and

  cloud-fan

  committed
  175d563

  Copy full SHA for 175d563

• [SPARK-49848][PYTHON][CONNECT] API compatibility check for Catalog

  Show description for c8bedb4
  itholic

  committed
  c8bedb4

  Copy full SHA for c8bedb4

• [SPARK-49996][SQL][TESTS] Upgrade mysql-connector-j to 9.1.0

  Show description for 74dbd9a
  

  panbingkun

  authored and

  yaooqinn

  committed
  74dbd9a

  Copy full SHA for 74dbd9a

• [SPARK-49770][SS][ROCKSDB HARDENING] Improve RocksDB SST file mapping management, and fix issue with reloading same version with existing snapshot

  Show description for 8405c9b
  

  2 peopleauthored and

  HeartSaVioR

  committed
  8405c9b

  Copy full SHA for 8405c9b

• [SPARK-49259][SS] Size based partition creation during kafka read

  Show description for f96a6f8
  

  subham611

  authored and

  HeartSaVioR

  committed
  f96a6f8

  Copy full SHA for f96a6f8

• [SPARK-49495][INFRA][FOLLOW-UP] Disable GitHub Pages workflow in forked repository

  Show description for 2469a6f
  HyukjinKwon

  committed
  2469a6f

  Copy full SHA for 2469a6f

• [SPARK-49998][SQL] Integrate _LEGACY_ERROR_TEMP_1252 into EXPECT_TABLE_NOT_VIEW

  Show description for 371a09a
  

  itholic

  authored and

  MaxGekk

  committed
  371a09a

  Copy full SHA for 371a09a

• [SPARK-49971][SQL] Assign proper error condition for _LEGACY_ERROR_TEMP_1097

  Show description for 0b7cc37
  

  itholic

  authored and

  MaxGekk

  committed
  0b7cc37

  Copy full SHA for 0b7cc37

• [SPARK-49997][SQL] Integrate _LEGACY_ERROR_TEMP_2165 into MALFORMED_RECORD_IN_PARSING

  Show description for 0dd6a2a
  

  itholic

  authored and

  MaxGekk

  committed
  0dd6a2a

  Copy full SHA for 0dd6a2a

• [SPARK-50009][INFRA] Exclude pandas/resource/testing from pyspark-core module

  Show description for b078c0d
  zhengruifeng

  committed
  b078c0d

  Copy full SHA for b078c0d

• [SPARK-49988][BUILD] Remove unused Hadoop dependency management

  Show description for f1f04b0
  

  pan3793

  authored and

  dongjoon-hyun

  committed
  f1f04b0

  Copy full SHA for f1f04b0

• [SPARK-50011][INFRA] Add a separate docker file for doc build

  Show description for 360ce0a
  

  zhengruifeng

  authored and

  dongjoon-hyun

  committed
  360ce0a

  Copy full SHA for 360ce0a

• [SPARK-50002][PYTHON][CONNECT] API compatibility check for I/O

  Show description for 24d9c97
  

  itholic

  authored and

  dongjoon-hyun

  committed
  24d9c97

  Copy full SHA for 24d9c97

• [SPARK-50003][PYTHON][DOCS] Add pie plot doc examples and correct area plot parameter type hints

  Show description for d144144
  

  xinrong-meng

  authored and

  dongjoon-hyun

  committed
  d144144

  Copy full SHA for d144144

• [SPARK-50008][PS][CONNECT] Avoid unnecessary operations in attach_distributed_sequence_column

  Show description for 6bcad77
  

  zhengruifeng

  authored and

  dongjoon-hyun

  committed
  6bcad77

  Copy full SHA for 6bcad77

• [SPARK-49958][PYTHON] Python API for string validation functions

  Show description for 120ae9a
  

  uros-db

  authored and

  MaxGekk

  committed
  120ae9a

  Copy full SHA for 120ae9a

• [SPARK-49936][BUILD] Upgrade datasketches-java to 6.1.1

  Show description for 6362e0c
  

  LuciferYang

  authored and

  dongjoon-hyun

  committed
  6362e0c

  Copy full SHA for 6362e0c

• [SPARK-49962][SQL] Simplify AbstractStringTypes class hierarchy

  Show description for 91becf1
  

  stefankandic

  authored and

  MaxGekk

  committed
  91becf1

  Copy full SHA for 91becf1

• [SPARK-50014][INFRA] Use grpcio* 1.67.0 in Python 3.13 image

  Show description for bdd3464
  dongjoon-hyun

  committed
  bdd3464

  Copy full SHA for bdd3464

Commits on Oct 18, 2024

• [SPARK-50019][INFRA] Install BASIC_PIP_PKGS except pyarrow in Python 3.13 image

  Show description for eb73143
  

  dongjoon-hyun

  authored and

  HyukjinKwon

  committed
  eb73143

  Copy full SHA for eb73143

• [SPARK-48773][FOLLOW-UP] spark.conf.set should not fail when setting spark.default.parallelism

  Show description for 78308da
  

  amaliujia

  authored and

  cloud-fan

  committed
  78308da

  Copy full SHA for 78308da

• [SPARK-49829][SS] Fix the bug on the optimization on adding input to state store in stream-stream join

  Show description for 75b8666
  

  HeartSaVioR

  and

  andrzejzera

  committed
  75b8666

  Copy full SHA for 75b8666

• [SPARK-49991][SQL] Make HadoopMapReduceCommitProtocol respect 'mapreduce.output.basename' to generate file names

  Show description for d90145d
  

  yaooqinn

  authored and

  dongjoon-hyun

  committed
  d90145d

  Copy full SHA for d90145d

• [SPARK-49802][SS] Add support for read change feed for map and list types used in stateful processors

  Show description for 8280249
  

  anishshri-db

  authored and

  HeartSaVioR

  committed
  8280249

  Copy full SHA for 8280249

• [SPARK-49411][SS] Communicate State Store Checkpoint ID between driver and stateful operators

  Show description for 5697df7
  

  siying

  authored and

  HeartSaVioR

  committed
  5697df7

  Copy full SHA for 5697df7

• [SPARK-49558][SQL] Add SQL pipe syntax for LIMIT/OFFSET and ORDER/SORT/CLUSTER/DISTRIBUTE BY

  Show description for ff5e0a1
  

  dtenedor

  authored and

  cloud-fan

  committed
  ff5e0a1

  Copy full SHA for ff5e0a1

• [SPARK-50011][INFRA][FOLLOW-UP] Refresh the image cache job

  Show description for 2724909
  zhengruifeng

  committed
  2724909

  Copy full SHA for 2724909

• [SPARK-50010][SQL] Expand implicit collation mismatch error

  Show description for 55c558a
  

  uros-db

  authored and

  MaxGekk

  committed
  55c558a

  Copy full SHA for 55c558a

• [SPARK-50018][SQL] Make AbstractStringType serializable

  Show description for af0cc8e
  

  uros-db

  authored and

  MaxGekk

  committed
  af0cc8e

  Copy full SHA for af0cc8e

• [SPARK-49558][SQL][FOLLOW-UP] Run ./build/mvn scalafmt:format to fix CI

  Show description for fae0c12
  HyukjinKwon

  committed
  fae0c12

  Copy full SHA for fae0c12

• [SPARK-49808][SQL] Fix a deadlock in subquery execution due to lazy vals

  Show description for 693d008
  

  zhengruifeng

  authored and

  cloud-fan

  committed
  693d008

  Copy full SHA for 693d008

• [SPARK-50016][SQL] Improve explicit collation mismatch error

  Show description for 25c55a8
  

  uros-db

  authored and

  MaxGekk

  committed
  25c55a8

  Copy full SHA for 25c55a8

• Revert "[SPARK-49909][SQL] Fix the pretty name of some expressions"

  Show description for b0414b1
  

  panbingkun

  authored and

  MaxGekk

  committed
  b0414b1

  Copy full SHA for b0414b1

• [SPARK-50007][SQL][SS] Provide default values for metrics on observe API when physical node is lost in executed plan

  Show description for 0705e10
  HeartSaVioR

  committed
  0705e10

  Copy full SHA for 0705e10

• [SPARK-50025][SQL] Integrate _LEGACY_ERROR_TEMP_1253 into EXPECT_VIEW_NOT_TABLE

  Show description for 7e9017e
  

  itholic

  authored and

  MaxGekk

  committed
  7e9017e

  Copy full SHA for 7e9017e

• [SPARK-49990][SQL] Improve performance of randStr

  Show description for 7e3b168
  

  panbingkun

  authored and

  MaxGekk

  committed
  7e3b168

  Copy full SHA for 7e3b168

• [SPARK-50004][SQL] Integrate _LEGACY_ERROR_TEMP_3327 into FIELD_NOT_FOUND

  Show description for 4c63672
  

  itholic

  authored and

  MaxGekk

  committed
  4c63672

  Copy full SHA for 4c63672

• [SPARK-50021][CORE][UI] Fix ApplicationPage to hide App UI links when UI is disabled

  Show description for ff47dd9
  

  dongjoon-hyun

  authored and

  MaxGekk

  committed
  ff47dd9

  Copy full SHA for ff47dd9

• [SPARK-49966][SQL] Use Invoke to implement JsonToStructs(from_json)

  Show description for a1fc7e6
  

  panbingkun

  authored and

  MaxGekk

  committed
  a1fc7e6

  Copy full SHA for a1fc7e6

• [SPARK-50026][SQL] Fix Identity Column metadata get

  Show description for ae87ce6
  

  zhipengmao-db

  authored and

  MaxGekk

  committed
  ae87ce6

  Copy full SHA for ae87ce6

• [SPARK-50022][CORE][UI] Fix MasterPage to hide App UI links when UI is disabled

  Show description for 9d0e31d
  dongjoon-hyun

  committed
  9d0e31d

  Copy full SHA for 9d0e31d

• Revert "[SPARK-50011][INFRA][FOLLOW-UP] Refresh the image cache job"

  Show description for aaecab3
  dongjoon-hyun

  committed
  aaecab3

  Copy full SHA for aaecab3

• Revert "[SPARK-50011][INFRA] Add a separate docker file for doc build"

  Show description for 6f710cd
  dongjoon-hyun

  committed
  6f710cd

  Copy full SHA for 6f710cd

• [SPARK-49846][SS] Add numUpdatedStateRows and numRemovedStateRows metrics for use with transformWithState operator

  Show description for b1d1f10
  

  anishshri-db

  authored and

  HeartSaVioR

  committed
  b1d1f10

  Copy full SHA for b1d1f10

• [SPARK-49550][BUILD] Upgrade Hadoop to 3.4.1

  Show description for f8d9222
  dongjoon-hyun

  committed
  f8d9222

  Copy full SHA for f8d9222

Commits on Oct 19, 2024

• [SPARK-50030][PYTHON][CONNECT] API compatibility check for Window

  Show description for 14ed86e
  

  itholic

  authored and

  HyukjinKwon

  committed
  14ed86e

  Copy full SHA for 14ed86e

• [SPARK-50023][PYTHON][CONNECT] API compatibility check for Functions

  Show description for 25b03f9
  

  itholic

  authored and

  HyukjinKwon

  committed
  25b03f9

  Copy full SHA for 25b03f9

Commits on Oct 20, 2024

• [SPARK-49530][PYTHON][CONNECT] Support kde/density plots

  Show description for d5550f6
  

  xinrong-meng

  authored and

  zhengruifeng

  committed
  d5550f6

  Copy full SHA for d5550f6

• [MINOR][DOCS] Add Development Version of docs in README.md

  Show description for 64218d7
  HyukjinKwon

  committed
  64218d7

  Copy full SHA for 64218d7

• [SPARK-50024][PYTHON][CONNECT] Switch to use logger instead of warnings module in client

  Show description for a31da6a
  

  nija-at

  authored and

  HyukjinKwon

  committed
  a31da6a

  Copy full SHA for a31da6a

• [MINOR][TESTS] Fix appname from sql.protobuf.functions to sql.connect.protobuf.functions

  Show description for 1ca2d56
  HyukjinKwon

  committed
  1ca2d56

  Copy full SHA for 1ca2d56

• [SPARK-50042][PYTHON] Upgrade numpy 2 for python linter

  Show description for 4883689
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  4883689

  Copy full SHA for 4883689

• [MINOR][DOCS][PYTHON] Clarify that profilers show the accumulated results

  Show description for 164a0f4
  HyukjinKwon

  committed
  164a0f4

  Copy full SHA for 164a0f4

• [SPARK-50040][PYTHON][TESTS] Make pysaprk-connect tests passing without optional dependencies

  Show description for ae75cac
  HyukjinKwon

  committed
  ae75cac

  Copy full SHA for ae75cac

• [SPARK-50039][CONNECT][PYTHON] API compatibility check for Grouping

  Show description for 76ea894
  

  itholic

  authored and

  HyukjinKwon

  committed
  76ea894

  Copy full SHA for 76ea894

• [SPARK-50038][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0008: MERGE_WITHOUT_WHEN

  Show description for f2f3099
  

  itholic

  authored and

  MaxGekk

  committed
  f2f3099

  Copy full SHA for f2f3099

• [SPARK-50027][SQL] Move Identity Column SQL parsing code to DataTypeAstBuilder

  Show description for 4508911
  

  zhipengmao-db

  authored and

  MaxGekk

  committed
  4508911

  Copy full SHA for 4508911

• [SPARK-50044][PYTHON] Refine the docstring of multiple math functions

  Show description for 32cc2dd
  

  zhengruifeng

  authored and

  MaxGekk

  committed
  32cc2dd

  Copy full SHA for 32cc2dd

Commits on Oct 21, 2024

• [SPARK-50001][PYTHON][PS][CONNECT] Adjust "precision" to be part of kwargs for box plots

  Show description for d2e3223
  xinrong-meng

  committed
  d2e3223

  Copy full SHA for d2e3223

• [SPARK-50037][SQL] Refactor AttributeSeq.resolve(...)

  Show description for f581bad
  

  vladimirg-db

  authored and

  cloud-fan

  committed
  f581bad

  Copy full SHA for f581bad

• [SPARK-49850][CONNECT][PYTHON] API compatibility check for Avro

  Show description for 537ca91
  itholic

  committed
  537ca91

  Copy full SHA for 537ca91

• [SPARK-49821][SS][PYTHON] Implement MapState and TTL support for TransformWithStateInPandas

  Show description for c1198fa
  

  bogao007

  authored and

  HeartSaVioR

  committed
  c1198fa

  Copy full SHA for c1198fa

• [SPARK-50048][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_2114: UNRECOGNIZED_STATISTIC

  Show description for 4ed908e
  

  itholic

  authored and

  MaxGekk

  committed
  4ed908e

  Copy full SHA for 4ed908e

• [SPARK-49851][CONNECT][PYTHON] API compatibility check for Protobuf

  Show description for 2632544
  itholic

  committed
  2632544

  Copy full SHA for 2632544

• [SPARK-50035][SS] Add support for explicit handleExpiredTimer function part of the stateful processor

  Show description for 738dfa3
  

  anishshri-db

  authored and

  HeartSaVioR

  committed
  738dfa3

  Copy full SHA for 738dfa3

• [SPARK-50029][SQL] Make StaticInvoke compatible with the method that return Any

  Show description for f9a5de4
  

  panbingkun

  authored and

  cloud-fan

  committed
  f9a5de4

  Copy full SHA for f9a5de4

• [SPARK-50053][SQL] Turn _LEGACY_ERROR_TEMP_2104 into INTERNAL_ERROR

  Show description for 3e597e2
  

  itholic

  authored and

  MaxGekk

  committed
  3e597e2

  Copy full SHA for 3e597e2

• [SPARK-50036][CORE][PYTHON] Include SPARK_LOG_SCHEMA in the context of REPL shell

  Show description for f86df1e
  

  gengliangwang

  and

  HyukjinKwon

  committed
  f86df1e

  Copy full SHA for f86df1e

• [SPARK-49967][SQL] Codegen Support for StructsToJson(to_json)

  Show description for b4eb034
  

  panbingkun

  authored and

  MaxGekk

  committed
  b4eb034

  Copy full SHA for b4eb034

• [SPARK-50052][PYTHON] Make NumpyArrayConverter support empty str ndarray

  Show description for 91ae102
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  91ae102

  Copy full SHA for 91ae102

Commits on Oct 22, 2024

• [SPARK-49552][CONNECT][FOLLOW-UP] Make 'randstr' and 'uniform' deterministic in Scala Client

  Show description for 70c9b1f
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  70c9b1f

  Copy full SHA for 70c9b1f

• [SPARK-50064][PYTHON][TESTS] Make pysaprk-ml-connect tests passing without optional dependencies

  Show description for 09b7bdf
  HyukjinKwon

  committed
  09b7bdf

  Copy full SHA for 09b7bdf

• [SPARK-50058][SQL] Factor out plan normalization functions to later use those in single-pass Analyzer testing

  Show description for d5419ab
  

  vladimirg-db

  authored and

  cloud-fan

  committed
  d5419ab

  Copy full SHA for d5419ab

• [SPARK-49849][CONNECT][PYTHON] API compatibility check for Structured Streaming Query Management

  Show description for 2c904e4
  

  itholic

  authored and

  HyukjinKwon

  committed
  2c904e4

  Copy full SHA for 2c904e4

• [SPARK-49944][DOCS] Fix broken main.js import and fix image links for streaming documentation

  Show description for e7cdb5a
  

  neilramaswamy

  authored and

  yaooqinn

  committed
  e7cdb5a

  Copy full SHA for e7cdb5a

• [SPARK-50051][PYTHON][CONNECT] Make lit works with empty numpy ndarray

  Show description for dc35ba8
  zhengruifeng

  committed
  dc35ba8

  Copy full SHA for dc35ba8

• [SPARK-49461][SS] Persistent Checkpoint ID to commit logs and read it back

  Show description for aaa7044
  

  WweiL

  authored and

  HeartSaVioR

  committed
  aaa7044

  Copy full SHA for aaa7044

• [SPARK-50057][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_1049: INVALID_ATTRIBUTE_NAME_SYNTAX

  Show description for 693ebd9
  

  itholic

  authored and

  MaxGekk

  committed
  693ebd9

  Copy full SHA for 693ebd9

• [SPARK-50047][BUILD] Upgrade jersey to 3.0.16

  Show description for ff0653f
  LuciferYang

  committed
  ff0653f

  Copy full SHA for ff0653f

• [SPARK-49708][BUILD] Upgrade RoaringBitmap to 1.3.0

  Show description for b142a19
  

  panbingkun

  authored and

  LuciferYang

  committed
  b142a19

  Copy full SHA for b142a19

• [SPARK-50051][PYTHON][FOLLOW-UP] Fix data type check for Numpy 1.x

  Show description for 61db251
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  61db251

  Copy full SHA for 61db251

• [SPARK-50059][CONNECT][PYTHON] API compatibility check for Structured Streaming I/O

  Show description for a376880
  

  itholic

  authored and

  HyukjinKwon

  committed
  a376880

  Copy full SHA for a376880

• [SPARK-50054][PYTHON][CONNECT] Support histogram plots

  Show description for a7b686b
  

  xinrong-meng

  authored and

  zhengruifeng

  committed
  a7b686b

  Copy full SHA for a7b686b

• [SPARK-50070][SQL] Integrate _LEGACY_ERROR_TEMP_0039 into UNSUPPORTED_SQL_STATEMENT

  Show description for 3015e3e
  

  itholic

  authored and

  MaxGekk

  committed
  3015e3e

  Copy full SHA for 3015e3e

• [SPARK-48413][SQL][FOLLOW-UP] Fix ALTER COLUMN with collation

  Show description for 6e3dcdb
  

  jovanm-db

  authored and

  MaxGekk

  committed
  6e3dcdb

  Copy full SHA for 6e3dcdb

• [SPARK-49934][SQL] Add implicit cast for accessing collated map with literal

  Show description for 3ec28ff
  

  stefankandic

  authored and

  MaxGekk

  committed
  3ec28ff

  Copy full SHA for 3ec28ff

• [SPARK-50031][SQL] Add the TryParseUrl expression

  Show description for abc4986
  

  jovanm-db

  authored and

  MaxGekk

  committed
  abc4986

  Copy full SHA for abc4986

• [SPARK-50065][SQL][TESTS] Upgrade MySQL & Mariadb docker image version

  Show description for 0083815
  

  panbingkun

  authored and

  MaxGekk

  committed
  0083815

  Copy full SHA for 0083815

Commits on Oct 23, 2024

• [SPARK-50050][PYTHON][CONNECT] Make lit accept str and bool type numpy ndarray

  Show description for 2bf41a6
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  2bf41a6

  Copy full SHA for 2bf41a6

• [SPARK-49985][SQL] Remove support for interval types in Variant

  Show description for 2957069
  

  harshmotw-db

  authored and

  HyukjinKwon

  committed
  2957069

  Copy full SHA for 2957069

• [SPARK-50064][PYTHON][TESTS][FOLLOW-UP] Make pysaprk-ml tests passing without optional dependencies

  Show description for 51e915a
  HyukjinKwon

  committed
  51e915a

  Copy full SHA for 51e915a

• [SPARK-50080][SQL][TESTS] Add benchmark cases for parquet adaptive bloom filter in BloomFilterBenchmark

  Show description for 2cb7a16
  

  yaooqinn

  authored and

  HyukjinKwon

  committed
  2cb7a16

  Copy full SHA for 2cb7a16

• [SPARK-50067][SQL] Codegen Support for SchemaOfCsv(by Invoke & RuntimeReplaceable)

  Show description for 369c40c
  

  panbingkun

  authored and

  MaxGekk

  committed
  369c40c

  Copy full SHA for 369c40c

• [SPARK-50078][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_0038: DUPLICATED_CTE_NAMES

  Show description for c8fb211
  

  itholic

  authored and

  MaxGekk

  committed
  c8fb211

  Copy full SHA for c8fb211

• [SPARK-49902][SQL] Catch underlying runtime errors in RegExpReplace

  Show description for 35d2ef9
  

  harshmotw-db

  and

  MaxGekk

  committed
  35d2ef9

  Copy full SHA for 35d2ef9

• [MINOR][PYTHON][DOCS] Fix the type hint for extract, date_part and datepart

  Show description for be18b7c
  zhengruifeng

  committed
  be18b7c

  Copy full SHA for be18b7c

• [SPARK-49638][SQL] Remove the ANSI config suggestion in INVALID_URL

  Show description for 5e9dec7
  

  jovanm-db

  authored and

  MaxGekk

  committed
  5e9dec7

  Copy full SHA for 5e9dec7

• [SPARK-50084][SQL] Assign appropriate error condition for _LEGACY_ERROR_TEMP_3168: MISSING_TIMEOUT_CONFIGURATION

  Show description for dcb4177
  

  itholic

  authored and

  MaxGekk

  committed
  dcb4177

  Copy full SHA for dcb4177

• [SPARK-50088][SQL] Refactor UnresolvedStarBase.expand

  Show description for 7066f06
  

  mihailoale-db

  authored and

  MaxGekk

  committed
  7066f06

  Copy full SHA for 7066f06

• [SPARK-50086][PYTHON][DOCS] Refine the docstring of multiple datetime functions

  Show description for d65c85e
  

  zhengruifeng

  authored and

  MaxGekk

  committed
  d65c85e

  Copy full SHA for d65c85e