Code

Apache spark latest pull#12

Code

Merged

rekhajoshm merged 151 commits intorekhajoshm:masterrekhajoshm/spark:masterfrom apache:masterapache/spark:masterCopy head branch name to clipboardFeb 9, 2018

+10,246-3,236

Commits

Commits on Jan 21, 2018

• [SPARK-23087][SQL] CheckCartesianProduct too restrictive when condition is false/null

  Show description for 121dc96
  

  mgaido91

  authored and

  gatorsmile

  committed
  121dc96

  Copy full SHA for 121dc96

• [SPARK-22119][ML] Add cosine distance to KMeans

  Show description for 4f43d27
  

  mgaido91

  authored and

  srowen

  committed
  4f43d27

  Copy full SHA for 4f43d27

• [SPARK-21293][SS][SPARKR] Add doc example for streaming join, dedup

  Show description for 2239d7a
  

  felixcheung

  authored and

  Felix Cheung

  committed
  2239d7a

  Copy full SHA for 2239d7a

Commits on Jan 22, 2018

• [SPARK-23169][INFRA][R] Run lintr on the changes of lint-r script and .lintr configuration

  Show description for 12faae2
  HyukjinKwon

  committed
  12faae2

  Copy full SHA for 12faae2

• [SPARK-20947][PYTHON] Fix encoding/decoding error in pipe action

  Show description for 602c6d8
  

  王晓哲

  authored and

  HyukjinKwon

  committed
  602c6d8

  Copy full SHA for 602c6d8

• [SPARK-22976][CORE] Cluster mode driver dir removed while running

  Show description for 11daeb8
  

  RussellSpitzer

  authored and

  jerryshao

  committed
  11daeb8

  Copy full SHA for 11daeb8

• [MINOR][SQL] Fix wrong comments on org.apache.spark.sql.parquet.row.attributes

  Show description for 8142a3b
  

  dongjoon-hyun

  authored and

  HyukjinKwon

  committed
  8142a3b

  Copy full SHA for 8142a3b

• [SPARK-23020][CORE] Fix races in launcher code, test.

  Show description for ec22897
  

  Marcelo Vanzin

  authored and

  cloud-fan

  committed
  ec22897

  Copy full SHA for ec22897

• [MINOR][DOC] Fix the path to the examples jar

  Show description for 60175e9
  

  tashoyan

  authored and

  jerryshao

  committed
  60175e9

  Copy full SHA for 60175e9

• [SPARK-23122][PYSPARK][FOLLOW-UP] Update the docs for UDF Registration

  Show description for 7328116
  gatorsmile

  committed
  7328116

  Copy full SHA for 7328116

• [SPARK-23170][SQL] Dump the statistics of effective runs of analyzer and optimizer rules

  Show description for 7880188
  gatorsmile

  committed
  7880188

  Copy full SHA for 7880188

• [MINOR][SQL][TEST] Test case cleanups for recent PRs

  Show description for 896e45a
  gatorsmile

  committed
  896e45a

  Copy full SHA for 896e45a

• [SPARK-23090][SQL] polish ColumnVector

  Show description for 5d680ca
  cloud-fan

  committed
  5d680ca

  Copy full SHA for 5d680ca

• [SPARK-7721][PYTHON][TESTS] Adds PySpark coverage generation script

  Show description for 87ffe7a
  HyukjinKwon

  committed
  87ffe7a

  Copy full SHA for 87ffe7a

• [SPARK-11630][CORE] ClosureCleaner moved from warning to debug

  Show description for 4327ccf
  

  rekhajoshm

  authored and

  srowen

  committed
  4327ccf

  Copy full SHA for 4327ccf

• [SPARK-23121][CORE] Fix for ui becoming unaccessible for long running streaming apps

  Show description for 446948a
  

  smurakozi

  authored and

  Marcelo Vanzin

  committed
  446948a

  Copy full SHA for 446948a

• [MINOR] Typo fixes

  Show description for 76b8b84
  

  jaceklaskowski

  authored and

  srowen

  committed
  76b8b84

  Copy full SHA for 76b8b84

• [SPARK-22389][SQL] data source v2 partitioning reporting interface

  Show description for 51eb750
  

  cloud-fan

  authored and

  gatorsmile

  committed
  51eb750

  Copy full SHA for 51eb750

Commits on Jan 23, 2018

• [SPARK-22274][PYTHON][SQL] User-defined aggregation functions with pandas udf (full shuffle)

  Show description for b2ce17b
  

  icexelloss

  authored and

  ueshin

  committed
  b2ce17b

  Copy full SHA for b2ce17b

• [SPARK-22465][FOLLOWUP] Update the number of partitions of default partitioner when defaultParallelism is set

  Show description for 96cb60b
  

  jiangxb1987

  authored and

  mridulm

  committed
  96cb60b

  Copy full SHA for 96cb60b

• [SPARK-20749][SQL][FOLLOW-UP] Override prettyName for bit_length and octet_length

  Show description for ee572ba
  

  gatorsmile

  authored and

  HyukjinKwon

  committed
  ee572ba

  Copy full SHA for ee572ba

• [SPARK-20664][SPARK-23103][CORE] Follow-up: remove workaround for .

  Show description for bdebb8e
  

  Marcelo Vanzin

  committed
  bdebb8e

  Copy full SHA for bdebb8e

• [SPARK-17088][HIVE] Fix 'sharesHadoopClasses' option when creating client.

  Show description for dc4761f
  

  Marcelo Vanzin

  committed
  dc4761f

  Copy full SHA for dc4761f

• [SPARK-22735][ML][DOC] Added VectorSizeHint docs and examples.

  Show description for 05839d1
  

  MrBago

  authored and

  mengxr

  committed
  05839d1

  Copy full SHA for 05839d1

• [SPARK-23192][SQL] Keep the Hint after Using Cached Data

  Show description for 613c290
  gatorsmile

  committed
  613c290

  Copy full SHA for 613c290

Commits on Jan 24, 2018

• [SPARK-23195][SQL] Keep the Hint of Cached Data

  Show description for 44cc4da
  gatorsmile

  committed
  44cc4da

  Copy full SHA for 44cc4da

• [SPARK-23197][DSTREAMS] Increased timeouts to resolve flakiness

  Show description for 15adcc8
  tdas

  committed
  15adcc8

  Copy full SHA for 15adcc8

• [SPARK-23177][SQL][PYSPARK] Extract zero-parameter UDFs from aggregate

  Show description for a3911cf
  

  viirya

  authored and

  HyukjinKwon

  committed
  a3911cf

  Copy full SHA for a3911cf

• [SPARK-21727][R] Allow multi-element atomic vector as column type in SparkR DataFrame

  Show description for f54b65c
  

  neilalex

  authored and

  Felix Cheung

  committed
  f54b65c

  Copy full SHA for f54b65c

• Revert "[SPARK-23195][SQL] Keep the Hint of Cached Data"

  Show description for 4e7b490
  gatorsmile

  committed
  4e7b490

  Copy full SHA for 4e7b490

• [SPARK-23174][BUILD][PYTHON] python code style checker update

  Show description for 7af1a32
  

  rekhajoshm

  authored and

  HyukjinKwon

  committed
  7af1a32

  Copy full SHA for 7af1a32

• [SPARK-23148][SQL] Allow pathnames with special characters for CSV / JSON / text

  Show description for de36f65
  

  Henry Robinson

  authored and

  HyukjinKwon

  committed
  de36f65

  Copy full SHA for de36f65

• [SPARK-22577][CORE] executor page blacklist status should update with TaskSet level blacklisting

  Show description for 0ec95bb
  

  attilapiros

  authored and

  squito

  committed
  0ec95bb

  Copy full SHA for 0ec95bb

• [SPARK-20906][SPARKR] Add API doc example for Constrained Logistic Regression

  Show description for e18d6f5
  

  felixcheung

  authored and

  Felix Cheung

  committed
  e18d6f5

  Copy full SHA for e18d6f5

• [SPARK-23020][CORE][FOLLOWUP] Fix Java style check issues.

  Show description for 8c273b4
  

  ueshin

  authored and

  Marcelo Vanzin

  committed
  8c273b4

  Copy full SHA for 8c273b4

• [SPARK-22837][SQL] Session timeout checker does not work in SessionManager.

  Show description for bbb87b3
  

  zuotingbing

  authored and

  gatorsmile

  committed
  bbb87b3

  Copy full SHA for bbb87b3

• [SPARK-23152][ML] - Correctly guard against empty datasets

  Show description for 840dea6
  

  tovbinm

  authored and

  srowen

  committed
  840dea6

  Copy full SHA for 840dea6

• [SPARK-22297][CORE TESTS] Flaky test: BlockManagerSuite "Shuffle registration timeout and maxAttempts conf"

  Show description for 0e178e1
  

  mpetruska

  authored and

  Marcelo Vanzin

  committed
  0e178e1

  Copy full SHA for 0e178e1

• [SPARK-23198][SS][TEST] Fix KafkaContinuousSourceStressForDontFailOnDataLossSuite to test ContinuousExecution

  Show description for bc9641d
  

  dongjoon-hyun

  authored and

  zsxwing

  committed
  bc9641d

  Copy full SHA for bc9641d

• [MINOR][SQL] add new unit test to LimitPushdown

  Show description for 6f0ba84
  

  heary-cao

  authored and

  gatorsmile

  committed
  6f0ba84

  Copy full SHA for 6f0ba84

Commits on Jan 25, 2018

• [SPARK-23129][CORE] Make deserializeStream of DiskMapIterator init lazily

  Show description for 45b4bbf
  

  caneGuy

  authored and

  cloud-fan

  committed
  45b4bbf

  Copy full SHA for 45b4bbf

• [SPARK-23208][SQL] Fix code generation for complex create array (related) expressions

  Show description for e29b08a
  

  hvanhovell

  authored and

  cloud-fan

  committed
  e29b08a

  Copy full SHA for e29b08a

• [SPARK-23163][DOC][PYTHON] Sync ML Python API with Scala

  Show description for 39ee2ac
  

  BryanCutler

  authored and

  Felix Cheung

  committed
  39ee2ac

  Copy full SHA for 39ee2ac

• [SPARK-21717][SQL] Decouple consume functions of physical operators in whole-stage codegen

  Show description for d20bbc2
  

  viirya

  authored and

  cloud-fan

  committed
  d20bbc2

  Copy full SHA for d20bbc2

• [SPARK-23112][DOC] Add highlights and migration guide for 2.3

  Show description for 8532e26
  

  Nick Pentreath

  committed
  8532e26

  Copy full SHA for 8532e26

• [SPARK-23081][PYTHON] Add colRegex API to PySpark

  Show description for 8480c0c
  

  huaxingao

  authored and

  HyukjinKwon

  committed
  8480c0c

  Copy full SHA for 8480c0c

Commits on Jan 26, 2018

• [SPARK-23032][SQL] Add a per-query codegenStageId to WholeStageCodegenExec

  Show description for e57f394
  

  rednaxelafx

  authored and

  gatorsmile

  committed
  e57f394

  Copy full SHA for e57f394

• [SPARK-23205][ML] Update ImageSchema.readImages to correctly set alpha values for four-channel images

  Show description for 7bd46d9
  

  smurching

  authored and

  srowen

  committed
  7bd46d9

  Copy full SHA for 7bd46d9

• [SPARK-23020][CORE] Fix race in SparkAppHandle cleanup, again.

  Show description for 70a68b3
  

  Marcelo Vanzin

  authored and

  cloud-fan

  committed
  70a68b3

  Copy full SHA for 70a68b3

• [SPARK-23200] Reset Kubernetes-specific config on Checkpoint restore

  Show description for d172181
  

  ssaavedra

  authored and

  jerryshao

  committed
  d172181

  Copy full SHA for d172181

• [SPARK-22799][ML] Bucketizer should throw exception if single- and multi-column params are both set

  Show description for cd3956d
  

  mgaido91

  authored and

  Nick Pentreath

  committed
  cd3956d

  Copy full SHA for cd3956d

• [SPARK-22797][PYSPARK] Bucketizer support multi-column

  Show description for c22eaa9
  

  zhengruifeng

  authored and

  Nick Pentreath

  committed
  c22eaa9

  Copy full SHA for c22eaa9

• [SPARK-22068][CORE] Reduce the duplicate code between putIteratorAsValues and putIteratorAsBytes

  Show description for 3e25251
  

  ConeyLiu

  authored and

  cloud-fan

  committed
  3e25251

  Copy full SHA for 3e25251

• [SPARK-23218][SQL] simplify ColumnVector.getArray

  Show description for dd8e257
  

  cloud-fan

  authored and

  gatorsmile

  committed
  dd8e257

  Copy full SHA for dd8e257

• Revert "[SPARK-22797][PYSPARK] Bucketizer support multi-column"

  Show description for a8a3e9b
  

  Nick Pentreath

  committed
  a8a3e9b

  Copy full SHA for a8a3e9b

• [SPARK-23207][SQL] Shuffle+Repartition on a DataFrame could lead to incorrect answers

  Show description for 94c67a7
  

  jiangxb1987

  authored and

  sameeragarwal

  committed
  94c67a7

  Copy full SHA for 94c67a7

Commits on Jan 27, 2018

• [SPARK-23242][SS][TESTS] Don't run tests in KafkaSourceSuiteBase twice

  Show description for 0737449
  zsxwing

  committed
  0737449

  Copy full SHA for 0737449

• [SPARK-23214][SQL] cached data should not carry extra hint info

  Show description for 5b5447c
  

  cloud-fan

  authored and

  gatorsmile

  committed
  5b5447c

  Copy full SHA for 5b5447c

• [MINOR][SS][DOC] Fix Trigger Scala/Java doc examples

  Show description for e7bc9f0
  

  dongjoon-hyun

  authored and

  srowen

  committed
  e7bc9f0

  Copy full SHA for e7bc9f0

• [SPARK-23245][SS][TESTS] Don't access lastExecution.executedPlan in StreamTest

  Show description for 6328868
  

  jose-torres

  authored and

  zsxwing

  committed
  6328868

  Copy full SHA for 6328868

• [SPARK-23233][PYTHON] Reset the cache in asNondeterministic to set deterministic properly

  Show description for 3227d14
  

  HyukjinKwon

  authored and

  gatorsmile

  committed
  3227d14

  Copy full SHA for 3227d14

Commits on Jan 28, 2018

• [SPARK-23248][PYTHON][EXAMPLES] Relocate module docstrings to the top in PySpark examples

  Show description for b8c32dc
  HyukjinKwon

  committed
  b8c32dc

  Copy full SHA for b8c32dc

• [SPARK-23166][ML] Add maxDF Parameter to CountVectorizer

  Show description for c40fda9
  

  ymazari

  authored and

  srowen

  committed
  c40fda9

  Copy full SHA for c40fda9

• [SPARK-23250][DOCS] Typo in JavaDoc/ScalaDoc for DataFrameWriter

  Show description for 686a622
  

  clchen28

  authored and

  srowen

  committed
  686a622

  Copy full SHA for 686a622

Commits on Jan 29, 2018

• [SPARK-23196] Unify continuous and microbatch V2 sinks

  Show description for 49b0207
  

  jose-torres

  authored and

  cloud-fan

  committed
  49b0207

  Copy full SHA for 49b0207

• [SPARK-23238][SQL] Externalize SQLConf configurations exposed in documentation

  Show description for 39d2c6b
  HyukjinKwon

  committed
  39d2c6b

  Copy full SHA for 39d2c6b

• [SPARK-23219][SQL] Rename ReadTask to DataReaderFactory in data source v2

  Show description for badf0d0
  

  gengliangwang

  authored and

  cloud-fan

  committed
  badf0d0

  Copy full SHA for badf0d0

• [SPARK-23199][SQL] improved Removes repetition from group expressions in Aggregate

  Show description for 54dd7cf
  

  heary-cao

  authored and

  gatorsmile

  committed
  54dd7cf

  Copy full SHA for 54dd7cf

• [SPARK-23059][SQL][TEST] Correct some improper with view related method usage

  Show description for fbce2ed
  

  xubo245

  authored and

  gatorsmile

  committed
  fbce2ed

  Copy full SHA for fbce2ed

• [SPARK-23223][SQL] Make stacking dataset transforms more performant

  Show description for 2d903cf
  hvanhovell

  committed
  2d903cf

  Copy full SHA for 2d903cf

• [SPARK-22221][DOCS] Adding User Documentation for Arrow

  Show description for 0d60b32
  

  BryanCutler

  authored and

  gatorsmile

  committed
  0d60b32

  Copy full SHA for 0d60b32

• [SPARK-22916][SQL][FOLLOW-UP] Update the Description of Join Selection

  Show description for e30b34f
  gatorsmile

  committed
  e30b34f

  Copy full SHA for e30b34f

• [SPARK-23209][core] Allow credential manager to work when Hive not available.

  Show description for b834446
  

  Marcelo Vanzin

  authored and

  squito

  committed
  b834446

  Copy full SHA for b834446

Commits on Jan 30, 2018

• [SPARK-22221][SQL][FOLLOWUP] Externalize spark.sql.execution.arrow.maxRecordsPerBatch

  Show description for f235df6
  

  BryanCutler

  authored and

  gatorsmile

  committed
  f235df6

  Copy full SHA for f235df6

• [SPARK-23088][CORE] History server not showing incomplete/running applications

  Show description for 31bd1da
  

  Paul Mackles

  authored and

  jerryshao

  committed
  31bd1da

  Copy full SHA for 31bd1da

• [SPARK-23207][SQL][FOLLOW-UP] Don't perform local sort for DataFrame.repartition(1)

  Show description for b375397
  

  jiangxb1987

  authored and

  cloud-fan

  committed
  b375397

  Copy full SHA for b375397

• [SPARK-23157][SQL] Explain restriction on column expression in withColumn()

  Show description for 8b98324
  

  Henry Robinson

  authored and

  gatorsmile

  committed
  8b98324

  Copy full SHA for 8b98324

• [SPARK-23138][ML][DOC] Multiclass logistic regression summary example and user guide

  Show description for 5056877
  

  sethah

  authored and

  Nick Pentreath

  committed
  5056877

  Copy full SHA for 5056877

• [SPARK-23260][SPARK-23262][SQL] several data source v2 naming cleanup

  Show description for 0a9ac02
  cloud-fan

  committed
  0a9ac02

  Copy full SHA for 0a9ac02

• [SPARK-23261][PYSPARK] Rename Pandas UDFs

  Show description for 7a2ada2
  

  gatorsmile

  authored and

  HyukjinKwon

  committed
  7a2ada2

  Copy full SHA for 7a2ada2

• [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

  Show description for 84bcf9d
  

  viirya

  authored and

  cloud-fan

  committed
  84bcf9d

  Copy full SHA for 84bcf9d

• [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*.py to .gitignore file.

  Show description for a23187f
  

  ueshin

  authored and

  HyukjinKwon

  committed
  a23187f

  Copy full SHA for a23187f

• [SPARK-23267][SQL] Increase spark.sql.codegen.hugeMethodLimit to 65535

  Show description for 31c00ad
  gatorsmile

  committed
  31c00ad

  Copy full SHA for 31c00ad

• [SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM

  Show description for 58fcb5a
  

  dilipbiswal

  authored and

  gatorsmile

  committed
  58fcb5a

  Copy full SHA for 58fcb5a

• [MINOR] Fix typos in dev/* scripts.

  Show description for 9623a98
  

  ashashwat

  authored and

  HyukjinKwon

  committed
  9623a98

  Copy full SHA for 9623a98

Commits on Jan 31, 2018

• [SPARK-23276][SQL][TEST] Enable UDT tests in (Hive)OrcHadoopFsRelationSuite

  Show description for 7786616
  

  dongjoon-hyun

  authored and

  gatorsmile

  committed
  7786616

  Copy full SHA for 7786616

• [SPARK-23274][SQL] Fix ReplaceExceptWithFilter when the right's Filter contains the references that are not in the left output

  Show description for ca04c3f
  gatorsmile

  committed
  ca04c3f

  Copy full SHA for ca04c3f

• [SPARK-23279][SS] Avoid triggering distributed job for Console sink

  Show description for 8c6a9c9
  jerryshao

  committed
  8c6a9c9

  Copy full SHA for 8c6a9c9

• [SPARK-23272][SQL] add calendar interval type support to ColumnVector

  Show description for 695f714
  cloud-fan

  committed
  695f714

  Copy full SHA for 695f714

• [SPARK-23112][DOC] Update ML migration guide with breaking and behavior changes.

  Show description for 161a3f2
  

  Nick Pentreath

  committed
  161a3f2

  Copy full SHA for 161a3f2

• [SPARK-23228][PYSPARK] Add Python Created jsparkSession to JVM's defaultSession

  Show description for 3d0911b
  

  jerryshao

  authored and

  HyukjinKwon

  committed
  3d0911b

  Copy full SHA for 3d0911b

• revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSet

  Show description for 48dd6a4
  cloud-fan

  committed
  48dd6a4

  Copy full SHA for 48dd6a4

• [SPARK-23249][SQL] Improved block merging logic for partitions

  Show description for 8c21170
  

  Glen Takahashi

  authored and

  cloud-fan

  committed
  8c21170

  Copy full SHA for 8c21170

• [SPARK-21525][STREAMING] Check error code from supervisor RPC.

  Show description for dd242ba
  

  Marcelo Vanzin

  committed
  dd242ba

  Copy full SHA for dd242ba

• [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

  Show description for 9ff1d96
  

  dilipbiswal

  authored and

  gatorsmile

  committed
  9ff1d96

  Copy full SHA for 9ff1d96

Commits on Feb 1, 2018

• [SPARK-23157][SQL][FOLLOW-UP] DataFrame -> SparkDataFrame in R comment

  Show description for f470df2
  

  Henry Robinson

  authored and

  HyukjinKwon

  committed
  f470df2

  Copy full SHA for f470df2

• [SPARK-23280][SQL] add map type support to ColumnVector

  Show description for 52e00f7
  cloud-fan

  committed
  52e00f7

  Copy full SHA for 52e00f7

• [SPARK-23247][SQL] combines Unsafe operations and statistics operations in Scan Data Source

  Show description for 2ac895b
  

  heary-cao

  authored and

  cloud-fan

  committed
  2ac895b

  Copy full SHA for 2ac895b

• [SPARK-23268][SQL] Reorganize packages in data source V2

  Show description for 56ae326
  

  gengliangwang

  authored and

  gatorsmile

  committed
  56ae326

  Copy full SHA for 56ae326

• [SPARK-21396][SQL] Fixes MatchError when UDTs are passed through Hive Thriftserver

  Show description for b2e7677
  

  atallahhezbor

  authored and

  gatorsmile

  committed
  b2e7677

  Copy full SHA for b2e7677

• [SPARK-23188][SQL] Make vectorized columar reader batch size configurable

  Show description for cc41245
  

  jiangxb1987

  authored and

  cloud-fan

  committed
  cc41245

  Copy full SHA for cc41245

• [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFrame().

  Show description for b6b50ef
  

  jiangxb1987

  authored and

  gatorsmile

  committed
  b6b50ef

  Copy full SHA for b6b50ef

• Revert "[SPARK-23200] Reset Kubernetes-specific config on Checkpoint restore"

  Show description for 4b7cd47
  jerryshao

  committed
  4b7cd47

  Copy full SHA for 4b7cd47

• [SPARK-22274][PYTHON][SQL][FOLLOWUP] Use assertRaisesRegexp instead of assertRaisesRegex.

  Show description for 07cee33
  

  ueshin

  authored and

  gatorsmile

  committed
  07cee33

  Copy full SHA for 07cee33

• [SPARK-23107][ML] ML 2.3 QA: New Scala APIs, docs.

  Show description for e15da5b
  

  yanboliang

  authored and

  Nick Pentreath

  committed
  e15da5b

  Copy full SHA for e15da5b

• [SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues.

  Show description for 8bb70b0
  ueshin

  committed
  8bb70b0

  Copy full SHA for 8bb70b0

• [SPARK-23280][SQL][FOLLOWUP] Enable MutableColumnarRow.getMap().

  Show description for 89e8d55
  ueshin

  committed
  89e8d55

  Copy full SHA for 89e8d55

• [SPARK-23202][SQL] Add new API in DataSourceWriter: onDataWriterCommit

  Show description for ffbca84
  

  gengliangwang

  authored and

  cloud-fan

  committed
  ffbca84

  Copy full SHA for ffbca84

• [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully

  Show description for ec63e2d
  

  zsxwing

  authored and

  cloud-fan

  committed
  ec63e2d

  Copy full SHA for ec63e2d

• [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--hiveconf" and ''--hivevar" variables since 2.0

  Show description for f051f83
  

  wangyum

  authored and

  gatorsmile

  committed
  f051f83

  Copy full SHA for f051f83

• [SPARK-23293][SQL] fix data source v2 self join

  Show description for 73da3b6
  

  cloud-fan

  authored and

  gatorsmile

  committed
  73da3b6

  Copy full SHA for 73da3b6

• [INFRA] Close stale PRs.

  Show description for 4bcfdef
  

  Marcelo Vanzin

  committed
  4bcfdef

  Copy full SHA for 4bcfdef

• [SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic

  Show description for 032c11b
  

  gerashegalov

  authored and

  Marcelo Vanzin

  committed
  032c11b

  Copy full SHA for 032c11b

Commits on Feb 2, 2018

• [SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot

  Show description for 90848d5
  

  viirya

  authored and

  cloud-fan

  committed
  90848d5

  Copy full SHA for 90848d5

• [SPARK-23020][CORE] Fix another race in the in-process launcher test.

  Show description for 969eda4
  

  Marcelo Vanzin

  authored and

  cloud-fan

  committed
  969eda4

  Copy full SHA for 969eda4

• [SPARK-23306] Fix the oom caused by contention

  Show description for b3a0428
  

  Zhan Zhang

  authored and

  cloud-fan

  committed
  b3a0428

  Copy full SHA for b3a0428

• [SPARK-23301][SQL] data source column pruning should work for arbitrary expressions

  Show description for 19c7c7e
  

  cloud-fan

  authored and

  gatorsmile

  committed
  19c7c7e

  Copy full SHA for 19c7c7e

• [SPARK-23312][SQL] add a config to turn off vectorized cache reader

  Show description for b9503fc
  cloud-fan

  committed
  b9503fc

  Copy full SHA for b9503fc

• [SPARK-23253][CORE][SHUFFLE] Only write shuffle temporary index file when there is not an existing one

  Show description for dd52681
  

  yaooqinn

  authored and

  squito

  committed
  dd52681

  Copy full SHA for dd52681

• [SPARK-23295][BUILD][MINOR] Exclude Waring message when generating versions in make-distribution.sh

  Show description for eefec93
  

  yaooqinn

  authored and

  srowen

  committed
  eefec93

  Copy full SHA for eefec93

Commits on Feb 3, 2018

• [SPARK-23064][SS][DOCS] Stream-stream joins Documentation - follow up

  Show description for eaf35de
  tdas

  committed
  eaf35de

  Copy full SHA for eaf35de

• [SQL] Minor doc update: Add an example in DataFrameReader.schema

  Show description for 3ff83ad
  

  rxin

  authored and

  gatorsmile

  committed
  3ff83ad

  Copy full SHA for 3ff83ad

• [SPARK-23317][SQL] rename ContinuousReader.setOffset to setStartOffset

  Show description for fe73cb4
  

  cloud-fan

  authored and

  gatorsmile

  committed
  fe73cb4

  Copy full SHA for fe73cb4

• [SPARK-23311][SQL][TEST] add FilterFunction test case for test CombineTypedFilters

  Show description for 63b49fa
  

  heary-cao

  authored and

  gatorsmile

  committed
  63b49fa

  Copy full SHA for 63b49fa

• [SPARK-23305][SQL][TEST] Test spark.sql.files.ignoreMissingFiles for all file-based data sources

  Show description for 522e0b1
  

  dongjoon-hyun

  authored and

  gatorsmile

  committed
  522e0b1

  Copy full SHA for 522e0b1

• [MINOR][DOC] Use raw triple double quotes around docstrings where there are occurrences of backslashes.

  Show description for 4aaa7d4
  

  ashashwat

  authored and

  gatorsmile

  committed
  4aaa7d4

  Copy full SHA for 4aaa7d4

• [SPARK-21658][SQL][PYSPARK] Revert "[] Add default None for value in na.replace in PySpark"

  Show description for 551dff2
  

  HyukjinKwon

  authored and

  gatorsmile

  committed
  551dff2

  Copy full SHA for 551dff2

Commits on Feb 4, 2018

• [SPARK-23256][ML][PYTHON] Add columnSchema method to PySpark image reader

  Show description for 715047b
  HyukjinKwon

  committed
  715047b

  Copy full SHA for 715047b

• [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperations.sql

  Show description for 6fb3fd1
  

  wangyum

  authored and

  gatorsmile

  committed
  6fb3fd1

  Copy full SHA for 6fb3fd1

Commits on Feb 5, 2018

• [SPARK-23307][WEBUI] Sort jobs/stages/tasks/queries with the completed timestamp before cleaning up them

  Show description for a6bf3db
  

  zsxwing

  authored and

  cloud-fan

  committed
  a6bf3db

  Copy full SHA for a6bf3db

• [SPARK-23310][CORE] Turn off read ahead input stream for unshafe shuffle reader

  Show description for 03b7e12
  

  Sital Kedia

  authored and

  sameeragarwal

  committed
  03b7e12

  Copy full SHA for 03b7e12

• [SPARK-23330][WEBUI] Spark UI SQL executions page throws NPE

  Show description for c2766b0
  

  jiangxb1987

  authored and

  Marcelo Vanzin

  committed
  c2766b0

  Copy full SHA for c2766b0

Commits on Feb 6, 2018

• [SPARK-23326][WEBUI] schedulerDelay should return 0 when the task is running

  Show description for f3f1e14
  

  zsxwing

  authored and

  cloud-fan

  committed
  f3f1e14

  Copy full SHA for f3f1e14

• [SPARK-23290][SQL][PYTHON] Use datetime.date for date type when converting Spark DataFrame to Pandas DataFrame.

  Show description for a24c031
  

  ueshin

  authored and

  cloud-fan

  committed
  a24c031

  Copy full SHA for a24c031

• [SPARK-23300][TESTS] Prints out if Pandas and PyArrow are installed or not in PySpark SQL tests

  Show description for 8141c3e
  HyukjinKwon

  committed
  8141c3e

  Copy full SHA for 8141c3e

• [SPARK-23334][SQL][PYTHON] Fix pandas_udf with return type StringType() to handle str type properly in Python 2.

  Show description for 63c5bf1
  

  ueshin

  authored and

  HyukjinKwon

  committed
  63c5bf1

  Copy full SHA for 63c5bf1

• [SPARK-23310][CORE][FOLLOWUP] Fix Java style check issues.

  Show description for 7db9979
  

  ueshin

  authored and

  Marcelo Vanzin

  committed
  7db9979

  Copy full SHA for 7db9979

• [SPARK-23312][SQL][FOLLOWUP] add a config to turn off vectorized cache reader

  Show description for ac7454c
  

  cloud-fan

  authored and

  gatorsmile

  committed
  ac7454c

  Copy full SHA for ac7454c

• [MINOR][TEST] Fix class name for Pandas UDF tests

  Show description for caf3044
  

  icexelloss

  authored and

  gatorsmile

  committed
  caf3044

  Copy full SHA for caf3044

• [SPARK-23315][SQL] failed to get output from canonicalized data source v2 related plans

  Show description for b96a083
  

  cloud-fan

  authored and

  gatorsmile

  committed
  b96a083

  Copy full SHA for b96a083

Commits on Feb 7, 2018

• [SPARK-23327][SQL] Update the description and tests of three external API or functions

  Show description for c36fecc
  gatorsmile

  committed
  c36fecc

  Copy full SHA for c36fecc

• [SPARK-23122][PYSPARK][FOLLOWUP] Replace registerTempTable by createOrReplaceTempView

  Show description for 9775df6
  

  gatorsmile

  authored and

  HyukjinKwon

  committed
  9775df6

  Copy full SHA for 9775df6

• [SPARK-23319][TESTS] Explicitly specify Pandas and PyArrow versions in PySpark tests (to skip or test)

  Show description for 71cfba0
  HyukjinKwon

  committed
  71cfba0

  Copy full SHA for 71cfba0

• [SPARK-23345][SQL] Remove open stream record even closing it fails

  Show description for 9841ae0
  

  viirya

  authored and

  gatorsmile

  committed
  9841ae0

  Copy full SHA for 9841ae0

• [SPARK-23092][SQL] Migrate MemoryStream to DataSourceV2 APIs

  Show description for 30295bf
  tdas

  committed
  30295bf

  Copy full SHA for 30295bf

Commits on Feb 8, 2018

• [SPARK-23319][TESTS][FOLLOWUP] Fix a test for Python 3 without pandas.

  Show description for a62f30d
  

  ueshin

  authored and

  HyukjinKwon

  committed
  a62f30d

  Copy full SHA for a62f30d

• Revert [SPARK-22279][SQL] Turn on spark.sql.hive.convertMetastoreOrc by default

  Show description for 3473fda
  

  gatorsmile

  authored and

  cloud-fan

  committed
  3473fda

  Copy full SHA for 3473fda

• [SPARK-23348][SQL] append data using saveAsTable should adjust the data types

  Show description for 7f5f5fb
  

  cloud-fan

  authored and

  gatorsmile

  committed
  7f5f5fb

  Copy full SHA for 7f5f5fb

• [SPARK-23268][SQL][FOLLOWUP] Reorganize packages in data source V2

  Show description for a75f927
  cloud-fan

  committed
  a75f927

  Copy full SHA for a75f927

• [SPARK-21860][CORE] Improve memory reuse for heap memory in HeapMemoryAllocator

  Show description for 76e019d
  

  10110346

  authored and

  cloud-fan

  committed
  76e019d

  Copy full SHA for 76e019d

• [SPARK-23336][BUILD] Upgrade snappy-java to 1.1.7.1

  Show description for 4df84c3
  

  wangyum

  authored and

  srowen

  committed
  4df84c3

  Copy full SHA for 4df84c3