Code

SPARK-2387: remove stage barrier#1328

Code

Closed

lirui-apache wants to merge 91 commits intoapache:masterapache/spark:masterfrom lirui-apache:removeStageBarrierlirui-apache/spark:removeStageBarrierCopy head branch name to clipboard

+569-153

Commits

Commits on May 5, 2014

• minor fix

  Show description for 163302d
  

  lirui

  committed
  163302d

  Copy full SHA for 163302d

Commits on May 7, 2014

• Merge branch 'master' of https://github.com/lirui-intel/spark

  

  lirui

  committed
  f81476d

  Copy full SHA for f81476d

Commits on May 8, 2014

• try to locate the point to remove the barrier

  

  lirui

  committed
  3124380

  Copy full SHA for 3124380

• apply upstream hot fix

  

  lirui

  committed
  8e625c0

  Copy full SHA for 8e625c0

Commits on May 9, 2014

• RemoveStageBarrier: support partial map outputs

  

  lirui

  committed
  1d5d0f0

  Copy full SHA for 1d5d0f0

Commits on May 11, 2014

• RemoveStageBarrier: build fix

  

  lirui

  committed
  c4f4054

  Copy full SHA for c4f4054

• RemoveStageBarrier: register map outputs progressively

  

  lirui

  committed
  444d2d9

  Copy full SHA for 444d2d9

Commits on May 12, 2014

• RemoveStageBarrier: increment epoch for progressive registration

  

  lirui

  committed
  2df1d4e

  Copy full SHA for 2df1d4e

• RemoveStageBarrier: fix check free CPUs

  

  lirui

  committed
  9f18dc7

  Copy full SHA for 9f18dc7

Commits on May 13, 2014

• RemoveStageBarrier: make reducers refresh map outputs less often

  

  lirui

  committed
  7af23c0

  Copy full SHA for 7af23c0

• RemoveStageBarrier: start reducers earlier

  

  lirui

  committed
  9a32a17

  Copy full SHA for 9a32a17

Commits on May 14, 2014

• RemoveStageBarrier: add log info

  

  lirui

  committed
  9ffb208

  Copy full SHA for 9ffb208

• RemoveStageBarrier: adjust sleep interval

  

  lirui

  committed
  ef3b043

  Copy full SHA for ef3b043

Commits on May 15, 2014

• RemoveStageBarrier: add a new iterator to manage partial map outputs

  

  lirui

  committed
  4213d63

  Copy full SHA for 4213d63

Commits on May 16, 2014

• RemoveStageBarrier: minor fixes

  

  lirui

  committed
  376230a

  Copy full SHA for 376230a

• RemoveStageBarrier: fix: reducers may fail due to very slow mappers

  

  lirui

  committed
  efd31ef

  Copy full SHA for efd31ef

• RemoveStageBarrier: add some log info

  

  lirui

  committed
  3cb944c

  Copy full SHA for 3cb944c

Commits on May 19, 2014

• RemoveStageBarrier: stage with a bigger ID should take precedence

  

  lirui

  committed
  641715e

  Copy full SHA for 641715e

Commits on May 23, 2014

• RemoveStageBarrier: track whether map output for a shuffle is partial in MapOutputTracker

  

  lirui

  committed
  b0c2df2

  Copy full SHA for b0c2df2

• RemoveStageBarrier: refine how we get the stage to pre-start

  

  lirui

  committed
  75d2744

  Copy full SHA for 75d2744

• RemoveStageBarrier: indicate the output is partial for progressive registration

  

  lirui

  committed
  b7f1f84

  Copy full SHA for b7f1f84

Commits on May 26, 2014

• add some debug info

  

  lirui

  committed
  be47408

  Copy full SHA for be47408

• add a new locality level for tasks with no preferred locations

  

  lirui

  committed
  c88014b

  Copy full SHA for c88014b

• re-compute pending list when new executor is added

  

  lirui

  committed
  133a356

  Copy full SHA for 133a356

• pendingTasksWithNoPrefs should only contain tasks that really have no preferred locations

  

  lirui

  committed
  7d92f9a

  Copy full SHA for 7d92f9a

Commits on May 27, 2014

• make the delay schedule configurable

  

  lirui

  committed
  c1de426

  Copy full SHA for c1de426

• clean up

  

  lirui

  committed
  e57e081

  Copy full SHA for e57e081

• do some refactor

  

  lirui

  committed
  fda0281

  Copy full SHA for fda0281

Commits on May 28, 2014

• RemoveStageBarrier: fix problem with consolidated shuffle file

  

  lirui

  committed
  781861d

  Copy full SHA for 781861d

Commits on May 29, 2014

• RemoveStageBarrier: should fail the pre-started stages if the parent stage gets re-submitted

  

  lirui

  committed
  679813b

  Copy full SHA for 679813b

• RemoveStageBarrier: fix issue with empty shuffle blocks

  

  lirui

  committed
  563d743

  Copy full SHA for 563d743

• RemoveStageBarrier: allow partial map output by default

  

  lirui

  committed
  2ab311e

  Copy full SHA for 2ab311e

• RemoveStageBarrier: make sure the feature is enabled before we use partial fetcher iterator

  

  lirui

  committed
  46da965

  Copy full SHA for 46da965

Commits on May 30, 2014

• RemoveStageBarrier: partialForShuffle may cause infinite loop

  

  lirui

  committed
  a89c93f

  Copy full SHA for a89c93f

• RemoveStageBarrier: cannot only depend on epoch to determine if the local map statuses are outdated

  

  lirui

  committed
  5cfbae8

  Copy full SHA for 5cfbae8

• RemoveStageBarrier: fix bug

  

  lirui

  committed
  2df5939

  Copy full SHA for 2df5939

Commits on Jun 3, 2014

• RemoveStageBarrier: adjust fetching order of CoGroupedRDD

  

  lirui

  committed
  6891d58

  Copy full SHA for 6891d58

Commits on Jun 4, 2014

• merge upstream master

  

  Rui Li

  committed
  104ebe3

  Copy full SHA for 104ebe3

• RemoveStageBarrier: make sure pre-started stage has lower priority to run

  

  lirui

  committed
  5dd28dc

  Copy full SHA for 5dd28dc

Commits on Jun 9, 2014

• RemoveStageBarrier: revert previous changes to CoGroupedRDD

  

  lirui

  committed
  6cdf2a3

  Copy full SHA for 6cdf2a3

Commits on Jun 11, 2014

• RemoveStageBarrier: sleep less waiting for new map outputs

  

  lirui

  committed
  af000f7

  Copy full SHA for af000f7

Commits on Jun 12, 2014

• RemoveStageBarrier: don't rely on epoch for updated map statuses

  

  lirui

  committed
  8349424

  Copy full SHA for 8349424

Commits on Jun 13, 2014

• RemoveStageBarrier: add a proxy to update partial map outputs periodically

  

  lirui

  committed
  28679b9

  Copy full SHA for 28679b9

• RemoveStageBarrier: remove verbose logs

  

  lirui

  committed
  71b87e5

  Copy full SHA for 71b87e5

Commits on Jun 16, 2014

• RemoveStageBarrier: don't increase epoch for partial map output registration

  

  lirui

  committed
  04f17e8

  Copy full SHA for 04f17e8

• RemoveStageBarrier: don't put partial outputs in cache

  

  lirui

  committed
  eafa476

  Copy full SHA for eafa476

Commits on Jun 17, 2014

• RemoveStageBarrier: block reducers waiting for new map outputs

  

  lirui

  committed
  7547686

  Copy full SHA for 7547686

• RemoveStageBarrier: bug fix

  

  lirui

  committed
  ca83d19

  Copy full SHA for ca83d19

• RemoveStageBarrier: add API to SchedulerBackend to tell if there's free slots

  

  lirui

  committed
  539f1a8

  Copy full SHA for 539f1a8

Commits on Jun 18, 2014

• RemoveStageBarrier: refine logs

  

  lirui

  committed
  6e10488

  Copy full SHA for 6e10488

• RemoveStageBarrier: fix the way we compute free slots

  

  lirui

  committed
  a48d592

  Copy full SHA for a48d592

Commits on Jun 23, 2014

• RemoveStageBarrier: when a task finishes, launch new tasks before populating the status update event

  

  lirui

  committed
  a418f03

  Copy full SHA for a418f03

Commits on Jun 24, 2014

• RemoveStageBarrier: make offer after successful/failed task is properly handled

  

  lirui

  committed
  3ced2bb

  Copy full SHA for 3ced2bb

Commits on Jun 25, 2014

• RemoveStageBarrier: handle failed task in a synchronized manner

  

  lirui

  committed
  5b0031a

  Copy full SHA for 5b0031a

Commits on Jun 27, 2014

• RemoveStageBarrier: add temp test code to detect deadlock

  

  lirui

  committed
  3c52c69

  Copy full SHA for 3c52c69

• RemoveStageBarrier: maintain support for asynchronous handling failed task

  

  lirui

  committed
  0473e3b

  Copy full SHA for 0473e3b

• RemoveStageBarrier: fix previously found problem

  

  lirui

  committed
  d267c9b

  Copy full SHA for d267c9b

• RemoveStageBarrier: fix test code

  

  lirui

  committed
  118914b

  Copy full SHA for 118914b

• RemoveStageBarrier: remove temp code

  

  lirui

  committed
  fe63024

  Copy full SHA for fe63024

• RemoveStageBarrier: add temp test code

  

  lirui

  committed
  a996c77

  Copy full SHA for a996c77

• RemoveStageBarrier: fix shuffle map stage fail over

  

  lirui

  committed
  cef517b

  Copy full SHA for cef517b

Commits on Jun 30, 2014

• RemoveStageBarrier: kill running tasks when resubmit failed stages

  

  lirui

  committed
  7d9a4a4

  Copy full SHA for 7d9a4a4

• RemoveStageBarrier: refine temp test code

  

  lirui

  committed
  5697b98

  Copy full SHA for 5697b98

• RemoveStageBarrier: fix test code

  

  lirui

  committed
  4f80b1d

  Copy full SHA for 4f80b1d

• RemoveStageBarrier: remove temp code

  

  lirui

  committed
  39ddb9d

  Copy full SHA for 39ddb9d

• RemoveStageBarrier: kill running tasks before resubmit failed stages

  

  lirui

  committed
  8cb8e4c

  Copy full SHA for 8cb8e4c

Commits on Jul 1, 2014

• RemoveStageBarrier: add temp test code

  

  lirui

  committed
  bc69fed

  Copy full SHA for bc69fed

• RemoveStageBarrier: fix test code

  

  lirui

  committed
  1e1907d

  Copy full SHA for 1e1907d

• RemoveStageBarrier: handle fetch failed task only if it comes from a running stage

  

  lirui

  committed
  930136d

  Copy full SHA for 930136d

• RemoveStageBarrier: kill tasks without interrupting the thread

  

  lirui

  committed
  b49cbdb

  Copy full SHA for b49cbdb

• RemoveStageBarrier: remove test code

  

  lirui

  committed
  6bcca9b

  Copy full SHA for 6bcca9b

Commits on Jul 2, 2014

• RemoveStageBarrier: use AKKA actor to access DAGScheduler's data structures

  

  lirui

  committed
  8fded0e

  Copy full SHA for 8fded0e

• RemoveStageBarrier: fix bug

  

  lirui

  committed
  aa2e0f2

  Copy full SHA for aa2e0f2

Commits on Jul 3, 2014

• RemoveStageBarrier: compute sorted task sets without holding a lock on TaskSchedulerImpl

  

  lirui

  committed
  0bbdb5d

  Copy full SHA for 0bbdb5d

• RemoveStageBarrier: make the updater sleep a little longer if maps are finishing slowly

  

  lirui

  committed
  d941899

  Copy full SHA for d941899

• RemoveStageBarrier: fix bug

  

  lirui

  committed
  12b8093

  Copy full SHA for 12b8093

Commits on Jul 4, 2014

• Revert "RemoveStageBarrier: fix bug"

  Show description for c74a876
  

  lirui

  committed
  c74a876

  Copy full SHA for c74a876

• RemoveStageBarrier: pre-start a stage if all of its parents' tasks have been launched

  

  lirui

  committed
  c313fe0

  Copy full SHA for c313fe0

• RemoveStageBarrier: code refactor

  

  lirui

  committed
  033ffc0

  Copy full SHA for 033ffc0

• RemoveStageBarrier: add some log

  

  lirui

  committed
  8a08a6c

  Copy full SHA for 8a08a6c

Commits on Jul 7, 2014

• RemoveStageBarrier: revert change about tracking waiting tasks

  

  lirui

  committed
  a8b5d75

  Copy full SHA for a8b5d75

• RemoveStageBarrier: code cleanup

  

  lirui

  committed
  f66a8eb

  Copy full SHA for f66a8eb

• merge upstream master branch

  

  Rui Li

  committed
  1521fef

  Copy full SHA for 1521fef

• RemoveStageBarrier: fix code style

  

  lirui

  committed
  9747d6b

  Copy full SHA for 9747d6b

Commits on Jul 8, 2014

• RemoveStageBarrier: minor fix

  

  lirui

  committed
  8f798d8

  Copy full SHA for 8f798d8

• RemoveStageBarrier: minor fix

  

  lirui

  committed
  1ab7a15

  Copy full SHA for 1ab7a15

• RemoveStageBarrier: let the reducer wake the updater

  

  lirui

  committed
  8417ffe

  Copy full SHA for 8417ffe

• RemoveStageBarrier: introduce a min interval to update map status

  

  lirui

  committed
  31c4634

  Copy full SHA for 31c4634

• RemoveStageBarrier: fix bug

  

  lirui

  committed
  e1c374c

  Copy full SHA for e1c374c

• RemoveStageBarrier: code clean up

  

  lirui

  committed
  a503508

  Copy full SHA for a503508

Commits on Jul 30, 2014

• fix style

  

  lirui

  committed
  85a5d85

  Copy full SHA for 85a5d85