-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-21253][CORE] Disable use DownloadCallback fetch big blocks #18466
Conversation
@jinxing64 how hard is it to fix this? if it's hard let's just disable it for 2.2 |
Test build #78901 has finished for PR 18466 at commit
|
@wangyum |
@cloud-fan |
Yes, I reproduce it by Yarn cluster, local mode can't reproduce, It seems |
Hi, @wangyum .
|
Sorry to bother you guys, but I'm just wondering if I missed something to see this bug.
|
@dongjoon-hyun I have not yet figured out the root cause of this issue. The major reason to disable it is this feature breaks old shuffle service. |
FYI, I'm fixing the root issue in #18472 |
Wow, great! |
It's great job ! 👍 |
Disable spark.reducer.maxReqSizeShuffleToMem because it breaks the old shuffle service. Credits to wangyum Closes #18466 Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Author: Yuming Wang <wgyumg@gmail.com> Closes #18467 from zsxwing/SPARK-21253. (cherry picked from commit 80f7ac3) Signed-off-by: Wenchen Fan <wenchen@databricks.com>
@dongjoon-hyun Try the following to reproduce, I missed
|
Thank you, @wangyum . I'll try. |
What changes were proposed in this pull request?
DownloadCallback has some issues cause
FetchFailedException
.Spark cluster can reproduce, local can't:
spark.reducer.maxReqSizeShuffleToMem=1K
:The error messages:
Immediately to release 2.2, how about disable
DownloadCallback
to fetch big blocks?How was this patch tested?
manual tests because need a spark cluster