Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MINOR] fix(client/netty): ShuffleServerGrpcNettyClient missing to send shuffleId and partitionIds for requirePreAllocation request (#1913) #2053

Merged
merged 1 commit into from
Aug 15, 2024

Conversation

maobaolong
Copy link
Member

What changes were proposed in this pull request?

Add partitionIds and shuffleId to RequireBufferRequest.

Why are the changes needed?

Without this changes, server cannot check limitHugePartition.

  public long requireBuffer(
      String appId, int shuffleId, List<Integer> partitionIds, int requireSize) {
    ShuffleTaskInfo shuffleTaskInfo = shuffleTaskInfos.get(appId);
    if (null == shuffleTaskInfo) {
      LOG.error("No such app is registered. appId: {}, shuffleId: {}", appId, shuffleId);
      throw new NoRegisterException("No such app is registered. appId: " + appId);
    }
    for (int partitionId : partitionIds) {
      long partitionUsedDataSize = getPartitionDataSize(appId, shuffleId, partitionId);
      if (shuffleBufferManager.limitHugePartition(
          appId, shuffleId, partitionId, partitionUsedDataSize)) {
        String errorMessage =
            String.format(
                "Huge partition is limited to writing. appId: %s, shuffleId: %s, partitionIds: %s, partitionUsedDataSize: %s",
                appId, shuffleId, partitionIds, partitionUsedDataSize);
        LOG.error(errorMessage);
        throw new NoBufferForHugePartitionException(errorMessage);
      }
    }
    return requireBuffer(appId, requireSize);
  }

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Test on local, start a rss cluster with netty, specific a small huge partition size, you can see NoBufferForHugePartitionException

(cherry picked from commit 7731998)

What changes were proposed in this pull request?

(Please outline the changes and how this PR fixes the issue.)

Why are the changes needed?

(Please clarify why the changes are needed. For instance,

  1. If you propose a new API, clarify the use case for a new API.
  2. If you fix a bug, describe the bug.)

Fix: # (issue)

Does this PR introduce any user-facing change?

(Please list the user-facing changes introduced by your change, including

  1. Change in user-facing APIs.
  2. Addition or removal of property keys.)

No.

How was this patch tested?

(Please test your changes, and provide instructions on how to test it:

  1. If you add a feature or fix a bug, add a test to cover your changes.
  2. If you fix a flaky test, repeat it for many times to prove it works.)

…nd shuffleId and partitionIds for requirePreAllocation request (apache#1913)

### What changes were proposed in this pull request?

Add partitionIds and shuffleId to `RequireBufferRequest`.

### Why are the changes needed?

Without this changes, server cannot check limitHugePartition.

```java
  public long requireBuffer(
      String appId, int shuffleId, List<Integer> partitionIds, int requireSize) {
    ShuffleTaskInfo shuffleTaskInfo = shuffleTaskInfos.get(appId);
    if (null == shuffleTaskInfo) {
      LOG.error("No such app is registered. appId: {}, shuffleId: {}", appId, shuffleId);
      throw new NoRegisterException("No such app is registered. appId: " + appId);
    }
    for (int partitionId : partitionIds) {
      long partitionUsedDataSize = getPartitionDataSize(appId, shuffleId, partitionId);
      if (shuffleBufferManager.limitHugePartition(
          appId, shuffleId, partitionId, partitionUsedDataSize)) {
        String errorMessage =
            String.format(
                "Huge partition is limited to writing. appId: %s, shuffleId: %s, partitionIds: %s, partitionUsedDataSize: %s",
                appId, shuffleId, partitionIds, partitionUsedDataSize);
        LOG.error(errorMessage);
        throw new NoBufferForHugePartitionException(errorMessage);
      }
    }
    return requireBuffer(appId, requireSize);
  }
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Test on local, start a rss cluster with netty, specific a small huge partition size, you can see NoBufferForHugePartitionException

(cherry picked from commit 7731998)
@maobaolong
Copy link
Member Author

@zhengchenyu Would you like to take a look at this PR?

@zhengchenyu
Copy link
Collaborator

LGTM+1

Copy link

github-actions bot commented Aug 15, 2024

Test Results

 2 370 files  ±0   2 370 suites  ±0   4h 40m 19s ⏱️ +32s
   918 tests ±0     917 ✅ ±0   1 💤 ±0  0 ❌ ±0 
10 655 runs  ±0  10 641 ✅ ±0  14 💤 ±0  0 ❌ ±0 

Results for commit 5815a86. ± Comparison against base commit 664c4f2.

♻️ This comment has been updated with latest results.

@zhengchenyu zhengchenyu merged commit 19fdc29 into apache:branch-0.9 Aug 15, 2024
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants