[ISSUE-448][Feature] shuffle server report storage info #449

advancedxy · 2022-12-27T11:20:48Z

What changes were proposed in this pull request?

ShuffleServer reports local storage info about itself.
This PR also defines a general message definition to extend remote distributed info.

Why are the changes needed?

To do better shuffle assignments and get more insight of shuffle server
This addresses #448

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added UTs.

common/src/main/java/org/apache/uniffle/common/storage/LocalStorageInfoUtils.java

codecov-commenter · 2022-12-28T11:16:46Z

Codecov Report

Merging #449 (f7fb582) into master (6ddf8a7) will increase coverage by 0.07%.
The diff coverage is 60.25%.

@@             Coverage Diff              @@
##             master     #449      +/-   ##
============================================
+ Coverage     58.71%   58.78%   +0.07%     
- Complexity     1651     1704      +53     
============================================
  Files           199      206       +7     
  Lines         11214    11471     +257     
  Branches        998     1024      +26     
============================================
+ Hits           6584     6743     +159     
- Misses         4237     4317      +80     
- Partials        393      411      +18

Impacted Files	Coverage Δ
...he/uniffle/coordinator/CoordinatorGrpcService.java	`2.08% <0.00%> (-0.02%)`	⬇️
...che/uniffle/server/storage/HdfsStorageManager.java	`93.75% <0.00%> (-1.49%)`	⬇️
...he/uniffle/server/storage/MultiStorageManager.java	`60.71% <0.00%> (-3.44%)`	⬇️
...e/uniffle/storage/common/StorageMediaProvider.java	`0.00% <0.00%> (ø)`
...org/apache/uniffle/common/storage/StorageInfo.java	`35.18% <35.18%> (ø)`
...g/apache/uniffle/common/storage/StorageStatus.java	`42.85% <42.85%> (ø)`
...rg/apache/uniffle/common/storage/StorageMedia.java	`43.47% <43.47%> (ø)`
...le/storage/common/DefaultStorageMediaProvider.java	`53.12% <53.12%> (ø)`
...he/uniffle/server/storage/LocalStorageManager.java	`85.87% <74.19%> (-2.49%)`	⬇️
...le/server/storage/StorageMediaFromEnvProvider.java	`89.65% <89.65%> (ø)`
... and 17 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

proto/src/main/proto/Rss.proto

zuston · 2023-01-03T08:24:37Z

server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java

@@ -336,6 +336,12 @@ public class ShuffleServerConf extends RssBaseConf {
      .withDescription("Threshold when flushing shuffle data to persistent storage, recommend value would be 256K, "
          + "512K, or even 1M");

+  public static final ConfigOption<String> STORAGE_TYPE_PROVIDER_ENV_KEY = ConfigOptions


Is this an internal config entry? If yes, could you mark it, rename from STORAGE_TYPE_PROVIDER_ENV_KEY to __INTERNAL_STORAGE_TYPE_PROVIDER_ENV_KEY ? WDYT

admin might change it. I'm not sure it falls into the internal config catalog?

do you have other suggestions?

proto/src/main/proto/Rss.proto

zuston

I think this is an initial PR to support better assignment, it's better to list all subtasks, and indicate the progress in the PR title, like [1/n] xxxx, [2/n] xxxxxx.

common/src/main/java/org/apache/uniffle/common/storage/SingleStorageType.java

common/src/main/java/org/apache/uniffle/common/storage/StorageStatus.java

proto/src/main/proto/Rss.proto

zuston · 2023-01-06T09:47:53Z

server/src/main/java/org/apache/uniffle/server/storage/LocalStorageManager.java

@@ -94,19 +101,26 @@ public class LocalStorageManager extends SingleStorageManager {
    // We must make sure the order of `storageBasePaths` and `localStorages` is same, or some unit test may be fail
    CountDownLatch countDownLatch = new CountDownLatch(storageBasePaths.size());
    AtomicInteger successCount = new AtomicInteger();
+    ServiceLoader<StorageTypeProvider> loader = ServiceLoader.load(StorageTypeProvider.class);


Why we need multiple storage type providers? I think one is enough.

I think one is not enough, especially when you need to provide your own provider logic.

Although, the current logic may not handle provider evaluation order properly. But we can modify that later.

I think one is not enough, especially when you need to provide your own provider logic.

Could you help provide some examples about needing multiple providers?

Could you help provide some examples about needing multiple providers?

Just like the DefaultStorageTypeProvider and StorageTypeFromEnvProvider. Some base dir might be indeed is a SSD disk but the default could only return HDD>

I can't agree this, we could use single provider to mark one storage media for one basePath, this is enough.

what if the DefaultStorageMediaProvider cannot provide accurate result?

The whole point of SPI is to provide an extend way to add plugins. So, the service loader must be a list?

server/src/main/java/org/apache/uniffle/server/storage/StorageTypeFromEnvProvider.java

storage/src/main/java/org/apache/uniffle/storage/common/DefaultStorageTypeProvider.java

advancedxy · 2023-01-08T13:29:04Z

I think this is an initial PR to support better assignment, it's better to list all subtasks, and indicate the progress in the PR title, like [1/n] xxxx, [2/n] xxxxxx.

The initial intention for this PR is to support shuffle server with LOCAL HDD disks better. You can see the parent and sub-issues from #448.

However this pr extends its scope a bit. I may open another issue to track this and related issues for better shuffle server assignments.

advancedxy · 2023-01-09T06:27:45Z

PLAT @zuston .

server/src/main/java/org/apache/uniffle/server/storage/StorageManager.java

common/src/main/java/org/apache/uniffle/common/storage/StorageTypePOJO.java

common/pom.xml

common/src/main/java/org/apache/uniffle/common/storage/StorageInfo.java

server/src/main/java/org/apache/uniffle/server/storage/StorageMediaFromEnvProvider.java

coordinator/src/main/java/org/apache/uniffle/coordinator/ServerNode.java

server/src/main/java/org/apache/uniffle/server/ShuffleServerConf.java

jerqi · 2023-01-10T11:06:52Z

If we introduce extra configuration option, we should add the document about this feature to explain how to use this feature.

advancedxy · 2023-01-11T01:38:05Z

If we introduce extra configuration option, we should add the document about this feature to explain how to use this feature.

Done. Please take another look when you have time.

jerqi

LGTM, let @zuston take a look

zuston

LGTM +1

a-li reviewed Dec 28, 2022

View reviewed changes

common/src/main/java/org/apache/uniffle/common/storage/LocalStorageInfoUtils.java Outdated Show resolved Hide resolved

advancedxy force-pushed the issue_448 branch from 170d3bc to adfdbbb Compare December 28, 2022 10:44

advancedxy marked this pull request as ready for review December 28, 2022 10:45

advancedxy changed the title ~~WIP: shuffle server report storage capability~~ [FEATURE] shuffle server report storage capability Dec 28, 2022

advancedxy changed the title ~~[FEATURE] shuffle server report storage capability~~ [Feature] shuffle server report storage capability Dec 28, 2022

advancedxy requested review from zuston and jerqi December 28, 2022 11:46

jerqi reviewed Dec 28, 2022

View reviewed changes

proto/src/main/proto/Rss.proto Outdated Show resolved Hide resolved

zuston reviewed Jan 3, 2023

View reviewed changes

advancedxy changed the title ~~[Feature] shuffle server report storage capability~~ [Feature] shuffle server report storage info Jan 6, 2023

jerqi changed the title ~~[Feature] shuffle server report storage info~~ [ISSUE-448][Feature] shuffle server report storage info Jan 6, 2023

zuston reviewed Jan 6, 2023

View reviewed changes

advancedxy marked this pull request as draft January 9, 2023 02:59

advancedxy commented Jan 9, 2023

View reviewed changes

server/src/main/java/org/apache/uniffle/server/storage/StorageManager.java Outdated Show resolved Hide resolved

advancedxy marked this pull request as ready for review January 9, 2023 08:51

jerqi reviewed Jan 9, 2023

View reviewed changes

common/src/main/java/org/apache/uniffle/common/storage/StorageTypePOJO.java Outdated Show resolved Hide resolved

jerqi linked an issue Jan 9, 2023 that may be closed by this pull request

[Subtask] Shuffle server shall report its local storage capability #448

Closed

3 tasks

[ISSUE-448] shuffle server report storage info

c082db1

advancedxy force-pushed the issue_448 branch from 7dc7766 to c082db1 Compare January 10, 2023 03:35

Fix import order

474ade4

jerqi reviewed Jan 10, 2023

View reviewed changes

common/pom.xml Show resolved Hide resolved

common/src/main/java/org/apache/uniffle/common/storage/StorageInfo.java Show resolved Hide resolved

jerqi reviewed Jan 10, 2023

View reviewed changes

server/src/main/java/org/apache/uniffle/server/storage/StorageMediaFromEnvProvider.java Show resolved Hide resolved

Fix equal of StorageInfo

de72bfb

jerqi reviewed Jan 10, 2023

View reviewed changes

advancedxy added 2 commits January 10, 2023 21:01

address comments.

36f0c83

address comments.

6a297e7

Fix test failure

f7fb582

jerqi approved these changes Jan 11, 2023

View reviewed changes

zuston approved these changes Jan 11, 2023

View reviewed changes

jerqi merged commit 19a8bac into apache:master Jan 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISSUE-448][Feature] shuffle server report storage info #449

[ISSUE-448][Feature] shuffle server report storage info #449

advancedxy commented Dec 27, 2022 •

edited

Loading

codecov-commenter commented Dec 28, 2022 •

edited

Loading

zuston Jan 3, 2023

advancedxy Jan 3, 2023

advancedxy Jan 9, 2023

zuston left a comment

zuston Jan 6, 2023

advancedxy Jan 8, 2023

zuston Jan 9, 2023

advancedxy Jan 9, 2023

zuston Jan 10, 2023

advancedxy Jan 10, 2023

advancedxy commented Jan 8, 2023

advancedxy commented Jan 9, 2023

jerqi commented Jan 10, 2023

advancedxy commented Jan 11, 2023

jerqi left a comment

zuston left a comment

[ISSUE-448][Feature] shuffle server report storage info #449

[ISSUE-448][Feature] shuffle server report storage info #449

Conversation

advancedxy commented Dec 27, 2022 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

codecov-commenter commented Dec 28, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zuston left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

advancedxy commented Jan 8, 2023

advancedxy commented Jan 9, 2023

jerqi commented Jan 10, 2023

advancedxy commented Jan 11, 2023

jerqi left a comment

Choose a reason for hiding this comment

zuston left a comment

Choose a reason for hiding this comment

advancedxy commented Dec 27, 2022 •

edited

Loading

codecov-commenter commented Dec 28, 2022 •

edited

Loading