This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 36
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
This PR did various things to improve profile API: First, the PR fixed the hang issue. Previously, when users run _profile/state or _profile on the multi-entity detector, the request hangs. The problem is due to incorrect maxResponseCount passed to MultiResponsesDelegateActionListener. Second, the PR fixes the multi-entity detector's wrong state issue. Previously, we can show the init state after an anomaly has shown up. We may have the problem because we read the most active entity's init progress in the cache for a detector's init_progress. But the entity already produced anomaly has been evicted out of the cache. This PR fixes the issue by double-checking the result index's non-zero RCF score for a multi-entity detector before reporting the init state. If there is any non-zero RCF score, we say running state instead of the initing state. Third, this PR adds more information to the entity level profile, including last_active_timestamp, last_sample_timestamp, init_progress, model, and state. Fourth, this PR adds models and total_size_in_bytes to the multi-entity detector level profile. This PR also fixes various "fail to return" issues in the rest API related transport action. We didn't return after sending channel responses. Later, when we use the channel to send back responses again, we get " java.lang.IllegalStateException: Channel is already closed." Testing done: 1. manual testing passes. 2. actively adding unit tests
- Loading branch information
Showing
41 changed files
with
1,953 additions
and
448 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
37 changes: 37 additions & 0 deletions
37
src/main/java/com/amazon/opendistroforelasticsearch/ad/AbstractProfileRunner.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| /* | ||
| * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"). | ||
| * You may not use this file except in compliance with the License. | ||
| * A copy of the License is located at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * or in the "license" file accompanying this file. This file is distributed | ||
| * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either | ||
| * express or implied. See the License for the specific language governing | ||
| * permissions and limitations under the License. | ||
| */ | ||
|
|
||
| package com.amazon.opendistroforelasticsearch.ad; | ||
|
|
||
| import com.amazon.opendistroforelasticsearch.ad.model.InitProgressProfile; | ||
|
|
||
| public abstract class AbstractProfileRunner { | ||
| protected long requiredSamples; | ||
|
|
||
| public AbstractProfileRunner(long requiredSamples) { | ||
| this.requiredSamples = requiredSamples; | ||
| } | ||
|
|
||
| protected InitProgressProfile computeInitProgressProfile(long totalUpdates, long intervalMins) { | ||
| float percent = Math.min((100.0f * totalUpdates) / requiredSamples, 100.0f); | ||
| int neededPoints = (int) (requiredSamples - totalUpdates); | ||
| return new InitProgressProfile( | ||
| // rounding: 93.456 => 93%, 93.556 => 94% | ||
| String.format("%.0f%%", percent), | ||
| intervalMins * neededPoints, | ||
| neededPoints | ||
| ); | ||
| } | ||
| } |
512 changes: 269 additions & 243 deletions
512
src/main/java/com/amazon/opendistroforelasticsearch/ad/AnomalyDetectorProfileRunner.java
Large diffs are not rendered by default.
Oops, something went wrong.
28 changes: 28 additions & 0 deletions
28
src/main/java/com/amazon/opendistroforelasticsearch/ad/DetectorModelSize.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| /* | ||
| * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"). | ||
| * You may not use this file except in compliance with the License. | ||
| * A copy of the License is located at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * or in the "license" file accompanying this file. This file is distributed | ||
| * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either | ||
| * express or implied. See the License for the specific language governing | ||
| * permissions and limitations under the License. | ||
| */ | ||
|
|
||
| package com.amazon.opendistroforelasticsearch.ad; | ||
|
|
||
| import java.util.Map; | ||
|
|
||
| public interface DetectorModelSize { | ||
| /** | ||
| * Gets all of a detector's model sizes hosted on a node | ||
| * | ||
| * @param detectorId Detector Id | ||
| * @return a map of model id to its memory size | ||
| */ | ||
| Map<String, Long> getModelSize(String detectorId); | ||
| } |
27 changes: 27 additions & 0 deletions
27
src/main/java/com/amazon/opendistroforelasticsearch/ad/EntityModelSize.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| /* | ||
| * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"). | ||
| * You may not use this file except in compliance with the License. | ||
| * A copy of the License is located at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * or in the "license" file accompanying this file. This file is distributed | ||
| * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either | ||
| * express or implied. See the License for the specific language governing | ||
| * permissions and limitations under the License. | ||
| */ | ||
|
|
||
| package com.amazon.opendistroforelasticsearch.ad; | ||
|
|
||
| public interface EntityModelSize { | ||
| /** | ||
| * Gets an entity's model sizes | ||
| * | ||
| * @param detectorId Detector Id | ||
| * @param entityModelId Entity's model Id | ||
| * @return the entity's memory size | ||
| */ | ||
| long getModelSize(String detectorId, String entityModelId); | ||
| } |
Oops, something went wrong.