Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.transform-internal-XXX index if frozen #82735

Open
kuisathaverat opened this issue Nov 5, 2020 · 0 comments
Open

.transform-internal-XXX index if frozen #82735

kuisathaverat opened this issue Nov 5, 2020 · 0 comments
Labels
Team:Defend Workflows “EDR Workflows” sub-team of Security Solution

Comments

@kuisathaverat
Copy link
Contributor

Kibana version:
8.0.0-SNAPSHOT

Elasticsearch version:
8.0.0-SNAPSHOT

Server OS version:
Elastic Cloud (Elasticsearch) + Kubernetes (Kibana)

Browser version:
Chrome 86

Browser OS version:
macOS 15

Original install method (e.g. download page, yum, from source, etc.):
Docker

Describe the bug:

We have detected that for some reason every few days the .transform-internal-XXX index if frozen, this breaks the fleet UI and the can see the following error in the Kibana logs

[cluster_block_exception] index [.transform-internal-005] blocked by: [FORBIDDEN/8/index write (api)]; response from /_transform/endpoint.metadata_current-default-0.16.1: {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [.transform-internal-005] blocked by: [FORBIDDEN/8/index write (api)];"}],"type":"runtime_exception","reason":"runtime_exception: Failed to persist transform configuration","caused_by":{"type":"cluster_block_exception","reason":"index [.transform-internal-005] blocked by: [FORBIDDEN/8/index write (api)];"}},"status":500}

When we check the index we see that it is frozen, this index does not have any alias neither ILM configured.

image

This cluster has about 5TB of disk space we are using about 3TB

related to #82580

Steps to reproduce:
there are not clear steps to replicate it.

Expected behavior:

The index should not be frozen.

Screenshots (if relevant):

image

Errors in browser console (if relevant):

[cluster_block_exception] index [.transform-internal-005] blocked by: [FORBIDDEN/8/index write (api)]; response from /_transform/endpoint.metadata_current-default-0.16.1: {"error":{"root_cause":[{"type":"cluster_block_exception","reason":"index [.transform-internal-005] blocked by: [FORBIDDEN/8/index write (api)];"}],"type":"runtime_exception","reason":"runtime_exception: Failed to persist transform configuration","caused_by":{"type":"cluster_block_exception","reason":"index [.transform-internal-005] blocked by: [FORBIDDEN/8/index write (api)];"}},"status":500}

Provide logs and/or server output (if relevant):

Any additional context:

@kuisathaverat kuisathaverat added the Team:Defend Workflows “EDR Workflows” sub-team of Security Solution label Nov 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Defend Workflows “EDR Workflows” sub-team of Security Solution
Projects
None yet
Development

No branches or pull requests

1 participant