Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Analyze cortex failure #282

Open
flipsed opened this issue Sep 28, 2023 · 0 comments
Open

Analyze cortex failure #282

flipsed opened this issue Sep 28, 2023 · 0 comments

Comments

@flipsed
Copy link

flipsed commented Sep 28, 2023

To avoid large queries, we've set a limit to the number of series a query can hit

./cortextool analyse prometheus \
    --address=https://internal-cortex.../ \
    --id=<redacted> \
    --key=<redacted> \
    --log.level=debug \
    --grafana-metrics-file="metrics-in-grafana.json"
DEBU[0025] in use cortex_request_duration_seconds_count 2 
DEBU[0025] in use cortex_ring_members 60                
DEBU[0025] in use cortex_s3_request_duration_seconds_count 0 
DEBU[0025] in use cs_error_counter 3                    
DEBU[0025] in use db_mat_view_query_duration_count 750  
cortextool: error: error querying count by (job) (db_model_duration_bucket): execution: expanding series: the query hit the max number of series limit (limit: 100000 series), try --help

is there anyway we can just skip this metrics rather than have the whole analysis fail?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant