Impossible to use variables in ilm_rollover_alias #10962
Comments
|
If you already have setup ILM, then you can continue using the the ILM fields are used at startup to create the write alias, install template, etc. All these actions are not related to events, since at that point there may be no events yet, so they can't use any event related information. |
|
Hello, as @jsvd suggests, the problem is the fact ILM requires Logstash to:
Unfortunately, at the moment this is not applicable for dynamic index names as otherwise Logstash would have to lookup into Elasticsearch before each The possible workarounds:
|
Thank you for your answer. To make it works it force you to use that kind of syntax this is not really what you might call a "dynamic parameter" But it works .. |
Thank you for your answer . I got the problem |
|
This is an issue that I have run into as well, I was going to report it today. Creating the index write_alias manually is acceptable, but it isn't ideal - beyond the issue of manually have to bootstrap every alias/index, if the write alias is deleted, data will automatically be pushed into a concrete index with the same name - making re-creation of the alias impossible until the entire pipeline is shutdown and data is no longer being received for the index. I'm going to look at templating the elasticsearch output file and generating it with static fields, but am curious whether there is any view towards changing this implementation? |
|
I feel like we should move this issue to the Elasticsearch Output repo (e.g. logstash-plugins/logstash-output-elasticsearch#858) |
Can you please enumerate what actions needs to be performed during bootstrapping a index with lifecycle. we are using daily index with YYYY.MM.DD in index name, does that mean bootstrapping has to be performed every day? |
|
Hello @JathinSanghvi - Bootstrapping means creating the first index and its associated write alias for the first time. It is documented at this page. E.g. in your case might be: The command above will create the index |
|
To help consolidate feedback (and votes) from the field on this request: If you end up in this issue, please vote in the parent issue in the logstash-output-elasticsearch repository: logstash-plugins/logstash-output-elasticsearch#858 |
|
As a drive by comment, if you setup your own ILM policy, without using Logstash and attach it via your template, then you can index using the alias as your index: elasticsearch {
hosts => ["https://d116qsdf.cheapdr.dd:9243/"]
index => "%{[@metadata][ilm_rollover_alias]}"
document_type => "_doc"
document_id => "%{[@metadata][hash]}"
action => "create"
}Then, as far as Logstash is concerned, you should just be indexing into a normal index. As long as you have ILM configured, it should be acting appropriately and automatically without involving Logstash. This does assume that you have a fixed number of indices coming through Logstash, but if you don't then you probably need to reconsider your workflow regardless of this issue. |
|
Dear all, We believe this issue is now solved with the use of DataStreams. Just wanted to note that DataStreams is now default for 8.x. More details on logstash-plugins/logstash-output-elasticsearch#858 (comment) As such, I'm going to go ahead and close this issue. If you believe this is still an issue, please re-open this bug report. cc @jsvd @flexitrev |
Hello I am using logstash with ILM and rollover alias and it seems the parameter
ilm_rollover_aliasdoes not support dynamic parameters.It joins one of the problems mentioned here :
logstash 7.0 broke template upload via manage_template (my custom index_patterns overwritten) #10687
Behaviour of ILM in particular when manage_template =>false #10596
I'm managing templates externally so I set
manage_template => falseBut , in my use case , i have many input and filter sections , and i have only one output section wich was configured like this in the past with a variable index name , when I was not using ILM
Which was working perfectly .
But now , if I want to do the same with the
ilm_rollover_aliaslike thisI get this error :
I don't know if there's a real reason why this feature is not supported.
Is there a plan to propose this feature in the future?
Thank you
The text was updated successfully, but these errors were encountered: