fluentbit not picking up file

[tracyliuzw]

1.When I collect data, I use the Flentdbit + Fluentd architecture, and the data is finally written to BigQuery. Fluentbit uses the tail plug-in, and the directory files I collect will generate a large number of log files in the unit of hours every hour, each time when the file is written to BigQuery Wait for an anomaly, feel field dislocation

---

2.Is there a limit on the length of the event or record?
The data I collected using the Tail plug-in was about 300 bytes long, but the collection failed

[2022/09/21 02:42:05] [ info] [fluent bit] version=1.9.8, commit=97a5e9dcf3, pid=2052
[2022/09/21 02:42:05] [debug] [engine] coroutine stack size: 98302 bytes (96.0K)
[2022/09/21 02:42:05] [ info] [storage] version=1.2.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128
[2022/09/21 02:42:05] [ info] [storage] backlog input plugin: storage_backlog.1
[2022/09/21 02:42:05] [ info] [cmetrics] version=0.3.6
[2022/09/21 02:42:05] [debug] [tail:tail.0] created event channels: read=440 write=584
[2022/09/21 02:42:05] [debug] [input:tail:tail.0] flb_tail_fs_stat_init() initializing stat tail input
[2022/09/21 02:42:05] [debug] [input:tail:tail.0] inode=1125899906846935 with offset=1203 appended as D:\log\log\battle_report.2022091300.log
[2022/09/21 02:42:05] [debug] [input:tail:tail.0] 1 new files found on path 'D:\log\log\battle_report.2022091300.log'
[2022/09/21 02:42:05] [debug] [storage_backlog:storage_backlog.1] created event channels: read=624 write=628
[2022/09/21 02:42:05] [ info] [input:storage_backlog:storage_backlog.1] queue memory limit: 15.3M
[2022/09/21 02:42:05] [debug] [emitter:re_emitted] created event channels: read=632 write=636
[2022/09/21 02:42:05] [debug] [stdout:stdout.0] created event channels: read=644 write=648
[2022/09/21 02:42:05] [ info] [sp] stream processor started
[2022/09/21 02:42:05] [ info] [output:stdout:stdout.0] worker #0 started
[2022/09/21 02:42:05] [debug] [input:tail:tail.0] inode=1125899906846935 file=D:\log\log\battle_report.2022091300.log promote to TAIL_EVENT
[2022/09/21 02:42:05] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2022/09/21 02:42:15] [debug] [input:tail:tail.0] 0 new files found on path 'D:\log\log\battle_report.2022091300.log'
[2022/09/21 02:42:25] [debug] [input:tail:tail.0] 0 new files found on path 'D:\log\log\battle_report.2022091300.log'
[2022/09/21 02:42:35] [debug] [input:tail:tail.0] 0 new files found on path 'D:\log\log\battle_report.2022091300.log'
[2022/09/21 02:42:45] [debug] [input:tail:tail.0] 0 new files found on path 'D:\log\log\battle_report.2022091300.log'

[patrick-stephens]

What is the actual problem here for 1.? I don't really understand the issue from what is written so please can you provide some reproducer of the issue or more details.

For 2, tail functions as per tail -f so it will only pick up new entries in a file since it started tailing - you need to configure read_from_head on if you want to read existing data already present in a file when the file is opened.
If you have configured a db setting then this will persist what data it has already sent: you do not want duplicate log entries in most cases. Have you done this?
https://docs.fluentbit.io/manual/pipeline/inputs/tail
Please provide your full configuration (as per the issue template) so we can see what options you are using.
I'm guessing this is on Windows but also check file permissions - the user Fluent Bit runs as needs to be able to read the file otherwise it cannot see it.

[tracyliuzw]

My business scenario: About 50 files are generated per hour in the tail directory. Data is occasionally lost in the first minute of each hour.
Can you give me some advice on how to configure the tail directory when there are many files in it
My configuration is as follows:

[SERVICE]
flush 5
daemon Off
log_level warn
Log_File /var/log/td-agent-bit/td-agent-bit.log
parsers_file parsers.conf
Decode_Field_as escaped_utf8 log
storage.path /var/log/td-agent-bit/storage/
storage.sync normal
storage.checksum Off
storage.max_chunks_up 103
storage.backlog.mem_limit 16M
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_PORT 2020
Health_Check On
HC_Errors_Count 5
HC_Retry_Failure_Count 5
HC_Period 5
[INPUT]
Name tail
Path /data/fjserver/log/gamelog/..log
Tag tp
Key tp
Path_Key filename
Buffer_Chunk_Size 24m
Buffer_Max_Size 32m
Refresh_Interval 10
Rotate_Wait 5
Ignore_Older 2h
Read_from_Head false
storage.type filesystem
storage.pause_on_chunks_overlimit on
Skip_Empty_Lines On
Skip_Long_Lines On
DB /var/log/td-agent-bit/db/tp.db
DB.sync normal
DB.journal_mode WAL
Mem_Buf_Limit 48m
Exit_On_Eof false
Inotify_Watcher true
[INPUT]
Name tail
Path /data/fjserver/log/serverlog/*.log
Db /var/log/td-agent-bit/db/tpserver.db
Tag tp.serverlog
Key tp
Mem_Buf_Limit 48m
Buffer_Chunk_Size 24m
Buffer_Max_Size 32m
Refresh_Interval 10
Rotate_Wait 5
Ignore_Older 2h
Read_from_Head false
storage.type filesystem
storage.pause_on_chunks_overlimit on
Skip_Empty_Lines On
Skip_Long_Lines On
Exit_On_Eof false
Inotify_Watcher true
[FILTER]
Name rewrite_tag
Match tp
Rule $filename /(\S+)/([a-zA-Z]+.\d{6})(\d{4}).log $TAG.$2 false
Emitter_Name re_emitted
Emitter_Mem_Buf_Limit 10M
Emitter_Storage.type filesystem
[FILTER]
Name record_modifier
Match .
Record hostname ${HOSTNAME}
[OUTPUT]
Name Forward
Match .
Upstream upstream.conf
Require_ack_response true
Send_options False
Compress gzip
Workers 2
storage.total_limit_size 48M

[tracyliuzw]

fluentd configuration file :
<filter tp.vip.*>
@type parser
key_name tp
reserve_data true
remove_key_name_field true
replace_invalid_sequence false
emit_invalid_record_to_error true

@type csv
##common##
types eventtype:string,eventtime:string,iggid:integer,f4:string,f5:string,f6:string,f7:string,f8:string
null_empty_string true
estimate_current_event true
##csv##
keys eventtype,eventtime,iggid,f4,f5,f6,f7,f8
delimiter "\t"
parser_type normal

<match tp.vip.*>

@type file
path /var/log/td-agent/buffer/tp/vip
timekey_use_utc true
chunk_limit_size 256MB
#chunk_limit_records 3000
total_limit_size 512MB
chunk_full_threshold 0.5
queued_chunks_limit_size 20
flush_at_shutdown true
flush_mode interval
flush_interval 5s
flush_thread_interval 1
flush_thread_count 1
flush_thread_burst_interval 1
delayed_commit_timeout 60
overflow_action block
retry_type exponential_backoff
retry_timeout 24h
retry_forever true
retry_max_times 20
retry_wait 2

@type bigquery_insert
auth_method json_key
json_key /etc/td-agent/style-saga-aa24aba4a06d.json
project style-saga
dataset gamedata
table vip_${yyyymm}
fetch_schema_table vip
fetch_schema true
auto_create_table true
ignore_unknown_values true
schema_cache_expire 600
allow_retry_insert_errors true
request_timeout_sec 120
request_open_timeout_sec 120
skip_invalid_rows true

[tracyliuzw]

When data is finally written to BigQuery, an error log is reported:
2022-09-20 22:00:24 -0500 [trace]: #5 enqueueing all chunks in buffer instance=2220
2022-09-20 22:00:24 -0500 [trace]: #13 enqueueing all chunks in buffer instance=4100
2022-09-20 22:00:24 -0500 [trace]: #13 enqueueing all chunks in buffer instance=5480
2022-09-20 22:00:24 -0500 [trace]: #3 enqueueing all chunks in buffer instance=5420
2022-09-20 22:00:24 -0500 [trace]: #12 enqueueing all chunks in buffer instance=5180
2022-09-20 22:00:24 -0500 [debug]: #12 insert rows project_id="style-saga" dataset="gamedata" table="dressup_202209" count=1
2022-09-20 22:00:24 -0500 [warn]: #12 insert errors project_id="style-saga" dataset="gamedata" table="dressup_202209" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fdf13c905f0 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fdf13e8fc98 @debug_info="", @location="eventtime", @message="Invalid datetime string \"8921\"", @Reason="invalid">], @index=0>]"
2022-09-20 22:00:24 -0500 [debug]: #12 taking back chunk for errors. chunk="5e92725249ac4b3c4d09a3c00c42891c"
2022-09-20 22:00:24 -0500 [trace]: #12 taking back a chunk instance=2720 chunk_id="5e92725249ac4b3c4d09a3c00c42891c"
2022-09-20 22:00:24 -0500 [trace]: #12 chunk taken back instance=2720 chunk_id="5e92725249ac4b3c4d09a3c00c42891c" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag=nil, variables={:yyyymm=>"202209"}, seq=0>
2022-09-20 22:00:24 -0500 [error]: #12 Hit limit for retries. dropping all chunks in the buffer queue. retry_times=0 records=1 error_class=Fluent::BigQuery::UnRetryableError error="failed to insert into bigquery(insert errors), and cannot retry"
2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-bigquery-2.3.0/lib/fluent/plugin/bigquery/writer.rb:99:in insert_rows' 2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-bigquery-2.3.0/lib/fluent/plugin/out_bigquery_insert.rb:102:in insert'
2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-bigquery-2.3.0/lib/fluent/plugin/out_bigquery_insert.rb:98:in write' 2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/output.rb:1180:in try_flush'
2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/output.rb:1501:in flush_thread_run' 2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin/output.rb:501:in block (2 levels) in start'
2022-09-20 22:00:24 -0500 [error]: #12 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-09-20 22:00:24 -0500 [trace]: #12 clearing queue instance=2720
2022-09-20 22:00:24 -0500 [debug]: #12 buffer queue cleared
2022-09-20 22:00:24 -0500 [trace]: #5 enqueueing all chunks in buffer instance=3020

[tracyliuzw]

help me ,Give me some advice!!!!!!!!!!!!!!!!!!!!!!!!!!!

[patrick-stephens]

It looks to me like the issue is with Fluentd sending to BigQuery so you probably want to drill down on that and raise in the Fluentd repository for that plugin where there will be expertise on that.

Is there some issue with the Fluent Bit side of things specifically? There is an output plugin already to send to BigQuery from Fluent Bit directly so does that work? https://docs.fluentbit.io/manual/pipeline/outputs/bigquery

The tail inputs you have seem to be ok but I can't really comment as you know the specific log files you have.

I did note one seems to have a strange path, is that right or did you mean a wildcard?

[INPUT]
Name tail
Path /data/fjserver/log/gamelog/..log
Tag tp

Also only the server logs have a DB set so the other will just read from the end (not the beginning) when Fluent Bit starts: only new data added after Fluent Bit is watching the file will be picked up. This is what I mean by tail -f, it functions as that does.

The server logs have a DB so will record which offset they got up to last and start from there:

[INPUT]
Name tail
Path /data/fjserver/log/serverlog/*.log
Db /var/log/td-agent-bit/db/tpserver.db
Tag tp.serverlog

Unrelated but you do not have to provide all configuration options, only the ones required or different from the defaults.
The Slack channel is likely better for discussion around best practice and the various options vs your requrements.

[tracyliuzw]

thank you your help !!!
sorry,Path /data/fjserver/log/gamelog/..log, I copy wrong, the correct is: the Path /data/fjserver/log/gamelog/..log
In the current two-tier architecture, Fluentbit is mainly put together with the production environment server. For fear of occupying too much production server resources during collection, the data is only processed slightly by forward to FluentD, and then processed by FluentD to Bigquery.
My original data is CSV. Is it possible that the fields are missing during the parsing process? I found that the fields are missing and misplaced in the error log of writing BigQuery, but the BigQuery fails to be written.

[tracyliuzw]

• The asterisk cannot be entered??

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.