Skip to content

DB.compare_filename causes files to be re-read from beginning when monitoring symlinks (e.g., glog) #11543

@zdyj3170101136

Description

@zdyj3170101136

Bug Report

Describe the bug

When DB.compare_filename is enabled and the tail input monitors symbolic links (e.g., glog-style log files), Fluent Bit fails to match the file against its database entry on restart. The file is treated as a new file, and with Read_from_Head True, the entire file is re-read from the beginning, causing duplicate log ingestion.

The root cause is in flb_tail_target_file_name_cmp() (plugins/in_tail/tail_file.h). When inserting into the database, the symlink path is stored as name (file->name). However, during the filename comparison, the DB-stored name is compared against file->real_name (the resolved target path obtained via readlink(/proc/pid/fd/N)) rather than file->name. For symlinks, these basenames differ, so the comparison always fails.

For example:

  • DB stores: name = "trading_system_main.INFO" (symlink), inode = 12345
  • On restart, comparison: basename("trading_system_main.INFO") vs basename("trading_system_main.INFO.20250303-120000.12345")mismatch
  • Result: existing DB entry deleted as stale, new entry inserted with offset 0 → full re-read

To Reproduce

  • Steps to reproduce the problem:
  1. Create a glog-style symbolic link: trading_system_main.INFO -> trading_system_main.INFO.20250303-120000.12345
  2. Configure tail input with DB.compare_filename true and Read_from_Head True to monitor the symlink via a glob pattern (e.g., *.INFO)
  3. Let Fluent Bit run and process some logs — the DB records the inode and symlink path
  4. Restart Fluent Bit
  5. Observe that the file is re-read from the beginning (duplicate logs emitted)

Expected behavior

On restart, Fluent Bit should recognize the file via its DB entry and resume reading from the previously recorded offset, without re-reading already-processed data.

Your Environment

  • Version used: 3.2.7 (also confirmed on latest master — the code in tail_file.h is identical)
  • Configuration:
ini
[INPUT]
Name tail
Path /path/to/logs/.INFO
Db /var/log/flb-position.db
DB.compare_filename true
Read_from_Head True

Your Environment

  • Version used: 3.2.7 (also confirmed on latest master — the code in tail_file.h is identical)
  • Configuration:

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions