elastic-agent-logging.md

Elastic-Agent logging

The Elastic-Agent process defines two log outputs:

• The "internal" core that is used by monitoring components and collected in the diagnostics. It's configuration is hardcoded. This output always logs to files in data/elastic-agent-<hash>/logs and uses the default configuration defined in elastic-agent-libs/logp.
• The "default" logger that is the user-configurable logger, it logs to the Elastic-Agent's root folder, it can also be configured to log to stderr. When running in a container environment, it defaults to logging to stderr.

Logger initialization

The logger initialization is not one of the first things done by the Elastic-Agent. Looking at the normal Elastic-Agent run, here is the stack trace from the logging initialization.

 0  0x00005e8768fea7ac in github.com/elastic/elastic-agent/pkg/core/logger.new
    at /devel/elastic-agent/pkg/core/logger/logger.go:83
 1  0x00005e8768fea54f in github.com/elastic/elastic-agent/pkg/core/logger.NewFromConfig
    at /devel/elastic-agent/pkg/core/logger/logger.go:65
 2  0x00005e876b511dba in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.runElasticAgent
    at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:151
 3  0x00005e876b5119e5 in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.run
    at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:138
 4  0x00005e876b51127e in github.com/elastic/elastic-agent/internal/pkg/agent/cmd.newRunCommandWithArgs.func1
    at /devel/elastic-agent/internal/pkg/agent/cmd/run.go:78

This means some log entries might not be collected by diagnostics or shipped to the monitoring output. Everything in the run function happens before the logger initialization.

elastic-agent/internal/pkg/agent/cmd/run.go

Lines 104 to 138 in 574aa5d

	func run(override cfgOverrider, testingMode bool, fleetInitTimeout time.Duration, modifiers ...component.PlatformModifier) error {
	// Windows: Mark service as stopped.
	// After this is run, the service is considered by the OS to be stopped.
	// This must be the first deferred cleanup task (last to execute).
	defer func() {
	service.NotifyTermination()
	service.WaitExecutionDone()
	}()
	if err := handleUpgrade(); err != nil {
	return fmt.Errorf("error checking for and handling upgrade: %w", err)
	}
	locker := filelock.NewAppLocker(paths.Data(), paths.AgentLockFileName)
	if err := locker.TryLock(); err != nil {
	return err
	}
	defer func() {
	_ = locker.Unlock()
	}()
	service.BeforeRun()
	defer service.Cleanup()
	// register as a service
	stop := make(chan bool)
	ctx, cancel := context.WithCancel(context.Background())
	var stopBeat = func() {
	close(stop)
	}
	defer cancel()
	go service.ProcessWindowsControlEvents(stopBeat)
	return runElasticAgent(ctx, cancel, override, stop, testingMode, fleetInitTimeout, false, nil, modifiers...)

Internal logging

The internal logging output is crucial for the Elastic-Agent self-monitoring and diagnostics. It is instantiated by MakeInternalFileOutput function that is called when a new logger is created. Its default configuration is:

• 10Mb per log file
• Maximum of 7 log files
• Rotated on startup
• ECS/JSON encoded
• UTC timestamps

There is also a second file output for events that is configured via agent.logging.event_data. It shares the same log level as the main logger and can only be configured for standalone agents. For Fleet-Managed agents it will always use the default values:

• 5Mb per log file
• Maximum of 2 log files
• Do not rotate on startup
• ECS/JSON encoded
• UTC timestamps

Default logging

The default logger is the easiest to discover because it's user configurable, logs to the Agent's root directory and can output to stderr. It's default configuration comes from

elastic-agent/pkg/core/logger/logger.go

Lines 132 to 148 in 574aa5d

	func DefaultLoggingConfig() *Config {
	cfg := logp.DefaultConfig(logp.DefaultEnvironment)
	cfg.Beat = agentName
	cfg.Level = DefaultLogLevel
	cfg.ToFiles = true
	cfg.Files.Path = paths.Logs()
	cfg.Files.Name = agentName
	cfg.Files.MaxSize = 20 * 1024 * 1024
	cfg.Files.Permissions = 0600 // default user only
	root, _ := utils.HasRoot() // error ignored
	if !root {
	// when not running as root, the default changes to include the group
	cfg.Files.Permissions = 0660
	}
	return &cfg
	}

and defaults to:

• 20Mb per log file
• Maximum of 7 log files
• Rotated on startup
• ECS/JSON are not explicitly set

Collecting logs for diagnostics

The Elastic-Agent will only collect data/elastic-agent-<hash>/logs. The functions responsible for collecting logs during diagnostics are:

• zipLogs
• zipLogsWithPath

zipLogsWithPath always appends the /logs to whatever path it receives.

Total footprint

Given the two log outputs and their default log rotation policies, the Elastic-Agent needs about 210Mb (20Mb x 7 + 10Mb x 7 = 210Mb) of disk for logging.