Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ntpd Error Log Caught by Log Analyzer Results in Test Cases Failure #19580

Merged
merged 3 commits into from
Oct 23, 2024

Conversation

wumiaont
Copy link
Contributor

Why I did it

In many mgmt feature test cases, it's found that there's an error log of "ERR ntpd[1902895]: statistics directory /var/log/ntpsec/ does not exist or is unwriteable, error Permission denied" shows up after ntpd is restarted which causes many test case failures as the error log is caught by log analyzer.

Work item tracking
  • Microsoft ADO (number only):

How I did it

Create this directory before ntpd service is started.

How to verify it

Verified on chassis with the fix and restarted ntpd service. No error log shows up.

Which release branch to backport (provide reason below if selected)

  • 201811
  • 201911
  • 202006
  • 202012
  • 202106
  • 202111
  • 202205
  • 202211
  • 202305
  • 202405

Tested branch (Please provide the tested image version)

master

@wumiaont wumiaont marked this pull request as ready for review July 15, 2024 15:41
@wumiaont wumiaont requested a review from lguohan as a code owner July 15, 2024 15:41
@wumiaont
Copy link
Contributor Author

/azpw run ms_conflict

@mssonicbld
Copy link
Collaborator

/AzurePipelines run ms_conflict

Copy link

No pipelines are associated with this pull request.

@rlhui rlhui requested a review from SuvarnaMeenakshi July 17, 2024 17:37
@rlhui
Copy link
Contributor

rlhui commented Jul 17, 2024

@SuvarnaMeenakshi please help review, thanks.

@SuvarnaMeenakshi
Copy link
Contributor

Did you try updating https://github.com/sonic-net/sonic-buildimage/blob/master/files/image_config/ntp/ntp-apparmor file so that it reflects all required files?
/etc/apparmor.d/usr.sbin.ntpd file on the device has various file permissions and currently this file has:
/var/log/ntp w,
/var/log/ntp.log w,
/var/log/ntpd w,
/var/log/ntpsec/clockstats* rwl,
/var/log/ntpsec/loopstats* rwl,
/var/log/ntpsec/peerstats* rwl,
/var/log/ntpsec/protostats* rwl,
/var/log/ntpsec/rawstats* rwl,
/var/log/ntpsec/sysstats* rwl,

@wumiaont
Copy link
Contributor Author

wumiaont commented Aug 1, 2024

Did you try updating https://github.com/sonic-net/sonic-buildimage/blob/master/files/image_config/ntp/ntp-apparmor file so that it reflects all required files? /etc/apparmor.d/usr.sbin.ntpd file on the device has various file permissions and currently this file has: /var/log/ntp w, /var/log/ntp.log w, /var/log/ntpd w, /var/log/ntpsec/clockstats* rwl, /var/log/ntpsec/loopstats* rwl, /var/log/ntpsec/peerstats* rwl, /var/log/ntpsec/protostats* rwl, /var/log/ntpsec/rawstats* rwl, /var/log/ntpsec/sysstats* rwl,

Currently the issue is that /var/log/ntpsec directory does not exist which causes error log thrown when ntpd is started every time. The change is trying to create this directory when it does not exist. If the directory already exist it will not do anything. I am not intended to modify permissions to any files inside /var/log/ntpsec.

@wumiaont
Copy link
Contributor Author

wumiaont commented Aug 1, 2024

It's found that /var/log/ntpsec has to have permission 777. If mode is set to 755 or 775, reset ntpd will have error log.

admin@ixre-egl-board29:/var/log$ sudo systemctl status ntpd
? ntpsec.service - Network Time Service
Loaded: loaded (/lib/systemd/system/ntpsec.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/ntpsec.service.d
??sonic-target.conf
Active: active (running) since Thu 2024-08-01 15:40:45 UTC; 16s ago
Docs: man:ntpd(8)
Process: 2920180 ExecStart=/usr/libexec/ntpsec/ntp-systemd-wrapper (code=exited, status=0/SUCCESS)
Main PID: 2920240 (ntpd)
Tasks: 1 (limit: 18982)
Memory: 2.4M
CGroup: /system.slice/ntpsec.service
??2920240 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -x -N -u ntpsec:ntpsec

Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: IO: Listen normally on 1 eth0 152.148.151.181:123
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: IO: Listen normally on 2 lo [::1]:123
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: IO: Listen normally on 3 eth0 [fc00:2::32]:123
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: IO: Listen normally on 4 eth0 [fe80::427c:7dff:febb:2771%7]:123
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: IO: Listening on routing socket on fd #21 for interface updates
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: INIT: MRU 10922 entries, 13 hash bits, 65536 bytes
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: INIT: Built with OpenSSL 3.0.9 30 May 2023, 30000090
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: INIT: Running with OpenSSL 3.0.13 30 Jan 2024, 300000d0
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: NTSc: Using system default root certificates.
Aug 01 15:40:45 ixre-egl-board29 ntpd[2920240]: statistics directory /var/log/ntpsec/ does not exist or is unwriteable, error Permission denied

Copy link
Contributor

@SuvarnaMeenakshi SuvarnaMeenakshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@SuvarnaMeenakshi
Copy link
Contributor

@wumiaont can you help fix conflict.

@wumiaont
Copy link
Contributor Author

/azpw ms_conflict

@SuvarnaMeenakshi
Copy link
Contributor

@rlhui can you help merge this

@abdosi
Copy link
Contributor

abdosi commented Oct 23, 2024

@rlhui @kperumalbfn @qiluo-msft : can you please merge this

@kperumalbfn kperumalbfn merged commit d4779a8 into sonic-net:master Oct 23, 2024
22 checks passed
mssonicbld pushed a commit to mssonicbld/sonic-buildimage that referenced this pull request Nov 6, 2024
…onic-net#19580)

Fix NTP error log which causes many test failure caught by log analyzer
@mssonicbld
Copy link
Collaborator

Cherry-pick PR to 202405: #20723

@mssonicbld
Copy link
Collaborator

@wumiaont cherry pick PR didn't pass PR checker. Please check!!!
#20723

mssonicbld pushed a commit that referenced this pull request Nov 15, 2024
…19580)

Fix NTP error log which causes many test failure caught by log analyzer
rkavitha-hcl pushed a commit to rkavitha-hcl/sonic-buildimage that referenced this pull request Nov 15, 2024
…onic-net#19580)

Fix NTP error log which causes many test failure caught by log analyzer
aidan-gallagher pushed a commit to aidan-gallagher/sonic-buildimage that referenced this pull request Nov 16, 2024
…onic-net#19580)

Fix NTP error log which causes many test failure caught by log analyzer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

8 participants