Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PMON service is started so late in the Master branch #19126

Closed
mlok-nokia opened this issue May 29, 2024 · 1 comment · Fixed by sonic-net/sonic-host-services#135 or #19657
Closed

PMON service is started so late in the Master branch #19126

mlok-nokia opened this issue May 29, 2024 · 1 comment · Fixed by sonic-net/sonic-host-services#135 or #19657
Assignees
Labels
P0 Priority of the issue Triaged this issue has been triaged

Comments

@mlok-nokia
Copy link
Contributor

mlok-nokia commented May 29, 2024

Description

On the master branch, PMON service is started so late. Please refer to the log below. It is about 6 or 7 minutes

ay  6 04:50:04.784589 ixre-egl-board15 NOTICE root: Stopping pmon docker
May  6 04:50:04.903390 ixre-egl-board15 INFO container: docker cmd: wait for pmon
May  6 04:50:04.981236 ixre-egl-board15 INFO systemd[1]: Stopping pmon.service - Platform monitor container...
May  6 04:50:05.265167 ixre-egl-board15 DEBUG container: read_data: config:True feature:pmon fields:[('set_owner', 'local'), ('no_fallback_to_local', False), ('state', 'disabled')] val:['local', False, 'enabled']
May  6 04:50:05.265787 ixre-egl-board15 DEBUG container: read_data: config:False feature:pmon fields:[('current_owner', 'none'), ('remote_state', 'none'), ('container_id', '')] val:['none', 'none', '']
May  6 04:50:05.266573 ixre-egl-board15 DEBUG container: container_stop: pmon: set_owner:local current_owner:none remote_state:none docker_id:pmon
May  6 04:50:05.280904 ixre-egl-board15 INFO container: docker cmd: stop for pmon
May  6 04:50:05.377694 ixre-egl-board15 INFO systemd[1]: pmon.service: Deactivated successfully.
May  6 04:50:05.378191 ixre-egl-board15 INFO systemd[1]: Stopped pmon.service - Platform monitor container.
May  6 04:50:08.053622 ixre-egl-board15 INFO dockerd[1174]: time="2024-05-06T04:50:08.053158626Z" level=error msg="Error setting up exec command in container pmon: Container 20a08df3c478eb3dd0e5342a71361470613853d3efc40852273dc1f4a94601c0 is not running"
May  6 04:53:52.473146 ixre-egl-board15 INFO featured: Updating feature 'pmon' systemd config file related to auto-restart ...
May  6 04:53:52.473537 ixre-egl-board15 INFO featured: Feature 'pmon' systemd config file related to auto-restart is updated!
May  6 04:53:52.941439 ixre-egl-board15 INFO featured: Feature is pmon delayed for port init
May  6 04:53:58.197403 ixre-egl-board15 INFO bgp0#supervisord 2024-05-06 04:53:58,196 INFO spawned: 'bgpmon' with pid 76
May  6 04:53:58.197957 ixre-egl-board15 INFO bgp0#supervisord 2024-05-06 04:53:58,197 INFO success: bgpmon entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
May  6 04:53:58.243800 ixre-egl-board15 INFO bgp1#supervisord 2024-05-06 04:53:58,243 INFO spawned: 'bgpmon' with pid 76
May  6 04:53:58.244021 ixre-egl-board15 INFO bgp1#supervisord 2024-05-06 04:53:58,243 INFO success: bgpmon entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
May  6 04:53:58.248714 ixre-egl-board15 INFO bgp0#bgpmon: bgpmon service started
May  6 04:53:58.303728 ixre-egl-board15 INFO bgp1#bgpmon: bgpmon service started
May  6 04:56:16.032746 ixre-egl-board15 NOTICE python3: :- publish: EVENT_PUBLISHED: {"sonic-events-host:event-down-ctr":{"ctr_name":"pmon","timestamp":"2024-05-06T04:56:16.32314Z"}}
May  6 04:57:00.313755 ixre-egl-board15 INFO featured: Running cmd: '['sudo', 'systemctl', 'unmask', 'pmon.service']'
May  6 04:57:00.755641 ixre-egl-board15 INFO featured: Running cmd: '['sudo', 'systemctl', 'enable', 'pmon.service']'
May  6 04:57:01.176663 ixre-egl-board15 INFO featured: Running cmd: '['sudo', 'systemctl', 'start', 'pmon.service']'
May  6 04:57:01.237212 ixre-egl-board15 INFO systemd[1]: Starting pmon.service - Platform monitor container...
May  6 04:57:01.890832 ixre-egl-board15 INFO pmon.sh[13426]: Starting existing pmon container with HWSKU Nokia-IXR7250E-36x400G
May  6 04:57:02.149033 ixre-egl-board15 DEBUG container: read_data: config:True feature:pmon fields:[('set_owner', 'local'), ('no_fallback_to_local', False), ('state', 'disabled')] val:['local', False, 'enabled']
May  6 04:57:02.149554 ixre-egl-board15 DEBUG container: read_data: config:False feature:pmon fields:[('current_owner', 'none'), ('remote_state', 'none'), ('container_id', '')] val:['none', 'none', '']
May  6 04:57:02.150284 ixre-egl-board15 DEBUG container: container_start: pmon: set_owner:local fallback:True remote_state:none server_connected:false
May  6 04:57:02.384032 ixre-egl-board15 INFO container: docker cmd: start for pmon
May  6 04:57:02.461131 ixre-egl-board15 INFO systemd[1]: Started pmon.service - Platform monitor container.
May  6 04:57:02.465856 ixre-egl-board15 INFO featured: Feature pmon is enabled and started

Steps to reproduce the issue:

Describe the results you received:

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

@arlakshm
Copy link
Contributor

arlakshm commented Jun 5, 2024

check if the delayed field is init_cfg.json help?

@rlhui rlhui added the Triaged this issue has been triaged label Jun 12, 2024
@abdosi abdosi added the P0 Priority of the issue label Jun 19, 2024
yejianquan pushed a commit to sonic-net/sonic-host-services that referenced this issue Jul 9, 2024
…Table. (#135)

What I did:
Added support to render template format of delayed flag on Feature Table.

Why I did:
To fix: sonic-net/sonic-buildimage#19126

How I verify:
Manual Verification
UT updated

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
mssonicbld pushed a commit to mssonicbld/sonic-host-services that referenced this issue Jul 9, 2024
…Table. (sonic-net#135)

What I did:
Added support to render template format of delayed flag on Feature Table.

Why I did:
To fix: sonic-net/sonic-buildimage#19126

How I verify:
Manual Verification
UT updated

Signed-off-by: Abhishek Dosi <abdosi@microsoft.com>
abdosi pushed a commit to sonic-net/sonic-host-services that referenced this issue Jul 11, 2024
…Table. (#135) (#137)

What I did:
Added support to render template format of delayed flag on Feature Table.

Why I did:
To fix: sonic-net/sonic-buildimage#19126

How I verify:
Manual Verification
UT updated

Signed-off-by: Abhishek Dosi abdosi@microsoft.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment