Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Mellanox] Stop pmon ahead of syncd #3505

Merged
merged 8 commits into from
Sep 27, 2019

Conversation

stephenxs
Copy link
Collaborator

- What I did
Issue Overview
shutdown flow
For any shutdown flow, which means all dockers are stopped in order, pmon docker stops after syncd docker has stopped, causing pmon docker fail to release sx_core resources and leaving sx_core in a bad state. The related logs are like the following:

INFO syncd.sh[23597]: modprobe: FATAL: Module sx_core is in use.
INFO syncd.sh[23597]: Unloading sx_core[FAILED]
INFO syncd.sh[23597]: rmmod: ERROR: Module sx_core is in use

config reload & service swss.restart
In the flows like "config reload" and "service swss restart", the failure cause further consequences:

  1. sx_core initialization error with error message like "sx_core: create EMAD sdq 0 failed. err: -16"
  2. syncd fails to execute the create switch api with error message "syncd_main: Runtime error: :- processEvent: failed to execute api: create, key: SAI_OBJECT_TYPE_SWITCH:oid:0x21000000000000, status: SAI_STATUS_FAILURE"
  3. swss fails to call SAI API "SAI_SWITCH_ATTR_INIT_SWITCH", which causes orchagent to restart. This will introduce an extra 1 or 2 minutes for the system to be available, failing related test cases.

reboot, warm-reboot & fast-reboot
In the reboot flows including "reboot", "fast-reboot" and "warm-reboot" this failure doesn't have further negative effects since the system has already rebooted. In addition, "warm-reboot" requires the system to be shutdown as soon as possible to meet the GR time restriction of both BGP and LACP. "fast-reboot" also requires to meet the GR time restriction of BGP which is longer than LACP. In this sense, any unnecessary steps should be avoided. It's better to keep those flows untouched.

summary
To summarize, we have to come up with a way to ensure:

  1. shutdown pmon docker ahead of syncd for "config reload" or "service swss restart" flow;
  2. don't shutdown pmon docker ahead of syncd for "fast-reboot" or "warm-reboot" flow in order to save time.
  3. for "reboot" flow, either order is acceptable.

Solution
To solve the issue, pmon shoud be stopped ahead of syncd stopped for all flows except for the warm-reboot.

- How I did it

  1. To stop pmon ahead of syncd stopped. This is done in /usr/local/bin/syncd.sh::stop() and for all shutdown sequence.
  2. Now pmon stops ahead of syncd so there must be a way in which pmon can start after syncd started. Another point that should be taken consideration is that pmon starting should be deferred so that services which have the logic of graceful restart in fast-reboot and warm-reboot have sufficient CPU cycles to meet their deadline.
    This is done by add "syncd.service" as "After" to pmon.service and startin /usr/local/bin/syncd.sh::wait()
    To start pmon automatically after syncd started.

- How to verify it
Test the following flows and ensure pmon and syncd started and stopped in the correct sequence:

  1. config reload
  2. service swss restart
  3. regular reboot
  4. warm-reboot
  5. fast-reboot

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Stephen Sun and others added 7 commits September 4, 2019 17:04
…onal branch to fix the issue that pmon not started after warm reboot
During system starting, pmon isn't supposed to start ahead of syncd starting in order to avoid racing condition between syncd and pmon.
Currently it is done by killing pmon if is alive when syncd is starting. However such implementation is still risky. Consider the following flow:
1. pmon is inactive when syncd.sh is checking. but syncd.sh is scheduled out somehow just ahead of "chipdown" called
2. systemd is switched in and starts pmon service
3. at this point, pmon and syncd are running simultaneously, critical section broken and racing condition formed
To prevent that issue, ony solution is to add syncd as "After" in pmon.service, which ensure that whenever pmon starts syncd has been started.
However, dong so requires to defer starting pmon.service after syncd.service has fully started otherwise a deadlock is formed as following:
1. syncd.sh starts pmon ahead of itself fully started, while
2. pmon not being able to start due to syncd, one of its "After", not fully started.
3. as a result, syncd and pmon have to wait for each other forever
To solve that, move starting pmon.service to "wait()" so that pmon is started after syncd fully started, breaking the deadlock.
…-start

[syncd.sh,pmon.service] Prevent pmon from starting ahead of syncd
@lguohan lguohan merged commit 7308d2e into sonic-net:master Sep 27, 2019
@stephenxs stephenxs deleted the stop-pmon-ahead-of-syncd branch September 27, 2019 08:35
mssonicbld added a commit that referenced this pull request Aug 29, 2024
…atically (#20069)

#### Why I did it
src/sonic-utilities
```
* d7788d4d - (HEAD -> 202311, origin/202311) Add lock to config reload/load_minigraph (#3475) (#3505) (10 hours ago) [Longxiang Lyu]
* b36b9b50 - [config] no op if Golden Config is invalid (#3367) (18 hours ago) [jingwenxie]
* f72699f7 - [config] Check golden config exist early if flag is set (#3169) (#3504) (22 hours ago) [mssonicbld]
```
#### How I did it
#### How to verify it
#### Description for the changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants