Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update swss_ready check to check per namespace swss service #1974

Merged
merged 2 commits into from
Dec 22, 2021

Conversation

SuvarnaMeenakshi
Copy link
Contributor

What I did

fixes sonic-net/sonic-buildimage#9411
on multi-asic platform, config reload CLI was not working without -f option.
This was because swss_ready function was checking status of swss.service and multi-asic platform will not have swss.service, it will have per-namespace swss service.

How I did it

Fix swss_ready function to check all swss services status running on the platform.

How to verify it

On multi-asic platform, before Fix:
admin@vlab-08:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks

After fix, on multi-asic platform, execute config load_minigraph to reload all services, immediately after that execute config reload. This should print swss not ready message.
Once swss services are up, execute config reload again and this time it should run without -f option.

admin@vlab-08:~$ sudo config load_minigraph -y
Disabling container monitoring ...
Stopping SONiC target ...
..
admin@vlab-08:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-08:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-08:~$ sudo config reload -y
Running command: rm -rf /tmp/dropstat-*
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen  -j /etc/sonic/config_db.json  -j /etc/sonic/init_cfg.json  --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen  -j /etc/sonic/config_db0.json  -j /etc/sonic/init_cfg.json  -n asic0  --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate -n asic0
Running command: /usr/local/bin/sonic-cfggen  -j /etc/sonic/config_db1.json  -j /etc/sonic/init_cfg.json  -n asic1  --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate -n asic1
Running command: /usr/local/bin/sonic-cfggen  -j /etc/sonic/config_db2.json  -j /etc/sonic/init_cfg.json  -n asic2  --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate -n asic2
Running command: /usr/local/bin/sonic-cfggen  -j /etc/sonic/config_db3.json  -j /etc/sonic/init_cfg.json  -n asic3  --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate -n asic3
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon

admin@vlab-08:~$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED       STATUS              PORTS     NAMES
81bb5eda49c7   docker-router-advertiser:latest      "/usr/bin/docker-ini…"   5 hours ago   Up 58 seconds                 radv
c15141bab5c8   docker-lldp:latest                   "/usr/bin/docker-lld…"   5 hours ago   Up 59 seconds                 lldp0
6e04586084ac   docker-lldp:latest                   "/usr/bin/docker-lld…"   5 hours ago   Up 59 seconds                 lldp2
f0a1003ad30e   docker-sonic-mgmt-framework:latest   "/usr/local/bin/supe…"   5 hours ago   Up 59 seconds                 mgmt-framework
d26c2cd31995   docker-sonic-telemetry:latest        "/usr/local/bin/supe…"   5 hours ago   Up 59 seconds                 telemetry
88f09b440b2e   docker-snmp:latest                   "/usr/local/bin/supe…"   5 hours ago   Up 58 seconds                 snmp
39149a821ced   docker-lldp:latest                   "/usr/bin/docker-lld…"   5 hours ago   Up About a minute             lldp3
70b18b3dc0ad   docker-lldp:latest                   "/usr/bin/docker-lld…"   5 hours ago   Up 59 seconds                 lldp1
32c8ab784421   docker-syncd-vs:latest               "/usr/local/bin/supe…"   5 hours ago   Up About a minute             syncd0
34301c9ec75c   docker-syncd-vs:latest               "/usr/local/bin/supe…"   5 hours ago   Up About a minute             syncd3
6e50ff68bceb   docker-teamd:latest                  "/usr/local/bin/supe…"   5 hours ago   Up About a minute             teamd1
1a280a802f56   docker-syncd-vs:latest               "/usr/local/bin/supe…"   5 hours ago   Up About a minute             syncd1
ad2e0746e4db   docker-teamd:latest                  "/usr/local/bin/supe…"   5 hours ago   Up About a minute             teamd0
5ec3f7d12756   docker-syncd-vs:latest               "/usr/local/bin/supe…"   5 hours ago   Up About a minute             syncd2
9934bbad6dbb   docker-teamd:latest                  "/usr/local/bin/supe…"   5 hours ago   Up About a minute             teamd3
dd968a9047cd   docker-teamd:latest                  "/usr/local/bin/supe…"   5 hours ago   Up About a minute             teamd2
f786f22755f4   docker-orchagent:latest              "/usr/bin/docker-ini…"   5 hours ago   Up About a minute             swss0
d289e9cfe24e   docker-orchagent:latest              "/usr/bin/docker-ini…"   5 hours ago   Up About a minute             swss3
1af61d7a515e   docker-orchagent:latest              "/usr/bin/docker-ini…"   5 hours ago   Up About a minute             swss1
2e41c739ab92   docker-orchagent:latest              "/usr/bin/docker-ini…"   5 hours ago   Up About a minute             swss2
854709d17784   docker-gbsyncd-vs:latest             "/usr/local/bin/supe…"   5 hours ago   Up About a minute             gbsyncd
13b6a70936a4   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   5 hours ago   Up About a minute             bgp2
0b79dd040480   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   5 hours ago   Up About a minute             bgp1
d399e4663437   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   5 hours ago   Up About a minute             bgp3
0a828098d038   docker-fpm-frr:latest                "/usr/bin/docker_ini…"   5 hours ago   Up About a minute             bgp0
e872fe014506   docker-lldp:latest                   "/usr/bin/docker-lld…"   5 hours ago   Up About a minute             lldp
8e76c59481e0   docker-platform-monitor:latest       "/usr/bin/docker_ini…"   5 hours ago   Up About a minute             pmon
edf8db5ce850   docker-database:latest               "/usr/local/bin/dock…"   5 hours ago   Up 5 hours                    database0
f0d325b5abfb   docker-database:latest               "/usr/local/bin/dock…"   5 hours ago   Up 5 hours                    database1
67515aad1976   docker-database:latest               "/usr/local/bin/dock…"   5 hours ago   Up 5 hours                    database2
060396d764c9   docker-database:latest               "/usr/local/bin/dock…"   5 hours ago   Up 5 hours                    database3
d4c131e48bdc   docker-database:latest               "/usr/local/bin/dock…"   5 hours ago   Up 5 hours                    database

Verified that there is no change on single asic VS:

admin@vlab-01:~$ sudo config load_minigraph -y
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen -H -m -j /etc/sonic/init_cfg.json   --write-to-db
Running command: pfcwd start_default
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Running command: config qos reload --no-dynamic-buffer
Running command: /usr/local/bin/sonic-cfggen  -d --write-to-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/Force10-S6000/buffers.json.j2,config-db -t /usr/share/sonic/device/x86_64-kvm_x86_64-r0/Force10-S6000/qos.json.j2,config-db -y /etc/sonic/sonic_version.yml
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon
Please note setting loaded from minigraph will be lost after system reboot. To preserve setting, run `config save`.
admin@vlab-01:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-01:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-01:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-01:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-01:~$ sudo config reload -y
SwSS container is not ready. Retry later or use -f to avoid system checks
admin@vlab-01:~$ sudo config reload -y
Running command: rm -rf /tmp/dropstat-*
Disabling container monitoring ...
Stopping SONiC target ...
Running command: /usr/local/bin/sonic-cfggen  -j /etc/sonic/config_db.json  -j /etc/sonic/init_cfg.json  --write-to-db
Running command: /usr/local/bin/db_migrator.py -o migrate
Running command: /usr/local/bin/sonic-cfggen -d -y /etc/sonic/sonic_version.yml -t /usr/share/sonic/templates/sonic-environment.j2,/etc/sonic/sonic-environment
Restarting SONiC target ...
Enabling container monitoring ...
Reloading Monit configuration ...
Reinitializing monit daemon

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
(cherry picked from commit 4457f05)
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
@abdosi
Copy link
Contributor

abdosi commented Dec 21, 2021

@anamehra can you try this fix.

@abdosi
Copy link
Contributor

abdosi commented Dec 21, 2021

/azp Azure.sonic-utilities

@azure-pipelines
Copy link

Command 'Azure.sonic-utilities' is not supported by Azure Pipelines.

Supported commands
  • help:
    • Get descriptions, examples and documentation about supported commands
    • Example: help "command_name"
  • list:
    • List all pipelines for this repository using a comment.
    • Example: "list"
  • run:
    • Run all pipelines or specific pipelines for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify specific pipelines to run.
    • Example: "run" or "run pipeline_name, pipeline_name, pipeline_name"
  • where:
    • Report back the Azure DevOps orgs that are related to this repository and org
    • Example: "where"

See additional documentation.

@abdosi
Copy link
Contributor

abdosi commented Dec 21, 2021

/azp

@azure-pipelines
Copy link

Supported commands
  • help:
    • Get descriptions, examples and documentation about supported commands
    • Example: help "command_name"
  • list:
    • List all pipelines for this repository using a comment.
    • Example: "list"
  • run:
    • Run all pipelines or specific pipelines for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify specific pipelines to run.
    • Example: "run" or "run pipeline_name, pipeline_name, pipeline_name"
  • where:
    • Report back the Azure DevOps orgs that are related to this repository and org
    • Example: "where"

See additional documentation.

@abdosi
Copy link
Contributor

abdosi commented Dec 21, 2021

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@anamehra
Copy link
Contributor

@anamehra can you try this fix.

@abdosi , tried on multi-asic LC and Sup, looks good:

root@sonic:/home/cisco# config reload
Clear current config and reload config in config_db format from the default config file(s) ? [y/N]: y
Running command: rm -rf /tmp/dropstat-*
Disabling container monitoring ...
Stopping SONiC target ...

@abdosi abdosi merged commit bb56fc2 into sonic-net:master Dec 22, 2021
@SuvarnaMeenakshi SuvarnaMeenakshi deleted the swss_ready2 branch December 22, 2021 01:15
judyjoseph pushed a commit that referenced this pull request Jan 9, 2022
What I did
fixes sonic-net/sonic-buildimage#9411
on multi-asic platform, config reload CLI was not working without -f option.
This was because swss_ready function was checking status of swss.service and multi-asic platform will not have swss.service, it will have per-namespace swss service.

How I did it
Fix swss_ready function to check all swss services status running on the platform.
@qiluo-msft
Copy link
Contributor

This commit could not be cleanly cherry-picked to 202012. Please submit another PR.

SuvarnaMeenakshi added a commit to SuvarnaMeenakshi/sonic-utilities that referenced this pull request Feb 8, 2023
…t#1974)

What I did
fixes sonic-net/sonic-buildimage#9411
on multi-asic platform, config reload CLI was not working without -f option.
This was because swss_ready function was checking status of swss.service and multi-asic platform will not have swss.service, it will have per-namespace swss service.

How I did it
Fix swss_ready function to check all swss services status running on the platform.

(cherry picked from commit bb56fc2)
@liuh-80 liuh-80 added the Rejected for 202012 Branch Label for a reject the cherry-pick request for 202012 branch label Feb 9, 2023
@liuh-80
Copy link
Contributor

liuh-80 commented Feb 9, 2023

Because 202012 not run on multi-asic device, add "Rejected for 202012 branch" label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

config reload reports SwSS container not running for multi-asic
7 participants