Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fdbshow]: Handle FDB cleanup gracefully #1918

Merged
merged 1 commit into from
Nov 17, 2021

Conversation

nazariig
Copy link
Collaborator

@nazariig nazariig commented Nov 10, 2021

Signed-off-by: Nazarii Hnydyn nazariig@nvidia.com

The race condition is caused by a blocking Redis call which gets the contents of the FDB entry from ASIC DB.
Since it has been implemented as a simple loop, there is no guarantee that entry will be present in DB when the contents are being read.

root@sonic:/home/admin# ./reproduce.sh
...
Tue Nov  9 15:58:49 UTC 2021: Query...
Tue Nov  9 15:58:52 UTC 2021: Query...
...
=> Read ASIC DB: start
=> Read ASIC DB: done
=> Read ASIC DB: start
Key 'ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000009cc","mac":"02:11:22:33:20:00","switch_id":"oid:0x21000000000000"}' unavailable in database '1'

The fdbshow script:

root@sonic:/home/admin# vim /usr/bin/fdbshow

...

class FdbShow(object):

    HEADER = ['No.', 'Vlan', 'MacAddress', 'Port', 'Type']
    FDB_COUNT = 0

    def __init__(self):
        super(FdbShow,self).__init__()
        self.db = SonicV2Connector(host="127.0.0.1")
        self.if_name_map, \
        self.if_oid_map = port_util.get_interface_oid_map(self.db)
        self.if_br_oid_map = port_util.get_bridge_port_map(self.db)
        self.fetch_fdb_data()
        return

    def fetch_fdb_data(self):
        """
            Fetch FDB entries from ASIC DB.
            FDB entries are sorted on "VlanID" and stored as a list of tuples
        """
        self.db.connect(self.db.ASIC_DB)
        self.bridge_mac_list = []

        fdb_str = self.db.keys('ASIC_DB', "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:*")
        if not fdb_str:
            return

        if self.if_br_oid_map is None:
            return

        oid_pfx = len("oid:0x")
        for s in fdb_str:
            fdb_entry = s.decode()
            fdb = json.loads(fdb_entry .split(":", 2)[-1])
            if not fdb:
                continue

            print("=> Read ASIC DB: start")
            ent = self.db.get_all('ASIC_DB', s, blocking=True)
            print("=> Read ASIC DB: done")
            br_port_id = ent[b"SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID"][oid_pfx:]
            ent_type = ent[b"SAI_FDB_ENTRY_ATTR_TYPE"]
            fdb_type = ['Dynamic','Static'][ent_type == "SAI_FDB_ENTRY_TYPE_STATIC"]

...

The reproduce.sh script:

root@sonic:/home/admin# cat reproduce.sh
#!/bin/bash

out=""
rc="0"

while [[ "${rc}" -eq "0" ]]; do
    echo "$(date): Query..."
    out="$(show mac)"
    rc="$?"
done

echo "${out}"

Closes #1866

What I did

How I did it

  • Removed blocking calls from fdbshow

How to verify it

  1. Run FDB test

Previous command output (if the output of a command-line utility has changed)

root@sonic:/home/admin# show mac
Key 'ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY:{"bvid":"oid:0x260000000009cc","mac":"02:11:22:33:20:00","switch_id":"oid:0x21000000000000"}' unavailable in database '1'

New command output (if the output of a command-line utility has changed)

root@sonic:/home/admin# show mac
No.    Vlan    MacAddress    Port    Type
-----  ------  ------------  ------  ------
Total number of entries 0

Signed-off-by: Nazarii Hnydyn <nazariig@nvidia.com>
@nazariig
Copy link
Collaborator Author

@ganglyu / @prsunny can you please approve?

@liat-grozovik
Copy link
Collaborator

@nazariig to which branches this fix is required? can you check it is cleanly cherry picked?

@liat-grozovik liat-grozovik merged commit c80321c into sonic-net:201911 Nov 17, 2021
stepanblyschak pushed a commit to stepanblyschak/sonic-utilities that referenced this pull request Apr 28, 2022
Commits on Oct 26, 2021
Remove exec from platform_reboot call to prevent reboot hang (sonic-net#1881) 066b5ad
  
Commits on Nov 17, 2021
[fdbshow]: Handle FDB cleanup gracefully. (sonic-net#1918) c80321c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants