Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Chassis] Errors seen with a config reload, config load_minigrph or reboot on LC #16787

Closed
mannytaheri opened this issue Oct 6, 2023 · 10 comments
Closed

[Chassis] Errors seen with a config reload, config load_minigrph or reboot on LC #16787

mannytaheri opened this issue Oct 6, 2023 · 10 comments
Assignees
@vmittal-msft
Labels
MSFT P0 Priority of the issue Triaged this issue has been triaged

Comments

@mannytaheri
Copy link

Description

We see many error messages when we do either a config reload, config load_minigraph or a reboot on a J2C+ line card in a multi asics chassis.

Steps to reproduce the issue:

  1. Do either a config reload, config load_minigraph or reboot on a J2C+ line card

Describe the results you received:

We see error messages like the following:
The messages get repeated for every port on the line card

Oct  6 16:45:50.072611 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072611 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:50.072633 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072633 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:50.072644 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072654 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:50.072671 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072671 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:50.072688 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072688 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:50.072705 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072705 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:50.072732 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct  6 16:45:50.072732 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
...
...
...
Oct  6 16:45:51.182566 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182578 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:51.182594 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182607 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:51.182624 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182633 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:51.182649 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182662 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:51.182678 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182691 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:51.182707 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182720 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct  6 16:45:51.182736 ixre-egl-board27 ERR swss0#orchagent: :- handleQueueTable: Port with alias:Ethernet280 not found
Oct  6 16:45:51.182748 ixre-egl-board27 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task

Describe the results you expected:

No Errors

Output of show version:

show version

SONiC Software Version: SONiC.20220532.45
SONiC OS Version: 11
Distribution: Debian 11.7
Kernel: 5.10.0-18-2-amd64
Build commit: 447280a621
Build date: Thu Sep 28 09:39:40 UTC 2023
Built by: cloudtest@7022e913c000000

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):
@mannytaheri
Copy link
Author

@vmittal-msft Have you seen this?

@saksarav-nokia
Copy link
Contributor

Also we are seeing the following errors after config reload

Oct 10 15:11:59.384636 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|0. Must contain 2 tokens
Oct 10 15:11:59.384648 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384669 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|1. Must contain 2 tokens
Oct 10 15:11:59.384669 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384694 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|2. Must contain 2 tokens
Oct 10 15:11:59.384707 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384726 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|3. Must contain 2 tokens
Oct 10 15:11:59.384726 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384752 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|4. Must contain 2 tokens
Oct 10 15:11:59.384767 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384782 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|5. Must contain 2 tokens
Oct 10 15:11:59.384797 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384814 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet256|6. Must contain 2 tokens
Oct 10 15:11:59.384821 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384843 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|0. Must contain 2 tokens
Oct 10 15:11:59.384843 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384874 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|1. Must contain 2 tokens
Oct 10 15:11:59.384874 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384898 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|2. Must contain 2 tokens
Oct 10 15:11:59.384916 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384929 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|3. Must contain 2 tokens
Oct 10 15:11:59.384949 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384956 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|4. Must contain 2 tokens
Oct 10 15:11:59.384976 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.384991 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|5. Must contain 2 tokens
Oct 10 15:11:59.385009 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Oct 10 15:11:59.385021 ixre-egl-board26 ERR swss0#orchagent: :- handleQueueTable: malformed key:ixre-egl-board26|asic1|Ethernet264|6. Must contain 2 tokens
Oct 10 15:11:59.385037 ixre-egl-board26 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task

@vmittal-msft vmittal-msft self-assigned this Oct 11, 2023
@vmittal-msft vmittal-msft added the Triaged this issue has been triaged label Oct 11, 2023
@saksarav-nokia
Copy link
Contributor

  1. config_db has QUEUE config applied to system ports, but handleQueueTable function doesn't handle the system port.
  2. orchagent adds the port with front panel names as alias for front panel ports and not as system ports, but config_db has QUEUE for system ports even for local ports. So getPort will fail
  3. Need to find out how "handleQueueTable: Port with alias:Ethernet0 not found" showing up since the config_db has systme ports and not local port names.

@saksarav-nokia
Copy link
Contributor

@vmittal-msft , tested with PR sonic-net/sonic-swss#2855 , but seeing errors reported in this issue
Oct 6 16:45:50.072611 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072611 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072633 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072633 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072644 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072654 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072671 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072671 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072688 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072688 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072705 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072705 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072732 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072732 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task

mssonicbld added a commit that referenced this issue Nov 22, 2023
@mssonicbld
[submodule] Update submodule sonic-swss to the latest HEAD automatica…
c14a8c1 
…lly (#17258)

#### Why I did it
src/sonic-swss
```
* 194566a7 - (HEAD -> master, origin/master, origin/HEAD) Fix the Orchagent Qos error messages reported in Issue #16787 (#2947) (6 hours ago) [saksarav-nokia]
```
#### How I did it
#### How to verify it
#### Description for the changelog
yxieca pushed a commit that referenced this issue Dec 13, 2023
@mssonicbld
[submodule] Update submodule sonic-swss to the latest HEAD automatica…
4ee9a5c 
…lly (#17455)

src/sonic-swss

* d839eec3 - (HEAD -> 202311, origin/202311) Add support for fabric monitor daemon (swss part). (#2920) (11 days ago) [jfeng-arista]
* 8dc0a856 - Add support for new Port SI parameters in PortsOA (#2929) (11 days ago) [Tomer Shalvi]
* 9458b855 - [hash]: Add ECMP/LAG hash algorithm to OA (#2953) (12 days ago) [Nazarii Hnydyn]
* dac3972d - [coppmgrd] Fix Copp processing logic by using Producer del instead of del from Table (13 days ago) [Vivek]
* f6a35e98 - [gcov]: Fix directory prefix issue for (#2969) (13 days ago) [Lawrence Lee]
* 14408ca3 - [Chassis][master][orchagent] : Added test case to verify WRED profile on system ports (#2954) (2 weeks ago) [vmittal-msft]
* 2ca3deb0 - [dash] fix DASH ACL Rule protocol use-after-free (#2958) (3 weeks ago) [Yakiv Huryk]
* b8841ecb - [orchagent]: Extend the SRv6Orch to support the programming of the L3Adj (#2902) (3 weeks ago) [Carmine Scarpitta]
* 194566a7 - Fix the Orchagent Qos error messages reported in Issue #16787 (#2947) (3 weeks ago) [saksarav-nokia]
@saksarav-nokia
Copy link
Contributor

@vmittal-msft , all these issues got fixed right?

@amitpawar12
Copy link

@vmittal-msft

Steps to reproduce:

  1. Issue 'ecnconfig -q 4 on' on the box. One can use option -q 3 as well to see this issue.
sudo ip netns exec asic0 ecnconfig -q 4 on
  1. Logs seen
Jul 15 20:49:06.801193 ixre-egl-board73 ERR swss0#orchagent: :- handleQueueTable: malformed key:Ethernet0|4. Must contain 4 tokens
Jul 15 20:49:06.801193 ixre-egl-board73 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Jul 15 20:49:06.801310 ixre-egl-board73 ERR swss0#orchagent: :- handleQueueTable: malformed key:Ethernet8|4. Must contain 4 tokens
Jul 15 20:49:06.801323 ixre-egl-board73 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task

Please let me know if the issue is fixed. Will open a new issue in case the above one is fixed.

Thanks,
-A

@vmittal-msft
Copy link
Contributor

@amitpawar12 i am looking into this.

@abdosi abdosi added the P0 Priority of the issue label Aug 28, 2024
@vmittal-msft
Copy link
Contributor

Original issue has been fixed in latest image. We are no more seeing -
Oct 6 16:45:50.072611 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072611 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072633 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072633 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072644 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072654 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072671 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072671 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072688 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072688 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072705 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072705 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
Oct 6 16:45:50.072732 ixre-egl-board27 ERR swss1#orchagent: :- handleQueueTable: Port with alias:Ethernet0 not found
Oct 6 16:45:50.072732 ixre-egl-board27 ERR swss1#orchagent: :- doTask: Failed to process invalid QOS task
...

@vmittal-msft
Copy link
Contributor

@amitpawar12 please open a separate issue for this -

Steps to reproduce:
Issue 'ecnconfig -q 4 on' on the box. One can use option -q 3 as well to see this issue.
sudo ip netns exec asic0 ecnconfig -q 4 on
Logs seen
Jul 15 20:49:06.801193 ixre-egl-board73 ERR swss0#orchagent: :- handleQueueTable: malformed key:Ethernet0|4. Must contain 4 tokens
Jul 15 20:49:06.801193 ixre-egl-board73 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task
Jul 15 20:49:06.801310 ixre-egl-board73 ERR swss0#orchagent: :- handleQueueTable: malformed key:Ethernet8|4. Must contain 4 tokens
Jul 15 20:49:06.801323 ixre-egl-board73 ERR swss0#orchagent: :- doTask: Failed to process invalid QOS task

@vmittal-msft
Copy link
Contributor

We are seeing different set of errors during LC reboot, i will open a separate issue for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vmittal-msft vmittal-msft

Labels
MSFT P0 Priority of the issue Triaged this issue has been triaged
Projects
SONiC Chassis
Archived in project
Milestone
No milestone
Development

No branches or pull requests
5 participants
@vmittal-msft @abdosi @saksarav-nokia @mannytaheri @amitpawar12