Add more debug information when PFC WD is triggered

[stephenxs]

What I did

Add more debug information when PFC WD is triggered

Why I did it

When PFC WD occurs, we need the relevant counters before and after. But PFC WD is checked at a very small period of time and it's very hard to collect the counters at that period, otherwise the data will be very large.
It will be helpful if the counters can be logged when the PFC WD is triggered.

How I verified it

Manually and regression test.

Details if related

1. The lua plugin to decide what counters to be added to the log. For vendors that do not provide it, nothing will be added.
2. PFC WD orchagent to handle the additional counters info, printing to syslog and publishing via event mechanism

[stephenxs]

Failed by a common issue which fails every PR

    def wait_for_result(
        polling_function: Callable[[], Tuple[bool, Any]],
        polling_config: PollingConfig = PollingConfig(),
        failure_message: str = None,
    ) -> Tuple[bool, Any]:
        """Run `polling_function` periodically using the specified `polling_config`.
    
        Args:
            polling_function: The function being polled. The function cannot take any arguments and
                must return a status which indicates if the function was succesful or not, as well as
                some return value.
            polling_config: The parameters to use to poll the polling function.
            failure_message: The message to print if the call times out. This will only take effect
                if the PollingConfig is set to strict.
    
        Returns:
            If the polling function succeeds, then this method will return True and the output of the
            polling function.
    
            If it does not succeed within the provided timeout, it will return False and whatever the
            output of the polling function was on the final attempt.
        """
        for _ in range(polling_config.iterations()):
            status, result = polling_function()
    
            if status:
                return (True, result)
    
            time.sleep(polling_config.polling_interval)
    
        if polling_config.strict:
            message = failure_message or f"Operation timed out after {polling_config.timeout} seconds with result {result}"
>           assert False, message
E           AssertionError: Operation timed out after 60 seconds with result {'arp_update': 'STOPPED', 'bgpd': 'STOPPED', 'buffermgrd': 'STOPPED', 'containercfgd': 'STOPPED', 'coppmgrd': 'STOPPED', 'fdbsyncd': 'STOPPED', 'fpmsyncd': 'STOPPED', 'gbsyncd': 'STOPPED', 'gearsyncd': 'STOPPED', 'intfmgrd': 'STOPPED', 'natmgrd': 'STOPPED', 'natsyncd': 'STOPPED', 'nbrmgrd': 'STOPPED', 'neighsyncd': 'STOPPED', 'orchagent': 'STOPPED', 'portmgrd': 'STOPPED', 'portsyncd': 'STOPPED', 'redis-chassis': 'STOPPED', 'redis-server': 'RUNNING', 'restore_neighbors': 'STOPPED', 'rsyslogd': 'RUNNING', 'sflowmgrd': 'STOPPED', 'start.sh': 'EXITED', 'staticd': 'STOPPED', 'syncd': 'FATAL', 'teammgrd': 'STOPPED', 'teamsyncd': 'STOPPED', 'tunnelmgrd': 'STOPPED', 'vlanmgrd': 'STOPPED', 'vrfmgrd': 'STOPPED', 'vxlanmgrd': 'STOPPED', 'zebra': 'STOPPED'}

[stephenxs]

/easycla

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

No commit pushedDate could be found for PR 2858 in repo sonic-net/sonic-swss

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[liat-grozovik]

@yxieca @neethajohn kindly reminder to review or assign someone

[neethajohn]

@stephenxs, has this change been tested against sonic-mgmt testcases? Do any of the existing testcase need modification?

[liat-grozovik]

@neethajohn @stephenxs can you follow up on this PR? i suggest we will backport it to 202305 as well. This is for debug purposes but it is the release maintainer call if needed or not.

[stephenxs]

Ack. I'm on it, enhancing sonic-mgmt test cases. 获取 Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ 发件人: Liat Grozovik ***@***.***> 发送时间: 星期三, 十月 11, 2023 12:31 上午 收件人: sonic-net/sonic-swss ***@***.***> 抄送: Stephen Sun ***@***.***>; Mention ***@***.***> 主题: Re: [sonic-net/sonic-swss] Add more debug information when PFC WD is triggered (PR #2858) @neethajohn<https://github.com/neethajohn> @stephenxs<https://github.com/stephenxs> can you follow up on this PR? i suggest we will backport it to 202305 as well. This is for debug purposes but it is the release maintainer call if needed or not. ― Reply to this email directly, view it on GitHub<#2858 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABJBIZBXIXBHZSCDM35BKFTX6VZ55AVCNFSM6AAAAAA2I47RLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONJVHAYDCMZTGQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[stephenxs]

/azpw run

[mssonicbld]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[StormLiangMS]

enhancement

[stephenxs]

enhancement

Hi @StormLiangMS
This is an enhancement but it is important. Currently, without this PR, there is no diagnosis information when a PFC watchdog is triggered, which makes it very difficult for customers/developers to understand what happened. In this sense, we would like to have it in 202305 as well.
If we decided not to take it into 202305, we may want to revert sonic-net/sonic-mgmt#10305 from 202305 as well.

[StormLiangMS]

@stephenxs could you update the test result with 202305?

[stephenxs]

Hi It passes all PFC WD test on 202305 I can attach a test log later 获取 Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ 发件人: StormLiangMS ***@***.***> 发送时间: 星期三, 十一月 8, 2023 6:24 下午 收件人: sonic-net/sonic-swss ***@***.***> 抄送: Stephen Sun ***@***.***>; Mention ***@***.***> 主题: Re: [sonic-net/sonic-swss] Add more debug information when PFC WD is triggered (PR #2858) @stephenxs<https://github.com/stephenxs> could you update the test result with 202305? ― Reply to this email directly, view it on GitHub<#2858 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABJBIZEJLJMOUWCV2TTUTO3YDNMUNAVCNFSM6AAAAAA2I47RLOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBRGU2TGMBWGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[StormLiangMS]

Hi @stephenxs thanks, is this a general change or only for Mellanox?

[stephenxs]

Hi @stephenxs thanks, is this a general change or only for Mellanox?

Hi @StormLiangMS
Currently, it's for Mellanox only.
Other vendors can add their own diagnosis info in their lua plug if they want to in the future.

[stephenxs]

A test log: pfc test.txt