[mirroring] mirror rule fails to apply after warm reboot

[stepanblyschak]

MirrorOrch and AclOrch don't process configuration correctly on warm restoration:
sonic_dump_r-qa-sw-eth-214_20200313_221021.tar.gz

Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet24 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet28 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet16 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet20 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet8 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet12 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet0 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet4 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- createBindAclTableGroup: Create ingress ACL table group and bind port Ethernet56 to it
Mar 13 22:08:28.704533 sonic NOTICE swss#orchagent: :- addAclTable: Created ACL table EVERFLOW oid:70000000005fb
Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- validateAddAction: Mirror rule reference mirror session that does not exists session1
Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_INGRESS_ACTION : session1'
Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Mar 13 22:08:40.016726 sonic NOTICE swss#orchagent: :- postBake: Start MirrorOrch post-baking
Mar 13 22:08:40.016726 sonic NOTICE swss#orchagent: :- createEntry: Created mirror session session1
Mar 13 22:08:40.016726 sonic NOTICE swss#orchagent: :- attach: Attached next hop observer of route 100.64.0.0/24 for destination IP 100.64.0.2
Mar 13 22:08:40.024539 sonic NOTICE swss#orchagent: :- updateNextHop: Updating mirror session session1 with route 100.64.0.0/24
Mar 13 22:08:40.024539 sonic NOTICE swss#orchagent: :- updateNextHop:     next hop IPs: 0.0.0.0@Ethernet64
Mar 13 22:08:40.024539 sonic NOTICE swss#orchagent: :- updateNextHop: Recover mirror session session1 with next hop 0.0.0.0@Ethernet64
Mar 13 22:08:40.024539 sonic NOTICE swss#orchagent: :- getNeighborInfo: Mirror session session1 neighbor is Ethernet64
Mar 13 22:08:40.036529 sonic NOTICE swss#orchagent: :- activateSession: Activated mirror session session1
Mar 13 22:08:40.036529 sonic NOTICE swss#orchagent: :- setWarmStartState: orchagent warm start state changed to restored
Mar 13 22:08:40.036529 sonic NOTICE swss#orchagent: :- warmRestoreAndSyncUp: Orchagent state restore done
Mar 13 22:08:40.036529 sonic NOTICE swss#orchagent: :- syncd_apply_view: Notify syncd APPLY_VIEW
Mar 13 22:08:40.036529 sonic NOTICE swss#orchagent: :- sai_redis_notify_syncd: sending syncd APPLY view

Adding more information about issue from our internal issue:

Steps to reproduce

1. Enable ISSU
2. Reboot switch
3. Assign ip addresses to interfaces connected to hosts.
4. Configure BGP neighboring between DUT and hosts.
5. Create Everflow table (ingress)
6. Create a session
7. Create ingress everflow rule
8. Send Flow-1 form Host-1 destined to Host-2. This traffic has to be mirrored by ingress everflow rule towards Host-3
9. Perform warm-reboot
10. During warm-rebooting verify that Flow-1 form Host-1 destined to Host-2 is mirrored by ingress everflow rule towards Host-3.Check traffic flow statistics. Both host-2 and Host-3 have to receive traffic from Host-1¶

Observed behavior
Everflow functionality doesn't work after warm-rebooting
Errors in syslog:
Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- validateAddAction: Mirror rule reference mirror session that does not exists session1
Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_INGRESS_ACTION : session1'
Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
Expected behavior
Everflow functionality should work after warm-rebooting.

[lguohan]

it is likely the mirroring is not added in the warm reboot test.

[lguohan]

what is the repro steps?

[JaredLius]

I also meet this issue. In the current design, the everflow rule must be set after mirror session created. but after warm-reboot. when restore the everflow rule, the mirror session didn't create yet, the aclorch check failed, print "Mar 13 22:08:40.016726 sonic ERR swss#orchagent: :- validateAddAction: Mirror rule reference mirror session that does not exists session1", and exit. It caused the configuration restore failed.

[tylerlinp]

[configuration]

{
    "ACL_TABLE": {
        "mirror": {
            "ports": [
                "Ethernet0" 
            ],
            "stage": "INGRESS",
            "type": "MIRROR" 
        }
    },
    "ACL_RULE": {
        "mirror|rule1": {
            "IP_TYPE": "IPV4ANY",
            "MIRROR_ACTION": "m1",
            "SRC_IP": "10.1.1.1/24" 
        }
    },
    "MIRROR_SESSION": {
        "m1": {
            "dst_ip": "34.1.1.4",
            "gre_type": "0x88be",
            "src_ip": "34.1.1.2" 
        }
    }
}

[result]

2651 Feb 28 01:16:37.708624 sonic ERR swss#orchagent: :- validateAddAction: Mirror rule reference mirror session that does not exists m1
2652 Feb 28 01:16:37.708877 sonic ERR swss#orchagent: :- doAclRuleTask: Unknown or invalid rule attribute 'MIRROR_ACTION : m1'
2653 Feb 28 01:16:37.709154 sonic ERR swss#orchagent: :- doAclRuleTask: Failed to create ACL rule. Rule configuration is invalid
2654 Feb 28 01:16:37.709154 sonic NOTICE swss#orchagent: :- postBake: Start MirrorOrch post-baking
2655 Feb 28 01:16:37.711207 sonic NOTICE swss#orchagent: :- createEntry: Created mirror session m1
2656 Feb 28 01:16:37.711207 sonic NOTICE swss#orchagent: :- attach: Attached next hop observer of route 34.1.1.0/24 for destination IP 34.1.1.4
2657 Feb 28 01:16:37.711666 sonic NOTICE swss#orchagent: :- updateNextHop: Updating mirror session m1 with route 34.1.1.0/24
2658 Feb 28 01:16:37.711666 sonic NOTICE swss#orchagent: :- updateNextHop:     next hop IPs: 0.0.0.0@Ethernet16
2659 Feb 28 01:16:37.711689 sonic NOTICE swss#orchagent: :- createEntry: Created mirror session m2
2660 Feb 28 01:16:37.711859 sonic NOTICE swss#orchagent: :- attach: Attached next hop observer of route 34.1.1.0/24 for destination IP 34.1.1.4
2661 Feb 28 01:16:37.712161 sonic NOTICE swss#orchagent: :- updateNextHop: Updating mirror session m2 with route 34.1.1.0/24
2662 Feb 28 01:16:37.712236 sonic NOTICE swss#orchagent: :- updateNextHop:     next hop IPs: 0.0.0.0@Ethernet16
2663 Feb 28 01:16:37.712303 sonic NOTICE swss#orchagent: :- warmRestoreValidation: There are pending consumer tasks after restore:
2664 Feb 28 01:16:37.712303 sonic NOTICE swss#orchagent: :- warmRestoreValidation: ACL_RULE|mirror|rule1|SET|IP_TYPE:IPV4ANY|MIRROR_ACTION:m1|SRC_IP:11
     0.1.1.1/24
2665 Feb 28 01:16:37.712325 sonic NOTICE swss#orchagent: :- setWarmStartState: orchagent warm start state changed to restored
2666 Feb 28 01:16:37.712325 sonic ERR swss#orchagent: :- warmRestoreAndSyncUp: Orchagent state restore failed
2667 Feb 28 01:16:37.712325 sonic ERR swss#orchagent: :- main: Failed to initialize orchstration daemon
2668 Feb 28 01:16:37.713569 sonic INFO swss#supervisord: orchagent terminate called without an active exception

[tylerlinp]

In our test, warm reboot failed randomly. I read the current code. MirrorOrch is freezed until postBake, so works cannot do in three iterations doTask, so warmRestoreValidation failed easily. Only if done all pending in MirrorOrch::postBake() it works. Creating ACL rule can redo so the failure log does not matter.