Calico (v3.0.1) calico/node unit test on SLES 12 SP2 frozen

[samding01]

Calico (v3.0.1) calico/node unit test on SLES 12 SP2 frozen, the problem is related to iptables.
The version is v.1.4.x,

Expected Behavior

Current Behavior

Running calico/node unit test by make st on SLES 12 SP2 is frozen.
The log shows there are ERRORs in felix log, which asks for higher version of iptables (current on SLES is v1.4.21)

  2018-02-08 20:23:32.972 [ERROR][78] table.go 778: Failed to program iptables, loading diags before panic. error=exit status 1 ipVersion=0x6 table="filter"^M
  ...

see st.log

When running unit test on RHEL 7.x or Ubuntu, no problem as the iptables version is v1.6.x.
The previous Calico versions do not have such a problem.

Possible Solution

Steps to Reproduce (for bugs)

1. under calico/calico_node folder, run make st

Context

Your Environment

• Calico version v3.0.1
• Orchestrator version (e.g. kubernetes, mesos, rkt):
• Operating System and version: SLES12 sp1/sp2/sp3
• Link to your project (optional):

[caseydavenport]

@fasaxc have we recently changed the version of iptables we require?

[fasaxc]

No, but it's possible that we've introduced a new module requirement (maybe xt_ipvs). @samding01 can you run the felix FV tests in the felix repo? make fv should do that. Those tests emit the felix logs so you should get a clearer error message (the calico ST tests only show a snippet of the logs, which in this case doesn't include the actual failing iptables command).

[samding01]

@fasaxc see test_sles12_2_fv.zip,
this is make fv result of felix
By the way, make fv is done x86_64 platform, all docker images/containers are downloaded

[guirish]

We are still observing this issue in v3.0.3. We too suspect that this issue is w.r.t IPVS. Commenting line 91 in github.com/projectcalico/felix/iptables/match_builder_test.go

@@ -88,6 +88,6 @@ var _ = DescribeTable("MatchBuilder",
        Entry("Protocol and ports", Match().Protocol("tcp").SourcePorts(1234).DestPorts(8080),
                "-p tcp -m multiport --source-ports 1234 -m multiport --destination-ports 8080"),
        // IPVS.
-       Entry("IPVSConnection", Match().IPVSConnection(), "-m ipvs --ipvs"),
+       //Entry("IPVSConnection", Match().IPVSConnection(), "-m ipvs --ipvs"),
        Entry("NotIPVSConnection", Match().NotIPVSConnection(), "-m ipvs ! --ipvs"),
 )

we were able to reduce the test failure from 150 to 31 with make fv command.

[fasaxc]

@samding01 @guirish Do the tests pass if you manually load the xt_ipvs kernel module before you start?

Can you reproduce on any of our officially supported distributions (Ubuntu or RH/CentOS) or is this SLES-specific?

[guirish]

@fasaxc this issue is SLES specific only , we did not observe it on RHEL(7.3 and 7.4) and Ubuntu(16.04 and 17.10) distros. Does it mean that Calico is NOT going to officially claim support of SLES?

[caseydavenport]

This appears down to IPVS support on the system. iptables should still work fine, as should Calico in iptables mode, and IPVS may well work too if the correct version of iptables is installed and the right kernel features are present.

In general, I think UTs are not guaranteed to pass on any given system, even in supported environments.