Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sending IPFIX flow records for Antrea flow exporter #825

Merged
merged 15 commits into from
Aug 11, 2020
Merged

Support sending IPFIX flow records for Antrea flow exporter #825

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
1d9ab19
Support IPFIX flow records for flow exporter feature:
srikartati Jun 9, 2020
628ef8f
Support UDP transport to send IPFIX flow records
srikartati Jun 17, 2020
2cc62b4
Flow exporter feature
srikartati Jun 23, 2020
9240e7e
Fixed memory lead issue and addressed new comments
srikartati Jul 1, 2020
7495f1e
Resolving conflicts with ToT and address template refresh issue
srikartati Jul 7, 2020
1fea8a4
Fixed starting record issue
srikartati Jul 14, 2020
d3fafae
Replaced personal ipfix library to vmware/go-ipfix library
srikartati Jul 15, 2020
6abc0f8
Added integration test to increase test coverage
srikartati Jul 20, 2020
52bb040
Address comments from 7/17
srikartati Jul 21, 2020
7eb03b5
Improve unit test coverage
srikartati Jul 27, 2020
a8ea1d5
Do not send double copies of flow records
srikartati Jul 30, 2020
090ef09
Addressed comments from 7/30
srikartati Aug 1, 2020
719f8e0
Address comments 8/6
srikartati Aug 7, 2020
841469c
Addressed comments
srikartati Aug 8, 2020
0ef571a
Addreseed the comments
srikartati Aug 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions build/images/ipfixcollector/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM ubuntu:18.04

LABEL maintainer="Antrea <projectantrea-dev@googlegroups.com>"
LABEL description="A Docker image based on Ubuntu 18.04 which contains a simple IPFIX collector to run flow exporter tests"

WORKDIR /ipfix

ADD https://svwh.dl.sourceforge.net/project/libipfix/libipfix/libipfix-impd4e_110224.tgz .

RUN apt-get update && \
apt-get install -y --no-install-recommends libc6-dev build-essential libpcap0.8-dev && \
tar -xvf libipfix-* && rm libipfix-* && \
cd libipfix_* && ./configure && make && make install && ldconfig && \
cp collector/ipfix_collector /usr/local/bin && \
cd .. && \
rm -rf libipfix_* && \
apt-get remove -y build-essential && \
rm -rf /var/cache/apt/* /var/lib/apt/lists/*

ENTRYPOINT "ipfix_collector"
17 changes: 17 additions & 0 deletions build/images/ipfixcollector/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# images/ipfixcollector

This Docker image is based on Ubuntu 18.04 which includes an IPFIX collector based on [libipfix](http://libipfix.sourceforge.net/), a C library.
In this image, IPFIX collector listening on tcp:4739 port.

libipfix package is downloaded from https://svwh.dl.sourceforge.net/project/libipfix/libipfix/libipfix-impd4e_110224.tgz
srikartati marked this conversation as resolved.
Show resolved Hide resolved

New version of the image can be built and pushed to Dockerhub using following instructions:

```bash
cd build/images/ipfixcollector
docker build -t antrea/ipfixcollector:latest .
docker push antrea/ipfixcollector:latest
```

The `docker push` command will fail if you do not have permission to push to the
`antrea` Dockerhub repository.
21 changes: 18 additions & 3 deletions build/yamls/antrea-aks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,21 @@ data:

# Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener.
#enablePrometheusMetrics: false

# Provide flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also enables
# the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto is given,
# we consider tcp as default.
#flowCollectorAddr: ""

# Provide flow poll interval as a duration string. This determines how often the flow exporter dumps connections from the conntrack module.
# Flow poll interval should be greater than or equal to 1s (one second).
# Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
#flowPollInterval: "5s"

# Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
# the flow collector.
# Flow export frequency should be greater than or equal to 1.
#flowExportFrequency: 12
antrea-cni.conflist: |
{
"cniVersion":"0.3.0",
Expand Down Expand Up @@ -770,7 +785,7 @@ metadata:
annotations: {}
labels:
app: antrea
name: antrea-config-24f6gdd4fb
name: antrea-config-62554ht95b
namespace: kube-system
---
apiVersion: v1
Expand Down Expand Up @@ -876,7 +891,7 @@ spec:
key: node-role.kubernetes.io/master
volumes:
- configMap:
name: antrea-config-24f6gdd4fb
name: antrea-config-62554ht95b
name: antrea-config
- name: antrea-controller-tls
secret:
Expand Down Expand Up @@ -1091,7 +1106,7 @@ spec:
operator: Exists
volumes:
- configMap:
name: antrea-config-24f6gdd4fb
name: antrea-config-62554ht95b
name: antrea-config
- hostPath:
path: /etc/cni/net.d
Expand Down
21 changes: 18 additions & 3 deletions build/yamls/antrea-eks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,21 @@ data:

# Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener.
#enablePrometheusMetrics: false

# Provide flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also enables
# the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto is given,
# we consider tcp as default.
#flowCollectorAddr: ""

# Provide flow poll interval as a duration string. This determines how often the flow exporter dumps connections from the conntrack module.
# Flow poll interval should be greater than or equal to 1s (one second).
# Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
#flowPollInterval: "5s"

# Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
# the flow collector.
# Flow export frequency should be greater than or equal to 1.
#flowExportFrequency: 12
antrea-cni.conflist: |
{
"cniVersion":"0.3.0",
Expand Down Expand Up @@ -770,7 +785,7 @@ metadata:
annotations: {}
labels:
app: antrea
name: antrea-config-24f6gdd4fb
name: antrea-config-62554ht95b
namespace: kube-system
---
apiVersion: v1
Expand Down Expand Up @@ -876,7 +891,7 @@ spec:
key: node-role.kubernetes.io/master
volumes:
- configMap:
name: antrea-config-24f6gdd4fb
name: antrea-config-62554ht95b
name: antrea-config
- name: antrea-controller-tls
secret:
Expand Down Expand Up @@ -1093,7 +1108,7 @@ spec:
operator: Exists
volumes:
- configMap:
name: antrea-config-24f6gdd4fb
name: antrea-config-62554ht95b
name: antrea-config
- hostPath:
path: /etc/cni/net.d
Expand Down
21 changes: 18 additions & 3 deletions build/yamls/antrea-gke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,21 @@ data:

# Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener.
#enablePrometheusMetrics: false

# Provide flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also enables
# the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto is given,
# we consider tcp as default.
#flowCollectorAddr: ""

# Provide flow poll interval as a duration string. This determines how often the flow exporter dumps connections from the conntrack module.
# Flow poll interval should be greater than or equal to 1s (one second).
# Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
#flowPollInterval: "5s"

# Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
# the flow collector.
# Flow export frequency should be greater than or equal to 1.
#flowExportFrequency: 12
antrea-cni.conflist: |
{
"cniVersion":"0.3.0",
Expand Down Expand Up @@ -770,7 +785,7 @@ metadata:
annotations: {}
labels:
app: antrea
name: antrea-config-8gtt9dfdgg
name: antrea-config-9gt8khtcth
namespace: kube-system
---
apiVersion: v1
Expand Down Expand Up @@ -876,7 +891,7 @@ spec:
key: node-role.kubernetes.io/master
volumes:
- configMap:
name: antrea-config-8gtt9dfdgg
name: antrea-config-9gt8khtcth
name: antrea-config
- name: antrea-controller-tls
secret:
Expand Down Expand Up @@ -1091,7 +1106,7 @@ spec:
operator: Exists
volumes:
- configMap:
name: antrea-config-8gtt9dfdgg
name: antrea-config-9gt8khtcth
name: antrea-config
- hostPath:
path: /etc/cni/net.d
Expand Down
21 changes: 18 additions & 3 deletions build/yamls/antrea-ipsec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,21 @@ data:

# Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener.
#enablePrometheusMetrics: false

# Provide flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also enables
# the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto is given,
# we consider tcp as default.
#flowCollectorAddr: ""

# Provide flow poll interval as a duration string. This determines how often the flow exporter dumps connections from the conntrack module.
# Flow poll interval should be greater than or equal to 1s (one second).
# Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
#flowPollInterval: "5s"

# Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
# the flow collector.
# Flow export frequency should be greater than or equal to 1.
#flowExportFrequency: 12
antrea-cni.conflist: |
{
"cniVersion":"0.3.0",
Expand Down Expand Up @@ -770,7 +785,7 @@ metadata:
annotations: {}
labels:
app: antrea
name: antrea-config-76fcd44cth
name: antrea-config-gf96hhfdg8
namespace: kube-system
---
apiVersion: v1
Expand Down Expand Up @@ -885,7 +900,7 @@ spec:
key: node-role.kubernetes.io/master
volumes:
- configMap:
name: antrea-config-76fcd44cth
name: antrea-config-gf96hhfdg8
name: antrea-config
- name: antrea-controller-tls
secret:
Expand Down Expand Up @@ -1135,7 +1150,7 @@ spec:
operator: Exists
volumes:
- configMap:
name: antrea-config-76fcd44cth
name: antrea-config-gf96hhfdg8
name: antrea-config
- hostPath:
path: /etc/cni/net.d
Expand Down
21 changes: 18 additions & 3 deletions build/yamls/antrea.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -722,6 +722,21 @@ data:

# Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener.
#enablePrometheusMetrics: false

# Provide flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also enables
# the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto is given,
# we consider tcp as default.
#flowCollectorAddr: ""

# Provide flow poll interval as a duration string. This determines how often the flow exporter dumps connections from the conntrack module.
# Flow poll interval should be greater than or equal to 1s (one second).
# Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
#flowPollInterval: "5s"

# Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
# the flow collector.
# Flow export frequency should be greater than or equal to 1.
#flowExportFrequency: 12
antrea-cni.conflist: |
{
"cniVersion":"0.3.0",
Expand Down Expand Up @@ -770,7 +785,7 @@ metadata:
annotations: {}
labels:
app: antrea
name: antrea-config-c6467gmm8c
name: antrea-config-mk822kf995
namespace: kube-system
---
apiVersion: v1
Expand Down Expand Up @@ -876,7 +891,7 @@ spec:
key: node-role.kubernetes.io/master
volumes:
- configMap:
name: antrea-config-c6467gmm8c
name: antrea-config-mk822kf995
name: antrea-config
- name: antrea-controller-tls
secret:
Expand Down Expand Up @@ -1091,7 +1106,7 @@ spec:
operator: Exists
volumes:
- configMap:
name: antrea-config-c6467gmm8c
name: antrea-config-mk822kf995
name: antrea-config
- hostPath:
path: /etc/cni/net.d
Expand Down
15 changes: 15 additions & 0 deletions build/yamls/base/conf/antrea-agent.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,3 +60,18 @@ featureGates:

# Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener.
#enablePrometheusMetrics: false

# Provide flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also enables
# the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto is given,
# we consider tcp as default.
#flowCollectorAddr: ""

# Provide flow poll interval as a duration string. This determines how often the flow exporter dumps connections from the conntrack module.
# Flow poll interval should be greater than or equal to 1s (one second).
# Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".
#flowPollInterval: "5s"

# Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
# the flow collector.
# Flow export frequency should be greater than or equal to 1.
#flowExportFrequency: 12
20 changes: 16 additions & 4 deletions cmd/antrea-agent/agent.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import (
"net"
"time"

"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/client-go/informers"
"k8s.io/klog/v2"

Expand All @@ -31,6 +32,8 @@ import (
"github.com/vmware-tanzu/antrea/pkg/agent/controller/noderoute"
"github.com/vmware-tanzu/antrea/pkg/agent/controller/traceflow"
"github.com/vmware-tanzu/antrea/pkg/agent/flowexporter/connections"
"github.com/vmware-tanzu/antrea/pkg/agent/flowexporter/exporter"
"github.com/vmware-tanzu/antrea/pkg/agent/flowexporter/flowrecords"
"github.com/vmware-tanzu/antrea/pkg/agent/interfacestore"
"github.com/vmware-tanzu/antrea/pkg/agent/metrics"
"github.com/vmware-tanzu/antrea/pkg/agent/openflow"
Expand Down Expand Up @@ -234,11 +237,20 @@ func run(o *Options) error {
if features.DefaultFeatureGate.Enabled(features.Traceflow) {
go ofClient.StartPacketInHandler(stopCh)
}
// Create connection store that polls conntrack flows with a given polling interval.

// Initialize flow exporter to start go routines to poll conntrack flows and export IPFIX flow records
if features.DefaultFeatureGate.Enabled(features.FlowExporter) {
ctDumper := connections.NewConnTrackDumper(nodeConfig, serviceCIDRNet, connections.NewConnTrackInterfacer())
connStore := connections.NewConnectionStore(ctDumper, ifaceStore)
go connStore.Run(stopCh)
connStore := connections.NewConnectionStore(
connections.InitializeConnTrackDumper(nodeConfig, serviceCIDRNet, agentQuerier.GetOVSCtlClient(), o.config.OVSDatapathType),
ifaceStore,
o.pollInterval)
pollDone := make(chan struct{})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you used an unbuffered channel, yet the code in the exporter looks like this:

		case <-ticker.C:
			// Waiting for expected number of pollDone signals from go routine(ConnectionStore.Run) is necessary because
			// IPFIX collector computes throughput based on flow records received interval. Number of pollDone
			// signals should be equal to export frequency before starting the export cycle.
			for i := uint(0); i < exp.exportFrequency; i++ {
				<-pollDone
			}

I don't understand how that works. Won't the polling code block until the exporter ticker ticks?
As indicated below I don't think the ticker in the exporter is necessary, but the channel should probably be buffered to accommodate o.config.FlowExportFrequency signals, especially if you keep that ticker.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, ticker is not needed anymore. Thanks for pointing it out.
@unbuffered channel: You are right. The initial poll cycles are messed up and behavior is not expected.
I removed the ticker and we can have unbuffered channel as we will not be waiting on bunch of signals simultaneously. Fixed this. Please take a look.

go connStore.Run(stopCh, pollDone)

flowExporter := exporter.NewFlowExporter(
flowrecords.NewFlowRecords(connStore),
o.config.FlowExportFrequency)
go wait.Until(func() { flowExporter.Export(o.flowCollector, stopCh, pollDone) }, 0, stopCh)
}

<-stopCh
Expand Down
14 changes: 14 additions & 0 deletions cmd/antrea-agent/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,18 @@ type AgentConfig struct {
// Enable metrics exposure via Prometheus. Initializes Prometheus metrics listener
// Defaults to false.
EnablePrometheusMetrics bool `yaml:"enablePrometheusMetrics,omitempty"`
// Provide the flow collector address as string with format <IP>:<port>[:<proto>], where proto is tcp or udp. This also
// enables the flow exporter that sends IPFIX flow records of conntrack flows on OVS bridge. If no L4 transport proto
// is given, we consider tcp as default.
// Defaults to "".
FlowCollectorAddr string `yaml:"flowCollectorAddr,omitempty"`
// Provide flow poll interval in format "0s". This determines how often flow exporter dumps connections in conntrack module.
// Flow poll interval should be greater than or equal to 1s(one second).
// Defaults to "5s". Follow the time units of duration.
FlowPollInterval string `yaml:"flowPollInterval,omitempty"`
// Provide flow export frequency, which is the number of poll cycles elapsed before flow exporter exports flow records to
// the flow collector.
// Flow export frequency should be greater than or equal to 1.
// Defaults to "12".
FlowExportFrequency uint `yaml:"flowExportFrequency,omitempty"`
}
Loading