IPFix data only on IPv6 ? #105

vigodeltoro · 2022-07-26T07:09:09Z

Hi there,

I capture IPFiX data on Port 2055, template will be send every 5 minutes... if I start goflow2 with the parameters:

goflow2 -loglevel="debug" -transport.file="./goflow2.output2" -listen="sflow://[IPV4_IP]:6343,netflow:///[IPV4_IP]:4739"

I see only packets with EtypeName:"IPv6" like that:

{"Type":"IPFIX","TimeReceived":1658818721,"SequenceNum":1466255583,"SamplingRate":1,"SamplerAddress":"111.111.111.111,"TimeFlowStart":1658818606,"TimeFlowEnd":1658818606,"TimeFlowStartMs":1658818606182,"TimeFlowEndMs":1658818606182,"Bytes":143,"Packets":1,"**SrcAddr":"::","DstAddr":"::"**,"Etype":34525,"Proto":17,"SrcPort":58635,"DstPort":27016,"InIf":0,"OutIf":0,"SrcMac":"b0:a8:6e:7c:07:c2","DstMac":"30:b6:4f:e0:22:ce","SrcVlan":0,"DstVlan":0,"VlanId":0,"IngressVrfID":1187,"EgressVrfID":0,"IPTos":32,"ForwardingStatus":0,"IPTTL":0,"TCPFlags":0,"IcmpType":0,"IcmpCode":0,"IPv6FlowLabel":0,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAS":0,"DstAS":0,"NextHop":"::","NextHopAS":0,"SrcNet":0,"DstNet":0,"**EtypeName":"IPv6"**,"ProtoName":"UDP","IcmpName":""}

And I see no IP addresses

But I know that these packets are IPv4 ( crosschecked with pmacct ). I ran goflow2 20 minutes, a template have to be send. That was crosschecked with pmacct as well. I made a traffic dump to compare the tools output..

I can see that goflow2 captures not all flows. Goflow2 around 11500 - PMACCT 12886.

My suspect is that goflow2 thinks that is all IPv6 and can't read the IP addresses if a IPFIX flow is IPv4.

If a flow is IPv6 then it works..

Do I something wrong ?

Does anybody has an idea ?

Help would be very appreciated..

thanks and best regards

Christian

The text was updated successfully, but these errors were encountered:

lspgn · 2022-07-27T05:17:40Z

Hi Christian,
Would you be able to send me the capture?
Which version are you using? And what is the device that sends captures?

Also you mention IPFIX on port 2055
but the -listen argument for NetFlow specifies port 4739.
Not sure if it's a typo?

Regarding the missing IPv6 address, you may be right

My suspect is that goflow2 thinks that is all IPv6 and can't read the IP addresses if a IPFIX flow is IPv4.

But I would need to have a look at the template.

Are you using an encapsulation?

vigodeltoro · 2022-07-27T07:32:06Z

Hi Louis,

I'm using GoFlow2 v1.1.0 (2022-03-26T23:35:31+0000) and the Docker Compose files of the last v1.1.0 release.

Yes you are right, IPFIX port is 2055 and I used 2055 for productive traffic and 4739 for replaying the capture with tcpreplay.. both, direct productive traffic and the replay has that issue..
In production the devices are Nokia switches which send the captures. The tcpreplay capture is from that devices.

Concerning the capture I have to ask If I'm allowed to send it to you.. because of the IPs and data privacy.
Meanwhile, do you can tell me how to get the template out of the capture ? If I extract the template I can send it to you.. that's no problem..

Concerning the encapsulation.. uff.. I have to ask to be 100% sure but as a first answer I would say no..

thanks a lot for your help
Best regards

Christian

vigodeltoro · 2022-07-27T10:37:47Z

Hi Louis,
I extracted the template from the capture with Wireshark:

`(1374 bytes) Obs-Domain-ID=16777216 [Data-Template:1910] [Options-Template:50710] [Data:50710]
[Data:1910] 2055 ✓
Frame 8782: 1416 bytes on wire (11328 bits), 1416 bytes captured (11328 bits)
Ethernet II, Src: VMware_9e:10:cf (00:50:56:9e:10:cf), Dst: VMware_ba:8c:11 (00:50:56:ba:8c:11)
Internet Protocol Version 4, Src: 10.100.0.1, Dst: 192.168.63.32
User Datagram Protocol, Src Port: 50034, Dst Port: 2055
Cisco NetFlow/IPFIX
Version: 10
Length: 1374
Timestamp: Jul 12, 2022 14:32:05.000000000 CEST
FlowSequence: 2643998971 (expected 2643998959)
Observation Domain Id: 16777216
Set 1 [id=2] (Data Template): 1910
FlowSet Id: Data Template (V10 [IPFIX]) (2)
FlowSet Length: 112
Template (Id = 1910, Count = 26)
Template Id: 1910
Field Count: 26
Field : ICMP_TYPE
0... .... .... .... = Pen provided: No
.000 0000 0010 0000 = Type: ICMP_TYPE (32)
Length: 2
Set 2 [id=3] (Options Template): 50710
FlowSet Id: Options Template (V10 [IPFIX]) (3)
FlowSet Length: 32
Options Template (Id = 50710) (Scope Count = 2; Data Count = 3)
Template Id: 50710
Total Field Count: 5
Scope Field Count: 2

/Users/cpetrasch/Downloads/ipfix_25k.pcap 25000 total packets, 25000 shown
Field (1/2) [Scope]: observationDomainId
0... .... .... .... = Pen provided: No
.000 0000 1001 0101 = Type: observationDomainId (149)
Length: 4
Field (2/2) [Scope]: templateId
0... .... .... .... = Pen provided: No
.000 0000 1001 0001 = Type: templateId (145)
Length: 2
Field (1/3): selectorAlgorithm
0... .... .... .... = Pen provided: No
.000 0001 0011 0000 = Type: selectorAlgorithm (304)
Length: 2
Field (2/3): samplingPacketInterval
0... .... .... .... = Pen provided: No
.000 0001 0011 0001 = Type: samplingPacketInterval (305)
Length: 4
Field (3/3): samplingPacketSpace
0... .... .... .... = Pen provided: No
.000 0001 0011 0010 = Type: samplingPacketSpace (306)
Length: 4
Padding: 0000
Set 3 [id=50710] (1 flows)
FlowSet Id: (Data) (50710)
FlowSet Length: 20
[Template Frame: 8782]
Flow 1
Set 4 [id=1910] (10 flows)
FlowSet Id: (Data) (1910)
FlowSet Length: 1194
[Template Frame: 8782]
Flow 1
Flow 2
Flow 3
Flow 4
Flow 5
Flow 6
Flow 7
Flow 8
Flow 9
Flow 10
[Expected Sequence Number: 2643998959]
[Previous Frame in Sequence: 8780]`

Does that help in the meantime ?

lspgn · 2022-07-27T16:14:49Z

Oh ok that makes sense:

Field (10/26): IP_DST_ADDR
0... .... .... .... = Pen provided: No
.000 0000 0000 1100 = Type: IP_DST_ADDR (12)
Length: 4
Field (11/26): IPV6_SRC_ADDR
0... .... .... .... = Pen provided: No
.000 0000 0001 1011 = Type: IPV6_SRC_ADDR (27)
Length: 16
Field (12/26): IPV6_DST_ADDR
/Users/cpetrasch/Downloads/ipfix_25k.pcap 25000 total packets, 25000 shown
0... .... .... .... = Pen provided: No
.000 0000 0001 1100 = Type: IPV6_DST_ADDR (28)

From my previous experimentations, Juniper and Cisco were producing two templates, one for IPv4, one for IPv6.
This is the first time I see a single one. Should be an "easy" fix (adding a check for IP_PROTOCOL_VERSION and making sure the fields are empty).
If you could just confirm with a sample extract of field 10-12 (you can obfuscate the IP): I'm just interested in the value of IPv6 when it's an IPv4 packet and vice versa.

Will try to get something this weekend.

Thanks for reporting!

lspgn · 2022-07-28T04:18:27Z

Would you be able to check this branch #106

vigodeltoro · 2022-07-28T06:48:56Z

Hi Louis,

Thanks for reporting!
No problem.. thanks to you for taking care of it :) and so fast..

I crosschecked the new branch. I startet the branch with the docker-compose file in the kcg directory. In my understanding the docker container will be built every time from the code ? Am I right ?
Because unfortunately your fix in the new branch didn't fix the issue.. same behavior, it finds IPv6 only
I ran my tcpreplay dump and production data for 15 minutes..

Here are the informations you asked for:

IPv4:

Flow 1
Source Mac Address: Cisco_24:00:bd (11:bc:0c:22:33:bd)
Destination Mac Address: JuniperN_4c:00:c0 (44:f7:ad:55:66:c0)
Ingress Physical Interface: 1342177292
Egress Physical Interface: 4082
Dot1q Vlan Id: 501
Dot1q Customer Vlan Id: 0
Post Dot1q Vlan Id: 0
Post Dot1q Customer Vlan Id: 42
SrcAddr: 111.111.111.111
DstAddr: 222.222.222.222
SrcAddr: ::
DstAddr: ::
Packets: 1
Octets: 1362
[Duration: 0.000000000 seconds (milliseconds)]
StartTime: Jul 12, 2022 14:31:50.032000000 CEST
EndTime: Jul 12, 2022 14:31:50.032000000 CEST
SrcPort: 2154
DstPort: 44158
TCP Flags: 0x00
00.. .... = Reserved: 0x0
..0. .... = URG: Not used
...0 .... = ACK: Not used
.... 0... = PSH: Not used
.... .0.. = RST: Not used
.... ..0. = SYN: Not used
.... ...0 = FIN: Not used
Protocol: TCP (6)
IPv6 Extension Headers: 0x00000000
IPv6 Next Header: 0
ipv6FlowLabel: 0
IP ToS: 0x00
IPVersion: 4
ICMP Type: 0x0000

**IPv6:**

Flow 7
Source Mac Address: JuniperN_fb:3f:00 (11:bf:11:fb:00:66)
Destination Mac Address: ExtremeN_bc:da:00 (22:33:2f:44:da:0b)
Ingress Physical Interface: 1342177368
Egress Physical Interface: 61
Dot1q Vlan Id: 0
Dot1q Customer Vlan Id: 4095
Post Dot1q Vlan Id: 0
Post Dot1q Customer Vlan Id: 2
SrcAddr: 0.0.0.0
DstAddr: 0.0.0.0
SrcAddr: 1111:222:3333:4444::555
DstAddr: 1111:222:000:8::666
Packets: 1
Octets: 238
[Duration: 0.000000000 seconds (milliseconds)]
    StartTime: Jul 12, 2022 14:31:51.048000000 CEST
    EndTime: Jul 12, 2022 14:31:51.048000000 CEST
SrcPort: 33004
DstPort: 7080
TCP Flags: 0x0000
    0000 .... .... .... = Zero (Header Length): 0x0
    .... 000. .... .... = Reserved: 0x0
    .... ...0 .... .... = ECN Nonce Sum: Not used
    .... .... 0... .... = CWR: Not used
    .... .... .0.. .... = ECN Echo: Not used
    .... .... ..0. .... = URG: Not used
    .... .... ...0 .... = ACK: Not used
    .... .... .... 0... = PSH: Not used
    .... .... .... .0.. = RST: Not used
    .... .... .... ..0. = SYN: Not used
    .... .... .... ...0 = FIN: Not used
Protocol: UDP (17)
IPv6 Extension Headers: 0x00000000
IPv6 Next Header: 17
ipv6FlowLabel: 0
IP ToS: 0x00
IPVersion: 6
ICMP Type: 0x0000
Ingress VRFID: 2
BGPNextHop: 0.0.0.0
BGPNextHop: ::

Does that help ?

Thanks a lot again for your help :)

best, Christian

vigodeltoro · 2022-07-28T06:59:14Z

A colleague of mine send me the documentation of the template, maybe that could help:

lspgn · 2022-07-28T15:18:58Z

Just in case:
You need to run

docker-compose build

Before starting the compose.

Without running go build or docker-compose the other solution would be to get the artifacts built here.

vigodeltoro · 2022-07-29T05:51:45Z

Ah.. okay.. it was my doubt that I'm doing sth. wrong..

Okay.. we are one step closer now.. cool :).. many thanks..

IPv4:

goflow_1 | {"Type":"IPFIX","ObservationPointID":0,"ObservationDomainID":33554432,"TimeReceived":1659072751,"SequenceNum":661958810,"SamplingRate":1,"SamplerAddress":"111.111.111.111","TimeFlowStart":1657629111,"TimeFlowEnd":1657629111,"TimeFlowStartMs":1657629111770,"TimeFlowEndMs":1657629111800,"Bytes":2940,"Packets":2,"SrcAddr":"::","DstAddr":"::","Etype":2048,"Proto":6,"SrcPort":443,"DstPort":16969,"InIf":0,"OutIf":0,"SrcMac":"c8:e7XX:XX:92:e6","DstMac":"68:ab:XX:XX:dd:5a","SrcVlan":0,"DstVlan":0,"VlanId":0,"IngressVrfID":32,"EgressVrfID":0,"IPTos":0,"ForwardingStatus":0,"IPTTL":0,"TCPFlags":16,"IcmpType":0,"IcmpCode":0,"IPv6FlowLabel":0,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAS":0,"DstAS":0,"NextHop":"","NextHopAS":0,"SrcNet":0,"DstNet":0,"EtypeName":"IPv4","ProtoName":"TCP","IcmpName":""}

IPv6:
{"Type":"IPFIX","ObservationPointID":0,"ObservationDomainID":33554432,"TimeReceived":1659072873,"SequenceNum":661955961,"SamplingRate":1,"SamplerAddress":"111.111.111.111","TimeFlowStart":1657629110,"TimeFlowEnd":1657629110,"TimeFlowStartMs":1657629110526,"TimeFlowEndMs":1657629110526,"Bytes":78,"Packets":1,"SrcAddr":"1111:1111:2222:3333","DstAddr":"1111:1111:2222:3333","Etype":34525,"Proto":6,"SrcPort":59092,"DstPort":995,"InIf":0,"OutIf":0,"SrcMac":"54:1e:56:a5:38:22","DstMac":"7c:25:86:40:2c:04","SrcVlan":0,"DstVlan":0,"VlanId":0,"IngressVrfID":32,"EgressVrfID":0,"IPTos":0,"ForwardingStatus":0,"IPTTL":0,"TCPFlags":0,"IcmpType":0,"IcmpCode":0,"IPv6FlowLabel":213364,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAS":0,"DstAS":0,"NextHop":"","NextHopAS":0,"SrcNet":0,"DstNet":0,"EtypeName":"IPv6","ProtoName":"TCP","IcmpName":""}

It seems that with that new build the flow Etypename will be recognized as IPv4 and IPv6 correctly but the IP addresses are still not read..

BTW: Sending you the capture is for data privacy reasons not possible.. But I can do the tests for you like we are doing at the moment...

Thanks a lot for your help.. really appreciated :)

lspgn · 2022-07-29T06:09:02Z

I think i know what happened. The fix checks for length whereas it should be testing for "zeroes". Will get a fix.

vigodeltoro · 2022-07-29T11:51:28Z

Hi Louis.. ah that could be 👍 .. thanks and have a nice weekend

lspgn · 2022-07-30T03:35:14Z

Could you try the artifacts from this build?

Made a few changes

vigodeltoro · 2022-08-01T06:21:05Z

Hi Louis..

it's working.. great work :)... thanks a lot:

{"Type":"IPFIX","ObservationPointID":0,"ObservationDomainID":33554432,"TimeReceived":1659333755,"SequenceNum":661958900,"SamplingRate":1,"SamplerAddress":"111.111.111.111","TimeFlowStart":1657629111,"TimeFlowEnd":1657629111,"TimeFlowStartMs":1657629111802,"TimeFlowEndMs":1657629111802,"Bytes":70,"Packets":1,"SrcAddr":"222.222.222.222","DstAddr":"333.333.333.333","Etype":2048,"Proto":6,"SrcPort":443,"DstPort":59960,"InIf":0,"OutIf":0,"SrcMac":"44:f4:XX:XX:10:a5","DstMac":"3c:08:XX:XX:2f:04","SrcVlan":0,"DstVlan":0,"VlanId":0,"IngressVrfID":32,"EgressVrfID":0,"IPTos":0,"ForwardingStatus":0,"IPTTL":0,"TCPFlags":0,"IcmpType":0,"IcmpCode":0,"IPv6FlowLabel":0,"FragmentId":0,"FragmentOffset":0,"BiFlowDirection":0,"SrcAS":0,"DstAS":0,"NextHop":"","NextHopAS":0,"SrcNet":0,"DstNet":0,"EtypeName":"IPv4","ProtoName":"TCP","IcmpName":""}

{"event_type": "purge", "mac_src": "44:f4:XX:XX:10:a5", "mac_dst": "3c:08:XX:XX:2f:04", "peer_ip_src": "111.111.111.111", "ip_src": "222.222.222.222", "ip_dst": "333.333.333.333", "port_src": 443, "port_dst": 59960, "tcp_flags": "0", "ip_proto": "6", "timestamp_start": "2022-07-12 14:31:51.802000", "timestamp_end": "2022-07-12 14:31:51.802000", "ip_prot_ver": "4", "in_phys_int": "1342177332", "dot1q_cvlan": "0", "dot1q_vlan": "0", "out_phys_int": "1342177282", "post_dot1q_cvlan": "0", "post_dot1q_vlan": "2", "in_vrfid": "32", "packets": 1, "bytes": 70}

IPv6 is working like a charm as well :)..

Even I had to check how to get "InIf" and "OutIf". Because this values are zero and pmacct reads that. But that are custom primitives in pmacct as well..

name=in_phys_int field_type=252 len=4 semantics=u_int
name=out_phys_int field_type=253 len=4 semantics=u_int

So I have to have a look how to deal with that in goflow2. If I do that, maybe I get a value for this fields as well.
If not, I would be happy to be allowed to contact you again ..

Thanks a lot for all that help.. very appreciated :) 👍
best,
Christian

lspgn · 2022-08-01T06:25:00Z

Cool!

For interface, I could likely implement mapping of:
ingressPhysicalInterface

vigodeltoro · 2022-08-01T07:45:17Z

Great.. that would be very helpful, if its not to much work and you could implement ingressPhysicalInterface/egressPhysicalInterface :+1 thanks..

May I ask you another question ? I don't want to steal your time.. It's no problem if it's not possible.. I was great that you helped us that bugfix :)..

During my replays/tests with my 25k UDP Stream test capture I figured out that pmacct finds 12885 flows in that capture after receiving the template ( fresh start of pmacct ) and goflow2 ( also after a fresh restart ) finds only 10787.

I investigated that behavior with one sample src address and figured out that pmacct sees 20 flows more for that sample src ip. Looking deeper it 17 of that 20 flows came from the same MAC address and the remaining 3 were a related looking Mac address.

I attached the Wireshark capture of one one that flow.. maybe you can have a look and see something suspicious why goflow2 didn't see that flows ? In my eyes it didn't look strange..

Flow 2
            Source Mac Address: Cisco_6b:e9:9a (a8:0c:xx:xx:e9:9a)
            Destination Mac Address: AristaNe_35:b5:7d (c0:d6:xx:xx:b5:7d)
            Ingress Physical Interface: 1342177336
            Egress Physical Interface: 21
Protocol Length Info
     CFLOW    1502   IPFIX flow
    Dot1q Vlan Id: 200
    Dot1q Customer Vlan Id: 0
    Post Dot1q Vlan Id: 0
    Post Dot1q Customer Vlan Id: 2
    SrcAddr: 111.111.111.111
    DstAddr: 222.222.222.222
    SrcAddr: ::
    DstAddr: ::
    Packets: 1
    Octets: 74
    [Duration: 0.000000000 seconds (milliseconds)]
        StartTime: Jul 12, 2022 14:31:50.518000000 CEST
        EndTime: Jul 12, 2022 14:31:50.518000000 CEST
    SrcPort: 51785
    DstPort: 8290
    TCP Flags: 0x0000
        0000 .... .... .... = Zero (Header Length): 0x0
        .... 000. .... .... = Reserved: 0x0
        .... ...0 .... .... = ECN Nonce Sum: Not used
        .... .... 0... .... = CWR: Not used
        .... .... .0.. .... = ECN Echo: Not used
        .... .... ..0. .... = URG: Not used
        .... .... ...0 .... = ACK: Not used
        .... .... .... 0... = PSH: Not used
        .... .... .... .0.. = RST: Not used
        .... .... .... ..0. = SYN: Not used
        .... .... .... ...0 = FIN: Not used
    Protocol: TCP (6)
    IPv6 Extension Headers: 0x00000000
    IPv6 Next Header: 0
    ipv6FlowLabel: 0
    IP ToS: 0x00
    IPVersion: 4
    ICMP Type: 0x0000
    Ingress VRFID: 2
    BGPNextHop: 0.0.0.0
    BGPNextHop: ::
    
    
    Thanks a lot for everything.. and best regards 
    Christian

lspgn · 2022-08-02T05:02:55Z

Regarding the following, I am missing important details to troubleshoot.

During my replays/tests with my 25k UDP Stream test capture I figured out that pmacct finds 12885 flows in that capture after receiving the template ( fresh start of pmacct ) and goflow2 ( also after a fresh restart ) finds only 10787.

I investigated that behavior with one sample src address and figured out that pmacct sees 20 flows more for that sample src ip. Looking deeper it 17 of that 20 flows came from the same MAC address and the remaining 3 were a related looking Mac address.

Could you describe how you count?
The total difference is 2098 at every run? What's the difference after a subsequent run (4196 2098?) without restarting the program?
How many UDP packets after the template are there? Is it possible that pmacct buffers the packets before the template?
Are they counted properly in the /metrics Prometheus endpoint?
For the 20 flows from one sampling IP: does the port change and where are they in the pcap? Is the sequence indicated, if so, are they decoded by GoFlow2 but not counted?
Try extracting them and replay only those 20 packets and the template.

lspgn · 2022-08-02T05:06:45Z

Regarding the interface index.
I'm still confused by why your device does not use:

ingressInterface (10)
egressInterface (14)

I have a solution that does not involve changing the code.
Could you create a mapping.yaml file containing:

ipfix:
  mapping:
    - field: 252
      destination: InIf
    - field: 253
      destination: OutIf

And run goflow -mapping mapping.yaml.

lspgn · 2022-08-02T05:15:17Z

One more question:

In production the devices are Nokia switches which send the captures

What is the model/series and possibly the version of the OS? I am trying to investigate the template.
I was able to track the doc to 7705 SAR Router Configuration Guide R9.0.R7.

vigodeltoro · 2022-08-02T07:20:57Z

Hi Louis..

the mapping solution works :) :+1 .. great.. thanks a lot..

Concerning the other issue..
I collected a 25k packet UDP tcpdump of our IPFix stream to disk and replay it to goflow2/pmacct. My expectings are that the count of IPFIX flows should be nearly the same when I send that replay to both tools.

I count by writing the output of goflow2 and pmacct to disk and do a grep for "IPFIX" | wc -l in goflow2 and a grep for another unique Key ( event_type ) in the pmacct file. If I don't restart the tools I got for the first run of goflow2 11430 flows ( pmacct 12886 ) , second run 27270 ( pmacct 38526 ). That means that pmacct finds 12712 more flows for that 2 runs.

The total difference is 2098 at every run? What's the difference after a subsequent run (4196 2098?) without restarting the >program?

No, it's swaying.. f. e. for a second test a restarted goflow2 and got 11246 flows in the first run and 27170 in second run, if I do a 3rd run with out restart, I got 28022 flows.. everything with the same pcap which is replayed.

How many UDP packets after the template are there?
The packet number with the template is 1980/25000, so I guess 23020 packets. I attached the file with the template with anonymized IPs + MAC

Is it possible that pmacct buffers the packets before the template?
IMHO not, because if I don't restart pmacct I finds much more flows than 12886 ( see above )

What is the model/series and possibly the version of the OS? I am trying to investigate the template.
anonymized.template.udp.txt

The model is Nokia 7750 and OS is 21.10 ..
Here is a link to the docs.. https://documentation.nokia.com/cgi-bin/dbaccessfilename.cgi/3HE17161AAADTQZZA01_V1_7450%20ESS%207750%20SR%207950%20XRS%20and%20VSR%20Router%20Configuration%20Guide%2021.10.R1.pdf

You can find the template on page 285

Concerning "Are they counted properly in the /metrics Prometheus endpoint?" and the extraction of the 20 flows I have to do my homework and come back to you..

Thanks lot for your help again :)

best regards
anonymized.template.udp.txt

lspgn · 2022-08-02T08:15:54Z

This seems to be a performance issue.
I would suggest replaying more slowly the pcap and increasing the amount of listeners on the same port -listen netflow://:2055,netflow://:2055 (until I get a better URI format).

The Prometheus page should give better insights on the decode metrics.

Thank you for the details on the device.

vigodeltoro · 2022-08-02T08:47:55Z

Ah.. that could be.. I will retry..

... I have retried with sending 10Mbit/s instead of 100 Mbit/s. all flows found.. ( 12886 ) great :D 👍 .. thanks..

Next step, I will try to increase performance by increasing the number of listeners

vigodeltoro · 2022-08-02T09:23:22Z

okay, I increased the listeners.. wow.. for 100Mbit/s of traffic I need 30 parallel listeners.. to write that 12886 flows in json format to file fast enough.

The server has no real load, I think maybe it's the IOPs.. anyway.. for production I'll switch the format to Protobuf and send to Kafka anyway. So I'm looking forward to this performance. If you are interested I can send you the results.

Thank you for all that help Louis.. It was really appreciated. Hope to meet you personally on a RIPE meeting to say a hello and invite you to a beer or sth. else :)

best regards
Christian

vigodeltoro · 2022-08-02T10:01:26Z

FYI:

JSON - 25k udp packets - 100 Mbit/s replay speed - 30 listener - 12886 - everything is there
At minimum 30 listeners for JSON needed

Protobuf - 25k udp packets - 100 Mbit/s replay speed - 1 listener - 12886 - everything is there

:) :) :) -- very happy

lspgn · 2022-08-03T01:22:27Z

Thank you :) .
Feel free to close unless you have more questions.
I'll test a bit more the PR and then release it.

lspgn · 2022-08-03T02:33:27Z

JFYI I'll be improving the socket repeat in #107 (using a count argument),

vigodeltoro · 2022-08-03T05:46:20Z

Solved.. thank you very much :)

lspgn added bug Something isn't working question Further information is requested labels Jul 27, 2022

lspgn mentioned this issue Jul 27, 2022

Get values of IPFIX IP addresses in human readable format.. #104

Closed

lspgn mentioned this issue Jul 28, 2022

fix single ip template case #106

Merged

vigodeltoro closed this as completed Aug 3, 2022

vigodeltoro mentioned this issue Mar 31, 2023

UDP packet drops and strange core usage behavior #152

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IPFix data only on IPv6 ? #105

IPFix data only on IPv6 ? #105

vigodeltoro commented Jul 26, 2022

lspgn commented Jul 27, 2022 •

edited

Loading

vigodeltoro commented Jul 27, 2022

vigodeltoro commented Jul 27, 2022

lspgn commented Jul 27, 2022 •

edited

Loading

lspgn commented Jul 28, 2022

vigodeltoro commented Jul 28, 2022

vigodeltoro commented Jul 28, 2022

lspgn commented Jul 28, 2022

vigodeltoro commented Jul 29, 2022 •

edited

Loading

lspgn commented Jul 29, 2022

vigodeltoro commented Jul 29, 2022

lspgn commented Jul 30, 2022

vigodeltoro commented Aug 1, 2022

lspgn commented Aug 1, 2022

vigodeltoro commented Aug 1, 2022 •

edited

Loading

lspgn commented Aug 2, 2022 •

edited

Loading

lspgn commented Aug 2, 2022 •

edited

Loading

lspgn commented Aug 2, 2022 •

edited

Loading

vigodeltoro commented Aug 2, 2022 •

edited

Loading

lspgn commented Aug 2, 2022

vigodeltoro commented Aug 2, 2022

vigodeltoro commented Aug 2, 2022

vigodeltoro commented Aug 2, 2022

lspgn commented Aug 3, 2022

lspgn commented Aug 3, 2022

vigodeltoro commented Aug 3, 2022

IPFix data only on IPv6 ? #105

IPFix data only on IPv6 ? #105

Comments

vigodeltoro commented Jul 26, 2022

lspgn commented Jul 27, 2022 • edited Loading

vigodeltoro commented Jul 27, 2022

vigodeltoro commented Jul 27, 2022

lspgn commented Jul 27, 2022 • edited Loading

lspgn commented Jul 28, 2022

vigodeltoro commented Jul 28, 2022

vigodeltoro commented Jul 28, 2022

lspgn commented Jul 28, 2022

vigodeltoro commented Jul 29, 2022 • edited Loading

lspgn commented Jul 29, 2022

vigodeltoro commented Jul 29, 2022

lspgn commented Jul 30, 2022

vigodeltoro commented Aug 1, 2022

lspgn commented Aug 1, 2022

vigodeltoro commented Aug 1, 2022 • edited Loading

lspgn commented Aug 2, 2022 • edited Loading

lspgn commented Aug 2, 2022 • edited Loading

lspgn commented Aug 2, 2022 • edited Loading

vigodeltoro commented Aug 2, 2022 • edited Loading

lspgn commented Aug 2, 2022

vigodeltoro commented Aug 2, 2022

vigodeltoro commented Aug 2, 2022

vigodeltoro commented Aug 2, 2022

lspgn commented Aug 3, 2022

lspgn commented Aug 3, 2022

vigodeltoro commented Aug 3, 2022

lspgn commented Jul 27, 2022 •

edited

Loading

lspgn commented Jul 27, 2022 •

edited

Loading

vigodeltoro commented Jul 29, 2022 •

edited

Loading

vigodeltoro commented Aug 1, 2022 •

edited

Loading

lspgn commented Aug 2, 2022 •

edited

Loading

lspgn commented Aug 2, 2022 •

edited

Loading

lspgn commented Aug 2, 2022 •

edited

Loading

vigodeltoro commented Aug 2, 2022 •

edited

Loading