No traffic shown on some macs #51

diversys · 2019-12-31T10:08:41Z

This is 9d93d34 on macOS 10.14.6
bandwhich starts but shows no traffic.
sudo /Users/diver/.cargo/bin/bandwhich

sudo /Users/diver/.cargo/bin/bandwhich -i en0 shows the same.

List of interfaces:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
EHC26: flags=0<> mtu 0
XHC20: flags=0<> mtu 0
EHC29: flags=0<> mtu 0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304
awdl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1484
en1: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
en2: flags=8963<UP,BROADCAST,SMART,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
vmnet1: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
vmnet8: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500

The text was updated successfully, but these errors were encountered:

mpereira · 2019-12-31T10:32:23Z

I get the same.

macOS: 10.14.5
rustc: 1.40.0 (73528e339 2019-12-16)
bandwhich: 0.6.0

berry · 2019-12-31T13:26:09Z

Also get the same.

MacOS: 10.14.6
Rustc: rustc 1.40.0 (73528e339 2019-12-16)
bandwhich: 0.6.0

Additional info. Using Macbook with wifi. Sudo does not help. Tried en0 and awdl0 and both did not work.

pdbradley · 2019-12-31T13:38:56Z

Same here. great tool and eager to use it. thanks.
Macbook Pro 10.15.2
bandwhich 0.6.0

munntjlx · 2019-12-31T16:03:54Z

Same here. Happy New Years??🙀

Dapacruz · 2019-12-31T16:20:30Z

I am having the same issue.

adraffy · 2019-12-31T16:33:24Z

Same, 10.15.2

capitalist · 2019-12-31T18:01:04Z

Same, 10.13.6
rustc 1.40.0
bandwhich 0.6.0

imsnif · 2019-12-31T18:05:48Z

The way I see it, this issue is likely caused by either:

Not getting any traffic on the network interface
Having an issue getting the output of lsof which is what we use on Mac to get process information

I somehow suspect it's the second... but I'm not certain.

For those experiencing this issue, while the tool is running, if in another terminal you run lsof -n -P -i4, do you get output? Would you be willing to paste it here if it doesn't contain anything private? Does it appear on the screen quickly, or is it a little sluggish?

I might ask for some more help debugging afterwards so we can get to the bottom of this issue :) Thanks!

diversys · 2019-12-31T18:12:17Z

lsof -n -P -i4 is instant, however lsof -n takes several seconds to appear.

COMMAND     PID  USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
loginwind   103 diver    7u  IPv4 0x1b167c2247b9834f      0t0  UDP *:*
UserEvent   377 diver    3u  IPv4 0x1b167c224a5e2f8f      0t0  UDP *:*
identitys   393 diver   19u  IPv4 0x1b167c224a78208f      0t0  UDP *:*
rapportd    403 diver    5u  IPv4 0x1b167c2247b9548f      0t0  UDP *:*
rapportd    403 diver    6u  IPv4 0x1b167c224a5e534f      0t0  UDP *:*
rapportd    403 diver    7u  IPv4 0x1b167c2249b3de4f      0t0  UDP *:*
sharingd    425 diver    4u  IPv4 0x1b167c224d93218f      0t0  UDP *:*
sharingd    425 diver    8u  IPv4 0x1b167c2249d0e84f      0t0  UDP *:*
sharingd    425 diver    9u  IPv4 0x1b167c2249d0e00f      0t0  UDP *:*
sharingd    425 diver   10u  IPv4 0x1b167c2249d1010f      0t0  UDP *:*
sharingd    425 diver   11u  IPv4 0x1b167c2249b3accf      0t0  UDP *:*
sharingd    425 diver   20u  IPv4 0x1b167c2247b9910f      0t0  UDP *:*
sharingd    425 diver   21u  IPv4 0x1b167c2247b993cf      0t0  UDP *:*
WiFiAgent   430 diver    6u  IPv4 0x1b167c224a5e42cf      0t0  UDP *:*
SystemUIS   458 diver   11u  IPv4 0x1b167c224d6e548f      0t0  UDP *:*
SystemUIS   458 diver   12u  IPv4 0x1b167c2250c6108f      0t0  UDP *:*
SystemUIS   458 diver   13u  IPv4 0x1b167c2250c6134f      0t0  UDP *:56330
SystemUIS   458 diver   14u  IPv4 0x1b167c2250c60dcf      0t0  UDP *:*
SystemUIS   458 diver   18u  IPv4 0x1b167c224a77f48f      0t0  UDP *:*
WiFiProxy   511 diver    5u  IPv4 0x1b167c2249b3c58f      0t0  UDP *:*
Google      536 diver   21u  IPv4 0x1b167c226cf39b17      0t0  TCP 192.168.1.2:49812->140.82.113.26:443 (ESTABLISHED)
Google      536 diver   22u  IPv4 0x1b167c225ab96e27      0t0  TCP 192.168.1.2:49951->140.82.114.25:443 (ESTABLISHED)
Google      536 diver   23u  IPv4 0x1b167c226984f49f      0t0  TCP 192.168.1.2:49933->140.82.114.25:443 (ESTABLISHED)
Google      536 diver   24u  IPv4 0x1b167c22487a849f      0t0  TCP 192.168.1.2:52501->173.194.76.188:5228 (ESTABLISHED)
Google      536 diver   25u  IPv4 0x1b167c225ab977af      0t0  TCP 192.168.1.2:49954->140.82.113.25:443 (ESTABLISHED)
Google      536 diver   27u  IPv4 0x1b167c2250c61e4f      0t0  UDP *:5353
Google      536 diver   30u  IPv4 0x1b167c22723abe27      0t0  TCP 192.168.1.2:61806->104.244.42.129:443 (ESTABLISHED)
Google      536 diver   37u  IPv4 0x1b167c2254c2949f      0t0  TCP 192.168.1.2:50166->192.30.253.124:443 (ESTABLISHED)
Google      536 diver   40u  IPv4 0x1b167c224d92df8f      0t0  UDP *:5353
Google      536 diver   60u  IPv4 0x1b167c224d92da0f      0t0  UDP *:5353
Google      536 diver   64u  IPv4 0x1b167c224d93168f      0t0  UDP *:5353
Adium     76849 diver   19u  IPv4 0x1b167c2250c5fd4f      0t0  UDP *:57684
Telegram  76914 diver   32u  IPv4 0x1b167c225081e137      0t0  TCP 192.168.1.2:49253->45.94.176.100:2286 (ESTABLISHED)

briantully · 2019-12-31T18:49:39Z

Same issue here. Without sudo prefix I get the following error:

bandwhich -i en0

Error: Failed to listen on network interface: No such file or directory (os error 2)

With sudo prefix sudo bandwhich -i en0 I get the bandwhich interface, but no traffic is captured/displayed.

While bandwhich is running, in another terminal tab I get many results when running lsof -n -P -i4, so it doesn't appear to be lsof that is the issue.

Mac 10.14.6
bandwhich 0.6.0
rustc 1.40.0

briantully · 2019-12-31T18:54:58Z

Output of lsof -n -P -i4 is instantaneous. However I noticed in the output the first line is a WARNING before it presents the results in a table.

Is it possible that bandwhich is parsing the output of lsof and expecting certain headers (e.g. COMMAND) and quietly failing when it sees the following warning as the first line in the lsof output?

lsof -n -P -i4

lsof: WARNING: can't stat() apfs file system /Volumes/com.apple.TimeMachine.localsnapshots/Backups.backupdb/MacBookPro-BrianTully (2)/2019-12-17-231200/Macintosh HD
      Output information may be incomplete.
      assuming "dev=300002d9" from mount table
COMMAND     PID        USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
loginwind   117 brian.tully    7u  IPv4 0x9f5f640908a36c2b      0t0  UDP *:*
...

Which seems to be related to a Time Machine Backup. Are others seeing the same warning as the first line in the lsof output?

grishy · 2019-12-31T19:07:19Z

Which seems to be related to a Time Machine Backup. Are others seeing the same warning as the first line in the lsof output?

No.

grishy · 2019-12-31T19:29:54Z

@imsnif Why not use syscall?
I had to develop a program that worked with the system (plugin for storage system).
As a result, I had to abandon the call of programs and do more than half by syscall ...

grishy · 2019-12-31T19:38:10Z

@briantully you can add -w to lsof for disable warnings
We can also add -F to simply parse just the values you need.

munntjlx · 2019-12-31T19:41:07Z

I don’t see any warnings on my side. Using Mojave 10.14.6 with currentest bandwhich. Sincerely, Thomas J. Munn Consulting Security Engineer LexisNexis On 12/31/19, 14:38, "Sergey" <notifications@github.com<mailto:notifications@github.com>> wrote: *** External email: use caution *** @briantully<https://github.com/briantully> you can add -w to lsof for disable warnings — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#51?email_source=notifications&email_token=ALVIUB4EGBKACMEJEBYAXUTQ3ONSHA5CNFSM4KBUKC72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEH4TKKI#issuecomment-569980201>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ALVIUB5HL7VR2M7RG2TTHZLQ3ONSHANCNFSM4KBUKC7Q>.

grishy · 2019-12-31T19:55:21Z

had to develop a program that worked with the system (plugin for storage system).
As a result, I had to abandon the call of programs and do more than half by syscall ...

I looked at how it looks on syscalls. Not very nice and maintained...

grishy · 2019-12-31T20:06:48Z

2. Having an issue getting the output of lsof which is what we use on Mac to get process information

Work fine.

grishy · 2019-12-31T20:25:27Z

Before update state we have empty utilization

grishy · 2019-12-31T20:35:48Z

I managed to show at least something

grishy · 2019-12-31T20:42:55Z

Could you explain the logic?

bandwhich/src/main.rs

Lines 197 to 226 in 9d93d34

    
           active_threads.push(display_handler); 
        
           let sniffer_threads = os_input 
        
               .network_interfaces 
        
               .into_iter() 
        
               .zip(os_input.network_frames.into_iter()) 
        
               .map(|(iface, frames)| { 
        
                   let name = format!("sniffing_handler_{}", iface.name); 
        
                   let running = running.clone(); 
        
                   let network_utilization = network_utilization.clone(); 
        
                   thread::Builder::new() 
        
                       .name(name) 
        
                       .spawn(move || { 
        
                           let mut sniffer = Sniffer::new(iface, frames); 
        
                           while running.load(Ordering::Acquire) { 
        
                               if let Some(segment) = sniffer.next() { 
        
                                   network_utilization.lock().unwrap().update(segment); 
        
                               } 
        
                           } 
        
                       }) 
        
                       .unwrap() 
        
               }) 
        
               .collect::<Vec<_>>(); 
        
           active_threads.extend(sniffer_threads); 
        
           for thread_handler in active_threads { 
        
               thread_handler.join().unwrap() 
        
           }

It feels like this code stops working after the first ui update. In fact, for an instant, something is drawn and immediately erased.

imsnif · 2020-01-01T08:53:18Z

Hey, @grishy - great work! Looks like you're on to something.

The utilization struct is where we record the network traffic information for our connections after sniffing the packets from the network card. If it's empty, it likely means we have a problem either sniffing them or recording them in this case.

The above code is where we iterate over segment size information from the network card and record it in the utilization struct.

If you're up to continue debugging this, I'd be very interested in finding out what happens here: https://github.com/imsnif/bandwhich/blob/master/src/network/sniffer.rs#L57
This method is supposed to be invoked after we process a segment. Is it invoked? Just once? Do you think you can tell where (if at all?) it gets stuck?

zhangxp1998 · 2020-01-01T16:44:26Z

Hey, @grishy - great work! Looks like you're on to something.

The utilization struct is where we record the network traffic information for our connections after sniffing the packets from the network card. If it's empty, it likely means we have a problem either sniffing them or recording them in this case.

The above code is where we iterate over segment size information from the network card and record it in the utilization struct.

If you're up to continue debugging this, I'd be very interested in finding out what happens here: https://github.com/imsnif/bandwhich/blob/master/src/network/sniffer.rs#L57
This method is supposed to be invoked after we process a segment. Is it invoked? Just once? Do you think you can tell where (if at all?) it gets stuck?

I added some print statements like this:

        println!("Polling...");
        let bytes = self.network_frames.next().ok()?;
        let packet = EthernetPacket::new(bytes)?;
        println!("Got packet!");

After running the program, "Got packet!" was never printed. Which means the pnet datalink receiver did not receive any packets.

imsnif · 2020-01-01T16:51:28Z

@zhangxp1998 - do you think "Got packet!" will also not be printed if you put it directly after bytes?

zhangxp1998 · 2020-01-01T16:52:48Z

@zhangxp1998 - do you think "Got packet!" will also not be printed if you put it directly after bytes?

Tried, wasn't printed.

zhangxp1998 · 2020-01-01T16:54:29Z

@zhangxp1998 - do you think "Got packet!" will also not be printed if you put it directly after bytes?

Maybe it has something to do with Mac's SIP ? I cannot restart my Mac at this moment, but if someone can run the bandwich program with SIP disabled that would be great.

imsnif · 2020-01-01T17:03:24Z

@zhangxp1998 - I more than suspect SIP is at least partly the issue here and would love it if someone could try.

Otherwise, this is just a guess: can you try commenting out this line and seeing if it helps? https://github.com/imsnif/bandwhich/blob/master/src/os/shared.rs#L37 (fair warning, you might have to kill -9 the process afterwards).

If it doesn't (and disabling SIP doesn't solve this), I'll check back with other options a little later. Thanks for the help!

zhangxp1998 · 2020-01-01T17:39:33Z

@zhangxp1998 - I more than suspect SIP is at least partly the issue here and would love it if someone could try.

Otherwise, this is just a guess: can you try commenting out this line and seeing if it helps? https://github.com/imsnif/bandwhich/blob/master/src/os/shared.rs#L37 (fair warning, you might have to kill -9 the process afterwards).

If it doesn't (and disabling SIP doesn't solve this), I'll check back with other options a little later. Thanks for the help!

Magically, removing the read timeout in shared.rs solved this issue. I suspect this is PNET's implementation quirk?

nimbius · 2020-01-03T15:47:03Z

confirmed seeing this issue on Mojave 10.14.6

imsnif · 2020-01-03T16:54:09Z

There's a fix for this issue, as well as the "getting an error without -i" issue in master right now.
I would prefer to test this before I release a new version, so @zhangxp1998, @grishy or anyone else who experienced this issue and is comfortable to check this.

Could you please:

Pull master of this repository
cargo build
sudo ./target/debug/bandwhich
make sure everything works and you see traffic
make sure you can easily quit the app with q or ctrl-c
sudo ./target/debug/bandwhich -i en0 (or your main network card)
make sure everything works and you see traffic
make sure you can easily quit the app with q or ctrl-c

Thanks very much! I'll release a version when I can confirm this works.

EDIT: the testing is mostly because I want to make sure the two fixes don't interfere with each other :)

munntjlx · 2020-01-03T18:54:18Z

It seems to work. The display changes quite quickly though so its kind of hard to see whats going on. But I definitely see stuff now without specifiying the interface. It doesn’t want to quit though, had to kill -9 the process as root.

munntjlx · 2020-01-03T18:55:24Z

It seems to work. The display changes quite quickly though so its kind of hard to see whats going on. But I definitely see stuff now without specifiying the interface. It doesn’t want to quit though, had to kill -9 the process as root. ctrl c didn't stop it.

imsnif · 2020-01-03T19:10:04Z

Thanks @munntjlx - I'm trying another option, just pushed to master. How about now?

munntjlx · 2020-01-03T19:16:31Z

It exits now. Any way I can update the polling cycle? It erases every second?

munntjlx · 2020-01-03T19:17:34Z

or perhaps an 'average' of the last 60 seconds, then print for a minute?

munntjlx · 2020-01-03T19:17:51Z

Works though! YeA!

imsnif · 2020-01-03T19:21:20Z

Thanks @munntjlx !

You have a very good point - right now it's mostly useful for prolonged traffic, otherwise it disappears too fast. This is actually the only issue I want to solve before a 1.0.0 release: #14

floriancargoet · 2020-01-03T19:39:19Z

I also confirm the issue is gone on master.

zhangxp1998 · 2020-01-03T19:49:51Z

Master no longer works for me, shows no traffic. But I can quit the program through q or CTRL+C.
See recording

imsnif · 2020-01-03T20:20:28Z

@zhangxp1998 - thanks for keeping tabs on this. I think we're going to go with 1s for this coming release as it seems to be working for more people. I really want to understand what's going on here and find a better solution. Would you be interested in investigating this deeper?

zhangxp1998 · 2020-01-03T20:22:40Z

@imsnif Yes of course :)

zhangxp1998 · 2020-01-03T21:31:03Z

Root cause found! @imsnif
It's a bug inside libpnet.

See bpf.rs, libpnet did not set the bit mask self.fd_set before each pselect call. The self.fd.fdth bit must be set for the pselect call to succeed. Libpnet sets this bit once when the receiver is initialized. However, if no packet is available, the kernel will clear self.fd_set, and next time they call pselect, the bit mask is not correctly set.

Why no read timeout fixes the issue?
The bug is only triggered if no packet is available, without a read timeout(or with a sufficiently long read timeout), the bit mask won't be cleared so this bug won't be triggered.

The solution is to add

libc::FD_SET(self.fd.fd, &mut self.fd_set as *mut libc::fd_set);

Before their pselect call. Unfortunately that has to be a separate pull request to libpnet. If you want, you can fix this bug locally, and release a version of bandwhich that's free of this bug. But there's nothing we can do in our codebase that will help this problem.

I fixed this issue locally, and rebuild bandwhich with read timeout of 10ms. I can see traffic with or without -I en0, and I can quit the app with q or CTRL+C.

UPDATE:
Pull request to libpnet created here.

imsnif · 2020-01-03T22:47:14Z

@zhangxp1998 - great work!! Very quick, too.
This is great news! Let's wait a short while to see if this gets merged, and if not we can temporarily fork libpnet and use it as a dep until it is.

zhangxp1998 · 2020-01-03T22:48:08Z

@imsnif I looked at pnet GitHub page. They might be seeking new maintainers and experiencing delays in PR. Some PRs are open since Mar 5th 2019. I don't think we will get a response promptly.

imsnif · 2020-01-03T22:52:38Z

Let's give them a short while - at least a few days. I see most of the PRs there are being at least commented on.

imsnif · 2020-01-05T11:00:01Z

There is a somewhat hacky workaround that should solve the problem in some cases (as indicated in this thread) released in 0.7.0. I hope to publish the actual fix with the dep (whether using libpnet or forking it) sometime this week. Leaving this open for now to track it.

JuxhinDB · 2020-01-08T20:42:47Z

Hey, just jumping in here to let you know that I merged the PR and bumped the version to 0.25.0 . Hope that helps

imsnif · 2020-01-08T21:25:04Z

Thanks very much @JuxhinDB !!

diversys changed the title ~~bandwhich shows no traffic~~ bandwhich shows no traffic on macOS Dec 31, 2019

imsnif mentioned this issue Dec 31, 2019

Mac (Mojave): Error: Failed to listen on network interface: No such file or directory #45

Closed

ebroto mentioned this issue Jan 1, 2020

error: Utf8Error #50

Closed

jbpratt mentioned this issue Jan 3, 2020

Add to Homebrew #9

Closed

imsnif changed the title ~~bandwhich shows no traffic on macOS~~ No traffic shown on some macs Jan 3, 2020

imsnif assigned zhangxp1998 Jan 3, 2020

zhangxp1998 mentioned this issue Jan 3, 2020

Fix a bug in calling pselect libpnet/libpnet#394

Merged

kontrafiktion mentioned this issue Jan 4, 2020

bandwhich 0.7.0 (new formula) Homebrew/homebrew-core#48504

Closed

5 tasks

imsnif mentioned this issue Jan 5, 2020

Fails to listen to network interface on Catalina #31

Closed

imsnif added the bug Something isn't working label Jan 5, 2020

This was referenced Jan 7, 2020

Fix bug that non-connected Udp sockets aren't displayed #82

Merged

chore(packages): use pnet fork until they fix the pselect bug #83

Merged

imsnif closed this as completed in #83 Jan 7, 2020

No traffic shown on some macs #51

No traffic shown on some macs #51

Comments

diversys commented Dec 31, 2019 • edited Loading

mpereira commented Dec 31, 2019

berry commented Dec 31, 2019

pdbradley commented Dec 31, 2019

munntjlx commented Dec 31, 2019

Dapacruz commented Dec 31, 2019 • edited Loading

adraffy commented Dec 31, 2019

capitalist commented Dec 31, 2019

imsnif commented Dec 31, 2019

diversys commented Dec 31, 2019 • edited Loading

briantully commented Dec 31, 2019

briantully commented Dec 31, 2019 • edited Loading

grishy commented Dec 31, 2019 • edited Loading

grishy commented Dec 31, 2019

grishy commented Dec 31, 2019 • edited Loading

munntjlx commented Dec 31, 2019 via email

grishy commented Dec 31, 2019

grishy commented Dec 31, 2019

grishy commented Dec 31, 2019

grishy commented Dec 31, 2019

grishy commented Dec 31, 2019 • edited Loading

imsnif commented Jan 1, 2020

zhangxp1998 commented Jan 1, 2020 • edited Loading

imsnif commented Jan 1, 2020

zhangxp1998 commented Jan 1, 2020 • edited Loading

zhangxp1998 commented Jan 1, 2020

imsnif commented Jan 1, 2020

zhangxp1998 commented Jan 1, 2020

nimbius commented Jan 3, 2020

imsnif commented Jan 3, 2020 • edited Loading

munntjlx commented Jan 3, 2020 via email • edited Loading

munntjlx commented Jan 3, 2020

imsnif commented Jan 3, 2020

munntjlx commented Jan 3, 2020

munntjlx commented Jan 3, 2020

munntjlx commented Jan 3, 2020

imsnif commented Jan 3, 2020

floriancargoet commented Jan 3, 2020

zhangxp1998 commented Jan 3, 2020 • edited Loading

imsnif commented Jan 3, 2020

zhangxp1998 commented Jan 3, 2020

zhangxp1998 commented Jan 3, 2020 • edited Loading

imsnif commented Jan 3, 2020

zhangxp1998 commented Jan 3, 2020 • edited Loading

imsnif commented Jan 3, 2020

imsnif commented Jan 5, 2020

JuxhinDB commented Jan 8, 2020

imsnif commented Jan 8, 2020

diversys commented Dec 31, 2019 •

edited

Loading

Dapacruz commented Dec 31, 2019 •

edited

Loading

diversys commented Dec 31, 2019 •

edited

Loading

briantully commented Dec 31, 2019 •

edited

Loading

grishy commented Dec 31, 2019 •

edited

Loading

grishy commented Dec 31, 2019 •

edited

Loading

grishy commented Dec 31, 2019 •

edited

Loading

zhangxp1998 commented Jan 1, 2020 •

edited

Loading

zhangxp1998 commented Jan 1, 2020 •

edited

Loading

imsnif commented Jan 3, 2020 •

edited

Loading

munntjlx commented Jan 3, 2020 via email •

edited

Loading

zhangxp1998 commented Jan 3, 2020 •

edited

Loading

zhangxp1998 commented Jan 3, 2020 •

edited

Loading

zhangxp1998 commented Jan 3, 2020 •

edited

Loading