WiFi shield packet loss causes cyclical noise spikes

[conorrussomanno]

See attached screenshots. You can see there is a spike of noise once every second or so. I'm switching back and forth between Dongle and WiFi shield, and it only appears when using WiFi, so it doesn't seem to be an environmental effect.

I'm using 1000Hz and 10ms latency.

[conorrussomanno]

Now this is only happening every once and a while for me. I'm not sure what is the root cause. It could be my hardware. I'm interested to see if anyone else experiences something similar using the WiFi Shield.

[andrewjaykeller]

Are these regularly spaced? Could you try updating your firmware? Maybe something got botched the first time?

[biomurph]

I don't see this problem on my wifi

[andrewjaykeller]

could you try putting EXT PWR to OFF and use two batteries?

[andrewjaykeller]

i think that might help with the noise

[andytheengineerguy]

Hi guys, was there any progress on this?

I'm also experiencing cyclical noise spikes, which vary between 1-4hz, and only appear occasionally. The occurance of these cyclical spikes seems loosely correlated with resetting the system through the OpenBCI GUI.

I'm not using a WiFi shield, just the regular cyton EEG sensing board. It had been working well up until tonight.

I was playing around with the BIAS and SRB pins before I saw this behaviour. Any ideas?

[andrewjaykeller]

Hi! There has been a lot of progress on this re: Cyton with WiFi shield. However this is not an issue we normally see with the Cyton over serial. Could you upload a screen shot of your spikes here?

[andytheengineerguy]

Hi AJ, thanks for the quick reply!

I actually just left work so cant take any more screenshots until I get in tomorrow morning (different time zone here in Aus), however I did take this photo showing what happens after repeatedly connecting-disconnecting the system through the OpenBCI GUI. For some reason, the frequency of the pulses get faster each time I disconnect-reconnect, and after 6-7 iterations this is what i end up with.

Its a strange waveform thats distinctly different from random noise. While showing this waveform im unable to sense any EEG data.

Have you seen this before? Apologies if Ive gone too off track with this issue - this is really confusing me haha. Will upload pulse screenshots in the morning.

[andytheengineerguy]

[andrewjaykeller]

That looks almost like a test signal, hmm is this issue still happening?

[biomurph]

Not the test signal. Some environmental noise?

[retiutut]

I am experiencing this issue also when using the wifi shield with 8 channels and using two separate batteries (4xAA and 3.7v 500mah).

[andrewjaykeller]

What firmware are you running on the WiFi shield?

[retiutut]

The WiFi shield firmware is v2.0.5 . Updated the Cyton firmware this morning and still having same problem:

[andrewjaykeller]

Have you tried UDP?

[retiutut]

I've mostly been using TCP. I did swap out the WiFi shield for a Daisy header and was able to see proper EEG over BLE using the dongle. I'll try UDP and UDPx3 later today or tomorrow.

[andrewjaykeller]

No doubt in my mind this is 100% due to using WiFi! This can happen on some routers I’ve seen, it should magically disappear with UDP. Probably won’t happen with TCP over WiFi direct too.

[retiutut]

To document: I'm using a current generation Airport Extreme mounted high on the wall. The LOS distance from the Airport to the WiFi shield was about 5 feet.

Had to put my iPhone in airplane mode to have uninterrupted upload for the Cyton firmware via Arduino IDE. Left it in airplane mode during these tests.

[retiutut]

This issue is still happening over UDP, rendering the WiFi shield unusable. Was not able to find the shield using WiFi direct in the GUI.

[andrewjaykeller]

Did you erase the network credentials?

[retiutut]

I erased the network credentials again, got 10 blinks in 2 seconds on boot, and connected to the Wifi shield. The GUI still only finds this:

[andrewjaykeller]

Weird! Which GUI version are you using.

[retiutut]

Using OpenBCI_GUI 3.2.0 standalone on Mac

[retiutut]

@aj-ptw What settings (hardware and software, apart from up-to-date firmware) would you recommend to resolve this issue? I was looking forward to the increased resolution over WiFi, but I get worried every time I have to change the hardware just to see if the WiFi shield works.

[andrewjaykeller]

@retiutut could you try on a different router? If not, maybe we can do a call and I show you how WiFi direct works. Skype me at pushtheworld_aj

[retiutut]

I think I have to try and update the Cyton firmware again. I'm having serious issues over BLE also. Here we have a series of eye blinks followed by rest and then a spike across all channels.....

[andrewjaykeller]

Hey @retiutut ! I have just finished a new gui version that will allow you to use the wifi direct for sure.

[retiutut]

I have identified the cause of the problem using the Processing version. I see packet loss correlations with these visible spikes when using an Airport Extreme. I was able to finally connect via wifi direct using a newer Mac Mini and I still see regular spikes when streaming. Here is a screenshot that documents this issue:

[wjcroft]

Richard, can you confirm, the packet loss and spikes have also been seen on Ganglion. I would assume. Is TCP more prone to this than UDP or UDPx3?

[retiutut]

As far as I know, the really pervasive spikes for little/no reason only happen with Cyton+WiFi or a Cyton with possible low battery.

To me, Ganglion+WiFi is really fun and stable. I've only been able to cause packet loss and spikes with Ganglion or Ganglion+WiFi by moving to another part of the house or even outside. Still if there is enough signal, a spike will show when a packet is dropped. A fix for these types of "hiccups" is in progress.

[wjcroft]

Had some exchanges with Jason on the thread discussing the power supply ringing. See the TI app note I posted there (and below). It exactly describes why the 100uF ceramic cap used on the bottom of the Ganglion, causes huge ringing and noise. Wish Joel or AJ had seen this before choosing that design. My guess is that the noise generated, is more disruptive to the Cyton than the Ganglion, due to the ADS1299 acting as kind of an antenna for high frequency noise pulses generated by the Shield. Components on the Ganglion are less complex and lower amplification, compared to Cyton.

http://openbci.com/forum/index.php?p=/discussion/2108/measured-shield-regulated-3-3v-supply-has-6v-dip-at-1-second-intervals-with-resonant-ringing#latest

TI app note: http://www.ti.com/lit/wp/snoa842/snoa842.pdf "Engineers note: Capacitors are key to voltage regulator design"

In summary, the LDO (low drop out) linear regulator as used on the Ganglion, plus the low ESR ceramic cap on the output, generate excessive ringing and noise due to the low ESR. Solution is to use a cap with both more microfarads, and more ESR. Electrolytic or tantalum looks promising. The previous 680uF electrolytic I happened to find on Amazon was an ultra low ESR; good for some applications, but not LDO linear regulator. Will reorder a more suitable cap for trial, using tips in the TI app note.

[jps2000]

I can not test it but it is important where tho place the cap. As it serves as a local battery it should be as close as possible to the load. In this case when the ESP is drawing current spikes the cap should be close to its VCC and GND pin. Ideally it is a tantal 100u or so having a low ESR. The cap close to the LDO regulator should be within data sheet recommendation ( 1-10u ).
In this case I see that the layout is poor.
I suggest to try a cap directly from pin 8 to the ground plane ( scratch away the solder resist)
You can even try to solder it to the shield of the ESP
Also remove the 100uf on the other side it can make it worse due to bigger current loops and resonances
Archambeault1.pdf

[furmanlukasz]

Hi, having the same issue over UDP but I notice that the signal goes right after disconnecting electrodes , the spikes go off, and the signal is nearly railed, so it is not the packet loose. I saw that the 'gold cup' electrodes can bee the trouble, any time spike goes up the same happens to the impedance of electrode(just click the 'om' icon to see if it happens the same in yours setup). They just far away from the desired quality to make good measure. I will test it over Ag electrodes when they arrived so I hope It will solve the problem. Maybe somebody can test it earlier. Just a hint. But true silver electrodes cost 10 times the ' Chinese gold cup' and have a screen in cable that can be grounded also. The screen between this two boards are also the good idea , perfectly can be grounded from one side cyton ground and the other side wifi ground.

[retiutut]

I think it's time to close this issue as it exists here on the OpenBCI_GUI Repo.

Packet interpolation is live as of GUI 4.1.7-beta.0, and it resolves most packet loss spikes.

While this does improve the data received and displayed in the GUI, it doesn't fix the hardware problems that cause cyclical noise spikes when pairing the WiFi shield with the Cyton or Cyton+Daisy, regardless of power configuration or WiFi connection settings.

To reiterate, I have tried two WiFi shields with Cyton, Cyton+Daisy, and two Ganglions, and this problem only occurs with increased number of channels when using Cyton or Cyton+Daisy with the WiFi shield.

As previously indicated in the conversation above, it seems a hardware fix is required, either to existing WiFi shields or in the development of a new board/shield of some kind.

[wjcroft]

Richard, I can understand why you want to close this as related to the GUI. And especially now that you have the interpolation going. But this thread documents a serious hardware issue that has yet to be solved.

So it needs to be kept open SOMEWHERE. Perhaps by referring back to this thread from an open issue on the Wifi Shield hardware or software repo. The material in this thread is documenting some of the engineering fixes required.

This is the related thread in the other repo:

OpenBCI/OpenBCI_WIFI#82
"WiFi Shield dropped packets"

I suggest changing the title on that thread from the original title, which was restricted to Direct mode. You might also consider changing the title on this issue thread to match.

William

[Joe-Westra]

Hi @retiutut and @wjcroft,

Thanks for providing this detailed overview of the cause of the noise spikes, they were maddening when unexplained. I am using a cyton+daisy+wifishield to transmit 16 channels at a minimum sample rate of 250hz. Is it correct that the only option to capture this stream reliably would therefore be to use the cyton+daisy+SD card and transfer the data via BLE at session termination?

Do you have any estimation of when a fix for the wifishield (or it's next iteration) might be available?

Thank you!

[wjcroft]

The shield power supply is helped, by running it on a separate AA battery pack. So TWO batteries, one on Cyton, one on Shield. Also UDPx3 is more robust than TCP, in general. Oddly some people have trouble with UDPx3, in which case fall back to TCP.

[Joe-Westra]

Thanks for the feedback @wjcroft, I'll let you know how successful we are when we get a chance to try the two separate battery packs.

[stellarpower]

Are we still expecting to see these sorts of issues? I don't see many references to Wifi development in the last few years. I have major regular noise artifacts in my datastream, and am going round an round in circles trying to find out what they are caused by:

Any time I think I have isolated the cause and pinned them down, they come back again. Always that distinctive shape. Is this still a known issue with the wifi mode of connection?

[wjcroft]

@stellarpower you are correct. The board you are using has the same issue documented here,

http://openbci.com/forum/index.php?p=/discussion/2108/measured-shield-regulated-3-3v-supply-has-6v-dip-at-1-second-intervals-with-resonant-ringing

The Wifi Shield works perfectly with Ganglion. Which leads one to believe that the firmware interaction between the Cyton mainboard and the shield, is over-stressing the ESP8266 processor on the shield. Causing power supply spikes when the ESP is doing extensive transmissions. Thus it is 'possible' that a firmware fix on Cyton might resolve this.

However the link above explores better power supply filtering on the shield as another potential route. Though not perfect, as you can see by reading the thread.

Unfortunately OpenBCI Staff are largely devoted to Galea hardware / software development at this time. Timeline uncertain as to when a Wifi Shield fix may be implemented. I have suggested the Cyton firmware route in the past. As this would avoid any hardware mods to the Shield board. And be far superior for existing Shield owners.

I assume you are using the UDPx3 mode, and separate battery power supplies (2 AA packs). This seemed to provide the most stable results. In part because the voltage is high enough (6V) to ride out any dips.

William Croft
OpenBCI Forum admin

[stellarpower]

Thanks for getting back to me so quickly.

I assume you are using the UDPx3 mode, and separate battery power supplies (2 AA packs). This seemed to provide the most stable results. In part because the voltage is high enough (6V) to ride out any dips.

I'm using an unofficial board, and both the wifi + daisy combined shield and the main board are powered by a rechargeable Li-ion (I think) battery. My inexpensive multimeter is reading 4V across the supply rail.

According to the description, the manufacturers had improved upon some of the original issues around the noise - I have asked and am waiting for clarification.

On the UDP mode - in the v5 GUI, this option has been removed. I believe it is being used by default, but am not sure how I can verify. Is there a raw command I can send?

Is there any known difference when using software other than the GUI? Running it on my laptop last night, the GUI was using a huge amount of CPU and drained the (laptop) battery within about 40 minutes. I have seen this bug that suggested similar artefacts ocurred when the software on the local side lagged and wasn't able to keep up with the signal. Is there any chance simply using a program in Brainflow to dump the data to e.g. CSV, and then analysing it later, would improve things at all? My next thought was to try this as using the GUI to stream the data is one variable I haven't yet removed from the equation.

So if it can be powered from 6V instead - are both boards then ambivalent to the voltage they're powered from, within reason? I would have assumed that this would need to be relatively exacting, or given that they are designed to run off batteries, do they in fact have internal voltage regulators that are more sophisticated than I would have assumed?

In terms of diagnosing this correctly, is there anything else that spikes looking like this could likely be? Should I be seeing packet drops when this occurs in the GUI widget? As it is currently not registering any at all, having fixed a previous issue in this regard.

And if this is definitely due to wifi issues, is there anything that can be done to mitigate this? One or two channels would be sufficient for the time being, if fully powering off some features reduced the draw causing the spikes. Otherwise it looks like I've bought an expensive doorstop.

Thanks for your help.

[wjcroft]

@stellarpower, thanks. Please do not post any more comments on this thread. Instead post further discussion on your Forum thread. You can quote the last comment you posted above. This thread is more devoted to the base issue with Cyton + Shield, and is not for troubleshooting specific workarounds.

https://openbci.com/forum/index.php?p=/discussion/3502/packet-loss-issue-cyton-wifi#latest

[stellarpower]

I'm happy to keep the discussion there, although am yet to receive any comments, but the main reason for adding here was to confirm if this issue is still reproducable, given there's been no activity in some time. Equally for #349, I need to do some more work on the graphing, but I'm not sure I'm necessarily seeing the same artefacts when the OpenBCI GUI is removed from the equation, in which case that may want to be re-opened, or linked to a fresh issue.

If it seems from the forum post that this is in fact due to the power spikes then I will add any new relevant details here.

Cheers

[philippitts]

I appreciate all the discussion here and on the forum. The WiFi shield has been pulled for a hardware update, and we don't have an ETA on its availability at this time. While OpenBCI does make some hardware designs open source, we cannot support third-party adaptations of this hardware as factors beyond our knowledge or control may have been changed.

As mentioned by @retiutut in this comment, the fixes for this issue appear to be hardware and firmware related for official devices sold by OpenBCI. I'm closing the issue as the fixes for this problem are not related to the GUI.

[stellarpower]

I don't know if you maintain open issues on hardware- and design- related threads in relevant repos, but if you do then I agree that the spikes on the power supply itself (as opposed to the fixed bug regarding packet loss, that should be seen in the packet loss widget, which I think is separate and was why this particular issue was originally opened - so the thread has become a bit tangential) would be best discussed there - and if so, a link here might be useful if anyone stumbles across this issue again and would be best forwarding comments towards ongoing discussion ;)

FWIF also appreciate it's perfectly reasonable not to be able to dedicate developer time to issues if these are limited to DIY or third-party fabrications of the board. I guess the odd thing may crop up within a third-party device that reflects on the designs or the software that would be beneficial to consider across the board, but in the general case that makes sense to me. I think trying to be inclusive of other implementations of the hardware - albeit on a second-class basis - where it seems that changes may be made that would affect all devices, and the issue hasn't been isolated just to that hardware, is more than a fair policy.

Cheers!