RedTide 650 support not working

[JuanCab]

spectrometer and system information

• model: RedTide 650
• operating system: macOS 10.12.5
• python version:3.5.1
• python-seabreeze version: 0.6.0
• installed-via: _conda

current problem

In a nutshell, the same code I have that works with an Ocean Optics USB2000 when applied to a RedTide 650 (which appears to be listed on the list of compatible devices) doesn't work due to a failure to connect to the spectrometer. The specific steps are below:

steps to reproduce

Specifically, although Apple's System Profiler clearly shows the RedTide connected (and I can access it via Vernier's Logger Pro software), when I do

import seabreeze.spectrometers as sb
devices = sb.list_devices()
print(devices)

I get a blank list.

[ap--]

With the new conda packages python-seabreeze defaults to the cseabreeze backend.
Maybe USB650 support doesn't work in cseabreeze... If so we need to file this upstream again.

In the meantime try installing python-seabreeze and pyUSB in conda:

conda install -c poehlmann python-seabreeze
pip install pyusb

(Note: the akode channel that we used before to install pyusb only ships it for 2.7. So we'll just install pyusb in conda via pip from pypi)

To check if pyusb was installed correctly run (it should not raise an exception):

import usb.core
usb.core.find()

and then:

import seabreeze
seabreeze.use("pyseabreeze")
import seabreeze.spectrometers as sb
devices = sb.list_devices()
print(devices)

I hope it'll work. I'll check in the meantime why the 650 isn't working with cseabreeze.

[JuanCab]

I will try this when I get back in the lab, but one quick question, is there a way to properly check if the cseabreeze interface works (for example, if devices = sb.list_devices() returns no devices) and then switch the interface to pyusb? I'm going to try an approach, but I suspect I can't simply re-import seabreeze.spectrometers.

[ap--]

Hmm. I think this might work on OSX and Linux:

# There is definitely a spectrometer connected, and the code
# should first try to connect via cseabreeze, and switch to
# pyseabreeze if it can't find the spectrometer
from seabreeze.backends import _use_cseabreeze
_lib = _use_cseabreeze()
if _lib is not None and _lib.device_list_devices():
    pass  # cseabreeze import works and it finds a spectrometer
else:
    # cseabreeze import failed or it didn't find a spectrometer
    import seabreeze
    seabreeze.use("pyseabreeze")

# continue normally
import seabreeze.spectrometers as sb
...

[ap--]

So as additional info:

I think reimporting seabreeze.spectrometers won't work,

but _use_cseabreeze and _use_pyseabreeze in seabreeze.backends return handles that have all the library methods including device_list_devices().
So it should be possible to do the check before importing seabreeze.spectrometers

[ap--]

Note to self: Maybe this should be default behavior...

[JuanCab]

Using your 'auto selection' code I was able to see the USB650, but I am getting errors when attempting to take a spectrum, even turning off dark and non-linearity corrections. The errors don't occur during every exposure, but happen about 80% of the time. Looks to be ultimately a USB overflow problem (based on reviewing error messages).

Juan

Traceback (most recent call last):
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/common.py", line 14, in decorated_func
    return func(*args, **kwargs)
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/spectrometer.py", line 87, in get_unformatted_spectrum
    timeout=int(self._INTEGRATION_TIME_MAX * 1e-3 + self.usbtimeout_ms))
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/communication.py", line 65, in usb_read_highspeed
    return self._device.read(self._ENDPOINT_MAP.highspeed_in, size, timeout=timeout)
  File "~/anaconda3/lib/python3.5/site-packages/usb/core.py", line 988, in read
    self.__get_timeout(timeout))
  File "~/anaconda3/lib/python3.5/site-packages/usb/backend/libusb1.py", line 833, in bulk_read
    timeout)
  File "~/anaconda3/lib/python3.5/site-packages/usb/backend/libusb1.py", line 936, in __read
    _check(retval)
  File "~/anaconda3/lib/python3.5/site-packages/usb/backend/libusb1.py", line 595, in _check
    raise USBError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBError: [Errno 84] Overflow
Traceback (most recent call last):
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/common.py", line 14, in decorated_func
    return func(*args, **kwargs)
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/spectrometer.py", line 87, in get_unformatted_spectrum
    timeout=int(self._INTEGRATION_TIME_MAX * 1e-3 + self.usbtimeout_ms))
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/communication.py", line 65, in usb_read_highspeed
    return self._device.read(self._ENDPOINT_MAP.highspeed_in, size, timeout=timeout)
  File "~/anaconda3/lib/python3.5/site-packages/usb/core.py", line 988, in read
    self.__get_timeout(timeout))
  File "~/anaconda3/lib/python3.5/site-packages/usb/backend/libusb1.py", line 833, in bulk_read
    timeout)
  File "~/anaconda3/lib/python3.5/site-packages/usb/backend/libusb1.py", line 936, in __read
    _check(retval)
  File "~/anaconda3/lib/python3.5/site-packages/usb/backend/libusb1.py", line 595, in _check
    raise USBError(_strerror(ret), ret, _libusb_errno[ret])
usb.core.USBError: [Errno 84] Overflow

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/common.py", line 14, in decorated_func
    return func(*args, **kwargs)
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/spectrometer.py", line 129, in get_formatted_spectrum
    self.get_unformatted_spectrum(tmp)
  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/common.py", line 23, in decorated_func
    raise SeaBreezeError(msg)
seabreeze.pyseabreeze.interfaces.common.SeaBreezeError: Error while reading raw spectrum.
Traceback (most recent call last):

  File "<ipython-input-17-710a9766b31a>", line 1, in <module>
    runfile('~/mycode/spec_controller.py', wdir='~/Documents/Research/Eclipse 2017/Experiments/Eclipse Software')

  File "~/anaconda3/lib/python3.5/site-packages/spyder/utils/site/sitecustomize.py", line 880, in runfile
    execfile(filename, namespace)

  File "~/anaconda3/lib/python3.5/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "~/mycode/spec_controller.py", line 28, in <module>
    lamb, intens = spec.exposure(dev, time, darkcorr=False, nlcorr=False, verbose=True)

  File "~/mycode/speclib.py", line 134, in exposure
    intens = spec.intensities(correct_dark_counts=darkcorr, correct_nonlinearity=nlcorr)

  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/spectrometers.py", line 150, in intensities
    out[transfered_N:])

  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/wrapper.py", line 123, in spectrometer_get_formatted_spectrum
    return device.interface.get_formatted_spectrum(out)

  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/common.py", line 14, in decorated_func
    return func(*args, **kwargs)

  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/spectrometer.py", line 129, in get_formatted_spectrum
    self.get_unformatted_spectrum(tmp)

  File "~/anaconda3/lib/python3.5/site-packages/seabreeze/pyseabreeze/interfaces/common.py", line 23, in decorated_func
    raise SeaBreezeError(msg)

SeaBreezeError: Error while reading raw spectrum.

[ap--]

Does this also occur when you don't use the "auto-select" code? So just:

import seabreeze
seabreeze.use("pyseabreeze")
import seabreeze.spectrometers as sb

And possible ways to proceed are:

1. We fix the seabreeze C library (and therefore cseabreeze)
   1. File an issue on sourceforge, and request USB650 support (Because, while they provide windows drivers and correct udev-rules, it's not implemented in the code)
   2. I can walk you through how to do it yourself if you're comfortable with C/C++.
2. We fix pyseabreeze
   1. I can walk you through how to do it yourself.

Anyways, it would be good to have a datasheet with the full usb command set and compare it in detail with the one for the USB2000 (Because I think it's almost identical to that one)

[JuanCab]

Not using the auto-select does in fact stabilize things. I can now perform exposures on the USB650 without issue.

I am comfortable with C (its been a few years, but it dominated my grad school days), less so with C++. Feel free to provide me with guidance on how to fix this if by fix you mean allowing auto-select to function. I think I should file an issue with seabreeze as well, but do you have a specific error to report other than USB 650 support is not functional?

[ap--]

To just fix the auto-select, I think we don't need to touch any of the C/C++ stuff...

I think my previous approach doesn't work because the cseabreeze backend is not unloaded correctly.

Maybe try:

import usb.core
import seabreeze
if usb.core.find(idVendor=0x2457, idProduct=0x1014):
    # If we find a connected USB650
    seabreeze.use("pyseabreeze")
else:
    seabreeze.use("cseabreeze")

import seabreeze.spectrometers as sb

But: Why don't you just use the pyseabreeze backend for all spectrometers?

And regarding the upstream ticket: it's just that USB650 support is missing.

[JuanCab]

I didn't want to try the new backend because I am in the middle of an experiment and my mentality is to keep everything the same. That said, I can't imagine there should be a difference, so that may be the easier solution here.

I'll try your solution to the auto-switching shortly and if that fails I will just switch to using pyusb.

[JuanCab]

FYI, I posted a trouble ticket for seabreeze, response was that they intentionally left out RedTide USB650 support. See https://sourceforge.net/p/seabreeze/tickets/32/

[JuanCab]

On a positive note, your "switching" code above does seem to do the trick.

[JuanCab]

I take it back, I am still getting USB overflow errors with the RedTide USB650, even when switching to using pyUSB exclusively. :/

And I have also discovered when I do manage to access the RedTide USB650, I often get a uniform 2409.00 (2417.15893453) counts in every channel with non-linearity corrections turned off (on). Clearly not real data. This only appears to occur with the RedTide USB650, my Ocean Optics USB2000 appears to be behaving OK, even when accessed via pyUSB.

Something strange is going on here. Could there be some issue with how python-seabreeze communicates with the RedTide USB650? I was initially getting "real" looking data, but now its either overflowing or clearly bogus data.

[ap--]

Regarding the sourceforge ticket: Well I guess that explains why I can't find a USB-communication datasheet for the USB650... Weird though that they intentionally leave out support for it...

Regarding the overflow issue: Without the hardware or a datasheet it's hard for me to help with this.
It could be, that some of the assumed parameters for the 650 in pyseabreeze/interfaces/defines.py or in pyseabreeze/interfaces/init.py are wrong.

Is your measuring protocol doing some asynchronous stuff? Or something else that might cause the sporadic failures?

[JuanCab]

I'm not doing anything asynchronous. Just attaching the spectrometer, taking a reading, and detaching the spectrometer. And as I said, it works with the Ocean Optics USB2000 even when accessing via pyusb. I am doing this in iPython, so maybe iPython doesn't close something it should.

[ap--]

And one last idea:
Maybe the USB650 is not a rebranded USB2000, but actually a rebranded USB4000...
(<tinfoil hat> The usb650 product page has a USB4000 tag... </tinfoil hat>)

import seabreeze
seabreeze.use("pyseabreeze")
# Monkey patch the USB650 to use the USB4000 protocol
from seabreeze.pyseabreeze.interfaces import USBInterfaces, USB4000
USBInterfaces[0x1014] = USB4000

import seabreeze.spectrometers as sb
...

No idea if this will work. But I guess it's worth one last try...

[JuanCab]

Pretty sure the USB650 is a rebranded USB2000, it has a sticker on the bottom indicating it is a USB2000.

[ap--]

Okay... Then I am out of ideas. Sorry.

[prissi]

The USB650 seems to return the spectrum in a different format.
Each 64 byte block first contains the low bytes and the next block 64 block the high bytes.
So two data points are actually constructed like this:

So for each 64 block one has to do: (j contains the offset in the raw data, i the spectrum point)

data[i] = raw[j]+256*raw[j+1+64]
i += 1
j += 1

And timeconst is actually 1000 less than for an USB2000+ ff and returns for maximum value 4095.

[ap--]

Hi @prissi

The current device implementation in the pyseabreeze backend is based on the USB2000 (not USB2000+) implementation.

python-seabreeze/src/seabreeze/pyseabreeze/devices.py

Lines 652 to 678 in ff806a8

	class USB650(SeaBreezeDevice):
	model_name = "USB650"
	# communication config
	transport = (USBTransport,)
	usb_product_id = 0x1014
	usb_endpoint_map = EndPointMap(ep_out=0x02, lowspeed_in=0x87, highspeed_in=0x82)
	usb_protocol = OOIProtocol
	# spectrometer config
	dark_pixel_indices = DarkPixelIndices.from_ranges()
	integration_time_min = 3000
	integration_time_max = 655350000
	integration_time_base = 1000
	spectrum_num_pixel = 2048
	spectrum_raw_length = (2048 * 2) + 1
	spectrum_max_value = 4095
	trigger_modes = TriggerMode.supported("NORMAL", "SOFTWARE", "HARDWARE")
	# features
	feature_classes = (
	sbf.eeprom.SeaBreezeEEPromFeatureOOI,
	sbf.spectrometer.SeaBreezeSpectrometerFeatureUSB650,
	sbf.rawusb.SeaBreezeRawUSBBusAccessFeature,
	sbf.continuousstrobe.SeaBreezeContinuousStrobeFeatureOOI,
	)

which uses the millisecond timescale the smaller max value and the following raw spectrum formatting:

python-seabreeze/src/seabreeze/pyseabreeze/features/spectrometer.py

Lines 164 to 175 in ff806a8

	class SeaBreezeSpectrometerFeatureOOI2K(SeaBreezeSpectrometerFeatureOOI):
	def get_intensities(self):
	tmp = self._get_spectrum_raw()
	# The byte order is different for some models
	N_raw = self._spectrum_raw_length - 1
	N_pix = self._spectrum_length
	idx = [(i // 2) % 64 + (i % 2) * 64 + (i // 128) * 128 for i in range(N_raw)]
	# high nibble not guaranteed to be pulled low
	tsorted = tmp[idx] & numpy.array((0xFF, 0x0F) * N_pix, dtype=numpy.uint8)
	ret = numpy.array(struct.unpack("<" + "H" * N_pix, tsorted), dtype=numpy.double)
	# sorted and parsed
	return ret * self._normalization_value

does this agree with your comment?

Cheers,
Andreas 😃

P.S.: Although I still am not entirely certain that the USB650 works with python-seabreeze, I believe that some USB650 require a firmware update to be supported. See: https://github.com/ap--/python-seabreeze/issues/48
It would be super interesting to know the firmware version of USB650 spectrometers and a report if they work with the pyseabreeze backend or not.

[prissi]

My knowledge of python and numpy is not good enough to answer that. But it seems similar. Seabreeze does not install on this machine, neither from Mingw python (where already numpy does not work) or it installs but does not work from anaconda. But I am not python specialist.

The USB650 is a recent model. OceanView does not report a firmware version, so I am not sure how to check for the firmware version. It does not work with the command for USB2000+, which I have.

Since the target computer is an Win7 machine without network access (due to old custom hardware ...), I am using the USB650 with the libusb from C directly and it works like an existing USB2000+, but with the different byte order as above. I arrived (at the old repo), because this is one of the very few places that had any info on the USB650, and the documentation here was enough to get it working.