Equation for Lux calculation does not reflect Library

[Rvice]

The calculations in this project (CircuitPython) reference the AdatFruit_TSL2591_Library, but the equations do not match.

https://github.com/adafruit/Adafruit_TSL2591_Library/blob/master/Adafruit_TSL2591.cpp#L220

This project does take into account the max_count for saturation and sensor overflow, but the source project does not.

[tannewt]

What should be changed in the CircuitPython library?

[Rvice]

Yes, but I am not sure which equation is right given the following open issue: adafruit/Adafruit_TSL2591_Library#24

[tannewt]

@caternuson Do you know what the correct equation is?

[caternuson]

@tannewt TLDR - no

It looks like the disagreement between the libraries is a result of this commit:
adafruit/Adafruit_TSL2591_Library@2b3b098
made after the CircuitPython library was written and was a response to this issue:
adafruit/Adafruit_TSL2591_Library#14
which was closed but has ongoing discussion. The additional issue linked above by @Rvice may be related.

I am having trouble finding a definitive lux equation for the TSL2591. The datasheet for the TSL2591 does not have one:
https://ams.com/documents/20143/36005/TSL2591_DS000338_6-00.pdf
Given that the TSL2591 is a high dynamic range variant, it's not clear if the information in the datasheets and application notes which cover the TSL2561 and TSL2581 (both replaced by TSL2571) apply. This particular AN seems to contain the necessary technical details and is even linked to from the TSL2591 product page:
https://ams.com/documents/20143/36005/AmbientLightSensors_AN000170_2-00.pdf
but the verbiage in it says it applies to the TSL2571 only. There is no mention anywhere in it of the TSL2591. Do we consider the link enough evidence that the information applies to the TSL2591?

[tannewt]

I'd be tempted to follow the AN since its linked. Maybe try it out and see?

[caternuson]

Let's ignore the disparity between the libraries for now and try to determine what is the most correct implementation for both.

@Rvice Where did you get your information to come up with the changes you made in #6 ?

[Rvice]

The datasheet table for ATIME Max Count in "Control Register (0x01)" and findings from issue #5 created.

[caternuson]

Thanks. Looks like the same info is contained in the summary table as well. For the TSL2591 there's this:

but for the TSL2561 there's a different behavior:

These values are used to check for digital and analog saturation prior to computing lux. Since the values are different, this check will be done differently between the TSL2591 and TSL2561. I believe this check is being done correctly in this library.

Now for lux...

Back to my question from above. I can't figure out where this came from:
https://github.com/adafruit/Adafruit_TSL2591_Library/blob/master/Adafruit_TSL2591.cpp#L284
and the equation currently being used:
https://github.com/adafruit/Adafruit_TSL2591_Library/blob/master/Adafruit_TSL2591.cpp#L290
is still being debated:
adafruit/Adafruit_TSL2591_Library#14 (comment)
Let me investigate the behavior of the various equations. Will report back here.

[caternuson]

Well, they (TSL2561 vs. TSL2591) behave differently. So anecdotally, I think the TSL2591 is different. I sent a request to AMS (they bought TAOS) to ask them for what they consider the proper lux equation for the TSL2591. This was via a web form accessed from here:
https://ams.com/tsl25911
Will give that a few days before continuing with the available documentation.

[ghost]

Has anyone taken an actual Lux meter and compared the values? I've been reading a little bit on these formulas and haven't seen anyone mention it.

[Rvice]

We've done a little with a lux meter and scale based on the difference of the reading. Also, clipping to 0 if the lux value is negative.

[microbuilder]

There is unfortunately a bit of complexity in deriving lux in the case of the TSL2591, partly because the required information isn't terribly clear in the datasheet.

Trying to do some reading on this (not being an expert in the domain myself!), there are some important points to consider talking about lux:

• First, lux is not a 'scientific' value based on a concrete physical phenomenon, it's an estimation based on a model of human vision and is calculated using the CIE Photopic Luminosity Function (part of CIE 1931). There is an improved CIE 1981 version (50 years later!) that should be used since it has a better response curve for low light, but most people still use the older CIE1931 curve.
• Lux is a photometric measurement, based on human vision and a model derived from a limited number of human test samples.
• Sensors can only accurately be used to measure radiometric units, which involves photons and quantifiable physics.
• There are, however, equivalent units for both photometric and radiometric values. In the case of lux they are:
  • Photometric unit: Illuminance (lux) = lumens/m^2
  • Equivalent radiometric unit: Irradiance (W/m^2)

The goal then, to derive lux, is to convert from irradiance (a radiometric measurement) to illuminance (a photometric value based on an estimation of human vision), which is done using the CIE 1931 photopic curve.

NOTE: To confuse matters further there are actually two luminosity functions defined by the CIE, the photopic curve for normal light levels, and the scotopic curve for low light levels, but we'll only look at the photopic curve for simplicity sake.

You can see the CIE 1931 photopic curve here, image taken from the wikipedia page for Luminous Function:

Normal human vision can detect wavelengths from 380-780nm, but our sense of 'brightness' is very heavily concentrated in the green to yellow range in the middle. It comes down to the design our our eyes which are made of of codes and rods.

Our eyes have three 'cones' that roughly are sensitive to red, green and blue wavelengths. The red cones (higher nm values) are most sensitive to light and are more abundant (64% red cones, 32% green cones and only 2% blue cones), but 'brightness' is more associated with the green cones than red and blue. (Bonus fact: Now you know why there are two green photodiodes to every 1 red and blue in bayer pattern sensors. 🙂).

This is why the photopic luminous curve that forms the basis of lux (the black curve above) is heavily biased by the green to yellow wavelengths (the peak in the middle).

Anyway ... enough botched anatomy I'm surely mischaracterizing. :P The problem is taking radiometric values that we can accurately measure and converting them to the equivalent photometric units based on this estimation of human vision.

The TSL2591 datasheet lists irradiance, the radiometric response, at two wavelengths:

At 4000K (what they are calling White light, though the idea of 'White' light is highly complex and a colorimetry tutorial in itself), with ATIME=100ms and GAIN=MAX (~9876x):

• 1uW/cm^2 of irradiance would give you a value of 6024 on CH0 and 1003 on CH1

At 850nm, which is OUTSIDE HUMAN VISION and in the non-visible infrared range, we would get:

• 1uW/cm^2 = 5338 on CH0 and 3474 on CH1

This makes sense since CH1 is more concentrated towards the IR range, as shown below:

In order to calculate lux, you will need to find a way to map the spectral responsivity of the sensor's photodiode(s) to the photopic curve mentioned earlier.

Interesting sidenote: An RGB sensor is actually more capable of delivering accurate 'lux' values in this situation, since it has three separate photodiodes and values to use when mapping radiometric sensor output to the photometric luminous curve!

While accurate 'lux' values don't 'really' exist since it's a somewhat arbitrary 'estimation' to begin with, to get the closest match we can to the photopic curve, it helps to know the spectral density of your light source. As the two datasheet values show, infrared heavy light will give different output than light more concentrated in the visual range, but both may have a very strong radiometric response.

Converting Radiance (Radiometric) to Irradiance (Photometric)

To try to make something useful out of this giant response, we can take the 4000K white LED light as a starting point, and calculate counts per uW/cm^2 based on the following values:

• 6024 on CH0 = 1uW/cm^2 = 0.000166002656 uW/cm^2 per lsb (at 100mS and max gain!)

It seems to me the best we can do to convert this to an equivalent photometric unit (lux) is to correlate the peak responsivity of the sensor, which looks to be ~650nm in the normalised spectral responsivity in the DS with the same position on the photopic curve.

Here is an overlay of the two curves:

I don't have an instant answer on the coefficients to map these two values, and it depends largely on the spectral composition of the light source you are measuring, but I think this at least explains all the science behind lux and how to calculate it since AMS apparently doesn't want to make life easy for us. :P

I'll have to think about this some more and follow up, but hopefully this gets the ball rolling towards better lux values, and establishes a solid 'scientific' bases for future calculations.

to be continued ...

PS: I'm not an expert in this domain, so please feel free to correct me on any mischaracterisations above if you're more knowledgable on the subject matter!

[caternuson]

Awesome info dump. Thanks.

Just to update on the approach I was taking - AMS never got back to me on my request for their lux equation. :(

[ghost]

It sounds like these sensors were never intended to accurately measure light in the first place. Maybe more effort should go into configuring the interrupt registers on the device. My TSL2591 works great as a light intensity switch when configured properly.

[microbuilder]

It sounds like these sensors were never intended to accurately measure light in the first place. Maybe more effort should go into configuring the interrupt registers on the device. My TSL2591 works great as a light intensity switch when configured properly.

I wouldn't go that far. These sensors have a fairly typical response curve, and with a combination of finding the right coefficient to match the photopic luminosity function, and probably a second 'device factor' coefficient that takes into account the physical characteristics of the sensor and enclosure you should be able to get 'reasonable' values. It just takes a fair amount of reading and some digging into the theory, but I think 90% of it is covered above.

That said ... RGB sensors are actually better suited to LUX conversions since you have a finer degree of control when 'selecting' values that fall within the photopic luminosity function.

[microbuilder]

Awesome info dump. Thanks.

Just to update on the approach I was taking - AMS never got back to me on my request for their lux equation. :(

AMS is kind of awful for support through public channels. Every time I've sent them a Q, however precise, I get an email to say that it was forwarded ... and then it falls into a black hole. Even when I send a second followup ping. :P But that's kind of par for the course, sadly. The only company with really good support (that I've worked with) is Nordiic, who set the bar extremely high.

I think the approach documented here is the right one, I just need some more time to work through it to find a good pair of coefficients, though there is also an element of the light source and the IR component. LED lights, for example, have almost no IR component, but natural sunlight has a huge component, which will change the weight of the output values. Just one more reason that a straight-forward 'accurate' lux conversion is anything but 'straight-forward' or 'accurate' when you dig into the theory and physics behind it. :)

[SimonMerrett]

Just in case anyone fancies having a go at mapping photopic luminosity function to the TSL2591 spectral response curves, I decided to extract the CH0 and CH1 XY curve points here. It makes a nice plot in Excel and I found a cool tool (see readme.md) into the bargain!

[microbuilder]

@SimonMerrett Nice find on the tool! I was going to write one myself, glad I won't have to. And thanks for sharing the data. I have my hands full with some new HW at the moment, but have saved a copy of this.

[SimonMerrett]

I was going to write one myself, glad I won't have to.

I was so excited I nearly submitted to the Hackaday tips line. Then I stopped and checked, and of course, Jenny List told us this tool existed nearly three years ago!

[SimonMerrett]

have saved a copy of this.

@microbuilder go and save the new .csv I committed, which contains both the CH0, CH1 and the photopic 1931/2005 and scoptopic xy values. Then get it into Excel or HeuristicLab's symbolic regression tool, depending on how hard you think the problem is!

[Llamero]

No need to reinvent the wheel; AMS provides datasheets on how to improve the accuracy of the sensor, even providing calibration software to calculate the coefficients: https://ams.com/tsl25911#tab/documents

Specifically, this is a good place to start: https://ams.com/documents/20143/36005/AmbientLightSensors_AN000173_2-00.pdf/ce3360f8-fb85-bc22-0bad-b60d3b31efc8

Note that there is no one size fits all equation, as it is dependent on the type of light you expect to encounter.

[kattni]

@caternuson Is this working looking into again?

[caternuson]

@kattni I think to really resolve this for the TSL2591 we would need calibration hardware.

The datasheet specific to the TSL2591 does not provide a specific formula. The Application Notes are all general, requiring the end user to actually follow the outlined processes to come up with results to fit their specific needs.

One solution may be to get rid of the lux parameter and only provide a light parameter?

[kattni]

@caternuson After looking into it, we'll keep lux for now and add a note that it isn't calibrated. I'll put in a PR and close this issue - we can reopen it if needed at a later date.

[rbpasker]

I've just done a bunch of work on getting this library to work in my application, and, although I am not an expert in light, I did figure out what seems to problematic.

The issue is that in low light at LOW gain, getLuminosity() will return something useless, like nan, max, or 0.0.

The trick to getting it to work in low light is to start at LOW gain, then if getLuminosity() returns something useless, raise the gain to MED and re-issue getLuminosity(), and then to HIGH or even MAX.

I've attached a sketch (renamed as .txt to satisfy github), which demonstrates how this works. Here's some output. As you can see in the first sample in very low light with gain=LOW (g=0), the luminosity is nan. When gain is set to MED (g=16), luminosity goes to 0.07, and then to HIGH (g=32), luminosity is 0.06

Anyway, feel free to close this issue again. But I just thought i would get my observation on the record for the next person coming along.

`
I: ----------------------------
I: 2 - g=0 l=nan i=0 f=0
I: 2 - g=16 l=0.07 i=1 f=3
I: 2 - g=32 l=0.06 i=18 f=48
I: ----------------------------
I: 2 - g=0 l=inf i=1 f=0
I: 2 - g=16 l=0.03 i=1 f=2
I: 2 - g=32 l=0.05 i=14 f=40
I: ----------------------------
I: 2 - g=0 l=nan i=0 f=0
I: 2 - g=16 l=0.03 i=1 f=2
I: 2 - g=32 l=0.05 i=13 f=37

[g=gain, l=luminosity, i=IR, f=full spectrum]
`

2591gain.txt