RS2_OPTION_EMITTER_ON_OFF not always working

[rolandgvc]

Required Info
Camera Model	D400
Firmware Version	05.12.12.100
Operating System & Version	Ubuntu 20.04
Kernel Version (Linux Only)	5.4.0-71-generic
Platform	PC
SDK Version	2.44
Language	C++
Segment	Robot

Issue Description

Having the depth sensor alternate on/off between frames is not always deterministic. Take for example the following code:

 rs2::pipeline rs_pipe;
  rs2::config cfg;
  cfg.enable_stream(RS2_STREAM_DEPTH, 0, RS2_FORMAT_Z16, 30);
  cfg.enable_stream(RS2_STREAM_INFRARED, 1, 1280, 720, RS2_FORMAT_Y8, 30);
  cfg.enable_stream(RS2_STREAM_INFRARED, 2, 1280, 720, RS2_FORMAT_Y8, 30);
  cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_RGB8, 30);
  rs2::pipeline_profile rs_profile = rs_pipe.start(cfg);

  // sensor options
  auto sensor = rs_profile.get_device().query_sensors()[0];
  sensor.set_option(RS2_OPTION_EMITTER_ENABLED, 1);
  sensor.set_option(RS2_OPTION_EMITTER_ON_OFF, 1);
  sensor.set_option(RS2_OPTION_FRAMES_QUEUE_SIZE, 1);
  
  while (true) {
    rs_pipe.wait_for_frames();
    rs2::depth_frame df = fs.get_depth_frame();
    if (df) {
      if (df.get_frame_metadata(RS2_FRAME_METADATA_FRAME_LASER_POWER_MODE)) {
        continue;
      }
    }
    const auto rs_left = fs.get_infrared_frame(1);
    display(rs_left);
  }

You would assume all the frames where the dot projections are visible would be skipped. Many times it happens that the RS2_FRAME_METADATA_FRAME_LASER_POWER_MODE flag will be a false negative.This can happen either at runtime (when none of the IR frames are hidden) or during operation (when some of the IR frames appear for a short periods of time).

Any ideas why this is happening?

[MartyG-RealSense]

Hi @rolandgvc May I first confirm which model of RealSense camera you are using, please? The per-frame emitter on-off mode was designed for camera models with a fast global shutter. This would include D430, D435, D435i and D455.

The link below, which provides information about this mode, contains a rapidly flashing image.

#3066

[rolandgvc]

Sorry, it's D455. #3066 this is the behaviour I am getting as well, just that it's not consistent.

[MartyG-RealSense]

Thanks very much for the clarification about the camera model.

It would be unnecessary to set RS2_OPTION_FRAMES_QUEUE_SIZE to 1 because '1' is the default value anyway. If you are using both depth and color streams then it may be worth setting it to '2' though, as described in the 'Latency Vs Performance' section of Intel's Frame Buffering Management documentation.

https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0

Does setting the frame queue size to '2' provide any improvement in performance, please?

[rolandgvc]

Unfortunately not. Could it be from the kernel and realsense version?

[MartyG-RealSense]

I do understand your situation, as I have handled a couple of past cases where the dot pattern has been visible on the "off" frame when using RS2_OPTION_EMITTER_ON_OFF, so only capturing frames where the emitter status was off did not prevent the chance of such frames having dots. My understanding is that it was related to the environmental lighting conditions at the time of capturing the image, and that the visibility could change at different times of day depending on the natural lighting level.

[rolandgvc]

Here is an example of the behaviour i'm encountering.
This is how the image looks like when RS2_OPTION_EMITTER_ON_OFF acts as expected.

This is what I'm encountering in 3/10 runs. In these frames, IR is ON at full power although df.get_frame_metadata(RS2_FRAME_METADATA_FRAME_LASER_POWER_MODE) is false.

[MartyG-RealSense]

I am not aware of a solution to this issue when using RS2_OPTION_EMITTER_ON_OFF, having studied the problem in the past cases where I have encountered this phenomenon.

If the scene that the camera is being used in is well lit then an alternative may be to turn the projector off completely. The 400 Series cameras can alternatively use ambient light in a scene instead of the dot pattern to analyze surfaces for depth information when there is a good level of light present in the location that the camera is being used in.

[MartyG-RealSense]

Hi @rolandgvc Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Thanks very much @rolandgvc for the update!