SR300 multible camera couses freezes in uvc-wmf.h

[Wollimayer]

Required Info
Camera Model	SR300
Firmware Version	3.15.0.0
Operating System & Version	Windows 10
Build System	VS2014

Currently I'm experimenting with 5 RS300 Cameras, each of them connected to it's own usb controler.
It's working fine for most situations, but does couse a program freeze at some points ( sometimes, but the more cameras are connected, the more often they occure )

Setting parameters via

void set_control(device & device, const extension_unit & xu, uint8_t ctrl, void *data, int len) (uvc-wmf.h)

can freeze the whole program without a chance of recovering.

This can be reproduced by attaching more than one camera and simple keep activating and deactivating the laser of one of the camera.

Once one of the RS300 did crash the context is no longer able to be created ( even after restarting the program)
It will freeze at

WinUsb_WritePipe(usb_interface_handle, endpoint, (PUCHAR)buffer, bufferLength, &lengthWritten, NULL); (uvc-wmf.cpp)

sometimes during the usb_synchronous_write but also at usb_synchronous_read .

These problems do not occure using the F200 cameras ( tried it with 7 F200 cameras without any problems)

[ddiakopoulos]

Could be firmware issue. Can you comment, @dorodnic?

[Wollimayer]

Is there a more stable Version ? With the RS DepthManager installer, 3.15.0.0 is the most current Firmeware version I could find.
@ddiakopoulos

[dorodnic]

3.15.0.0 is the most up-to-date FW. I will try to reproduce the issue and get back to you.

[dorodnic]

Could you please send the exact program you are using to reproduce? Also, what is the average time to reproduction with 5 cameras? I made a demo that switches laser power of the first camera between 0 and 1 in a loop and ran it overnight with 6 connected SR300 cameras, but it is still running fine.
I have a hunch about what could be causing this but I need to see it to be sure.

[Wollimayer]

@dorodnic usualy it does occure within a minute here. I have written a small library to handle the cameras. I will extract the code into one program in order to make this reproducable without all the overhead. Just a minute.

In the meantime, can you provide your testcode, so I can be sure, that it is not just a problem with one of the cameras?

UPDATE:

Here is the code. I tried it with 3 cameras now and it seemed to work quite ok. Now i attached 6 cameras and it did crash after about a minute.
After it crashed once, it was not able to create the context anymore and needed a complete reboot.

`
rs::context* ctx;

int main(int argumentNumber, char* arguments[])
{
bool running = true;
rs_error* _error;
rs_log_to_console(rs_log_severity::RS_LOG_SEVERITY_DEBUG, &_error);

ctx = new rs::context();

for (int cameraIndexInit = 0; cameraIndexInit < ctx->get_device_count(); ++cameraIndexInit) {

    auto & dev = *(ctx->get_device(cameraIndexInit));
    dev.enable_stream(rs::stream::depth, rs::preset::best_quality);
    dev.enable_stream(rs::stream::color, rs::preset::best_quality);
    dev.set_option(rs::option::f200_laser_power, 0);
    dev.set_option(rs::option::f200_motion_range, 80);
    dev.set_option(rs::option::f200_accuracy, 1);
    dev.set_option(rs::option::f200_filter_option, 6);
    dev.start();
}


while (running) {

    for (int cameraIndex = 0; cameraIndex < ctx->get_device_count(); ++cameraIndex) {
        auto & dev = *(ctx->get_device(cameraIndex));


        auto depthIntrinsic = dev.get_stream_intrinsics(rs::stream::depth);

        dev.set_option(rs::option::f200_laser_power, 15);
        for (int wait = 0; wait < 5; ++wait) dev.wait_for_frames();

        const uint16_t * depth = reinterpret_cast<const uint16_t *>(dev.get_frame_data(rs::stream::depth));
        cv::Mat_<ushort> depthImage = cv::Mat_<ushort>(depthIntrinsic.height, depthIntrinsic.width, const_cast<uint16_t *>(depth), depthIntrinsic.width * 2).clone();
        dev.set_option(rs::option::f200_laser_power, 0);

        //cv::imshow("Depth Image : " + std::to_string(cameraIndex + 1), depthImage);
        //auto key = cv::waitKey(10);
        //if (key != -1)
        //  running = false;
    }

}

for (int cameraIndexStop = 0; cameraIndexStop < ctx->get_device_count(); ++cameraIndexStop) {
    auto & dev = *(ctx->get_device(cameraIndexStop));
    dev.stop();
    dev.disable_stream(rs::stream::depth);
    dev.disable_stream(rs::stream::color);

}

return EXIT_SUCCESS;

}`

[dorodnic]

@Wollimayer thanks, I was doing something slightly different. You wrote "This can be reproduced by attaching more than one camera and simple keep activating and deactivating the laser of one of the camera." so I was toggling laser of just the first camera. Now the scenario is clear, I'll experiment with it once I get to the lab.
One question - you mentioned that you need to reboot in order to recover. Does the problem persists even after you disconnect and reconnect all of the cameras?
Just to make sure we don't get any interference from the default RealSense driver, please stop RealSenseDCMSR300 service from the Task Manager.

[Wollimayer]

@dorodnic : reconnecting the Camera manually will also solve the problem.
I closed the depth manager now, but the error does still occur.

[dorodnic]

@Wollimayer I ran the exact program you've submitted with 6 attached cameras for over an hour but still no reproduction.
At this point it is reasonable to assume there might be a problem with one of the cameras or with the host. Since both replacement cameras and a PC capable of handling 6 USB3 devices are hard to come by, lets assume it is not a problem with the setup and try to exhaust all other options first.
In that case there are still couple of things to be checked:

1. Could you please collect a dump file (you can do it from the Task Manager) of the sample when it is first getting stuck on set_control and also when you try to reopen it (stuck on WinUsb_WritePipe). If you could upload those together with your realsense-d.pdb it might give us some extra information.
2. I know you wrote Windows 10, but if by chance you are using an older Windows version, there was a very similar bug in usbvideo.sys that was patched in KB3075872. (You would also need KB2919355)
3. Could you add the following lines:
   WinUsb_ResetPipe(usb_interface_handle, 0x81); WinUsb_ResetPipe(usb_interface_handle, 0x01);
   to uvc-wmf.cpp, right after:
   if(!WinUsb_Initialize(usb_file_handle, &usb_interface_handle)) { LOG_ERROR("Last Error: " << GetLastError()); throw std::runtime_error("could not initialize winusb"); }
   This will not stop the problem from happening, but it might help with recovery (no need to reset). This is my best guess without seeing the issue, if it changes anything please let me know.
4. The exact setup I was able to find for the test was:
   Windows 8.1 64-bit (with the relevant patches)
   Intel(R) Core(TM) i5-4670, 16GB
   6 x SR300 PRQ 3.10.10.0
   2 x Renesas USB 3.0 eXtensible Host Controller - 0100 (Microsoft)
   Perhaps this could give you additional direction for further tests.

[Wollimayer]

Point 1 :
Here is the dmp file ( had to zip it )
http://www.file-upload.net/download-11559625/github-issue-129.zip.html
And the exe file together with the pdb.
github-issue-129-exe-and-pdb.zip

The dmp was captured when the program crashed on setting up the context.I try to get one of the crash during set_control .

UPDATE:
This dmp was captured when it did crash on set Parameter
http://www.file-upload.net/download-11559838/github-issue-129setParameter.zip.html

Point 2 :
I'm using the windows 10 enterprise version with an i5-4460 and 8Gb of ram.
However i did upgrade to windows 10 as it was running more stable with the f200 ( probabily becouse of the mentionend fixed not installed with my version of windows 8.1)
The crashe occured with 3 SR300 3.15.0.0 ( not 3.10.0.0 )

Point 3 :
I will change this and try to reproduce the error.

[ddiakopoulos]

@Wollimayer any updates on your end?

[Wollimayer]

Point 3 :
It seems to improve the systems ability to recover after an error. But the problem does still exists ( as expected) .

I also did some test on a simple capture program that will :

• start a single camera
• capture some images
• stop the camera
• wait for 500 ms
  Repeat for each camera.

This seems to improve the stability of the whole process but is really slow and does produces errors an rare occasions.

@dorodnic Are the dmp files of any help for you ?

[dorodnic]

Unfortunately, not enough. The files didn't capture original first-chance exception but rather the error as it was being re-thrown from librealsense API wrapper. If you are using procdump you can capture first-chance exceptions by running: procdump -e 1 process_name.exe dump_name.dmp
Do you see in the log something along the lines of: IKsControl::KsProperty(...) returned 0x???
Anything else suspicious in the logs?
Also, in your original submission you mentioned that the program would freeze on set_control. Could you provide dump file or full call-stacks of the running threads when it does that?

[Wollimayer]

@dorodnic I will try to reproduce the errors using procdump and upload it as soon as possible.

Is there a general guidline for using multible cameras in order to keep the system stable ?
E.g. are there any operations that need extra sleep time after execution or are known to couse problems using multicamera systems ?