multi camera HW sync demo

[bloyl]

Is the demo application used in this white paper (https://realsense.intel.com/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper_rev1.1.pdf)

available as part of the sdk? maybe its somewhere else?
Thanks
Luke

[MartyG-RealSense]

The multi cam paper was originally only available to Intel's customers who have a Non Disclosure Agreement (NDA). So the app is probably only available to NDA customers.

#1789

[bloyl]

I see.

They just emailed out a blog entry (https://realsense.intel.com/multiple-depth-cameras-setup-guide/?cid=em-elq-38512&utm_source=elq&utm_medium=email&utm_campaign=38512&elq_cid=3078062) that links to the white paper and here, so I image this question is going to come up a fair amount soon :)

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
hi @bloyl,

i don't think we have this app release, but let me check on this.

[zuru]

After reading the white paper again, it seems like HW sync is only supported when all the devices are connected to the same PC? Is that indeed the case?

We have created a distributed system where each device is connected to an Intel NUC and are trying to HW sync them but without any success up to now (see #2148). The reason seems to be that when setting one device as slave, it keeps acquiring frames even when no HW signal is triggered. Could it be as mentioned above that this happens because the master and slave are not connected on the same PC? If yes, can this somehow be circumvented? Because connecting a larger number of devices will not be possible without networking involved.

[MartyG-RealSense]

The setup examples given in the multiple camera paper (all cams connected to a hub, or all cams connected directly to USB ports) do seem to indicate a preference for all cameras being connected to the same PC.

As the paper suggests, the ideal setup is to use multiple individual USB ports instead of a hub in order to avoid more than one device being shared on the same USB controller. The paper correctly points out though that some PC devices take the shortcut of pairing individual ports on a shared internal hub instead of each having their own USB controller.

I would suppose that if a slave camera is connected to a different PC to the master then the PC that the master was connected to would not know of the slave camera's existence as a USB device.

USB supports the chaining of up to 127 devices on the same PC, in theory. In practice, you could not actually connect 127 cameras because you would need to connect together a pile of USB hubs, and each of those hubs claims some of the 127 addresses even before a device is attached to the hub. So you could probably chain 50+ cameras together on the same PC, but in practice the performance would probably be so slow as to be unusable.

The ideal in mass-camera setups would be to only have several cameras at a time active, and cycle through the cameras sequentially, turning them on and off in batches as capture snapshots are taken until the cycle gets back to the first cameras again and begins the cycle anew. On the old RealSense camera models, this was called Enumeration. I'm not sure this would be feasible for camera syncing though.

It might become more feasible on modern, faster USB standards like USB 3.1, but there would still be the issue of the poorer performance on a hub's shared USB controllers compared to connecting each to its own individual USB controller.

[zuru]

Thanks for the detailed answer MartyG.

• Therefore, unfortunately we cannot have HW sync between networked devices (i.e. devices plugged in on different PCs and network connected via Ethernet and hardware connected with a syncing cable) currently.
• Is there a possibility of this being supported sooner or later? Because for live multi-view streaming, as you said it is impractical - if not infeasible - to use a single PC for >4 Intel RealSense devices.
• Concluding, our only choice for HW synced live multi-view capture currently are USB extension/repeater cables that are connected on separate USB controllers (e.g. adding PCIe USB cards) on the same PC, and hardware connected between themselves using a signalling cable.

If you could confirm my understanding as mentioned above, it would be great.

Many thanks !

[MartyG-RealSense]

I an not an Intel employee and so have no knowledge of Intel's RealSense development plans - Dorodnic or RealSense-Customer-Engineering will be better able to comment on that.

The cable supplier Newnex offers the ability to use 400 Series cameras with fiber optic extension up to a range of 100 meters.

https://itm-components.co.uk/blogs/news/newnex-usb-extension-for-the-intel-realsense-d400-series-camera

There is such a thing as USB Over Network / USB Over Ethernet. Though if all cameras have to be physically connected to each other in the same location with sync cabling, this would suggest that even if you could mass-connect to RealSense cameras on numerous different machines, you could gather data from them but they would not hardware-sync.

However, the sync cable can be up to 10 feet long, so in theory, you could put multiple PCs together in the same room using USB Over Ethernet / Network, with all the cameras within a 10 m distance of that batch of machines. An efficient, tidy way to do this might be a server-rack of PCs, or at least some kind of shelving.

[zuru]

Sorry that RealSense in your username confused me ;)

We already have a tidy setup with 4 Intel NUCs mounted on tripods and each one with a D415. However, as mentioned we connect them to the central workstation via Ethernet networking and transmit the steams the central PC for processing. And, our hypothesis (partly confirmed by you), is that we cannot achieve HW sync (we have them connected to a master device via daisy chain) as they are not co-located (in the USB sense) on the same central PC. Nonetheless, we have measured the syncing cable signal and it is correctly reaching to all destinations (slaves).

It would be great if we could a confirmation from the RealSense development or technical support team.

[MartyG-RealSense]

'MartyG' was already taken. Since my name is already established over on the Intel RealSense support forum where I also do support (there's two forums, this GitHub development team one and the Intel Support one), it made sense to use the 'RealSense' addition in the name to establish (1) who I am, and (b) my specialization. Apologies for any confusion caused!

I'm sure the Intel guys will be able to give you excellent further information. Good luck!

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
hi @bloyl,

no plan to release multi cam HW sync app or example, but plan to integrate into realsense viewer.

hi @zuru,

would like to know if you are NDA customer to intel? if so, you can file this question to Zendesk for direct support.

[zuru]

Unfortunately we are not NDA Intel customers. Is there any other way to receive feedback about this?

[agrunnet]

The sync cable and the commands for whether a camera is space or master is all that is needed to enable HW sync. You can do this from different PCs. But of course sync cable DOES need to connect cameras. You can make a wireless version yourself if you have a longer range as it is simply a sync pulse that needs to be shared.
However you will need to properly collect frames together using the frame counters (which will be offset for each camera but in sync).

When you set a camera to slave it will still acquire frames at the right frame rate even if there is no sync input signal. It tries to sync but if there is no signal that comes in at the right time it gives up and triggers itself.

[zuru]

As mentioned in #2121 and #2148 we do have sync cables connecting the cameras, we are monitoring the pulses and they are indeed arriving. However, acquiring color and depth frames robustly is an issue, and in the 4 camera case, it will rarely be accomplished. In addition, after testing with two cameras facing a screen with a millisecond clock, we did not observe HW synchronization level timings and the frame numbers were unrelated. We can provide more detailed proof (e.g. videos, code, etc.) if required.

Regarding the last point, there is a contradiction. If the slave (which starts acquiring frames at a different time than the master in the distributed case) starts grabbing frames even when not signaled, then there will be no frame number correspondences. Perhaps this can be guaranteed in the centralized case (having the master and slave connected on the same computer), starting the master and then the slave and acquiring frames using the blocking or polling calls, and not asynchronously (which is the best performing way to go about it).

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
hi Zuru,

yes i think if you can have a short video can help me to understand your issue more.
i just did a simple test to have 2xD415 HW synced and connect to two PC (1 D415 to 1 PC), looking at a millisecond clock and stop both D415 at the same time, i do capture same millisecond shown on the clock, seems you have good result with this too. but you mentioned to acquiring color and depth frame robustly, this may be relative to the resolution, so need you to provide the resolution you are testing.
and you say you test 2 cameras facing to clock and frame numbers were unrelated, are they connect to same PC, if yes, this is not the case i see here, i can have exactly frame number from both camera.

[vladsterz]

@RealSense-Customer-Engineering
It would be really helpful if you could share some sample code, so that people could figure out by themselves if they're doing something wrong.

[MartyG-RealSense]

Hi vladsterz, if you are seeking some sample scripts for accessing and controlling devices in a multi-camera setup, there are a number of sample scripts posted on the discussion linked to below (read downward from the point that is linked to).

#2219 (comment)

[vladsterz]

@MartyG-RealSense thank you for your response, but I am interested in the case of a distributed system (i.e. 2 cameras and 2 machines) rather than a centralized one (i.e. 2 cameras on the same machine).

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
hi @vladsterz,

we plan to have some sample code for multi-cam, but so far still suggest to start validation with Realsense viewer first, so we can make sure all HW setting is good then continue to debug.
for distributed system, per my test with realsense viewer, it's working good, so i think you just need to implement communication between systems then the script MartyG mentioned should work.

[supriyasathya]

Hi,

1. Can D415 and D435 be connected in multi-cam configuration?
2. Is there any documentation of multi-camera calibration? I am trying to use two-cameras connected to one PC in an inward configuration. (I read the section on multi-camera support in the white paper on dynamic calibration, but there is only mention of calibration of one camera at a time, and not the system)

Thanks

[MartyG-RealSense]

1. Intel Support member Sahira says that D435 multi camera sync support has been added in the latest 5.10.3 firmware. If that's so, I guess either the D415 or the D435 could act as the master in the master-slave pairing of the two cameras.

https://communities.intel.com/message/562088#562088

2. Intel offers a camera calibration system for $1500 USD in their online Click store that supports multiple devices.

https://click.intel.com/realsense-d400-camera-oem-calibration.html

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @bloyl,

Still need any support for this topic?