Timestamp for time received time (backend_time?), how to access source

[timprepscius]

Hey there again,

For whatever reason, I need to access the time generator which is used to mark a frame's arrival. Is this considered the backend_time? I need to ask this same timestamp generator to generate a new timestamp at my choosing. I use this to quantify error in your system and measure clock skews.

Could you tell me a line number for where the timestamp is generated?

Also, can you confirm that you are marking receive time when the first packet from the usb has arrived, and not when the last packet arrives?

Thanks

[MartyG-RealSense]

I believe that the instruction frame.get_timestamp() returns the timestamp of the frame, in milliseconds since the device was started.

The discussion in the link below may be useful. In it, @ev-mp provides explanations of the types of timestamp and provides a reply to someone who was seeking to query the timestamp system to correlate it with their host system.

#2188

I would think (not certain) that timestamps would be handled by the file ds5-timestamp.cpp:

https://github.com/IntelRealSense/librealsense/blob/master/src/ds5/ds5-timestamp.cpp

[dorodnic]

Hi @timprepscius
Do you know which backend you are using? (Windows = Media Foundation, Linux = V4L2, rest and when explicitly enabled = RSUSB)
For V4L2, backend timestamp is generated using system monotonic clock, so clock_gettime(CLOCK_MONOTONIC, &mono_time);
More generally - I think rs2_get_time is what you are looking for.

[timprepscius]

Ok, I'm reading through your code which pertains to time. I'm looking where the frame_object is constructed.

in v4l2 I see:
timestamp = monotonic_to_realtime(timestamp);

in mf-uvc I see:
frame_object f{ current_length, metadata_size, byte_buffer, metadata, monotonic_to_realtime(llTimestamp/10000.f) };

I haven't found the rsusb version yet.

Why are you using realtime instead of monotonic? I need to change you to monotonic. Will I be breaking anything?

I wish that instead of monotonic_to_realtime you had:

1. kept monotonic
2. added an additional realtime stamp (I don't need it, but someone might)
3. provided a standard rs2_query_monotonic_clock()

your rs2_get_time goes to environment time which goes to a realtime clock.

This is what I think I'm seeing.
Is this about right?

I am going to experiment with changing:

1. monotonic_to_realtime to just return
2. modify the os_time_service to use the chrono::steady_clock

[dorodnic]

Backend timestamp is backend (OS) specific. I don't think there is specific reason for each implementation, but we also do not guaranty backend timestamp API will be consistent between backends. I'm open to changing how backend timestamp is generated. @ev-mp?

[ev-mp]

Having backend and host time use the same EPOCH is useful to profile kernel->user space transition bottlenecks.
Changing the backend_ts to monotonic is an API-breaking change. On a technical level there are also certain issues with its implementation on Linux. Also as @dorodnic noted the implementation is OS-specific.
But I think it would be OK to consider adding monotonic_ts to the frame metadata as an additional field if there is a demand.

W.r.t rs2_query_monotonic_clock:

• In general Librealsense should not be used as a wrapper to OS functionality, the same as with open/close files.
• We already expose rs2_get_time which might be extended to rs2_get_time(Realtime/Monotonic). This however, will require a certain amount of work especially with playback device.

So I think that introducing this attribute via metadata route is quite straight-forward. Extending the API is more complicated, imho.

[timprepscius]

Maybe I can provide a little bit more detail of my use case:

This originated with a strategy to synchronize kinectsV2 on multiple machines.
The MS library for kinect didn't provide any good timestamps. I modified the freenect library to provide a received timestamp (not wall clock) when the first packet arrived.

I have put in logging of when your (realsense) os_time_service is queried, and what timestamp arrives on a frame, and it seems as though the frame timestamp is in fact the first timestamp, which is good.

Both the Kinect and the Realsense provide a timestamp originating in their device. This is probably driven by the usb clock driving the polling (atleast the kinect seemed to be). The USB clock has noticeable skew from the PC clock. And each computer I buy has different changing skew rates. This meant that Kinect Device timestamp was not just an "offset" of the PC timestamp. And of course, each PC timestamp was not just an offset of another PC timestamp.

You all already know all of this though. I found it really surprising when I first started figuring out how much skew there really was.

To overcome all these skew problems, I make time_received be a monotonic time. I then maintain a (sample windowed) linear-regression between the [Realsense/Kinect]DeviceTime -> the ReceivedTime(LibraryHostSteadyClock).
Further more I establish a master-time (through exactly the mechanism you just imagined I use), and I maintain a (sample windowed) linear-regression between HostTime(LibraryHostSteadyClock) and MasterTime.

Given these equations, I can, mostly, given a [Realsense/Kinect]DeviceTime on any computer, provide a master clock time, with specificity better than 1/90th of a second.

In my situation, I would like for your library to:

1. Provide a "realsense_steady_clock()" which is of highest resolution, and which I can access at arbitrary times (in order to provide data for the linear-regression to MasterTime).
2. Use realsense_steady_clock() to timestamp a frame coming in.
3. I would love to get access to those rtt pings you are sending to the realsense to measure time. I would use a stddev to calculate the mean.

I have not done #3, because I have so many other things to do at the moment.
However the changes (kludges) for #1/#2 are:

timprepscius@65d9f3c

Done babbling- hope this is of use.

[MartyG-RealSense]

@timprepscius As a follow-up, I have added the 'Enhancement' (feature request) label to this case to reflect the 3 feature requests that you made in your last comment above, Thanks!

[MartyG-RealSense]

Hi @timprepscius As changes to rs-frame.hpp **were merged into the SDK in late May, I will close this case now. Thanks for your wish-list!

[WasabiFan]

Despite this issue being closed, this issue has not otherwise been resolved, as far as I can tell. The OP above showed their own custom patches to the library to add some of the missing features. This does not mean the feature has been implemented, or is available to others.

I suggest that either this issue is reopened or a new one is created, to track the availability of a monotonic timestamp on each frame and getter to compare against.