Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature camera stream #22

Merged
merged 4 commits into from
Feb 10, 2025
Merged

Feature camera stream #22

merged 4 commits into from
Feb 10, 2025

Conversation

Jegp
Copy link
Contributor

@Jegp Jegp commented Jan 26, 2025

This PR fixes #1 by adding support for Inivation camera streaming.

NOTE: this also adds a flag enforce_monotonic_timestamps to the file encoders to allow timestamps arriving out-of-order. That is, unfortunately, common for camera packets (at least from Inivation).
This PR sets the flag to false by default. Partly because it'll be hard to configure this differently for different kinds of input, partly because I'm not sure I understand the need for the check. Are there specific readers/parsers that don't accept timestamps out of order? Care to comment @aMarcireau?

@aMarcireau
Copy link
Contributor

@Jegp This is a fantastic addition!

It might be nice to add the drivers dependency as a pip extra (so that users can use pip install faery[drivers] if they need camera support).

Format support for non-monotonic timestamps varies:

  • Event Stream (.es) does not support them at all (since it encodes relative time deltas with varying-bit-size unsigned integers).
  • Prophesee formats (DAT and EVT) technically support them (at the binary level) but they use limited-precision timestamps that overflow often. Our decoder assumes monotonic timestamps to detect these overflows.
  • Aedat and CSV allow them.

Besides file formats, most algorithms implicitly assume that timestamps are monotonic. Not doing so makes quite a few operations much more complicated (time deltas used in time surfaces can be negative and break look-up tables, tracking algorithms need to sort the events since they rely on forward Bayesian models, regularization needs to buffer more data and sort non-monotonic events to ensure consistency, and so on).

Overall, I am tempted to treat non-monotonic timestamps as bug that should be fixed upstream in the Inivation drivers (they most likely indicate some sort of buffer corruption). As a workaround, we could either sort the events by timestamp (potentially bad for latency since one needs to store a copy of the current buffer before dispatching it, in case events in a future buffer have smaller timestamps) or "freeze time" until it increases again (the timestamp sequence [0, 3, 10, 8, 9, 12, 15] would become [0, 3, 10, 10, 10, 12, 15]). Note that I am readily doing the later in most decoders.

One exception to monotonicity are multiplexed streams (for instance events + frames or events + triggers). Whilst guaranteed monotonicity would be nice to have for such streams (to implement hybrid frame/events algorithms, for instance), it can be hard to implement without sorting data. It is also somewhat ill-defined since frames have multiple timestamps (frame trigger, integration start, integration end, and readout end). This is even worse in the case of a rolling shutter (where technically one needs one timestamp per row to properly describe the frame). At any rate, this is beyond the current scope of Faery (which focuses on events) so I think that the aedat decoder and inivation driver should dump the frame data as it arrives from the source and only check the monotonicity of events.

@Jegp
Copy link
Contributor Author

Jegp commented Feb 3, 2025

Thank you for your detailed comments @aMarcireau! I see that it's a good idea to ensure the ordering. If you agree to retain the flag, I definitely think it should be true by default.

I like the idea of "freeze time", but isn't that costly? Particularly because it can't really be paralleled.

Alternatively, wouldn't it be faster to do something like this?

events = ... # Assume an array of timestamps
min_timestamp = events[0]
events = min_timestamp + np.clip(events - min_timestamp, 0)

I btw agree that this should happen in the event drivers and definitely not be a part of Faery. For now. It may be nice to have a filter in the future.
For this PR, I'll revert the flag to true and push the changes, if you agree. I'll also remove the feature from .es, .dat, and .evt to avoid ambiguous timestamps when overflowing.

@Jegp Jegp merged commit c9b4be7 into main Feb 10, 2025
90 checks passed
@Jegp Jegp deleted the feature-stream branch February 10, 2025 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Inivation drivers
2 participants