First and last seen event

[liyiy]

Changes

Return the first/last seen event with last_seen=true or first_seen=true for #4267

url example: ?properties=%7B%7D&event=watched_movie&orderBy=%5B"timestamp"%5D&last_seen=true

Checklist

• All querysets/queries filter by Organization, by Team, and by User
• Django backend tests
• Jest frontend tests
• Cypress end-to-end tests
• Migrations are safe to run at scale (e.g. PostHog Cloud) – present proof if not obvious
• Frontend/CSS is usable at 320px (iPhone SE) and decent at 360px (most phones)

[liyiy]

@EDsCODE I don't have clickhouse on my computer I might need some help with the clickhouse version of this even though it seems pretty straightforward 🤔

[mariusandra]

I wonder if it wouldn't be better in the long run if we add first_seen_at and last_seen_at fields to the event definition table, and update them in the plugin server on ingestion (with some throttling built in).

Saying this because it might be especially costly with Clickhouse to go back to the beginning of your billions of events, and find the first occurrence of something. Or it might not be, I don't know to be honest :).

[EDsCODE]

I wonder if it wouldn't be better in the long run if we add first_seen_at and last_seen_at fields to the event definition table, and update them in the plugin server on ingestion (with some throttling built in).

Was about to suggest this because the first seen timestamp will never change so we don't need to keep calculating it

Does the last seen timestamp need to be completely up to date?

[liyiy]

I wonder if it wouldn't be better in the long run if we add first_seen_at and last_seen_at fields to the event definition table, and update them in the plugin server on ingestion (with some throttling built in).

Was about to suggest this because the first seen timestamp will never change so we don't need to keep calculating it

Does the last seen timestamp need to be completely up to date?

Yes I think the last seen timestamp should also be completely up to date. This would still be updated on the plugin-server, correct? Seems like all I should do here is just add those fields to the event definition model instead of this then

[mariusandra]

There are a few complexities here:

• We have existing posthog installations that need these values populated. However setting these in a migration is a no-go since this can take approximately forever, and that will crash most upgrade deployments.
• The plugin server should write the last_seen_at fields. These writes shouldn't happen once per event, as at this volume we're talking a lot of postgres traffic. They should somehow be buffered and synced every so often.

Nothing some code won't fix though :)

[paolodamico]

Seems like this will change based on Eric's & Marius's feedback. Tag me if you need another review 😉

[liyiy]

Closing while I update