Possibility to set last message ID on first subscribe

[FZambia]

This pull request contains code to provide last message ID manually on first subscribe to channel to restore missed messages on page open. This way it's possible to save last message ID in localStorage and then restore from that position. I.e. sth like:

var lastMessageID = localStorage.getItem("lastMessageID");

var sub = centrifuge.subscribe(channel, handleMessage, lastMessageID);

function handleMessage(message) {
    localStorage.setItem("lastMessageID", message.uid);
    ...
}

Not sure that its needed though... Because on first subscribe history should be loaded from backend. But in theory this is interesting. Open for discussion.

[Inpassor]

I think this is a very necessary feature.
I definitely need the user to download not all the available history from the backend, but only what he missed.

[FZambia]

This feature can theoretically make users misuse Centrifugo - i.e. only relying on Centrifugo to recover missed messages - and this is in general bad practice because Centrifugo does not have any guarantees about that - only best effort. On the other hand some apps that do not require much performance and reliability can benefit. @Inpassor thanks for feedback on this, hope we collect more opinions with time.

[Inpassor]

In my opinion, in any case, you rely on a centrifuge to restore messages. But there will be a choice - to restore all available messages, or just missed. Don't care about guarantees, at least something will be recovered.

[gazugafan]

Sorry to drag this back to life, but coming from NCHAN this seems like a really useful missing feature. The way I used it in NCHAN was like...

1. At the server, on page request, retrieve the current state from the database--including the last message ID
2. Send this initial state to the browser. Eventually the browser will load this page with the initial state, and then subscribe to a channel for state updates. It could take several seconds between retrieving the current state from the database and subscribing to the channel--so some updates could be missed. BUT! When subscribing to the channel, we also send the last message ID that we had retrieved. As long as the missed messages are still in NCHAN's history, we're guaranteed to get exactly the right update messages that apply after the initial load.

... and we're only talking a few seconds of possible missed messages while the page loads, so of course we'll still have them with any reasonable history limits configured.

Hope this can get merged into v2 at some point! Until then, any suggestions on accomplishing something like this for now?

[FZambia]

@gazugafan hello!

When subscribing to the channel, we also send the last message ID that we had retrieved. As long as the missed messages are still in NCHAN's history, we're guaranteed to get exactly the right update messages that apply after the initial load.

Does this mean that message IDs in your database and NCHAN cache match? Centrifugo act more like Kafka when incremental message offset in channel (may call it ID I suppose) assigned internally by Centrifugo itself at the moment of publication. Though it's pretty possible to get a last offset for a channel in Centrifugo and will be available as part of API soon in Centrifugo v3.

[gazugafan]

Yes, the message ID in the NCHAN cache ends up matching what's stored in the application database. This is because NCHAN returns the new message ID when you publish a message. So, everytime we publish a message from the backend, we get the message ID and immediately save it back to the database.

Though it's pretty possible to get a last offset for a channel in Centrifugo and will be available as part of API soon in Centrifugo v3.

This seems to be the missing piece of the puzzle. There's currently no way to get the last offset for a channel after publishing?

[FZambia]

NCHAN returns the new message ID when you publish a message. So, everytime we publish a message from the backend, we get the message ID and immediately save it back to the database.

It's also possible to return a published offset in Centrifugo (looks like one more thing worth adding to v3).

Some caveats to be aware of:

1. In Centrifugo offset incremented per-channel (not globally). So inside one channel example1 offset grows like 1, 2, 3, 4 etc. In another channel example2 you can also have same incremented offsets. Is it OK for your use case that offset is incremental per channel?
2. In Centrifugo position is determined using 2 fields actually - offset I was talked about and epoch. Epoch - is a string identifier for a current channel history stream generation (since history stream can be lost in case of memory engine for example after server restart, or if you are not persisting data in Redis while using Redis engine etc). It's possible to omit epoch when recovering automatically or manually and rely only on offset - but in this case there is a small possibility you will be recovering from the newly created stream and thus the behavior can be undefined (I think acceptable for many applications though since correct state saved in main database anyway).

There's currently no way to get the last offset for a channel after publishing?

Currently no. If you feel that this all can be a good fit for your app could you open an issue in Centrifugo? Maybe this is possible to introduce in terms of Centrifugo v2 - not sure which version to aim at the moment. But this all seems very reasonable to have.

[gazugafan]

Is it OK for your use case that offset is incremental per channel?

This is preferable!

In Centrifugo position is determined using 2 fields actually

Makes sense!

If you feel that this all can be a good fit for your app could you open an issue in Centrifugo?

Will do, thanks so much! ❤

[FZambia]

Closing in favour of newly created issue. Code here is for v1 so should be rewritten anyway.