Feature: Add flag to waitanyinvoice to immediately return or not.

[jonasnick]

Modified by @ZmnSCPxj:

Add immediate argument to waitanyinvoice to return with an error if no invoice beyond the given lastpay_index has been paid. This allows the requested behavior of applications to batch-process invoices that came in while the application was not running (see text below the divide) or for a crontab process that regularly polls for new incoming payments, and so on.

---

Original text by @jonasnick

As far as I can see right now there's no way to get the latest waitanyinvoice index. It may be useful for applications to get the index on startup and then call waitanyinvoice only on future invoices. Or yet other applications (my case) want to catch up with the latest received invoice before starting the web interface.

[ZmnSCPxj]

If you design your application in that way, then if your application dies while lightningd lives on, then:

• Your application dies.
• lightningd receives a payment, incrementing the current waitanyinvoice index.
• Your application starts up and queries the latest waitanyinvoice index.
• Your user which managed to pay while your application was dead complains to your QA department.

Or yet other applications (my case) want to catch up with the latest received invoice before starting the web interface.

Rather than expose the latest waitanyinvoice index (and tempt application designers to the above misdesign), we can just add an immediate flag. If no invoice currently exists with the given lastpay_index and the immediate flag is set, return with a failure. Then your application startup can instead poll using the immediate flag being set and process those payments before starting the web interface, then once a call to waitanyinvoice immediate=true fails it continues starting up.

[jonasnick]

If you design your application in that way,

To be clear, this is not what I suggested in the OP. However, I agree that exposing the waitanyinvoice index might result in broken designs. I prefer the immediate flag idea because then the "implicit" waitanyinvoice index can not change out from under the application.

[rustyrussell]

So, in the long run I prefer to move to a generic "watch" subscribe-style interface, by providing one or more serial numbers on each "list"able object. e.g. watch invoice-change-serial 0 would return a new serial number when a new invoice is paid, and listinvoices would accept 'prev-invoice-paid-serial=' to get that invoice.

This gives several advantages:

1. We can use the same subscription mechanism to watch for creation or change in almost anything we want to support. Not just for invoices (the obvious case) but for channel creation or changes, peer changes, etc.
2. By also implementing 'limit=' on the list functions, we get pagination. You can ask to listinvoice prev-invoice-create-serial=0 limit=100 then take the last one and use that as the next prev-invoice-create-serial argument. You only need to process 100 at a time and not miss any.
3. By implementing a single-shot watch, we avoid flooding; it's up to the caller to then use list* to get the information on one or more changes.

But meanwhile, this change is useful.