Speed up enabling buttons to open form in Enketo

[lognaturel]

Currently it takes several seconds (20+) for web form buttons (Preview and +New) to be enabled in these cases:

• New draft is created
• Existing draft is updated
• Form is published for the first time

The following do NOT take time to be enabled:

• New draft is published to an existing form

The "existing draft is updated" case is the one that adds a lot of friction. I believe it is identical to the "new draft is published" case. That is, both have a stable Enketo ID that can be reused. The Enketo ID is determined by Enketo Express using the host URL and the form_id. The host URL stays stable within a draft editing session (because the draft token is stable). I believe in the "existing draft is updated" case we should be able to keep the buttons enabled like we do for the "new draft is published" case.

It would also be valuable to speed up the "new draft is created" and "form is published for the first time" cases. My current understanding of why this is slow is that:

• Enketo's response which includes the formID is processed by a worker
• Frontend polls for the enketoID every 3s

Enketo's response is going to be near-instant in almost all cases, especially using EE 6.

I understand the desire not to couple form creation too tightly with preview generation. Could we do something like have backend block on Enketo's response with an aggressive timeout (500ms or 1s) and fall back to the existing async model?

[matthew-white]

I'm going to try to address the "existing draft is updated" case by copying-forward the Enketo ID from draft to draft. We already copy-forward the draft token and form attachments from draft to draft.

For the "new draft is created" case and maybe also the "form is published for the first time" case, I'm planning to implement the following:

Could we do something like have backend block on Enketo's response with an aggressive timeout (500ms or 1s) and fall back to the existing async model?

With these changes, it might not be necessary to change the worker or the Frontend polling. We have getodk/central-backend#533 to think about adjusting the worker. We might also tune the Frontend polling, in part because that polling can be noisy during local development.

[matthew-white]

It turns out that the "new draft is created" case actually becomes two subcases:

• new draft is created for a new form
• new draft is created for an existing form

In the former case, Forms.createNew() generates the draft token and creates the draft. In the latter, it's Forms.createVersion().

I made progress on this issue in getodk/central-backend#782. I've addressed these two cases:

• existing draft is updated
• new draft is created for an existing form

I'm planning to continue working on this issue during .3 in order to address the remaining cases:

• new draft is created for a new form
• form is published for the first time

Once those changes are made, part of me thinks that we could remove the Frontend polling altogether. If the vast majority of Enketo IDs can be generated during the request rather than by a worker, it would be great if we could remove this source of complexity in Frontend.

@lognaturel, I know you've seen the latest behavior in action on the QA server. Do you think that you tried out both of the cases that I addressed in getodk/central-backend#782? I'm thinking that as long as you've taken a look, we probably don't need the QA team to also verify.

[lognaturel]

Sorry I missed that tag! You probably saw my comment in Slack saying it's changed my life. 🌟 Looking forward to the next couple of cases!

[matthew-white]

@getodk/testers, this issue is ready to be verified, on either staging or the QA server. Previously, it was common to see a message like "Preview has not finished processing for this Form." However, now Enketo should be enabled right away. When a new form is created, or a new draft is created, or a draft is published, preview and submission should be available in Enketo right away.

[dbemke]

There is a descending order of submission order numbers applied after using the "Refresh" button in Draft submission table and Submissions table. The issue occurs both with submission via Enketo and Collect. (The issue doesn’t occur on sandbox)

Steps to reproduce:

• Upload a draft version and go to the "Testing" tab
• Click +New
• Submit the form
• Click the "Refresh" button
• Click +New
• Submit the form
• Click the "Refresh" button
• Check the order numbers of submissions
• Refresh the page
• Check the order numbers of submissions

The order is correct if you:

• e.g. add 2 first submissions and then click the Refresh button
• refresh the page

[dbemke]

Another thing that I'm not sure if it's related to the PR and I tried to figure out why/when it happens but haven't found reproducible steps yet.. maybe someone will have some ideas when it happens:
I tried the open form in Enketo in undeleted forms. In one of my project I had some forms that were deleted some time ago and I clicked "undelete". "Web form isn't available..." stays all the time even after refreshing or waiting some time. Publishing makes it possible to use Enketo. I encountered it only twice in this project both undeleted forms were some selects and after deleting opened in draft view. I'm not able to reproduce it with other forms yet.
https://staging.getodk.cloud/#/projects/24/forms/selectColumns/draft/testing

[matthew-white]

There is a descending order of submission order numbers applied after using the "Refresh" button in Draft submission table and Submissions table.

I don't think this would have been affected by the changes here, but I think this is probably related to getodk/central-frontend#741. I'll leave a comment there about this.

[matthew-white]

I tried the open form in Enketo in undeleted forms. In one of my project I had some forms that were deleted some time ago and I clicked "undelete". "Web form isn't available..." stays all the time even after refreshing or waiting some time.

I see 4 other form drafts in this state that were created this year. They were all created between 2023-08-10 09:16:29 and 2023-08-10 09:28:45 UTC. Looking at Slack, it looks like the QA server ran out of disk space around that time. If Enketo is down for a little while, then Web Forms won't be made available for forms or form drafts created during that time. However, as you've seen, they can be made available once Enketo is back up, by republishing the form. It should be rare for Enketo to be down.

[matthew-white]

There is a descending order of submission order numbers applied after using the "Refresh" button in Draft submission table and Submissions table.

Would you mind double-checking this? It should be fixed now.

[dbemke]

Thank you Matthew for help I wouldn't figure that one on my own. The descending order is fixed.

[dbemke]

Tested with success!

[srujner]

Tested with success!