[3.x] Timestamped input buffering - prevent stalling and improve timing

[lawnjelly]

Input was resulting in stalling and ANRs on Android and possibly other platforms. This PR separates buffered input into two parts to prevent stalls. It also timestamps input events and applies them on the relevant logical physics tick, rather than attempting to apply them "realtime".

This PR is quite a significant refactor to input buffering in order to make it threadsafe, store timestamps, and offer accurate agile input on physics ticks on platforms with separate input thread. It also offers a legacy mode (with a project setting) in case of regressions, this can be removed once any bugs have been worked out, which should make the code simpler.

N.B.

This PR was written immediately prior to #76399 being merged, which may have addressed some of the similar concerns for ANRs. This PR is thus more concerned with agile input than fixing ANRs.

Explanation

The fix in #76399 was to deal with the problem that processing input events (which could take an extended amount of time, for say a load or reading a web address etc) was locking the input buffer. It took a simple approach of micro-unlock / relock for each input event.

This PR is a bit more involved, it changes the buffer into a more thread safe structure.

• Two mirror incoming buffers for incoming events (and events are timestamped as they come in)
• The active mirror is swapped quickly with a quick lock on flush, so there is no need for microlocking as in Allow concurrent buffering and dispatch of input events #76399
• The incoming read buffer is copied across to a master buffer, which is thus sorted by timestamp

When flushing the master buffer can thus be flushed up to a desired timestamp. This enables us to flush on physics ticks up to the logical timestamp of the physics tick (rather than realtime of when the tick is processed).

This means input can be spread out correctly over logical time - if a frame has e.g. 8 physics ticks, an on off keypress can be assigned to say ticks 2 and 3, instead of being only applied if the key happens to be pressed at the frame time. This is essentially a (hopefully! 😄 ) more accurate version of @RandomShaper 's agile input.

Accumulating

In AGILE mode, accumulating input (e.g. mouse motion) is deferred until flushing, which means that accumulation can be done accurately on a per tick basis. This can potentially give perfectly accurate input for say first person shooters, while still using accumulation to reduce the number of calls to user code and thus increase efficiency. In FRAME mode (and OS that do not support AGILE), accumulation still takes place correct to the nearest frame, as before.

Input thread

There is one snag, for the input timestamps to be meaningful, we ideally want input to come in free flow from the OS. This so far only occurs on Android, where there is a separate thread for input. On most of the other platforms, the OS message pump is done once per frame, so the end result is the input gets bunched up like the current status quo.

However, if we have the input backend working correctly with timestamps / agile input, that does leave the option open to now improve the input accuracy on other platforms in later PRs.

We can either investigate using a separate input thread (ideally) or perhaps resampling the input queue from the OS multiples times per frame (in single threaded fashion). The latter is not ideal but could lead to some greater input resolution.

Notes

• Adds a temporary legacy_event_flushing project setting for more backward compatibility (but even so, input is quite different so will need testing).
• I'll also do a PR for master, which may be able to support more platforms for AGILE.

Demo Project

Can be run on desktop and Android.
Note that it will need to have export template compiled for android (or use the artifact), and set the custom template to be used by the export.
AgileInputTest.zip
Note that the frame_delay_ms is used to limit the frames per second, and this also has the effect of limiting the frames per second in the editor (this is nothing to do with the PR).
It will run at 1tps, and the physics ticks will run 8 per frame.

When running:

• Pressing return or clicking mouse will show the press and release ticks on desktop. Note that these will be bunched on the frame tick, because there is no agile input available.
• On Android, touch the screen and move around. Note that the input ticks can be between the frame ticks, and agile input is working.

Status

• Now tested and working on Android.
• Tested and working on Linux with multiple message pumps on the main thread, although I have decided to leave this feature to a future PR:

Supporting multiple pumps offers some advantages but some disadvantages on platforms that don't have a separate input thread.

The reason is that the message pumps will tend to occur bunched up in time (probably before physics ticks), but this timing will be variable on different computers, depending on their GPU / CPU power. Variable input timing is generally something that might not be a good thing in games because it can make testing difficult with variable gameplay. But on the other hand it could be better than nothing in apps that seek to e.g. draw using the mouse.

[Riteo]

BTW Wayland and X11 use a separate event thread already which is mostly input so both could already and very probably should be adapted to this system.

[lawnjelly]

Incidentally fixing up the timing accuracy for the input suggests two improvements that should probably be done in follow up PRs. These are not necessary to get started with agile input (after all it is already available in a basic form), but they will greatly improve the input timing.

Further work

Jitter Fix

Jitter fix is a problem, as it liberally changes the timing of individual physics ticks, shifting them before and after frames. This means the wall clock timestamps of input events won't line up in any sensible way with the physics ticks themselves.

Jitter fix is also a problem for fixed timestep interpolation, which is why it is forcibly set to 0.0 when interpolation is in use. Even better though would be to add an additional path for a simple reference fixed timestep implementation, something I attempted to introduce as part of #30798 . Even though semi-fixed is controversial, having a reference timestep implementation without jitter fix would be invaluable in removing a whole host of potential timing bugs.

This would require review independently and is best done as a separate PR, that perhaps enforces reference timestep when agile input is in use.

Delta smoothing

When delta smoothing is in use, each frame can be compressed or expanded in time (to form smoothed time) relative to wall clock time. This is ideally used to modify the first frame timings to make agile input as smooth as possible. This is relatively simple but needs to be done after removing jitter fix, because jitter fix creates a significant confounding factor.