[MM-62925] introduce websocket client-side ping

[davidkrauser]

This PR is a continuation of #8619. The PR was migrated to a branch on the upstream repository because E2E tests can't execute on branches from forked repositories.

Summary

This PR introduces new functionality on the client side to send PING messages over the websocket. If the server doesn't respond within PING_INTERVAL (currently 30 seconds), the connection is closed and re-created. This will allow us to find broken connections more quickly.

Ticket Link

https://mattermost.atlassian.net/browse/MM-62925

This is related to a similar change in the webapp client: mattermost/mattermost#30293

Testing at the Application Layer

This change can be manually verified using a mitmproxy that intermittently drops websocket messages. This script will drop every other PONG message from the server, starting with the first:

import time
from mitmproxy import ctx

class WebSocketDropper:
    def __init__(self):
        self.drop_pong = True

    def websocket_message(self, flow):
        if not "pong" in flow.websocket.messages[-1].content.decode('utf-8'):
            return
        
        if self.drop_pong:
            ctx.log.info("Dropping server PONG response")
            flow.websocket.messages[-1].drop()
            self.drop_pong = False
        else:
            self.drop_pong = True

addons = [WebSocketDropper()]

This proxy can be run with:

mitmproxy --set verbosity=3 -s PATH_TO_MITMPROXY_PYTHON_SCRIPT --listen-port 8080 --mode reverse:http://localhost:8065

That will start delivering requests on localhost:8080 to a local mattermost instance on the default port.

Additionally to get around some CORS checks, the local instance needs to have its base URL set to http://LOCAL_HOST_IP:8080 in the system console. The mobile app should connect to http://LOCAL_HOST_IP:8080.

With everything up and running, you'll see:

• After 30 seconds, the client sends a PING, but the server PONG message is dropped
• After another 30 seconds, the websocket will be closed and re-opened (since the PONG was never received).
• After another 30 seconds, the client will successfully complete a PING/PONG interaction with the server
• Repeat

Testing at the Network Layer

We can also verify that the connection is re-established quickly after a failed attempt to send a PING message. This can happen if the packets are being rejected.

To test on a Mac:

1. Set up a standard development instance of mattermost running on localhost at port 8065
2. Open the webapp in a web browser (at localhost:8065)
3. Configure the admin user and set the base url to http://LOCAL_HOST_IP:8065 in the system console.
4. Login to the mobile app at http://LOCAL_HOST_IP:8065.
5. In a terminal, execute sudo pfctl -e && echo "block return proto tcp to port 8065" | sudo pfctl -f -. This will block all tcp traffic to port 8065.
6. Watch the mobile app report that it has lost it's connection with the server.
7. In a terminal, execute sudo pfctl -d. This will start allowing tcp traffic again
8. Verify a the mobile app connection is re-established.

Checklist

• Added or updated unit tests (required for all new features)
• Have run E2E tests by adding label E2E iOS tests for PR.

Release Note

Added a client-side ping to the websocket to detect broken connections more quickly.

[davidkrauser]

@enahum and @larkox this is a new PR with the same contents as #8619 that you already reviewed. The old PR was using a branch from a forked repo, which doesn't work with the E2E testing infrastructure.

Would you mind lending your approvals on this PR, too? 🙂

[davidkrauser]

@yasserfaraazkhan and @larkox - the E2E android tests had failures on the last run here, and the check reported an overall failure. The E2E iOS tests also had failures on the last run, but the check reported an overall success. Is that expected?

Additionally, I looked back at the last few merged PRs to see if the E2E android test failures were expected, and it appears that we did not run them. Should I be running them for my PR? And if so, are the results currently stable and expected to succeed on the master branch? Trying to figure out if there are legitimate issues with my changes that need to be addressed 🙂

[yasserfaraazkhan]

@davidkrauser I see the android build step failed with error android:exported needs to be explicitly specified for element <activity#androidx.test.core.app.InstrumentationActivityInvoker$BootstrapActivity>. It had been fixed here. I'll take a look at it separately.

For now, we can use iOS e2e tests on PR. There are few flaky failing tests. Usually, if the overall % is around 90. That should be good. We will get to fixing or disabling the flaky tests in future.

[lindalumitchell]

@yasserfaraazkhan I see you've already been taking a look at this one, but I was added as a reviewer. Should I remove myself and add you instead?

[yasserfaraazkhan]

@lindalumitchell I've only reviewed the e2e tests. You can try some more scenarios.