program: fall back from data server to multiplexer

[wchargin]

Summary:
Under --load_fast=auto, if the data server exits without writing a
port file, we now gracefully degrade to the multiplexer reading path.
This subsumes and expands upon the checks from #4786, with semantics now
defined by the data server as in #4793.

This depends on the --error-file data server flag added in #4793,
which is not yet released. To handle this, we add a mechanism for gating
flags against data server versions. As discussed in #4689, this
raises a concern of what to do when the data server is chosen by a
mechanism other than the Python package. We resolve this by simply
treating such data servers as bleeding-edge. If you link in a data
server yourself, you should use it with a copy of tb-nightly built
from the same Git commit, or your mileage may vary.

See also #4786, which first proposed this functionality and offered some
alternative mechanisms.

Test Plan:
Set your GOOGLE_APPLICATION_CREDENTIALS to an invalid JSON file, and
try running with --load_fast=auto with release v0.4.0 of the data
server and also with latest head (//tensorboard/data/server:install).
Note with --verbosity 0 that with the old server, --error-file is
not passed and the data server has to fall back to anonymous creds,
whereas with the new server, --error-file is passed and TensorBoard
correctly falls back to the multiplexer.

wchargin-branch: data-server-fallback

[nfelt]

Optional, but I find the control flow here to have gotten pretty convoluted. What about just putting server_ingester.get_server_binary(), ingester = server_ingester.SubprocessServerDataIngester(), and ingester.start() all into a single common helper (say start_data_server_and_ingester()) and then splitting out the two cases for "auto" and "true"? It does mean you have a broader try-except block but since the exceptions are typed that doesn't seem like the end of the world, and it would simplify the logic to:

if flags.load_fast == "true":
    try:
        return start_data_server_and_ingester()
    except server_ingester.NoDataServerError as e:
        sys.stderr.write("Option --load_fast=true not available" ... )
        sys.exit(1)
    except server_ingester.DataServerStartupError as e:
        sys.stderr.write(_DATA_SERVER_STARTUP_ERROR_MESSAGE_TEMPLATE % e)
        sys.exit(1)

if flags.load_fast == "auto":
    try:
        return start_data_server_and_ingester()
    except (server_ingester.NoDataServerError, server_ingester.DataServerStartupError) as e:
        logger.info("Data server error: %s; falling back to multiplexer", e)

ingester = local_ingester.LocalDataIngester(flags)
ingester.start()
return ingester

[wchargin]

I like it; thanks. Done.

[nfelt]

FWIW, it seems like we still want this logic if we want to avoid behavior with a data server <0.5.0 regressing to the way it was before #4786, i.e. where auto will try to use the data server even if the logdir is not compatible. I suppose if we see this PR as just supplanting #4786 entirely (such that if you want graceful fallback you need to be on 0.5.0+ rather than having the logic replicated here) that's fine, but just pointing it out.

[wchargin]

Fair enough. I was intending to supplant, but there’s not much harm in
keeping it for a few more days. Done.