[FlightRPC] Authentication Redesign

[asfimport]

Redesign the authentication process to:

• Decouple authentication from the handshake
• Provide better integration with standard HTTP authentication mechanisms.
• Simplify the handshake process:

  • Make this single-stage rather than potentially multi-stage.

  • Remove auth payloads.

    Please see this document and PR

Reporter: James Duong / @jduo
Assignee: James Duong / @jduo

Related issues:

• [FlightRPC] Handshake Extension (Dependency)
• [FlightRPC] Make Handshake optional (relates to)
• [FlightRPC][C++][Python] Header-based authentication (relates to)
• [FlightRPC] Deprecate old authentication mechanism (is depended upon by)

PRs and other links:

• GitHub Pull Request #7994

_{Note: This issue was originally created as ARROW-9804. Please see the migration documentation for further details.}

[asfimport]

James Duong / @jduo:
The work done here to decouple auth from the handshake makes it much easier to implement ARROW-7579.

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 7994
#7994