Token refresh failing for office365

[sshafiullah]

Hi,
this worked a treat for me for integration with gmail, but I am running into issues with the office365 setup. I followed the instructions, created an app and I was able to generate the token, but the refresh seems to fail:

TokenStore::Refresh: code=400, response={"error":"invalid_request","error_description":"AADSTS900144: The request body must contain the following parameter: 'scope'. Trace ID: 02cde8c3-be9e-45c0-b767-e4c211760100 Correlation ID: 271fa28c-34ce-473f-aaab-e84bfd32c2e8 Timestamp: 2024-08-01 17:44:37Z","error_codes":[900144],"timestamp":"2024-08-01 17:44:37Z","trace_id":"02cde8c3-be9e-45c0-b767-e4c211760100","correlation_id":"271fa28c-34ce-473f-aaab-e84bfd32c2e8","error_uri":"https://login.microsoftonline.com/error?code=900144"}
sasl-xoauth2: TokenStore::Refresh: request failed
sasl-xoauth2: Client::DoStep: new state 0 and err -5
smtp: 8F557F44: to=<sshafiullah@yahoo.com>, relay=smtp.office365.com[40.104.46.34]:587, delay=2180, delays=2179/0.08/1.1/0, dsn=4.7.0, status=deferred (SASL authentication failed; cannot authenticate to server smtp.office365.com[40.104.46.34]: bad protocol / cancel)

I have made sure 'Allow public client flows' is enabled. I did however perform the browser authentication part from another machine since the smtp server is headless. I know it needed to be from the same machine for gmail but was not sure if that is the case for office 365 as well.

[tarickb]

My own setup is based on Gmail so I don't get to test integration with 365 very often, but I did set up a test account just now and everything still seems to be working fine in the token refresh flow. Can you elaborate a little on what your setup looks like? You generated the initial token with sasl-xoauth2-tool get-token, right? Can you try sasl-xoauth2-tool test-token-refresh? That should help narrow down the problem to a sasl-xoauth2 issue vs. a problem with your Postfix setup.

[sshafiullah]

Hi,
thanks for taking the time to look at this. To be honest I am unfamiliar with Azure Cloud services so it is probably a setting I am missing. Here is a little bit more context for the problem:

OS: ubuntu 24.04
Postfix: 3.8.6-1build2

Postfix configurations:
`smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
append_dot_mydomain = no
readme_directory = no
compatibility_level = 3.6

smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_tls_security_level=may
smtp_tls_CApath=/etc/ssl/certs
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = shahriar-worknode2.ec2.internal
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = $myhostname, shahriar-worknode2, localhost.localdomain, , localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all

relayhost = [smtp.office365.com]:587
smtp_tls_security_level = encrypt
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_sasl_auth_enable = yes
smtp_sasl_mechanism_filter = xoauth2
smtp_sasl_security_options = noanonymous
smtp_sasl_tls_security_options = noanonymous
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_use_tls = yes`

sasl_passwd:
[smtp.office365.com]:587 outlook_6d93615f23b383e7@outlook.com:/etc/tokens/outlook_6d93615f23b383e7@outlook.com

sasl-xoauth2.conf (actual client ID replaced with x's):
{ "client_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "client_secret": "", "token_endpoint": "https://login.microsoftonline.com/consumers/oauth2/v2.0/token", "always_log_to_syslog": "yes", "log_full_trace_on_failure": "yes" }

Permissions on tokens directory:
ls -l /var/spool/postfix/etc/tokens/ total 12 -rw------- 1 postfix postfix 4111 Aug 11 13:39 outlook_6d93615f23b383e7@outlook.com

API Permissions:

When I initially create the token, I get an error if I do not specify a --tenent value (I've replaced the actual client ID with x's):
sasl-xoauth2-tool get-token outlook /var/spool/postfix/etc/tokens/outlook_6d93615f23b383e7@outlook.com --client-id="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" --use-device-flow Traceback (most recent call last): File "/usr/bin/sasl-xoauth2-tool", line 342, in <module> main() File "/usr/bin/sasl-xoauth2-tool", line 335, in main args.func(args) File "/usr/bin/sasl-xoauth2-tool", line 228, in subcommand_get_token get_token_outlook( File "/usr/bin/sasl-xoauth2-tool", line 205, in get_token_outlook tokens = outlook_get_initial_tokens_by_device_flow(client_id, tenant) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/bin/sasl-xoauth2-tool", line 189, in outlook_get_initial_tokens_by_device_flow raise Exception("Failed to create device flow. Ensure that public client flows are enabled for the application. Flow: %s" % json.dumps(flow, indent=4)) Exception: Failed to create device flow. Ensure that public client flows are enabled for the application. Flow: { "error": "unauthorized_client", "error_description": "AADSTS700016: Application with identifier 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' was not found in the directory 'Microsoft Accounts'. This can happen if the application has not been installed by the administrator of the tenant or consented to by any user in the tenant. You may have sent your authentication request to the wrong tenant. Trace ID: 98bd0fa3-7aa1-4342-b2c9-346e37a63801 Correlation ID: 078f9011-d94b-4412-bf1b-c43d4103ecd9 Timestamp: 2024-08-11 13:37:18Z", "error_codes": [ 700016 ], "timestamp": "2024-08-11 13:37:18Z", "trace_id": "98bd0fa3-7aa1-4342-b2c9-346e37a63801", "correlation_id": "078f9011-d94b-4412-bf1b-c43d4103ecd9", "error_uri": "https://login.microsoftonline.com/error?code=700016", "interval": 5, "expires_in": 1800, "expires_at": 1723385239.0101411, "_correlation_id": "078f9011-d94b-4412-bf1b-c43d4103ecd9" }

When I place my --tenant id value, I am able to generate the token but for some reason when I try to send an email, or run sasl-xoauth2-tool test-token-refresh, I get the following error message. This occurs even if ran a few seconds after generating the initial token:

sasl-xoauth2-tool test-token-refresh /var/spool/postfix/etc/tokens/outlook_6d93615f23b383e7@outlook.com Config check passed. auth failed: 2024-08-11 13:56:34: TokenStore::Read: file=/var/spool/postfix/etc/tokens/outlook_6d93615f23b383e7@outlook.com 2024-08-11 13:56:34: TokenStore::Read: refresh=0.AbcAzKym1iO0z0GieQfs6M9DU0StNLx_gKFKoGdb-wqRH1a3AFg.AgABAwEAAAApTwJmzXqdR4BN2miheQMYAgDs_wUA9P9dYuu4Dbs6wXUJvITuMxP3tcooJ1nvBQDLbBGzKDcfXuNBJDVJC9TeriIzj8PjmAyXjFKeNwiD2P6hqNxktpZurVrFwYgAQDO3reA7F-17WDduJ9TvQJchY2Pt3fQIyE4p9_ua2j2CT0yKwq_7UzC8JeFfAOnPVznVLIiVkdFA_6U8ovBsbcCTd0wZOaV0kdLCq5lxS86Oacr3oYTmvZu_A5BWjgUGj_eg2Mujv_WtS_ML3wDh9AJ46yaGMM03sgEbsq0ZiAFk3WLyGjrRz6kVlCnzb3HK9X_t8RWDYSfupnc1Mgb2GA9Z2Stman_hElPHGxIdkfaJ0CG_94YWdwJWLL3tEMYfK4gKLM6SQujve_l5Fb0NbYyiyg6ZYjN10AuJrID9uxhKzslTtGK-WbhYtLTCDmOQBZQEpJpmwmykt9SlesNr6t53uQlSHWJrtHZN5iNIiiDgx6wJ1tWfByxZxvJBlpCvjGu8u-SBvpL9KXSZRSvHWZ9V3bzK4WyDL_8K5VnxmBBSv6wVyFt3wtuilL_66Tchv1aYNg0uSZzELnJdtq5r5y1pc09z6F622ogpkNE0-bgkbOmZJwff6b8cMo5HDKNEDX1Z54ECmmgqsTH_FH2iGCjCIKEfA2fmxvVXt40SylgOHqhvu0rCSufjMGNMrHoUc3xZEHO5Etjwdabbduxhn-X9PhwRMx5BS9v7-jvix_oNvqWsW2ESIGrygRboRpvV_IfsBMPh7EvolrbY-210vQaEHIUH2H94eLSB7Y8jaAcE13t9v57P0mFj4vyGUwqlsT_QCAQ6reD8zRUBOcXN-qI1jZV8toPcjyvl_w6GudVa-cmFcgtnbqDzIJ-YziQgi0eqBnbIBXwi1EYuNFAAPMJVP9PYN2yNteB4nhr 2024-08-11 13:56:34: TokenStore::Refresh: attempt 1 2024-08-11 13:56:34: TokenStore::Refresh: token_endpoint: https://login.microsoftonline.com/consumers/oauth2/v2.0/token 2024-08-11 13:56:34: TokenStore::Refresh: request: client_id=bc34ad44-807f-4aa1-a067-5bfb0a911f56&client_secret=&grant_type=refresh_token&refresh_token=0.AbcAzKym1iO0z0GieQfs6M9DU0StNLx_gKFKoGdb-wqRH1a3AFg.AgABAwEAAAApTwJmzXqdR4BN2miheQMYAgDs_wUA9P9dYuu4Dbs6wXUJvITuMxP3tcooJ1nvBQDLbBGzKDcfXuNBJDVJC9TeriIzj8PjmAyXjFKeNwiD2P6hqNxktpZurVrFwYgAQDO3reA7F-17WDduJ9TvQJchY2Pt3fQIyE4p9_ua2j2CT0yKwq_7UzC8JeFfAOnPVznVLIiVkdFA_6U8ovBsbcCTd0wZOaV0kdLCq5lxS86Oacr3oYTmvZu_A5BWjgUGj_eg2Mujv_WtS_ML3wDh9AJ46yaGMM03sgEbsq0ZiAFk3WLyGjrRz6kVlCnzb3HK9X_t8RWDYSfupnc1Mgb2GA9Z2Stman_hElPHGxIdkfaJ0CG_94YWdwJWLL3tEMYfK4gKLM6SQujve_l5Fb0NbYyiyg6ZYjN10AuJrID9uxhKzslTtGK-WbhYtLTCDmOQBZQEpJpmwmykt9SlesNr6t53uQlSHWJrtHZN5iNIiiDgx6wJ1tWfByxZxvJBlpCvjGu8u-SBvpL9KXSZRSvHWZ9V3bzK4WyDL_8K5VnxmBBSv6wVyFt3wtuilL_66Tchv1aYNg0uSZzELnJdtq5r5y1pc09z6F622ogpkNE0-bgkbOmZJwff6b8cMo5HDKNEDX1Z54ECmmgqsTH_FH2iGCjCIKEfA2fmxvVXt40SylgOHqhvu0rCSufjMGNMrHoUc3xZEHO5Etjwdabbduxhn-X9PhwRMx5BS9v7-jvix_oNvqWsW2ESIGrygRboRpvV_IfsBMPh7EvolrbY-210vQaEHIUH2H94eLSB7Y8jaAcE13t9v57P0mFj4vyGUw 2024-08-11 13:56:34: TokenStore::Refresh: code=400, response={"error":"invalid_request","error_description":"AADSTS900144: The request body must contain the following parameter: 'scope'. Trace ID: 4ea55645-8875-43e5-a292-f7dd3f55c300 Correlation ID: b4c79506-060f-4523-b512-d6c0cd5223dc Timestamp: 2024-08-11 13:56:34Z","error_codes":[900144],"timestamp":"2024-08-11 13:56:34Z","trace_id":"4ea55645-8875-43e5-a292-f7dd3f55c300","correlation_id":"b4c79506-060f-4523-b512-d6c0cd5223dc","error_uri":"https://login.microsoftonline.com/error?code=900144"} 2024-08-11 13:56:34: TokenStore::Refresh: request failed Token refresh failed.

[tarickb]

Thanks for the detailed logs and context! I suspect, but I'm not 100% sure, that if you're having to specify --tenant to generate the token in the first place, then you probably also need to use a different token-refresh endpoint. I briefly allude to this in the README with a pointer to Microsoft's docs (because the URLs and the guidance seem to change). You could try this value for token_endpoint in /etc/sasl-xoauth2.conf instead:

https://login.microsoftonline.com/<YOUR TENANT ID>/oauth2/v2.0/token

[sshafiullah]

When I change the endpoint to use the tenant ID, I am able to generate an initial token like before, but I am get the following error when I attempt to send out a message:

2024-08-11T18:13:26.691145+00:00 shahriar-worknode2 sasl-xoauth2: TokenStore::Refresh: attempt 1
2024-08-11T18:13:26.691204+00:00 shahriar-worknode2 sasl-xoauth2: TokenStore::Refresh: token_endpoint: https://login.microsoftonline.com/d6a6accc-b423-41cf-a279-07ece8cf4353/oauth2/v2.0/token
2024-08-11T18:13:26.691228+00:00 shahriar-worknode2 sasl-xoauth2: TokenStore::Refresh: request: client_id=bc34ad44-807f-4aa1-a067-5bfb0a911f56&client_secret=&grant_type=refresh_token&refresh_token=0.AbcAzKym1iO0z0GieQfs6M9DU0StNLx_gKFKoGdb-wqRH1a3AFg.AgABAwEAAAApTwJmzXqdR4BN2miheQMYAgDs_wUA9P8KBEDUHPyDCxtldYAaASFeDg8MhsM_Kis55ZEYUkRvpA-2yMCZLsYXNJ6Rx-0KnTNMDX8tB5WW8r1qe1MojiqVcJRN6KwNPSYodDOBy6guhbhu6l96aWennfuZVgkbOOfPiqgdFkDf_KG5tTIHeKNtK_58WH5NGVDwLNe1wOWvz9WQbvCi6onZWUTtPAWt3YBVmEEK3Tsb2lieu7YwQmxfLxOEq1TdC0Rxet8lXsBvQ2iyW6bjSsHHNbYae6xIHJ3DI7_CUzWB6kY_Mfw1MgieStdQR5k5JAgHbPUtLCw9L4wJWCP1A32LoOL2LIPq7ZYzE0LdLsfWed9XGnWvu5_X0Vhk4OkJk43ORl-b37d_kTwJQMp1zAvajHX32tcpE3l_oTHOluVWLZXh8WRdExXj8W1QebdLyAV6PNDidva-JiDp7uxzXo53d6ZEy6Yl-BO75ZZpio3v2TxHNOtkcGPL8zsY0fQFLV1UDnFe12HmDQZXvT5roPGIT5eid7LiMhDdSvUk_CyQ59YNJsetjuPdoNPgI6-6n6VnVHEjHCnXtN2W8aAUst4dc_Ez7VsTn_bgxXiUyORYjsaXa9Vr0RPWgzXb9wOb9lvI3_AXlxHsgVtxDjgzGNkIk-31koJlcdtpm5wccpwoFPrrvgH0SN29fxYRyesDFEATxxqbPMzvmhv5ZMO7VYqnE0rhiiIAylNh4014fF_oY03Qgpw15BKk7c56d8vjpt53ahfYpX4v_TlhjLYNXDukxH8rXS2XbDP3rdbL7lFGMcg85NdFlvrltxOapW
2024-08-11T18:13:26.867117+00:00 shahriar-worknode2 postfix/smtpd[214789]: disconnect from localhost[127.0.0.1] ehlo=1 mail=1 rcpt=1 data=1 quit=1 commands=5
2024-08-11T18:13:27.418854+00:00 shahriar-worknode2 sasl-xoauth2: TokenStore::Refresh: code=400, response={"error":"invalid_grant","error_description":"AADSTS65001: The user or administrator has not consented to use the application with ID 'bc34ad44-807f-4aa1-a067-5bfb0a911f56' named 'Postfix OAuth'. Send an interactive authorization request for this user and resource. Trace ID: 8c9482c0-d89f-418a-be9a-e28af443c900 Correlation ID: 7182161d-71fe-4a79-828b-23754d58d5c4 Timestamp: 2024-08-11 18:13:27Z","error_codes":[65001],"timestamp":"2024-08-11 18:13:27Z","trace_id":"8c9482c0-d89f-418a-be9a-e28af443c900","correlation_id":"7182161d-71fe-4a79-828b-23754d58d5c4","suberror":"consent_required"}
2024-08-11T18:13:27.418881+00:00 shahriar-worknode2 sasl-xoauth2: TokenStore::Refresh: request failed
2024-08-11T18:13:27.418895+00:00 shahriar-worknode2 sasl-xoauth2: Client::DoStep: new state 0 and err -5
2024-08-11T18:13:27.418921+00:00 shahriar-worknode2 smtp: 4A664164F: SASL authentication failed; cannot authenticate to server smtp.office365.com[52.96.189.18]: bad protocol / cancel
2024-08-11T18:13:27.419096+00:00 shahriar-worknode2 sasl-xoauth2: Client: destroyed
2024-08-11T18:13:27.419166+00:00 shahriar-worknode2 smtp: connect to smtp.office365.com[2603:1036:303:3c19::2]:587: Network is unreachable
2024-08-11T18:13:27.430493+00:00 shahriar-worknode2 smtp: 4A664164F: to=<sshafiullah@yahoo.com>, relay=none, delay=33, delays=31/0.06/2/0, dsn=4.4.1, status=deferred (connect to smtp.office365.com[2603:1036:303:3c19::2]:587: Network is unreachable)

seems it wants another interactive authorization even after we use the --device-flow option to create the initial token

[tarickb]

Does the email address you have specified in /etc/sasl_passwd match the account you were signed into when you opened the device-flow link and authorized the application? As part of the authorization flow, were you asked if you wanted to allow Postfix OAuth to send mail?

[sshafiullah]

I am using my outlook.com email address for this. It is the same one in my sasl_passwd entry. Is there anything I need to do to link my outlook.com account with my azure cloud account?
I am also using a browser on another machine to perform the authentication since the postfix server is headless. Would that cause an issue?

[tarickb]

You may need to update the supported account types for your "Postfix OAuth" app registration to include consumer accounts:

At that point you may need to change your token_endpoint back to whatever you were using before, or possibly the "common" endpoint instead: https://login.microsoftonline.com/common/oauth2/v2.0/token

[sshafiullah]

This worked! It seems that when we register the app at first, we need to specify one of the bottom 2 options below in order for outlook.com accounts to work:

Note that we cannot specify that option after the app is registered already.

Then I used the 'common' endpoint without specifying a tenant to initiate the token. Thank you... you are magnificent!

[tarickb]

Oh awesome -- very glad to hear that worked!