> On Jun 14, 2020, at 8:54 PM, Kevin J. McCarthy <ke...@8t8.us> wrote:
>
> I made a commit to master adding this yesterday. I did some testing on
> Gmail, but haven't gotten around to testing Microsoft platforms yet. I could
> definitely use some help there.
With the attached mutt_oauth2.py script and a corresponding app registration I
successfully connected mutt to:
- Gmail account
- Microsoft consumer account (e.g., outlook.com)
- Microsoft work/school account (Office365 under an Azure organizational
tenant)
An obstacle for users is the needed "app registration". With Google it didn't
strike me as obvious how to create a test registration. As for a Microsoft
work/school account, my institution won't even allow end users to get to the
app registration screen, nor do my institutional security admins seem willing
to create an in-house app registration for me to use, feeling that the software
vendor should have obtained one and coded it into their product (as is the case
with Microsoft Outlook, Apple Mail, and Mozilla Thunderbird). I might succeed
in convincing them to approve an unofficial registration that I create myself
using a consumer account, but only mutt maintainers would be able to create an
official mutt registration verified by the cloud mail provider as belonging to
"mutt.org". I suspect my institution is more likely to approve an official
registration.
Ideally the mutt maintainers create an official registration and hardcode it
into the script distributed with mutt, but this is surely an unreasonable
expectation once one considers all the different mail providers on the planet.
Perhaps for the time being this could be limited to specific mail providers
requested by people on the mutt mailing list? Taking Thunderbird as an example,
the version 78.0b2 source has hardcoded registrations for several major
providers---see the OAuth2-related files in comm/mailnews/base/util. A source
comment reveals they are exploring eventually replacing the hardcoded
registrations with some type of "dynamic registration" that some mail providers
might support. I don't know how that mechanism would work and whether it would
be an option for mutt, by which I mean I don't know whether anyone would be
able to use that mechanism or whether some entity like Mozilla Foundation will
be brokering access to such a mechanism on behalf of Thunderbird.
Here's how to create a Microsoft registration (whether done individually by
each end user, or done once and for all by the mutt maintainers). Go to
portal.azure.com, log in with a Microsoft account (get a free one at
outlook.com), then search for "app registration", and add a new registration.
On the initial form that appears, put a name like "Mutt", and leave the other
two questions at default (allow any type of account, no redirect URI), then
more carefully go through each screen:
Branding
- End users might leave these fields blank
- For official registration, verify the publisher domain by adding a
retrievable file to "mutt.org"
Authentication:
- Platform "Mobile and desktop"
- Redirect URI
"https://login.microsoftonline.com/common/oauth2/nativeclient";
- Any kind of account
- Enable public client (allow device code flow)
API permissions:
- Microsoft Graph, Delegated, "offline_access"
- Microsoft Graph, Delegated, "IMAP.AccessAsUser.All"
- Microsoft Graph, Delegated, "POP.AccessAsUser.All"
- Microsoft Graph, Delegated, "SMTP.Send"
- Microsoft Graph, Delegated, "User.Read"
Overview:
- Take note of the Application ID (a.k.a. Client ID), you'll need it shortly
End users who aren't able to get to the app registration screen within
portal.azure.com for their work/school account can temporarily use an incognito
browser window to create a free outlook.com account and use that to create the
app registration.
Edit the client ID into the mutt_oauth2.py script. Run "mutt_oauth2.py --help"
to learn script usage. To obtain the initial set of tokens, run the script
specifying a name for a disposable token storage file, as well as
"--authorize", for example using this naming scheme:
mutt_oauth2.py use...@myschool.edu.tokens --verbose --authorize
The script will ask questions and provide some instructions. For the flow
question, pick one; you can go back later and delete your token file and start
over with the other flow, to see which one you like more. Depending on the
OAuth2 provider and how the app registration was configured, both flows might
not work, but trying them is the best way to figure out what works and which
one you prefer. In the "authcode" flow you paste a complicated URL into a
browser, then manually extract a "code" parameter from a subsequent URL in the
browser and paste that back to the script; in the "devicecode" flow you go to a
simple URL and just enter a short code, no additional steps.
If you're stuck at this point because a web browser screen says your
institution admins must grant approval, either request that approval, or as a
temporary punt for end-user experimentation you could "borrow" the Mozilla
Thunderbird registrations mentioned earlier (but heed the comment in their
source warning their experimental registration might eventually disappear).
You'll need the client_id, client_secret, and redirect_uri. Before trying mutt
with this borrowed registration, I suggest first configuring Thunderbird itself
for your work/school account using OAuth2 (called "Modern Auth" in Microsoft
marketing lingo), as you might need to seek approval from your institutional
admins to get that to work. Once Thunderbird itself is working, any
OAuth2-capable IMAP/POP/SMTP client using the same registration should also
work (how could the server possibly tell the difference?).
Once you've succeeded authorizing mutt_oauth2.py to obtain tokens, try one of
the following to see whether IMAP/POP/SMTP are working:
mutt_oauth2.py use...@myschool.edu.tokens --verbose --test
mutt_oauth2.py use...@myschool.edu.tokens --verbose --debug --test
Without optional parameters, the script simply returns an access token
(possibly first conducting a behind-the-scenes URL retrieval using a stored
refresh token to obtain an updated access token). This is how the script will
be used by mutt. Your muttrc would look something like:
set imap_user="use...@myschool.edu"
set folder="imap://outlook.office365.com/"
set smtp_url="smtp://${imap_user}@smtp.office365.com:587/"
set imap_authenticators="xoauth2"
set imap_oauth_refresh_command="/path/to/script/mutt_oauth2.py
${imap_user}.tokens"
set smtp_authenticators=${imap_authenticators}
set smtp_oauth_refresh_command=${imap_oauth_refresh_command}
I didn't navigate creating my own registration at Google, instead tested the
script against Google by borrowing the Thunderbird registration and using the
authcode flow. Using a Microsoft consumer account I could easily create my own
registration, and tested both the authcode flow and the devicecode flow. For a
Microsoft work/school account I needed a registration approved by my
institution, so I again borrowed the Thunderbird registration and could test
the authcode flow.
Security considerations and roadmap for improvement:
Any client mechanism for OAuth2 will want to somehow store the long-term
refresh token, otherwise the user will have to go through the entire
authorization gauntlet upon each client invocation. Ideally the tokens would be
kept in an encrypted store that must first be unlocked by the user in a way
that limits which client can access the unlocked tokens---instead of
reinventing the wheel here, leveraging OS-provided crypto stores is appealing
but doing so in a platform-independent way may be challenging at present. These
considerations aren't specific to mutt, but in the case of mutt there would be
increased opportunity for security and improved user experience as follows. At
a minimum, mutt itself could hold in memory the current access token, reusing
it until it expires, relying on the external script only at time of such
expiration. That way a user could choose not to store refresh tokens at all
(and servers might not even hand them out in the first place), and no tokens
will leak. Even better if mutt can also keep the optional refresh token in
memory, simply to be passed back and forth between mutt and the script, thus
allowing the script to maintain some "state" between invocations, thereby
allowing mutt sessions to go longer than the life of one access token (an
hour?), yet still not leaking tokens. Finally, somehow allowing the script to
interact with the user would be nice, as then the script could usually just
return a new access token if the script already has a valid refresh token, but
otherwise immediately interact with the user to complete an authcode flow or
devicecode flow---from the user perspective all of this would be occurring
seamlessly inside mutt. Whether refresh tokens get stored on disk by the script
could then be user-configurable, and how they are stored/retrieved could be
improved as better options for that become available (for one thing, allowing
the script to interact means users could be prompted for a PIN to unlock
tokens). Eventually the entire script logic could be moved inside mutt, but
perhaps reasonable to wait until many more cloud providers offer OAuth2 and the
subtle differences between their implementations have been identified---it is
easier for users to play with script variations to experiment connecting to
their cloud provider than to mess around with mutt source code. For the time
being, the proposed roadmap would keep most of the script separate but merely
move the token state inside mutt and allow the script to interact with the
user. Would that be hard to implement?
I hope this is useful to someone. Feedback/comments/questions/improvements are
always welcome.
--Alex
#!/usr/bin/python3
#
# Mutt OAuth2 token management script, version 2020-07-01
#
# Copyright (C) 2020 Alexander Perlis
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License as
# published by the Free Software Foundation; either version 2 of the
# License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301, USA.
import sys
import json
import argparse
import urllib.parse
import urllib.request
import imaplib
import poplib
import smtplib
import base64
import secrets
import hashlib
import time
from datetime import timedelta
from datetime import datetime
from pathlib import Path
registrations = {
'google': {
'authorize_endpoint': 'https://accounts.google.com/o/oauth2/auth',
'devicecode_endpoint': 'https://oauth2.googleapis.com/device/code',
'token_endpoint': 'https://accounts.google.com/o/oauth2/token',
'redirect_uri': 'urn:ietf:wg:oauth:2.0:oob',
'imap_endpoint': 'imap.gmail.com',
'pop_endpoint': 'pop.gmail.com',
'smtp_endpoint': 'smtp.gmail.com',
'sasl_method': 'OAUTHBEARER',
'scope': 'https://mail.google.com/',
'client_id': '',
'client_secret': '',
},
'microsoft': {
'authorize_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/authorize',
'devicecode_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/devicecode',
'token_endpoint': 'https://login.microsoftonline.com/common/oauth2/v2.0/token',
'redirect_uri': 'https://login.microsoftonline.com/common/oauth2/nativeclient',
'tenant': 'common',
'imap_endpoint': 'outlook.office365.com',
'pop_endpoint': 'outlook.office365.com',
'smtp_endpoint': 'smtp.office365.com',
'sasl_method': 'XOAUTH2',
'scope': 'offline_access https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send',
'client_id': '',
'client_secret': '',
},
}
ap = argparse.ArgumentParser(epilog='''
Run with "--verbose --authorize" to get started or whenever all tokens have expired.
To truly start over from scratch, first delete TOKENFILE. Use "--verbose --test" to
test the IMAP/POP/SMTP endpoints. Programs like mutt that just need an access token
should call this script without optional parameters.''')
ap.add_argument('-v', '--verbose', action='store_true', help='increase verbosity')
ap.add_argument('-d', '--debug', action='store_true', help='enable debug output')
ap.add_argument('tokenfile', help='persistent token storage')
ap.add_argument('-a', '--authorize', action='store_true', help='manually authorize new refresh token')
ap.add_argument('-t', '--test', action='store_true', help='test IMAP/POP/SMTP endpoints')
args = ap.parse_args()
f = Path(args.tokenfile)
if not f.exists(): f.touch(mode=0o600)
if 0o600 != 0o777 & f.stat().st_mode:
print('Token file has unsafe mode. Suggest deleting and starting over.')
sys.exit(1)
token = {}
with open(args.tokenfile, 'r') as f:
try: token = json.load(f)
except json.JSONDecodeError: pass
if args.debug: print('Obtained from token file:', json.dumps(token))
if not token:
if args.verbose: print('NOTICE: Empty or malformed token file.')
if not args.authorize:
print('You must run script with "--authorize" at least once.')
sys.exit(1)
print('Available app and endpoint registrations:', *registrations)
token['registration'] = input('OAuth2 registration: ')
token['oauthflow'] = input('Preferred OAuth2 flow ("authcode" or "devicecode"): ')
token['email'] = input('Account e-mail address: ')
token['access_token'] = ''
token['access_token_expiration'] = ''
token['refresh_token'] = ''
with open(args.tokenfile, 'w') as f: json.dump(token, f)
if token['registration'] not in registrations:
print(f'ERROR: Unknown registration "{token["registration"]}". Delete token file and start over.')
sys.exit(1)
registration = registrations[token['registration']]
baseparams = {'client_id': registration['client_id']}
# Microsoft uses 'tenant' but Google does not
if 'tenant' in registration: baseparams['tenant'] = registration['tenant']
def access_token_valid():
a = token['access_token_expiration']
return a and datetime.now() < datetime.fromisoformat(a)
def update_tokens(r):
token['access_token'] = r['access_token']
token['access_token_expiration'] = (datetime.now() + timedelta(seconds=int(r['expires_in']))).isoformat()
if 'refresh_token' in r: token['refresh_token'] = r['refresh_token']
with open(args.tokenfile, 'w') as f: json.dump(token, f)
if args.verbose: print(f'NOTICE: Obtained new access token, expires {token["access_token_expiration"]}.')
if args.authorize:
p = baseparams.copy();
p['scope'] = registration['scope']
if token['oauthflow'] == 'authcode':
verifier = secrets.token_urlsafe(90)
challenge = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest())[:-1]
p.update({'login_hint': token['email'],
'response_type': 'code',
'redirect_uri': registration['redirect_uri'],
'code_challenge': challenge,
'code_challenge_method': 'S256'})
print(registration["authorize_endpoint"]+'?'+urllib.parse.urlencode(p, quote_via=urllib.parse.quote))
authcode = input('Visit displayed URL to retrieve authorization code. Enter code from server (might be in browser address bar): ')
for k in 'response_type', 'login_hint', 'code_challenge', 'code_challenge_method': del p[k]
p.update({'grant_type': 'authorization_code',
'code': authcode,
'client_secret': registration['client_secret'],
'code_verifier': verifier})
try:
response = urllib.request.urlopen(registration['token_endpoint'],
urllib.parse.urlencode(p).encode())
except urllib.error.HTTPError as err:
print(err.code, err.reason)
response = err
response = response.read()
if args.debug: print(response)
response = json.loads(response)
if 'error' in response:
print(response['error'])
if 'error_description' in response: print(response['error_description'])
sys.exit(1)
elif token['oauthflow'] == 'devicecode':
try:
response = urllib.request.urlopen(registration['devicecode_endpoint'],
urllib.parse.urlencode(p).encode())
except urllib.error.HTTPError as err:
print(err.code, err.reason)
response = err
response = response.read()
if args.debug: print(response)
response = json.loads(response)
if 'error' in response:
print(response['error'])
if 'error_description' in response: print(response['error_description'])
sys.exit(1)
print(response['message'])
del p['scope']
p.update({'grant_type': 'urn:ietf:params:oauth:grant-type:device_code',
'client_secret': registration['client_secret'],
'device_code': response['device_code']})
interval = int(response['interval'])
print('Polling...', end='', flush=True)
while True:
time.sleep(interval)
print('.', end='', flush=True)
try:
response = urllib.request.urlopen(registration['token_endpoint'],
urllib.parse.urlencode(p).encode())
except urllib.error.HTTPError as err:
# Not actually always an error, might just mean "keep trying..."
response = err
response = response.read()
if args.debug: print(response)
response = json.loads(response)
if 'error' not in response: break
if response['error'] == 'authorization_declined':
print(' user declined authorization.')
sys.exit(1)
if response['error'] == 'expired_token':
print(' too much time has elapsed.')
sys.exit(1)
if response['error'] != 'authorization_pending':
print(response['error'])
if 'error_description' in response: print(response['error_description'])
sys.exit(1)
print()
else:
print(f'ERROR: Unknown OAuth2 flow "{token["oauthflow"]}. Delete token file and start over.')
sys.exit(1)
update_tokens(response)
if not access_token_valid():
if args.verbose: print('NOTICE: Invalid or expired access token; using refresh token to obtain new access token.')
if not token['refresh_token']:
print(f'ERROR: No refresh token. Manually run "{sys.argv[0]} tokenfile --authorize".')
sys.exit(1)
p = baseparams.copy()
p.update({'client_secret': registration['client_secret'],
'refresh_token': token['refresh_token'],
'grant_type': 'refresh_token'})
try:
response = urllib.request.urlopen(registration['token_endpoint'],
urllib.parse.urlencode(p).encode())
except urllib.error.HTTPError as err:
print(err.code, err.reason)
response = err
response = response.read()
if args.debug: print(response)
response = json.loads(response)
if 'error' in response:
print(response['error'])
if 'error_description' in response: print(response['error_description'])
print(f'Perhaps refresh token invalid. Try starting over with "{sys.argv[0]} tokenfile --authorize"')
sys.exit(1)
update_tokens(response)
if not access_token_valid():
print('ERROR: No valid access token. This should not be able to happen.')
sys.exit(1)
if args.verbose: print('Access Token: ', end='')
print(token['access_token'])
def build_sasl_string(user,host,port,token):
if registration['sasl_method'] == 'OAUTHBEARER':
return f'n,a={user},\1host={host}\1port={port}\1auth=Bearer {token}\1\1'
elif registration['sasl_method'] == 'XOAUTH2':
return f'user={user}\1auth=Bearer {token}\1\1'
else: print(f'Unknown SASL method {registration["sasl_method"]}.'); sys.exit(1)
if args.test:
errors = False
imap_conn = imaplib.IMAP4_SSL(registration['imap_endpoint'])
sasl_string = build_sasl_string(token['email'], registration['imap_endpoint'], 993, token['access_token'])
if args.debug: imap_conn.debug = 4
try:
imap_conn.authenticate(registration['sasl_method'], lambda _: sasl_string.encode())
if args.verbose: print('IMAP authentication succeeded')
except imaplib.IMAP4.error as e:
print('IMAP authentication FAILED (does your account allow IMAP?):', e)
errors = True
pop_conn = poplib.POP3_SSL(registration['pop_endpoint'])
sasl_string = build_sasl_string(token['email'], registration['pop_endpoint'], 995, token['access_token'])
if args.debug: pop_conn.set_debuglevel(2)
try:
# poplib doesn't have an auth command taking an authenticator object
# Microsoft requires a two-line SASL for POP
pop_conn._shortcmd('AUTH ' + registration['sasl_method'])
pop_conn._shortcmd(base64.standard_b64encode(sasl_string.encode()).decode())
if args.verbose: print('POP authentication succeeded')
except poplib.error_proto as e:
print('POP authentication FAILED (does your account allow POP?):', e)
errors = True
# SMTP_SSL would be simpler but Microsoft does not answer on port 465.
smtp_conn = smtplib.SMTP(registration['smtp_endpoint'], 587)
sasl_string = build_sasl_string(token['email'], registration['smtp_endpoint'], 587, token['access_token'])
smtp_conn.ehlo('test')
smtp_conn.starttls()
smtp_conn.ehlo('test')
if args.debug: smtp_conn.set_debuglevel(2)
try:
smtp_conn.auth(registration['sasl_method'], lambda _ = None: sasl_string)
if args.verbose: print('SMTP authentication succeeded')
except smtplib.SMTPAuthenticationError as e:
print('SMTP authentication FAILED:', e)
errors = True
if errors: sys.exit(1)
