Migrate the application to Python 3 (#4251)

* Make core app compatible with Python 3

No backward compatibility with Python 2.7 is kept.
This commit mostly contains changes made with 2to3 and manual
tweaking when necessary.

* Use Python 3.7 as base docker image

Since it is not possible to change redash/base:debian to Python 3
without breaking future relases, its Dockerfile is temporarly
copied here.

* Upgrade some requirements to newest versions

Some of the older versions were not compatible with Python 3.

* Migrate tests to Python 3

* Build frontend on Python 3

* Make the HMAC sign function compatible with Python 3

In Python 3, HMAC only works with bytes so the strings and the
float used in the sign function need to be encoded.
Hopefully this is still backward compatible with already generated
signatures.

* Use assertCountEqual instead of assertItemsEqual

The latter is not available in Python 3.
See https://bugs.python.org/issue17866

* Remove redundant encoding header for Python 3 modules

* Remove redundant string encoding in CLI

* Rename list() functions in CLI

These functions shadow the builtin list function which is
problematic since 2to3 adds a fair amount of calls to the builtin
list when it finds dict.keys() and dict.values().

Only the Python function is renamed, from the perspective of the
CLI nothing changes.

* Replace usage of Exception.message in CLI

`message` is not available anymore, instead use the string
representation of the exception.

* Adapt test handlers to Python 3

* Fix test that relied on dict ordering

* Make sure test results are always uploaded (#4215)

* Support encoding memoryview to JSON

psycopg2 returns `buffer` objects in Python 2.7 and `memoryview`
in Python 3. See #3156

* Fix test relying on object address ordering

* Decode bytes returned from Redis

* Stop using e.message for most exceptions

Exception.message is not available in Python 3 anymore, except
for some exceptions defined by third-party libraries.

* Fix writing XLSX files in Python 3

The buffer for the file should be made of bytes and the actual
content written to it strings.

Note: I do not know why the diff is so large as it's only a two
lines change. Probably a white space or file encoding issue.

* Fix test by comparing strings to strings

* Fix another exception message unavailable in Python 3

* Fix export to CSV in Python 3

The UnicodeWriter is not used anymore. In Python 3, the interface
provided by the CSV module only deals with strings, in and out.
The encoding of the output is left to the user, in our case
it is given to Flask via `make_response`.

* (Python 3) Use Redis' decode_responses=True option (#4232)

* Fix test_outdated_queries_works_scheduled_queries_tracker (use utcnow)

* Make sure Redis connection uses decoded_responses option

* Remove unused imports.

* Use Redis' decode_responses option

* Remove cases of explicit Redis decoding

* Rename helper function and make sure it doesn't apply twice.

* Don't add decode_responses to Celery Redis connection URL

* Fix displaying error while connecting to SQLite

The exception message is always a string in Python 3, so no
need to try to decode things.

* Fix another missing exception message

* Handle JSON encoding for datasources returning bytes

SimpleJSON assumes the bytes it receives contain text data, so it
tries to UTF-8 encode them. It is sometimes not true, for instance
the SQLite datasource returns bytes for BLOB types, which typically
do not contain text but truly binary data.

This commit disables SimpleJSON auto encoding of bytes to str and
instead uses the same method as for memoryviews: generating a
hex representation of the data.

* Fix Python 3 compatibility with RQ

* Revert some changes 2to3 tends to do (#4261)

- Revert some changes 2to3 tends to do when it errs on the side of caution regarding dict view objects.

- Also fixed some naming issues with one character variables in list comprehensions.

- Fix Flask warning.

* Upgrade dependencies

* Remove useless `iter` added by 2to3

* Fix get_next_path tests (#4280)

* Removed setting SERVER_NAME in tests setup to avoid a warning.

* Change get_next_path to not return empty string in case of a domain only value.

* Fix redirect tests:

Since version 0.15 of Werkzeug it uses full path for fixing the location header instead of the root path.

* Remove explicit dependency for Werkzeug

* Switched pytz and certifi to unbinded versions.

* Switch to new library for getting country from IP

`python-geoip-geolite2` is not compatible with Python 3, instead
use `maxminddb-geolite2` which is very similar as it includes
the geolite2 database in the package .

* Python 3 RQ modifications (#4281)

* show current worker job (alongside with minor cosmetic column tweaks)

* avoid loading entire job data for queued jobs

* track general RQ queues (default, periodic and schemas)

* get all active RQ queues

* call get_celery_queues in another place

* merge dicts the Python 3 way

* extend the result_ttl of refresh_queries to 600 seconds to allow it to continue running periodically even after longer executions

* Remove legacy Python flake8 tests

.circleci/config.yml

@@ -1,18 +1,13 @@
 version: 2.0
 
-flake8-steps: &steps
-  - checkout
-  - run: sudo pip install flake8
-  - run: ./bin/flake8_tests.sh
 jobs:
   python-flake8-tests:
     docker:
       - image: circleci/python:3.7.0
-    steps: *steps
-  legacy-python-flake8-tests:
-    docker:
-      - image: circleci/python:2.7.15
-    steps: *steps
+    steps:
+      - checkout
+      - run: sudo pip install flake8
+      - run: ./bin/flake8_tests.sh
   backend-unit-tests:
     environment:
       COMPOSE_FILE: .circleci/docker-compose.circle.yml
@@ -44,6 +39,7 @@ jobs:
             mkdir -p /tmp/test-results/unit-tests
             docker cp tests:/app/coverage.xml ./coverage.xml
             docker cp tests:/app/junit.xml /tmp/test-results/unit-tests/results.xml
+          when: always
       - store_test_results:
           path: /tmp/test-results
       - store_artifacts:
@@ -63,8 +59,8 @@ jobs:
       - image: circleci/node:8
     steps:
       - checkout
-      - run: sudo apt install python-pip
-      - run: sudo pip install -r requirements_bundles.txt
+      - run: sudo apt install python3-pip
+      - run: sudo pip3 install -r requirements_bundles.txt
       - run: npm install
       - run: npm run bundle
       - run: npm test
@@ -99,8 +95,8 @@ jobs:
     steps:
       - setup_remote_docker
       - checkout
-      - run: sudo apt install python-pip
-      - run: sudo pip install -r requirements_bundles.txt
+      - run: sudo apt install python3-pip
+      - run: sudo pip3 install -r requirements_bundles.txt
       - run: .circleci/update_version
       - run: npm run bundle
       - run: .circleci/docker_build
@@ -109,7 +105,6 @@ workflows:
   build:
     jobs:
       - python-flake8-tests
-      - legacy-python-flake8-tests
       - backend-unit-tests
       - frontend-lint
       - frontend-unit-tests:

Dockerfile

@@ -8,11 +8,40 @@ COPY client /frontend/client
 COPY webpack.config.js /frontend/
 RUN npm run build
 
-FROM redash/base:debian
+FROM python:3.7-slim
+
+EXPOSE 5000
 
 # Controls whether to install extra dependencies needed for all data sources.
 ARG skip_ds_deps
 
+RUN useradd --create-home redash
+
+# Ubuntu packages
+RUN apt-get update && \
+  apt-get install -y \
+    curl \
+    gnupg \
+    build-essential \
+    pwgen \
+    libffi-dev \
+    sudo \
+    git-core \
+    wget \
+    # Postgres client
+    libpq-dev \
+    # for SAML
+    xmlsec1 \
+    # Additional packages required for data sources:
+    libssl-dev \
+    default-libmysqlclient-dev \
+    freetds-dev \
+    libsasl2-dev && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
+
+WORKDIR /app
+
 # We first copy only the requirements file, to avoid rebuilding on every file
 # change.
 COPY requirements.txt requirements_bundles.txt requirements_dev.txt requirements_all_ds.txt ./

bin/bundle-extensions

@@ -1,9 +1,8 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
+#!/usr/bin/env python3
 """Copy bundle extension files to the client/app/extension directory"""
 import logging
 import os
-from pathlib2 import Path
+from pathlib import Path
 from shutil import copy
 from collections import OrderedDict as odict
 

bin/get_changes.py

@@ -1,9 +1,10 @@
-#!/bin/env python
-from __future__ import print_function
+#!/bin/env python3
+
 import sys
 import re
 import subprocess
 
+
 def get_change_log(previous_sha):
     args = ['git', '--no-pager', 'log', '--merges', '--grep', 'Merge pull request', '--pretty=format:"%h|%s|%b|%p"', 'master...{}'.format(previous_sha)]
     log = subprocess.check_output(args)
@@ -33,4 +34,4 @@ def get_change_log(previous_sha):
     changes = get_change_log(previous_sha)
 
     for change in changes:
-        print(change)
+        print(change)

bin/release_manager.py

@@ -1,4 +1,4 @@
-from __future__ import print_function
+#!/usr/bin/env python3
 import os
 import sys
 import re

bin/upgrade

@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 import urllib
 import argparse
 import os
@@ -27,7 +27,7 @@ def run(cmd, cwd=None):
 
 
 def confirm(question):
-    reply = str(raw_input(question + ' (y/n): ')).lower().strip()
+    reply = str(input(question + ' (y/n): ')).lower().strip()
 
     if reply[0] == 'y':
         return True

migrations/versions/65fc9ede4746_add_is_draft_status_to_queries_and_.py

@@ -1,11 +1,11 @@
 """Add is_draft status to queries and dashboards
 
 Revision ID: 65fc9ede4746
-Revises: 
+Revises:
 Create Date: 2016-12-07 18:08:13.395586
 
 """
-from __future__ import print_function
+
 from alembic import op
 import sqlalchemy as sa
 
@@ -26,7 +26,7 @@ def upgrade():
         op.execute("UPDATE dashboards SET is_draft = false")
     except ProgrammingError as e:
         # The columns might exist if you ran the old migrations.
-        if 'column "is_draft" of relation "queries" already exists' in e.message:
+        if 'column "is_draft" of relation "queries" already exists' in str(e):
             print("Can't run this migration as you already have is_draft columns, please run:")
             print("./manage.py db stamp {} # you might need to alter the command to match your environment.".format(revision))
             exit()

migrations/versions/969126bd800f_.py

@@ -5,7 +5,7 @@
 Create Date: 2018-01-31 15:20:30.396533
 
 """
-from __future__ import print_function
+
 import simplejson
 from alembic import op
 import sqlalchemy as sa

redash/__init__.py

@@ -2,8 +2,6 @@
 import logging
 import os
 import sys
-import urllib
-import urlparse
 
 import redis
 from flask_mail import Mail
@@ -41,6 +39,7 @@ def setup_logging():
 setup_logging()
 
 redis_connection = redis.from_url(settings.REDIS_URL)
+rq_redis_connection = redis.from_url(settings.RQ_REDIS_URL)
 mail = Mail()
 migrate = Migrate()
 statsd_client = StatsClient(host=settings.STATSD_HOST, port=settings.STATSD_PORT, prefix=settings.STATSD_PREFIX)

redash/app.py

@@ -10,7 +10,7 @@ def __init__(self, *args, **kwargs):
         kwargs.update({
             'template_folder': settings.STATIC_ASSETS_PATH,
             'static_folder': settings.STATIC_ASSETS_PATH,
-            'static_path': '/static',
+            'static_url_path': '/static',
         })
         super(Redash, self).__init__(__name__, *args, **kwargs)
         # Make sure we get the right referral address even behind proxies like nginx.

redash/authentication/__init__.py

@@ -2,7 +2,7 @@
 import hmac
 import logging
 import time
-from urlparse import urlsplit, urlunsplit
+from urllib.parse import urlsplit, urlunsplit
 
 from flask import jsonify, redirect, request, url_for
 from flask_login import LoginManager, login_user, logout_user, user_logged_in
@@ -33,8 +33,8 @@ def sign(key, path, expires):
     if not key:
         return None
 
-    h = hmac.new(str(key), msg=path, digestmod=hashlib.sha1)
-    h.update(str(expires))
+    h = hmac.new(key.encode(), msg=path.encode(), digestmod=hashlib.sha1)
+    h.update(str(expires).encode())
 
     return h.hexdigest()
 
@@ -93,7 +93,7 @@ def hmac_load_user_from_request(request):
             calculated_signature = sign(query.api_key, request.path, expires)
 
             if query.api_key and signature == calculated_signature:
-                return models.ApiUser(query.api_key, query.org, query.groups.keys(), name="ApiKey: Query {}".format(query.id))
+                return models.ApiUser(query.api_key, query.org, list(query.groups.keys()), name="ApiKey: Query {}".format(query.id))
 
     return None
 
@@ -118,7 +118,7 @@ def get_user_from_api_key(api_key, query_id):
             if query_id:
                 query = models.Query.get_by_id_and_org(query_id, org)
                 if query and query.api_key == api_key:
-                    user = models.ApiUser(api_key, query.org, query.groups.keys(), name="ApiKey: Query {}".format(query.id))
+                    user = models.ApiUser(api_key, query.org, list(query.groups.keys()), name="ApiKey: Query {}".format(query.id))
 
     return user
 
@@ -271,4 +271,10 @@ def get_next_path(unsafe_next_path):
     parts[1] = ''  # clear netloc
     safe_next_path = urlunsplit(parts)
 
+    # If the original path was a URL, we might end up with an empty
+    # safe url, which will redirect to the login page. Changing to 
+    # relative root to redirect to the app root after login.
+    if not safe_next_path:
+        safe_next_path = './'
+
     return safe_next_path

redash/authentication/account.py

@@ -48,7 +48,7 @@ def send_verify_email(user, org):
     }
     html_content = render_template('emails/verify.html', **context)
     text_content = render_template('emails/verify.txt', **context)
-    subject = u"{}, please verify your email address".format(user.name)
+    subject = "{}, please verify your email address".format(user.name)
 
     send_mail.delay([user.email], subject, html_content, text_content)
 
@@ -57,7 +57,7 @@ def send_invite_email(inviter, invited, invite_url, org):
     context = dict(inviter=inviter, invited=invited, org=org, invite_url=invite_url)
     html_content = render_template('emails/invite.html', **context)
     text_content = render_template('emails/invite.txt', **context)
-    subject = u"{} invited you to join Redash".format(inviter.name)
+    subject = "{} invited you to join Redash".format(inviter.name)
 
     send_mail.delay([invited.email], subject, html_content, text_content)
 
@@ -67,7 +67,7 @@ def send_password_reset_email(user):
     context = dict(user=user, reset_link=reset_link)
     html_content = render_template('emails/reset.html', **context)
     text_content = render_template('emails/reset.txt', **context)
-    subject = u"Reset your password"
+    subject = "Reset your password"
 
     send_mail.delay([user.email], subject, html_content, text_content)
     return reset_link
@@ -76,6 +76,6 @@ def send_password_reset_email(user):
 def send_user_disabled_email(user):
     html_content = render_template('emails/reset_disabled.html', user=user)
     text_content = render_template('emails/reset_disabled.txt', user=user)
-    subject = u"Your Redash account is disabled"
+    subject = "Your Redash account is disabled"
 
     send_mail.delay([user.email], subject, html_content, text_content)

redash/cli/__init__.py

@@ -1,4 +1,4 @@
-from __future__ import print_function
+
 
 import click
 import simplejson
@@ -53,7 +53,7 @@ def status():
 @manager.command()
 def check_settings():
     """Show the settings as Redash sees them (useful for debugging)."""
-    for name, item in current_app.config.iteritems():
+    for name, item in current_app.config.items():
         print("{} = {}".format(name, item))
 
 

redash/cli/data_sources.py

@@ -1,4 +1,4 @@
-from __future__ import print_function
+
 from sys import exit
 
 import click
@@ -15,11 +15,11 @@
 manager = AppGroup(help="Data sources management commands.")
 
 
-@manager.command()
+@manager.command(name='list')
 @click.option('--org', 'organization', default=None,
               help="The organization the user belongs to (leave blank for "
               "all organizations).")
-def list(organization=None):
+def list_command(organization=None):
     """List currently configured data sources."""
     if organization:
         org = models.Organization.get_by_slug(organization)
@@ -99,11 +99,11 @@ def new(name=None, type=None, options=None, organization='default'):
             print("{}. {}".format(i + 1, query_runner_name))
 
         idx = 0
-        while idx < 1 or idx > len(query_runners.keys()):
+        while idx < 1 or idx > len(list(query_runners.keys())):
             idx = click.prompt("[{}-{}]".format(1, len(query_runners.keys())),
                                type=int)
 
-        type = query_runners.keys()[idx - 1]
+        type = list(query_runners.keys())[idx - 1]
     else:
         validate_data_source_type(type)
 
@@ -119,7 +119,7 @@ def new(name=None, type=None, options=None, organization='default'):
 
         options_obj = {}
 
-        for k, prop in schema['properties'].iteritems():
+        for k, prop in schema['properties'].items():
             required = k in schema.get('required', [])
             default_value = "<<DEFAULT_VALUE>>"
             if required:

redash/cli/groups.py

@@ -1,4 +1,4 @@
-from __future__ import print_function
+
 from sys import exit
 
 from sqlalchemy.orm.exc import NoResultFound
@@ -36,7 +36,7 @@ def create(name, permissions=None, organization='default'):
             permissions=permissions))
         models.db.session.commit()
     except Exception as e:
-        print("Failed create group: %s" % e.message)
+        print("Failed create group: %s" % e)
         exit(1)
 
 
@@ -67,7 +67,7 @@ def change_permissions(group_id, permissions=None):
         models.db.session.add(group)
         models.db.session.commit()
     except Exception as e:
-        print("Failed change permission: %s" % e.message)
+        print("Failed change permission: %s" % e)
         exit(1)
 
 
@@ -80,10 +80,10 @@ def extract_permissions_string(permissions):
     return permissions
 
 
-@manager.command()
+@manager.command(name='list')
 @option('--org', 'organization', default=None,
         help="The organization to limit to (leave blank for all).")
-def list(organization=None):
+def list_command(organization=None):
     """List all groups"""
     if organization:
         org = models.Organization.get_by_slug(organization)