Skip to content

Commit

Permalink
Update to Superset version 3.1.0
Browse files Browse the repository at this point in the history
We update the build process so as to build from the upstream superset
source, rather than just installing packages from pypi.

Add the superset[trino,hive,postgresql,spark] extras in order to
generate a more complete set of dependencies.

In addition to this, we have temporarily applied the patch
to fix this issue: apache/superset#25636

Bug: T335356
Change-Id: I9a47ae07a106a1481fdffe03f670ced4c46b262d
  • Loading branch information
tullis committed Jan 24, 2024
1 parent 5175ad7 commit 580b45a
Show file tree
Hide file tree
Showing 238 changed files with 41 additions and 24 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
virtualenv
venv
superset_upstream
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/Flask-1.1.4-py2.py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/Flask-2.2.5-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/Jinja2-3.1.2-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/Markdown-3.4.1-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/PyHive-0.6.2-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/PyHive-0.7.0-py3-none-any.whl
Binary file not shown.
Binary file removed artifacts/bullseye/PyJWT-1.7.1-py2.py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/PyJWT-2.4.0-py3-none-any.whl
Binary file not shown.
Binary file modified artifacts/bullseye/PyMeeus-0.5.12-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/PySocks-1.7.1-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified artifacts/bullseye/WTForms_JSON-0.3.5-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/aiosignal-1.3.1-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/alembic-1.9.2-py3-none-any.whl
Binary file not shown.
Binary file removed artifacts/bullseye/amqp-2.6.1-py2.py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/amqp-5.1.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/apispec-6.3.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/attrs-22.2.0-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/attrs-23.1.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/backoff-2.2.1-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/cachelib-0.4.1-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/cattrs-23.2.1-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/celery-5.2.2-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/click-7.1.2-py2.py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/click-8.1.3-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/dnspython-2.3.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified artifacts/bullseye/func_timeout-4.3.5-py3-none-any.whl
Binary file not shown.
Binary file modified artifacts/bullseye/future-0.18.3-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/gunicorn-20.1.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/h11-0.14.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/holidays-0.10.3-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/humanize-4.5.0-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/idna-3.2-py3-none-any.whl
Binary file not shown.
Binary file removed artifacts/bullseye/idna-3.4-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/kombu-5.2.4-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/limits-3.4.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/mdurl-0.1.2-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/packaging-23.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified artifacts/bullseye/pgsanity-0.2.9-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file modified artifacts/bullseye/pylibmc-1.6.1-cp39-cp39-linux_x86_64.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/pyspnego-0.7.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/redis-4.4.2-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/redis-4.5.4-py3-none-any.whl
Binary file not shown.
Binary file removed artifacts/bullseye/requests-2.28.2-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/rich-13.3.4-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/selenium-4.8.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file added artifacts/bullseye/shortid-0.1.2-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/sniffio-1.3.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/trino-0.321.0-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/trino-0.327.0-py3-none-any.whl
Binary file not shown.
Binary file removed artifacts/bullseye/trio-0.22.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/tzlocal-4.2-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/tzlocal-5.2-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/vine-1.3.0-py2.py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/wsproto-1.2.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed artifacts/bullseye/zipp-3.12.0-py3-none-any.whl
Binary file not shown.
Binary file added artifacts/bullseye/zipp-3.15.0-py3-none-any.whl
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
6 changes: 3 additions & 3 deletions build_frontend.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,13 @@ set -ex
apt-get update
apt-get install -y curl ca-certificates gpg

# Taken from https://deb.nodesource.com/setup_14.x
# Taken from https://deb.nodesource.com/setup_16.x
keyring='/usr/share/keyrings'
node_key_url='https://deb.nodesource.com/gpgkey/nodesource.gpg.key'
local_node_key="$keyring/nodesource.gpg"
curl -s $node_key_url | gpg --dearmor | tee $local_node_key >/dev/null
echo "deb [signed-by=$local_node_key] https://deb.nodesource.com/node_14.x buster main" > /etc/apt/sources.list.d/nodesource.list
echo "deb-src [signed-by=$local_node_key] https://deb.nodesource.com/node_14.x buster main" >> /etc/apt/sources.list.d/nodesource.list
echo "deb [signed-by=$local_node_key] https://deb.nodesource.com/node_16.x bullseye main" > /etc/apt/sources.list.d/nodesource.list
echo "deb-src [signed-by=$local_node_key] https://deb.nodesource.com/node_16.x bullseye main" >> /etc/apt/sources.list.d/nodesource.list

apt-get update
apt-get install -y nodejs
Expand Down
15 changes: 10 additions & 5 deletions build_wheels.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@

set -ex

export SUPERSET_VERSION=3.1.0

# source common variables
source $(dirname $0)/profile.sh

Expand All @@ -20,12 +22,13 @@ sudo apt-get --yes install \
libffi-dev \
libsasl2-dev \
libldap2-dev \
default-libmysqlclient-dev
default-libmysqlclient-dev \
pkg-config

# This will be set to the mtime of frozen-requirements.txt
# This makes it more likely that wheels built with the same versions
# will be byte for byte identical.
export SOURCE_DATE_EPOCH=$(stat -c %Y ${deploy_dir}/frozen-requirements.txt)
export SOURCE_DATE_EPOCH=$(stat -c %Y /superset_upstream/requirements/base.txt)

# Build in a virtualenv, but install wheel files to $wheels_dir.
build_venv=/tmp/superset-build-venv-${SOURCE_DATE_EPOCH}
Expand All @@ -37,8 +40,10 @@ test -e $build_venv && rm -rf $build_venv
rm -rf $wheels_dir
mkdir -p $wheels_dir

cd /superset_deploy/superset_upstream

# Update pip to be able to install the manylinux2010 and later wheel format.
$build_venv/bin/pip install --upgrade pip==22.3.1
$build_venv/bin/pip install --upgrade pip==23.2.1
# Package pip itself as a wheel, so that pip can install the new wheels upon deployment
$build_venv/bin/pip wheel -w $wheels_dir pip==22.3.1
$build_venv/bin/pip wheel --trusted-host pypi.org --trusted-host files.pythonhosted.org -w $wheels_dir -r $deploy_dir/$requirements_file
$build_venv/bin/pip wheel -w ../$wheels_dir pip==23.2.1
$build_venv/bin/pip wheel --trusted-host pypi.org --trusted-host files.pythonhosted.org -w ../$wheels_dir -r requirements/base.txt -r /superset_deploy/$requirements_file
2 changes: 1 addition & 1 deletion create_virtualenv.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,6 @@ virtualenv --python python3 --never-download $venv
# Instead, just install all wheels in the wheels dir.

# First upgrade pip to pip-22 so that newer wheels will be supported
$venv/bin/pip install --no-index --find-links $wheels_dir $wheels_dir/pip-22.3.1-py3-none-any.whl
$venv/bin/pip install --no-index --force-reinstall --find-links $wheels_dir $wheels_dir/pip-23.2.1-py3-none-any.whl

$venv/bin/pip install --no-deps --no-index --find-links $wheels_dir $wheels_dir/*.whl
16 changes: 2 additions & 14 deletions frozen-requirements.txt
Original file line number Diff line number Diff line change
@@ -1,16 +1,4 @@
pyhive[kerberos,presto]==0.6.2
pyhive[kerberos,presto]==0.7.0
gunicorn[gevent]
apache-superset[presto,mysql,druid]==1.5.3
apache-superset[hive,presto,mysql,druid,trino,spark,postgres]
pylibmc==1.6.1
trino

# Pinning markupsafe because of this issue:
# https://github.com/pallets/markupsafe/issues/282
# If superset upgrades flask or jinja, it may become unnecessary.
markupsafe==2.0.1

# Pinning importlib-metadata because of issues with version 5.0 and above
importlib-metadata==4.13.0

# Pinning cryptography because of issues with version 39.0.0 and above
cryptography==38.0.4
25 changes: 24 additions & 1 deletion prep_env_for_docker.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,29 @@

set -xe

export SUPERSET_VERSION=3.1.0

apt-get update
apt-get upgrade -y
apt-get install -y sudo lsb-release libkrb5-dev libmemcached-dev
apt-get install -y sudo lsb-release libkrb5-dev libmemcached-dev git ca-certificates curl

# n.b The following source download and verify is disabled because the checksums are currently wrong - See: https://github.com/apache/superset/issues/25333
# Download the superset source from Apache's repository and verify its authenticity
# curl --remote-name-all https://downloads.apache.org/superset/${SUPERSET_VERSION}/apache-superset-${SUPERSET_VERSION}-source.tar.gz
# curl --remote-name-all https://downloads.apache.org/superset/${SUPERSET_VERSION}/apache-superset-${SUPERSET_VERSION}-source.tar.gz.{asc,sha512}
# curl https://downloads.apache.org/superset/KEYS | gpg --import &&
# gpg --verify apache-superset-${SUPERSET_VERSION}-source.tar.gz.asc
# cat apache-superset-${SUPERSET_VERSION}-source.tar.gz.sha512 | sha512sum --check --strict
# Extract the superset source
#tar xzvf apache-superset-${SUPERSET_VERSION}-source.tar.gz
cd /superset_deploy

git config --global --add safe.directory /superset_deploy

git clone --branch ${SUPERSET_VERSION} --single-branch https://github.com/apache/superset.git superset_upstream ||
( cd superset_upstream; git checkout ${SUPERSET_VERSION} )

# The following is a temporary workaround for the issues identifed in https://phabricator.wikimedia.org/T335356#9478404
# It can be removed as soon as https://github.com/apache/superset/pull/26782 is approved by the upstream project and released.
cd /superset_deploy/superset_upstream
curl https://github.com/brouberol/superset/commit/6cab1dcd13f0466f86e3e5139e7e3e0cbe5e58ae.patch | git apply -v --index

0 comments on commit 580b45a

Please sign in to comment.