two questions about Dockerizing #108
Replies: 4 comments 28 replies
-
|
Hey @paulkaefer , not yet.. however v1.5.0 will include support for pulling the config information from environment variables or a For syncers, we have a secondary install process (so we don't eagerly install all dependencies when building the initial environment). Snowflake's got some issues with SQLAlchemy 2.0 support that I had to monkeypatch in the 1.5.0 series as well. It might make sense for us to get on a call to resolve these issues for you, but I'd probably also recommend trying to install from 1.5.0 beta instead of working on 1.4.x series, since it'll just be more complicated to get working 😅 WDYT ? |
Beta Was this translation helpful? Give feedback.
-
|
@boonhapus a call would be excellent; I will work with you to schedule it. |
Beta Was this translation helpful? Give feedback.
-
|
@paulkaefer with 1.5.0 released , I think we can more meaningfully talk about Dockerizing or serverless approaches to CS Tools. I'll offer an example for GitHub Actions (that I'm currently still tinkering with...) to get us started. Maybe you can share a (public friendly) Dockerfile and we can work through it together? It's been close to 5 years since I touched dockerfiles directly... I'm a Portainer snob 😄 https://github.com/thoughtspot/cs_tools/blob/master/.github/workflows/fetch-bi-data.yaml 👉 click to show .yamlname:
Extract data with CS Tools.
on:
workflow_dispatch:
schedule:
# Runs every day at 5:15 AM UTC
- cron: "15 5 * * *"
jobs:
extract_data_from_thoughtspot:
# Configure Environment Variables for CS Tools configuration
env:
CS_TOOLS_THOUGHTSPOT__URL: ${{ secrets.THOUGHTSPOT_URL }}
CS_TOOLS_THOUGHTSPOT__USERNAME: ${{ secrets.THOUGHTSPOT_USERNAME }}
CS_TOOLS_THOUGHTSPOT__SECRET_KEY: ${{ secrets.THOUGHTSPOT_SECRET_KEY }}
# CS_TOOLS_TEMP_DIR: ...
runs-on: ubuntu-latest
steps:
- name: Get 7 days ago
run: echo "days_ago_7=$(date -d "-7 days" +'%Y-%m-%d')" >> $GITHUB_ENV
- name: Get 1 days ago
run: echo "days_ago_1=$(date -d "-1 days" +'%Y-%m-%d')" >> $GITHUB_ENV
- name: Check out the repository main branch
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v4
with:
python-version: 3.12
- name: Install a specific version of CS Tools
run: python -m pip install -e .[cli]
# --config ENV: tells CS Tools to pull the information from environment variables.
- name: Grab N-7 to N-1 days of BI Server Data
run: "cs_tools tools searchable bi-server --from-date $days_ago_7 --to-date $days_ago_1 --syncer ${{ secrets.SYNCER_DECLARATIVE_STRING }} --config ENV:"The idea behind this is the mandatory properties on a configuration file are really just the authentication details.. so url, username, and secret.
in addition to specifying the You'll also see that I've hid the Syncer config in an environment variable as well. This is primarily so I don't need to have a second file to mess around with... but with Docker you can connect volumes and reference files from them instead of a pure serverless environment like Github Actions. |
Beta Was this translation helpful? Give feedback.
-
|
@boonhapus this is exciting! Your GitHub Actions example looks straightforward & could be something we explore down the road. One comment, though: if it's scheduled to run daily, do you really want it to pull Here's our current Dockerfile. You'll notice I don't have the FROM python:3.11.4-slim as base
USER root
WORKDIR /home/thoughtspot
RUN apt-get update -y && apt-get upgrade -y
RUN pip install https://github.com/thoughtspot/cs_tools/archive/v1.5.0.zip
COPY bi-server.toml .
COPY gather.toml .
COPY process_thoughtspot_credentials.py .
ENTRYPOINT ["bash"] |
Beta Was this translation helpful? Give feedback.
-
You're correct @paulkaefer , it's overkill. ThoughtSpot typically is pretty good at ensuring data loads are timely and complete, but this give me some buffer against that problem if it were to ever occur. The key here is using the new
CS Tools is actually multiple python packages in one. Your What responsibility is your |
Beta Was this translation helpful? Give feedback.
-
|
@boonhapus makes sense.
|
Beta Was this translation helpful? Give feedback.
-
|
@paulkaefer yeah exactly. If you've got some better way of setting envvars, I'd probably recommend it over using python to set the container's vars. Also I think your final line should read something like.. Replacing the Try it out and let me know! |
Beta Was this translation helpful? Give feedback.
-
|
@boonhapus I'm getting an error when trying to install the |
Beta Was this translation helpful? Give feedback.
-
|
@paulkaefer try this variation .. You might need to wrap it in quotes, depends on how docker handles the args.. |
Beta Was this translation helpful? Give feedback.
-
|
@boonhapus ok, it seems like the click to see full error message |
Beta Was this translation helpful? Give feedback.
-
|
Next step would be to try to replicate (based on same config) in Docker. Should be able to find your direct config details by running I may introduce a breaking change in your workflow in 1.5.2 or 1.5.3 for that encrypted password though. I'll detail this in release notes so you know when to change the envvar. |
Beta Was this translation helpful? Give feedback.
-
|
@boonhapus ok, the duplication answer sounds good to me. Copying my local config in (and changing the temp path) moves it along... I saw the emoji tracker. It errored after that, so I pip installed 1.5.1 (will update the Dockerfile, too). Now I get |
Beta Was this translation helpful? Give feedback.
-
|
That sounds odd @paulkaefer ... can you share your latest docker file? Naturally please remove any sensitive details. |
Beta Was this translation helpful? Give feedback.
-
latest DockerfileFROM python:3.11.4-slim as base
USER root
WORKDIR /home/thoughtspot
RUN apt-get update -y && apt-get upgrade -y
# REDACTED: update /etc/ssl/certs
RUN pip install --upgrade pip
RUN pip install "cs_tools[cli] @ https://github.com/thoughtspot/cs_tools/archive/v1.5.1.zip"
# if we don't set this, `cs_tools --version` will have errors:
RUN export CI=dev
COPY bi-server.toml .
COPY gather.toml .
# copies snowflake username/password from env var to .toml files:
COPY process_thoughtspot_credentials.py .
ENTRYPOINT ["bash"] |
Beta Was this translation helpful? Give feedback.
-
|
Hm. Let's get a call together and close this one out @paulkaefer . I'll shoot you an email. |
Beta Was this translation helpful? Give feedback.
-
cs_tools config createthat will bypass the two prompts? I tried--silentbut that doesn't seem to do it.cs_tools tools searchablewithin the container, I get the below message. If I manually installplatformdirs==3.11.0it gets farther. Is this a bug?Beta Was this translation helpful? Give feedback.
All reactions