feat(object storage): add unused object storage #9846
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hazzadous
reviewed
May 19, 2022
hazzadous
reviewed
May 19, 2022
docker-compose.dev.yml
Outdated
| ports: | ||
| - '19000:19000' | ||
| - '19001:19001' | ||
| volumes: |
There was a problem hiding this comment.
For dev compose files, could we remove these volumes? I believe the Dockerfile already specifies that /data is a volume and should create an anonymous volume for us? Otherwise we might need to add ./object_storage to gitignore
There was a problem hiding this comment.
object_storage is in ./gitignore already... but no point having things in the docker-compose file that are unnecessary? 👍
hazzadous
reviewed
May 19, 2022
| ports: | ||
| - '19000:19000' | ||
| - '19001:19001' | ||
| volumes: |
There was a problem hiding this comment.
For hobby, I'd add a named volume as with zookeeper and reference this rather than ./object_storage, to be in line with the other services.
hazzadous
reviewed
May 19, 2022
hazzadous
reviewed
May 19, 2022
hazzadous
reviewed
May 19, 2022
hazzadous
reviewed
May 19, 2022
hazzadous
suggested changes
May 19, 2022
posthog/storage/object_storage.py
Outdated
| if not s3_client: | ||
| s3_client = client( | ||
| "s3", | ||
| endpoint_url=f"http://{OBJECT_STORAGE_HOST}:{OBJECT_STORAGE_PORT}", |
There was a problem hiding this comment.
Nit but I think we should simply provide the whole endpoint_url as {OBJECT_STORAGE_ENDPOINT}. Hardcoding http is already a warning flag to me
guidoiaquinti
approved these changes
May 19, 2022
hazzadous
approved these changes
May 19, 2022
fuziontech
added a commit
that referenced
this pull request
May 20, 2022
* master: chore: start stack once in cloud tests (#9879) feat(apps): frontend apps (#9831) chore: Fix snapshots on master (#9885) chore(apps): rename plugins to apps (#9755) refactor: Remove constance library dependency, use json-encoded model (#9852) chore(clickhouse): avoid creating kafka_events, events_mv (#9863) fix(insights): Fix timezone date issues (#9678) refactor(plugin-server): refactor the event pipeline (#9829) feat(object storage): add unused object storage (#9846) fix: make kafka health check timeout test reliable (#9857) fix: query elements from start of day (#9827)
alexkim205
pushed a commit
that referenced
this pull request
May 23, 2022
* feat(object_storage): add unused object storage with health checks * only prompt debug users if object storage not available at preflight * safe plugin server health check for unused object storage * explicit object storage settings * explicit object storage settings * explicit object storage settings * downgrade pip tools * without spaces? * like this? * without updating pip? * remove object_storage from dev volumes * named volume on hobby * lazily init object storage * simplify conditional check * reproduced error locally * reproduced error locally * object_storage_endpoint not host and port * log more when checking kafka and clickhouse * don't filter docker output * add kafka to hosts before starting stack? * silly cloud tests (not my brain)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
see #9294
For several reasons we want to add object storage
Changes
This adds object storage to docker-compose files and adds it to health checks, preflight, and instance status
Without breaking for anyone not running it, and without using it for anything
Preflight
debug set to true, object store not running
user sees a preflight warning
debug set to false, object store not running
no warning for the user
object store running
Plugin Server
on startup, with object storage enabled and running
on startup, with object storage enabled and not running
on startup, with object storage disabled (regardless of if minio is running)
Django
Instance status
How did you test this code?
running it locally with object storage enabled but the minio container both running and not running