Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

audiusd in audius-compose #10859

Merged
merged 22 commits into from
Dec 30, 2024
Merged

audiusd in audius-compose #10859

merged 22 commits into from
Dec 30, 2024

Conversation

alecsavvy
Copy link
Contributor

@alecsavvy alecsavvy commented Dec 28, 2024

Description

Meat and potatoes:

  • removes the mediorum dev go cluster and starts up dev nodes in the same way stage and production nodes start up.
  • modifies the core cluster to be an audiusd cluster with nodes 2-4 running mediorum to match the previously removed dev instances.
  • modifies the container names from core-$node_type-* to be audiusd-*.
  • modifies the mediorum test suite to point to audiusd-* hostnames by default.
  • modifies the mediorum startup defaults to account for the past dev instances.
  • fixes dev validator registration but registering on ethereum before any peering happens.

Side dishes:

  • adds a make static-deps which build dependencies that don't need to be built all the time, poa and eth ganache. We can add more to this command later.
  • switches eth-ganache and poa-ganache to be pulled from dockerhub instead of being built.
  • rename the Dockerfile.cpp to Dockerfile.deps because vscode was thinking this was an actual C++ file and failing the gls because it thought it required certain build flags.
  • changes this pattern RUN go mod graph | awk '{if ($1 !~ "@") print $2}' | xargs go get to go mod download which will cache correctly or bust if go.mod || go.sum change. Previously this would bust the docker cache early.
  • changes the Dockerfile.dev discovery image to use a python slim base, this significantly speeds up build times. In particular all the wheels. See here for more details. We should move the prod image over as well but didn't want to touch that here.
  • Modified the way the rust version gets installed into Dockerfile.dev for discovery so if the pulled file matches the previous one docker will consider it cached and move on. This was busting the cache early before the pip requirements installation so that explains why wheels were being built all the time.
  • Removed that pesky register.py that runs when discovery starts up. This has been failing for a long time and is replaced by core registering itself on dev.
  • Removed the indirection of the dev-tools/setup.py which dynamically set some env vars for dev discovery. There's some further improvements we can make here.
  • Adds the full elasticsearch health check, elastic search was running out of space locally and this helped.
  • modifies relay startup to not rebuild twice.
  • noops openresty in dev, sets it to a busybox container (~1mb) that doesn't even get included during audius-compose up. We do need the entry in there for the root discovery compose file to extend from.

How Has This Been Tested?

  • lots and lots of failed audius-compose builds
  • ci
  • super cached and fast audius-compose up

Copy link

changeset-bot bot commented Dec 28, 2024

⚠️ No Changeset found

Latest commit: b4f7e73

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Copy link

gitguardian bot commented Dec 28, 2024

⚠️ GitGuardian has uncovered 7 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
11648675 Triggered Generic High Entropy Secret 43ce675 pkg/mediorum/mediorum.go View secret
11648676 Triggered Generic High Entropy Secret 43ce675 cmd/core/infra/dev_config/discovery-one.docker.env View secret
11648678 Triggered Generic High Entropy Secret b588087 cmd/core/infra/dev_config/content-three.docker.env View secret
11648679 Triggered Generic High Entropy Secret 43ce675 pkg/mediorum/mediorum.go View secret
11648680 Triggered Generic High Entropy Secret b588087 cmd/core/infra/dev_config/content-one.docker.env View secret
11648681 Triggered Generic High Entropy Secret 43ce675 pkg/mediorum/mediorum.go View secret
9412812 Triggered Generic Password 35e2ba4 dev-tools/compose/docker-compose.blockchain.yml View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@alecsavvy alecsavvy marked this pull request as ready for review December 28, 2024 21:42
@alecsavvy alecsavvy requested a review from phelpsdb December 28, 2024 22:49
@@ -245,3 +245,33 @@ release-aa-backfill:
@DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build -t audius/audio-analysis-backfill:latest -f ./cmd/audio-analysis-backfill/Dockerfile .
@docker push audius/audio-analysis-backfill:latest

.PHONY: static-deps
static-deps:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment as to why this is needed i.e. why buildx does not work here

@endline endline merged commit 04efb42 into main Dec 30, 2024
11 checks passed
@endline endline deleted the as/audiusd-audius-compose branch December 30, 2024 17:53
audius-infra pushed a commit that referenced this pull request Dec 31, 2024
[9318508] Upgrade audiusd to bookworm, move off essentia (#10867) endline
[937a873] SLA rollup improvements, fixes (#10868) Danny
[a38e4b6] Fix search for old clients (#10869) Raymond Jacobson
[b555bc5] add dev to healthz (#10862) alecsavvy
[373ce1d] add custom legacy profile to core and mediorum (#10866) alecsavvy
[c0b91c5] fix ci: add back mediorum and core in legacy docker compose files (#10864) alecsavvy
[8ef27ef] Remove go pkg/infra (#10863) endline
[04efb42] audiusd in audius-compose (#10859) alecsavvy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants