Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(devex): pnpm workspaces #27267

Closed
wants to merge 26 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 40 additions & 34 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -1,49 +1,55 @@
*
!bin
!requirements.txt
!requirements-dev.txt
!posthog
!plugin-server
!staticfiles
!manage.py
!gunicorn.config.py
!babel.config.js
!posthog.json
!package.json
!pnpm-lock.yaml
!webpack.config.js
!playwright.config.ts
!tailwind.config.js
!.devcontainer
!.kearc
!.storybook
!tsconfig.json
!.npmrc
!babel.config.js
!bin
!ee
!eslint-rules
!frontend/@posthog
!frontend/*.*
!frontend/public
!frontend/src
!frontend/types
!frontend/public
!frontend/*.*
!ee
!.devcontainer
!gunicorn.config.py
!hogvm
!manage.py
!package.json
!patches
!playwright.config.ts
!plugin-server
!plugin-server/.eslintrc.js
!plugin-server/.prettierrc
!plugin-server/package.json
!plugin-server/pnpm-lock.yaml
!plugin-server/tsconfig.json
!plugin-server/tsconfig.eslint.json
!plugin-server/src
!plugin-server/.eslintrc.js
!plugin-server/.prettierrc
!share/GeoLite2-City.mmdb
!hogvm/python
!unit.json.tpl
!plugin-transpiler/src
!plugin-server/tsconfig.eslint.json
!plugin-server/tsconfig.json
!plugin-transpiler/*.*
!test-runner-jest.config.js
!test-runner-jest-environment.js
!patches
!plugin-transpiler/src
!pnpm-lock.yaml
!pnpm-workspace.yaml
!posthog
!posthog.json
!requirements-dev.txt
!requirements.txt
!rust
!share/GeoLite2-City.mmdb
!staticfiles
!tailwind.config.js
!test-runner-jest-environment.js
!test-runner-jest.config.js
!tsconfig.json
!unit.json.tpl
!webpack.config.js
frontend/@posthog/*/node_modules
hogvm/typescript/node_modules
plugin-server/node_modules
rust/.env
rust/.github
rust/docker
rust/target
rust/cyclotron-node/dist
rust/cyclotron-node/node_modules
rust/cyclotron-node/index.node
rust/cyclotron-node/node_modules
rust/docker
rust/target
7 changes: 5 additions & 2 deletions .github/workflows/ci-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,12 @@ jobs:
restore-keys: |
${{ runner.os }}-pnpm-cypress-

- name: Install package.json dependencies with pnpm
- name: Build and install npm dependencies
if: needs.changes.outputs.shouldTriggerCypress == 'true'
run: pnpm install --frozen-lockfile
run: |
pnpm install --frozen-lockfile
cd hogvm/typescript && pnpm compile
cd ../../rust/cyclotron-node && pnpm build

- name: Stop/Start stack with Docker Compose
# these are required checks so, we can't skip entire sections
Expand Down
12 changes: 10 additions & 2 deletions .github/workflows/ci-frontend.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,11 @@ jobs:

- name: Install package.json dependencies with pnpm
if: needs.changes.outputs.frontend == 'true'
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile --filter .

- name: Install @posthog/hogvm
if: needs.changes.outputs.frontend == 'true'
run: cd hogvm/typescript && pnpm install --frozen-lockfile --filter . && pnpm run compile

- name: Check formatting with prettier
if: needs.changes.outputs.frontend == 'true'
Expand Down Expand Up @@ -153,7 +157,11 @@ jobs:

- name: Install package.json dependencies with pnpm
if: needs.changes.outputs.frontend == 'true'
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile --filter .

- name: Install @posthog/hogvm
if: needs.changes.outputs.frontend == 'true'
run: cd hogvm/typescript && pnpm install --frozen-lockfile --filter . && pnpm run compile

- name: Test with Jest
# set maxWorkers or Jest only uses 1 CPU in GitHub Actions
Expand Down
94 changes: 3 additions & 91 deletions .github/workflows/ci-hog.yml
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,7 @@ jobs:
- name: Run HogVM TypeScript tests
run: |
cd hogvm/typescript
pnpm install --frozen-lockfile
pnpm install --frozen-lockfile --filter .
pnpm run test

- name: Run Hog tests
Expand Down Expand Up @@ -191,99 +191,11 @@ jobs:
node-version: 18
registry-url: https://registry.npmjs.org
- name: Install package.json dependencies
run: cd hogvm/typescript && pnpm install
run: cd hogvm/typescript && pnpm install --frozen-lockfile --filter .

- name: Publish the package in the npm registry
run: cd hogvm/typescript && npm publish --access public
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Sleep 60 seconds to allow npm to update the package
run: sleep 60

update-versions:
name: Update versions in package.json
runs-on: ubuntu-24.04
needs: release-hogvm
if: always() # This ensures the job runs regardless of the result of release-hogvm
steps:
- name: Checkout the repository
uses: actions/checkout@v4
with:
fetch-depth: 1
repository: ${{ github.event.pull_request.head.repo.full_name }}
ref: ${{ github.event.pull_request.head.ref }}
token: ${{ secrets.POSTHOG_BOT_GITHUB_TOKEN }}

- name: Install pnpm
uses: pnpm/action-setup@v4
- name: Set up Node 18
uses: actions/setup-node@v4
with:
node-version: 18
registry-url: https://registry.npmjs.org

- name: Check for version mismatches
id: check-mismatch
run: |
# Extract committed version
HOGVM_VERSION=$(jq -r '.version' hogvm/typescript/package.json)

# Compare dependencies in package.json
MAIN_VERSION=$(jq -r '.dependencies."@posthog/hogvm"' package.json | tr -d '^')
PLUGIN_VERSION=$(jq -r '.dependencies."@posthog/hogvm"' plugin-server/package.json | tr -d '^')

echo "HOGVM_VERSION=$HOGVM_VERSION"
echo "MAIN_VERSION=$MAIN_VERSION"
echo "PLUGIN_VERSION=$PLUGIN_VERSION"

# Set output if mismatches exist
if [[ "$HOGVM_VERSION" != "$MAIN_VERSION" || "$HOGVM_VERSION" != "$PLUGIN_VERSION" ]]; then
echo "mismatch=true" >> "$GITHUB_ENV"
else
echo "mismatch=false" >> "$GITHUB_ENV"
fi

- name: Update package.json versions
if: env.mismatch == 'true'
run: |
VERSION=$(jq ".version" hogvm/typescript/package.json -r)

retry_pnpm_install() {
local retries=0
local max_retries=20 # 10 minutes total
local delay=30

while [[ $retries -lt $max_retries ]]; do
echo "Attempting pnpm install (retry $((retries+1))/$max_retries)..."
pnpm install --no-frozen-lockfile && break
echo "Install failed. Retrying in $delay seconds..."
sleep $delay
retries=$((retries + 1))
done

if [[ $retries -eq $max_retries ]]; then
echo "pnpm install failed after $max_retries attempts."
exit 1
fi
}

# Update main package.json
mv package.json package.old.json
jq --indent 4 '.dependencies."@posthog/hogvm" = "^'$VERSION'"' package.old.json > package.json
rm package.old.json
retry_pnpm_install

# Update plugin-server/package.json
cd plugin-server
mv package.json package.old.json
jq --indent 4 '.dependencies."@posthog/hogvm" = "^'$VERSION'"' package.old.json > package.json
rm package.old.json
retry_pnpm_install

- name: Commit updated package.json files
if: env.mismatch == 'true'
uses: EndBug/add-and-commit@v9
with:
add: '["package.json", "pnpm-lock.yaml", "plugin-server/package.json", "plugin-server/pnpm-lock.yaml", "hogvm/typescript/package.json"]'
message: 'Update @posthog/hogvm version in package.json'
default_author: github_actions
github_token: ${{ secrets.POSTHOG_BOT_GITHUB_TOKEN }}
30 changes: 25 additions & 5 deletions .github/workflows/ci-plugin-server.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ jobs:
cache: pnpm

- name: Install package.json dependencies with pnpm
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile --filter .

- name: Check formatting with prettier
run: pnpm prettier:check
Expand Down Expand Up @@ -163,9 +163,19 @@ jobs:
cache: pnpm
cache-dependency-path: plugin-server/pnpm-lock.yaml

- name: Install package.json dependencies with pnpm
- name: Build npm dependencies
if: needs.changes.outputs.plugin-server == 'true'
run: cd plugin-server && pnpm i
run: |
cd plugin-server
pnpm install --frozen-lockfile --filter .

cd ../hogvm/typescript
pnpm install --frozen-lockfile --filter .
pnpm compile

cd ../../rust/cyclotron-node
pnpm install --frozen-lockfile --filter .
pnpm build

- name: Wait for Clickhouse, Redis & Kafka
if: needs.changes.outputs.plugin-server == 'true'
Expand Down Expand Up @@ -255,11 +265,21 @@ jobs:
cache: pnpm
cache-dependency-path: plugin-server/pnpm-lock.yaml

- name: Install package.json dependencies with pnpm
- name: Build npm dependencies
if: needs.changes.outputs.plugin-server == 'true'
run: |
cd plugin-server
pnpm install --frozen-lockfile
pnpm install --frozen-lockfile --filter .

cd ../hogvm/typescript
pnpm install --frozen-lockfile --filter .
pnpm compile

cd ../../rust/cyclotron-node
pnpm install --frozen-lockfile --filter .
pnpm build

cd ../../plugin-server
pnpm build

- name: Wait for Clickhouse, Redis & Kafka
Expand Down
17 changes: 14 additions & 3 deletions .github/workflows/storybook-chromatic.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,15 @@ jobs:
cache: pnpm

- name: Install dependencies and Chromatic
run: pnpm i -D chromatic
run: |
pnpm install --frozen-lockfile --filter .
pnpm install chromatic -w

- name: Install npm dependencies
run: |
cd hogvm/typescript
pnpm install --frozen-lockfile --filter .
pnpm run compile

- name: Publish to Chromatic
uses: chromaui/action@v11
Expand Down Expand Up @@ -115,10 +123,13 @@ jobs:
cache: pnpm

- name: Install package.json dependencies with pnpm
run: pnpm install --frozen-lockfile
run: pnpm install --frozen-lockfile --filter .

- name: Install @posthog/hogvm
run: cd hogvm/typescript && pnpm install --frozen-lockfile --filter . && pnpm run compile

- name: Install CI utilities with pnpm
run: pnpm install http-server wait-on
run: pnpm install http-server wait-on -w

- name: Build Storybook
run: pnpm build-storybook --test --quiet # Silence since progress logging results in a massive wall of spam
Expand Down
6 changes: 6 additions & 0 deletions .github/workflows/storybook-deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,12 @@ jobs:
cache: pnpm
cache-dependency-path: posthog/pnpm-lock.yaml

- name: Install npm packages
run: |
cd posthog/hogvm/typescript
pnpm install --frozen-lockfile --filter .
pnpm run compile

- name: Install dependencies (pnpm)
run: cd posthog && pnpm i

Expand Down
1 change: 1 addition & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
shared-workspace-lockfile=false
Loading
Loading