Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create hourly spellbook and move all subprojects in new directory structure #6422

Merged
merged 23 commits into from
Jul 26, 2024

Conversation

0xRobin
Copy link
Collaborator

@0xRobin 0xRobin commented Jul 24, 2024

  1. Split out hourly models into its' own new 'hourly_spellbook' dbt subproject
├── _project
│   ├── balancer
│   ├── cow_protocol
│   ├── ens
│   ├── lido
│   └── safe
└── _sector
    ├── abi
    ├── account_abstraction
    ├── blobs
    ├── cex
    ├── lending
    ├── perpetual
    ├── prices
    ├── rollup_economics
    └── staking

  1. This PR also moves the following lineages to the tokens subproject:
- prices.tokens
- transfers
- balances

3.This PR puts all the subprojects in a clean directory structure and consolidates all the remaining models in daily_spellbook.

@0xRobin
Copy link
Collaborator Author

0xRobin commented Jul 24, 2024

ok CI's seems to actually be running, so if you prefer to do everything in one go we can just merge this in main @jeff-dude.

@0xRobin 0xRobin changed the title Move all subprojects in new directory structure Create hourly spellbook and move all subprojects in new directory structure Jul 24, 2024
@0xRobin 0xRobin changed the base branch from hourly-subproject to main July 24, 2024 11:47
@0xRobin 0xRobin marked this pull request as ready for review July 24, 2024 11:52
@0xRobin
Copy link
Collaborator Author

0xRobin commented Jul 24, 2024

files with actual changes (ignores renamed files) in:

>  git diff --diff-filter=r --name-only main

.github/workflows/commit_manifest.yml
.github/workflows/daily_spellbook.yml
.github/workflows/dbt_run.yml
.github/workflows/dex.yml
.github/workflows/hourly_spellbook.yml
.github/workflows/nft.yml
.github/workflows/prices_check.yml
.github/workflows/solana.yml
.github/workflows/spellbook.yml
.github/workflows/tokens.yml
dbt_subprojects/README.md
dbt_subprojects/daily_spellbook/models/dao/ens/_schema.yml
dbt_subprojects/hourly_spellbook/README.md
dbt_subprojects/hourly_spellbook/models/_sector/cex/_schema.yml
dbt_subprojects/tokens/models/prices/prices_schema.yml
macros/models/_project/balancer/balancer_bpt_prices_macro.sql
macros/models/_project/balancer/balancer_liquidity_macro.sql
macros/models/_project/balancer/balancer_token_balance_changes_daily_agg_macro.sql
macros/models/_sector/add_pool_price_usd.sql
macros/models/_sector/fungible/fungible_transfers.sql
macros/models/_sector/social/enrich_social_trades.sql
models/moonwell/base/moonwell_base_schema.yml
scripts/ensure_cluster.sh
seeds/.gitkeep
sources/_subprojects_outputs/hourly_spellbook/_sources.yml
sources/_subprojects_outputs/tokens/transfers.yml
tests/.gitkeep

@0xRobin 0xRobin requested a review from jeff-dude July 24, 2024 12:02
@0xRobin 0xRobin added ready-for-review this PR development is complete, please review dune team created by dune team labels Jul 24, 2024
@jeff-dude jeff-dude self-assigned this Jul 24, 2024
@jeff-dude jeff-dude added in review Assignee is currently reviewing the PR and removed ready-for-review this PR development is complete, please review labels Jul 24, 2024
@jeff-dude
Copy link
Member

tried to update documentation in multiple spots to make it as obvious as possible

@jeff-dude
Copy link
Member

i think i'll give this a go tomorrow morning. keeping track of thoughts / plan:

  • turn off modified steps to be safe, only changes should be config / refs --> sources
  • update dbt cloud
    • subproject value in each project settings to include dbt_subprojects/
    • clean out old legacy job, update to just an hourly job with same run commands as other projects
  • submit / merge PR for clean up script to handle new subproject directory structure
  • communicate internally of the merge, link to gh discussion, mention if any downtime in spells refresh schedule it's to ensure all jobs are back to normal after

@jeff-dude
Copy link
Member

i'm creating a mess here with merge conflicts, but i got sidetracked today with a bug fix on amount_usd due to bad price, then went down rabbit hole of a bunch of merges.

i guess we'll have to fix a bunch of open PRs after anyway, so maybe easier to do here anyway 🤷

@0xRobin 0xRobin force-pushed the move-daily-and-subprojects branch from 19ef27d to b7c29d0 Compare July 26, 2024 09:13
@0xRobin
Copy link
Collaborator Author

0xRobin commented Jul 26, 2024

no worries, these conflicts are pretty easy to resolve.

@jeff-dude jeff-dude removed the in review Assignee is currently reviewing the PR label Jul 26, 2024
@jeff-dude jeff-dude merged commit d2ae743 into main Jul 26, 2024
1 of 7 checks passed
@jeff-dude jeff-dude deleted the move-daily-and-subprojects branch July 26, 2024 13:11
@github-actions github-actions bot locked and limited conversation to collaborators Jul 26, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
dune team created by dune team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants