Update Pipelines folder structure #31525

erohmensing · 2023-10-17T22:43:56Z

Overview

This is an update to the folder/file structure of our pipelines project.

The goal is through this change to improve readability and context sharing for new and existing developers.

However, This is NOT an update to any logic or code. That is out of scope for now.

The goal is to merge this fast so we dont have too much rebase hell

Changes

These are the broad high level changes and why

1. The colocation of pipelines and commands

Currently and in the future we believe the pipeline and the command concepts map 1:1.

Ideally we would be able to merge the abstractions (like in aircmd) but for now we believe a good first step is to hold them in the same location.

This allows

The folder structure to show what commands are available
Create a clear pairing between the pipeline code that is run and the cli arguments that feed them

2. A top level folder for CLI entry points

Even though commands are located beside their respective pipelines the entry point to the airbyte-ci should be as close to the root as possible and visible

This allows

any new developer know where to start tracing code.
A single area to collect CLI helpers like telemetry and custom click classes

3. The reduction of large files into smaller single purpose files

This is to improve reading and tracing code.

We've split

bases.py into models
utils.py into helpers
environments.py into dagger/actions and dagger/containers

4. Steps get more generic as you go up the file structure

The idea is that there is a taxonomy of steps that should be separated based on their wish to be reused.

Step Base Class now found at models/steps.py
Generic Steps intended to be reused (e.g. PoetryRunStep, SimpleDockerStep) now found at pipeline/steps
Single Purpose General Steps (e.g. GitPushChanges) now found at pipeline/steps
Specialize Pipeline specific steps (e.g. BuildConnectorDistributionTar) found at pipeline/connectors/build/steps

Separating them on this taxonomy lets developers infer how wide spread their usage or if the step is intended to work in any pipeline vs just a specific set of pipelines

5. Containers separated from actions

Environments.py had a mix of two different type signatures

Container -> Container: These are functions that install something, analogous to dockerfile commands
Context|str|any -> Container: These are functions that create/initialize a container, analogous to dockerfile base image

We broke this into actions and containers respectively. This is to make this distinction clearer at a glance and potentially lean on the fact that the developer may be used to a distinction like this if theyve used docker before

File Structure

.
├── README.md
├── pipelines
│   ├── __init__.py
│   ├── cli
│   │   ├── airbyte_ci.py
│   │   ├── dagger_pipeline_command.py
│   │   ├── dagger_run.py
│   │   └── telemetry.py
│   ├── consts.py
│   ├── dagger
│   │   ├── __init__.py
│   │   ├── actions
│   │   │   ├── __init__.py
│   │   │   ├── connector
│   │   │   │   ├── hooks.py
│   │   │   │   └── normalization.py
│   │   │   ├── python
│   │   │   │   ├── __init__.py
│   │   │   │   ├── common.py
│   │   │   │   ├── pipx.py
│   │   │   │   └── poetry.py
│   │   │   ├── remote_storage.py
│   │   │   ├── secrets.py
│   │   │   └── system
│   │   │       ├── __init__.py
│   │   │       ├── common.py
│   │   │       └── docker.py
│   │   └── containers
│   │       ├── __init__.py
│   │       ├── internal_tools.py
│   │       ├── java.py
│   │       └── python.py
│   ├── hacks.py
│   ├── helpers
│   │   ├── __init__.py
│   │   ├── connectors
│   │   │   ├── __init__.py
│   │   │   ├── metadata_change_helpers.py
│   │   │   └── modifed.py
│   │   ├── gcs.py
│   │   ├── git.py
│   │   ├── github.py
│   │   ├── sentry_utils.py
│   │   ├── slack.py
│   │   ├── steps.py
│   │   └── utils.py
│   ├── internal_tools
│   │   └── internal.py
│   ├── models
│   │   ├── bases.py
│   │   ├── contexts.py
│   │   ├── reports.py
│   │   └── steps.py
│   └── pipeline
│       ├── __init__.py
│       ├── connectors
│       │   ├── __init__.py
│       │   ├── commands.py
│       │   ├── context.py
│       │   ├── pipeline.py
│       │   ├── builds
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   ├── pipeline.py
│       │   │   └── steps
│       │   ├── bump_version
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   └── pipeline.py
│       │   ├── format
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   ├── pipeline.py
│       │   │   └── steps
│       │   ├── list
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   └── pipeline.py
│       │   ├── migrate_to_base_image
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   └── pipeline.py
│       │   ├── publish
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   ├── context.py
│       │   │   └── pipeline.py
│       │   ├── reports.py
│       │   ├── test
│       │   │   ├── __init__.py
│       │   │   ├── commands.py
│       │   │   ├── pipeline.py
│       │   │   └── steps
│       │   └── upgrade_base_image
│       │       ├── __init__.py
│       │       ├── commands.py
│       │       └── pipeline.py
│       ├── metadata
│       │   ├── __init__.py
│       │   ├── commands.py
│       │   └── pipeline.py
│       ├── steps
│       │   ├── __init__.py
│       │   ├── docker.py
│       │   ├── git.py
│       │   ├── gradle.py
│       │   ├── no_op.py
│       │   └── poetry.py
│       └── test
│           ├── __init__.py
│           ├── commands.py
│           └── pipeline.py
├── poetry.lock
├── pyproject.toml
└── tests
    ├── __init__.py
    ├── conftest.py
    ├── test_actions
    │   └── test_environments.py
    ├── test_bases.py
    ├── test_builds
    │   ├── dummy_build_customization.py
    │   └── test_python_connectors.py
    ├── test_commands
    │   ├── __init__.py
    │   └── test_groups
    │       ├── __init__.py
    │       └── test_connectors.py
    ├── test_gradle.py
    ├── test_publish.py
    ├── test_steps
    │   ├── __init__.py
    │   └── test_simple_docker_step.py
    ├── test_tests
    │   ├── __init__.py
    │   ├── test_common.py
    │   └── test_python_connectors.py
    ├── test_utils.py
    └── utils.py

36 directories, 105 files

vercel · 2023-10-17T22:44:03Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Oct 18, 2023 11:29pm

erohmensing · 2023-10-17T23:49:54Z

airbyte-ci/connectors/pipelines/pipelines/dagger/actions/secrets.py

@@ -42,9 +41,12 @@ async def download(context: ConnectorContext, gcp_gsm_env_variable_name: str = "
    Returns:
        Directory: A directory with the downloaded secrets.
    """
+    # temp - fix circular import


@bnchrch this is the one circular import I just couldn't figure out how to implement in a decent way. It's also imported here. Maybe you can take a crack at it

alafanechere

Very cool changes! Thank you!!!!!!
Just minor suggestions.
Feel free to merge if the unit test pass.
A pre-release + a java and python connector test run would also be reassuring (feel free to ignore if you tested this locally).

alafanechere · 2023-10-18T18:11:18Z

airbyte-ci/connectors/pipelines/pipelines/pipeline/connectors/list/commands.py

+
+@connectors.command(cls=DaggerPipelineCommand, help="List all selected connectors.")
+@click.pass_context
+def list(


This command is not a pipeline but only printing stuff. Do you think a subpackage like utils_commands would make sense?

good point - for what its worth, the test command (running tests for airbyte ci) is not a pipeline either, nor will the new format be. I don't think these are utils though, maybe it is just a naming problem.

@bnchrch you talked about pipelines, not commands, being the first class citizens here, somehow, right? I can't remember how that conversation went

🤔 good call out.

I dont have a good answer.

This feels like one of those puzzle pieces that can't fit

If we leave it as is, theres a command with no pipeline under pipeline

If we move it to util_commands the folder structure no longer matches the airbyte-ci hierarchy

Currently Im thinking we leave it as is but perhaps find a new name for pipeline

maybe even airbyte-ci?

Currently Im thinking we leave it as is but perhaps find a new name for pipeline
maybe even airbyte-ci?

This was going to be my exact suggestion, i say we go for it 😄

airbyte-ci/connectors/pipelines/pipelines/dagger/containers/__init__.py

airbyte-ci/connectors/pipelines/pipelines/internal_tools/internal.py

erohmensing · 2023-10-18T19:48:39Z

Current dependencies on/for this PR:

master
- PR Update Pipelines folder structure #31525 👈
  - PR [don't merge] trigger test runs for refactor pr #31586

This comment was auto-generated by Graphite.

…ete old binaries to get this to work

…tainers to internal tools, reports to own module and connector report to pipelines/connectors

…rs commands

erohmensing · 2023-10-18T22:20:50Z

Some successful javapython connector test runs here: #31586 (after fixing something so definitely worth testing!)

Successful prerelease here: https://github.com/airbytehq/airbyte/actions/runs/6567115837

pedroslopez

Nice! I kept having to hop around and this feels like would make things much clearer!

pedroslopez · 2023-10-18T22:26:13Z

airbyte-ci/connectors/pipelines/pipelines/pipeline/connectors/builds/__init__.py

nit: why is this folder plural (builds)? I think I would've expected it to line up with the command build if i understand the explanation correctly

Good catch! created an exception in our gitignore and renamed

just kidding, that still messes with our pipeline exclude logic.

went for build_image instead

sentry-io · 2023-10-19T11:10:47Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ **ExecuteTimeoutError: Request timed out. Try setting a higher value in 'execute_timeout' config for this dagger.Connec...** pipelines.airbyte_ci.connectors.pipeline in run...` View Issue
‼️ DaggerError: Dagger Command test failed. pipelines.cli.dagger_pipeline_command in invoke View Issue
‼️ QueryError: sync /runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-paypal-transaction: fai... pipelines.airbyte_ci.connectors.context in get_... View Issue
‼️ QueryError: file size 260503070 exceeds limit 134217728 pipelines.helpers.utils in get_container_output View Issue
‼️ ExecError: process "git diff --diff-filter=MADRT --name-only origin/master...748211ba7708b3b7ce163001dc59d4f... pipelines.helpers.git in get_modified_files_in_... View Issue

_{Did you find this useful? React with a 👍 or 👎}

Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: erohmensing <erohmensing@users.noreply.github.com> Co-authored-by: bnchrch <bnchrch@users.noreply.github.com>

erohmensing commented Oct 17, 2023

View reviewed changes

bnchrch requested review from evantahler, pedroslopez and alafanechere October 18, 2023 00:53

bnchrch changed the title ~~Ella ben/pipelines/refactor folder struct~~ Update Pipelines folder structure Oct 18, 2023

alafanechere approved these changes Oct 18, 2023

View reviewed changes

erohmensing marked this pull request as ready for review October 18, 2023 19:42

erohmensing force-pushed the ella-ben/pipelines/refactor-folder-struct branch from c795238 to ddf6e74 Compare October 18, 2023 19:48

bnchrch and others added 15 commits October 18, 2023 15:00

First refactor pass

0d1c6a2

fix script paths. when running install --force, you might have to del…

d2b8802

…ete old binaries to get this to work

add bases back, fix some imports

d72aeb9

fix some more imports

7a9e590

move internal tools containers to containers. fix some more

f9cbac0

bit of a hacky fix for circular imports on this one - come back to this

d9c3c73

move lots of stuff - some steps to pipelines/steps, internal tool con…

a54dc06

…tainers to internal tools, reports to own module and connector report to pipelines/connectors

fixes

b8ff63a

airbyte ci help finally works :D

586372c

run formatter

01340fe

fix extra numbers

cd5aef3

add python licenses to airbyte ci

41ee5e2

Bump version

a66caba

Add more inits

86b2fec

move list of internal tool paths to const file

a42f2f2

erohmensing force-pushed the ella-ben/pipelines/refactor-folder-struct branch from ddf6e74 to a42f2f2 Compare October 18, 2023 20:01

erohmensing marked this pull request as draft October 18, 2023 20:01

erohmensing mentioned this pull request Oct 18, 2023

[don't merge] trigger test runs for refactor pr #31586

Closed

erohmensing marked this pull request as ready for review October 18, 2023 20:13

Automated Commit - Formatting Changes

9f4ca1e

bnchrch and others added 16 commits October 18, 2023 13:23

Update step imports

b4185a6

Automated Commit - Formatting Changes

c64a438

Fix imports

e4ba25d

Fix imports

1b621a2

example: add subcommands from other files manually. also make test work

1d44e62

add connectors publish

555c2e5

Fix imports

e5275b7

Fix imports

b4f4c7b

add all connectors commands and fix truncated help texts for connecto…

299f99f

…rs commands

Fix imports test_utils

4449bc6

Fix imports test_common

4be0aa7

fix jinja template reference

a4ca6e5

actually fix the jinja template

7506f2e

Fix imports test_utils

894a3e2

Fix imports test_publish

60f15f3

Fix imports test_connectors

c23224a

pedroslopez approved these changes Oct 18, 2023

View reviewed changes

bnchrch and others added 6 commits October 18, 2023 22:33

Automated Commit - Formatting Changes

6dffb72

rename pipeline to airbyte_ci

549d0e3

Update .gitignore

7e40db2

Rename builds to build

e495869

Rename build to build_image

eb6dc66

Automated Commit - Formatting Changes

aab73de

bnchrch enabled auto-merge (squash) October 18, 2023 23:35

bnchrch merged commit a53a020 into master Oct 18, 2023

bnchrch deleted the ella-ben/pipelines/refactor-folder-struct branch October 18, 2023 23:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Pipelines folder structure #31525

Update Pipelines folder structure #31525

erohmensing commented Oct 17, 2023 •

edited by bnchrch

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

erohmensing Oct 17, 2023

alafanechere left a comment •

edited

Loading

alafanechere Oct 18, 2023

erohmensing Oct 18, 2023

bnchrch Oct 18, 2023

erohmensing Oct 18, 2023

erohmensing commented Oct 18, 2023 •

edited

Loading

erohmensing commented Oct 18, 2023

pedroslopez left a comment

pedroslopez Oct 18, 2023 •

edited

Loading

bnchrch Oct 18, 2023

bnchrch Oct 18, 2023

sentry-io bot commented Oct 19, 2023 •

edited

Loading

Update Pipelines folder structure #31525

Update Pipelines folder structure #31525

Conversation

erohmensing commented Oct 17, 2023 • edited by bnchrch Loading

Overview

Changes

1. The colocation of pipelines and commands

2. A top level folder for CLI entry points

3. The reduction of large files into smaller single purpose files

4. Steps get more generic as you go up the file structure

5. Containers separated from actions

File Structure

vercel bot commented Oct 17, 2023 • edited Loading

erohmensing Oct 17, 2023

Choose a reason for hiding this comment

alafanechere left a comment • edited Loading

Choose a reason for hiding this comment

alafanechere Oct 18, 2023

Choose a reason for hiding this comment

erohmensing Oct 18, 2023

Choose a reason for hiding this comment

bnchrch Oct 18, 2023

Choose a reason for hiding this comment

erohmensing Oct 18, 2023

Choose a reason for hiding this comment

erohmensing commented Oct 18, 2023 • edited Loading

erohmensing commented Oct 18, 2023

pedroslopez left a comment

Choose a reason for hiding this comment

pedroslopez Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

bnchrch Oct 18, 2023

Choose a reason for hiding this comment

bnchrch Oct 18, 2023

Choose a reason for hiding this comment

sentry-io bot commented Oct 19, 2023 • edited Loading

Suspect Issues

erohmensing commented Oct 17, 2023 •

edited by bnchrch

Loading

vercel bot commented Oct 17, 2023 •

edited

Loading

alafanechere left a comment •

edited

Loading

erohmensing commented Oct 18, 2023 •

edited

Loading

pedroslopez Oct 18, 2023 •

edited

Loading

sentry-io bot commented Oct 19, 2023 •

edited

Loading