Code

Rfix/pipeline v2 extract comments#68

Code

Closed

adrianbr wants to merge 66 commits intomasterdlt-hub/dlt:masterfrom rfix/pipeline-v2-extract-2dlt-hub/dlt:rfix/pipeline-v2-extract-2Copy head branch name to clipboard

+9,039-5,036

Commits

Commits on Sep 20, 2022

• initial implementation of plugabble data writers and buffered writer

  rudolfix

  committed
  6eba24d

  Copy full SHA for 6eba24d

• experimental implementation of extraction pipe, source, resources and associated typing

  rudolfix

  committed
  47e4766

  Copy full SHA for 47e4766

• adds schema utils to diff tables

  rudolfix

  committed
  20f6180

  Copy full SHA for 20f6180

• adds hard linking and path validation to file storage

  rudolfix

  committed
  ddd06a9

  Copy full SHA for ddd06a9

• implements pipe, resource, source and extractor without tests

  rudolfix

  committed
  15d29b5

  Copy full SHA for 15d29b5

• adds pathvalidate to deps

  rudolfix

  committed
  533f576

  Copy full SHA for 533f576

• adds additional caps to loader clients, fixes tests to use new writers

  rudolfix

  committed
  3db4695

  Copy full SHA for 3db4695

• refactoring extractor and normalizer code to introduce new data writers and extracted file jsonl format

  rudolfix

  committed
  be15172

  Copy full SHA for be15172

• makes schema name validation part of name normalization, function to normalize all names in tables

  rudolfix

  committed
  059e77d

  Copy full SHA for 059e77d

• several name cleanips

  rudolfix

  committed
  e9b70dd

  Copy full SHA for e9b70dd

Commits on Sep 24, 2022

• adds close data writer operation and written items count

  rudolfix

  committed
  fcaaef6

  Copy full SHA for fcaaef6

• adds explicit table name to json normalizer func signature

  rudolfix

  committed
  40c5acc

  Copy full SHA for 40c5acc

• adds performance improvements to Schema, 3x normalization speed gained by just removing runtime protocol check

  rudolfix

  committed
  b07fb54

  Copy full SHA for b07fb54

• removes runtime protocol check from schema altogether

  rudolfix

  committed
  d8d8563

  Copy full SHA for d8d8563

• implements data item storage and adds to normalize and load storages as mixins

  rudolfix

  committed
  f98ddbb

  Copy full SHA for f98ddbb

• rewrites Normalize to work with new extracted jsonl files and use buffered data writers

  rudolfix

  committed
  4d3861b

  Copy full SHA for 4d3861b

• adds decorator for Pool workers to pass configuration types across process boundary

  rudolfix

  committed
  52eebb9

  Copy full SHA for 52eebb9

• simplifies ExtractorStorage by moving data items storage out

  rudolfix

  committed
  693f89d

  Copy full SHA for 693f89d

Commits on Sep 28, 2022

• refactors configurations: uses dataclass based instances, no production configs, snake case field names, recursive binding to values

  rudolfix

  committed
  1ce4735

  Copy full SHA for 1ce4735

• adds init values and embedded config tests

  rudolfix

  committed
  277f8e4

  Copy full SHA for 277f8e4

Commits on Sep 30, 2022

• reorganizes configuration module

  rudolfix

  committed
  400ca1a

  Copy full SHA for 400ca1a

Commits on Oct 1, 2022

• moves configurations into specs folder

  rudolfix

  committed
  b53ed23

  Copy full SHA for b53ed23

• moves file_storage to storages, writes lists of items into jsonl files between extract and normalize stages

  rudolfix

  committed
  c0cc1d6

  Copy full SHA for c0cc1d6

• data item may be of any type, not dict loaded from json

  rudolfix

  committed
  80233c0

  Copy full SHA for 80233c0

Commits on Oct 5, 2022

• adds resolving configs via providers with variable namespaces, provides lookup traces

  rudolfix

  committed
  f7b5434

  Copy full SHA for f7b5434

• moves config tests

  rudolfix

  committed
  f16285b

  Copy full SHA for f16285b

Commits on Oct 12, 2022

• adds config inject decorator, basic tests, applies decorator to buffered writers and pipe

  rudolfix

  committed
  a80b2f5

  Copy full SHA for a80b2f5

• adds injection container with tests

  rudolfix

  committed
  1e0695f

  Copy full SHA for 1e0695f

• adds config providers framewrok, injectable config and basic tests

  rudolfix

  committed
  81eb730

  Copy full SHA for 81eb730

• adds namespaces, pipeline name, injectable namespace config to config resolver, typing improvements and tests

  rudolfix

  committed
  f08b8d5

  Copy full SHA for f08b8d5

Commits on Oct 13, 2022

• removes legacy extractor

  rudolfix

  committed
  0d126d5

  Copy full SHA for 0d126d5

• moves pipe, extract and sources code into extract

  rudolfix

  committed
  47e5029

  Copy full SHA for 47e5029

Commits on Oct 14, 2022

• adds config injection to normalize, passes instantiated configs to workers

  rudolfix

  committed
  5c76bcb

  Copy full SHA for 5c76bcb

• removes per component version, uses package version in logging

  rudolfix

  committed
  afcabae

  Copy full SHA for afcabae

• applies config injection to storages, adds dictionary config provider for testing

  rudolfix

  committed
  4e4645c

  Copy full SHA for 4e4645c

Commits on Oct 16, 2022

• adds create pipeline examples

  rudolfix

  committed
  c476254

  Copy full SHA for c476254

Commits on Oct 18, 2022

• improves config resolution: last exception is preserved, resolved configs are not resolved again, external and injected namespaces are compacted, injected configs renamed to contexts

  rudolfix

  committed
  ec20591

  Copy full SHA for ec20591

• implements new instantiation mechanics for destinations

  rudolfix

  committed
  71fcddc

  Copy full SHA for 71fcddc

• implements first version of pipeline v2 and applies config injection mechanics everywhere

  rudolfix

  committed
  3f40da6

  Copy full SHA for 3f40da6

Commits on Oct 20, 2022

• adding pseudo code samples for api v2: create pipeline, working with credentials

  rudolfix

  committed
  47277d6

  Copy full SHA for 47277d6

Commits on Oct 23, 2022

• adds more code samples

  rudolfix

  committed
  6b65451

  Copy full SHA for 6b65451

• adds general usage samples

  rudolfix

  committed
  b17e513

  Copy full SHA for b17e513

• adds example to general usage

  rudolfix

  committed
  f20871e

  Copy full SHA for f20871e

Commits on Oct 24, 2022

• general usage doc completed + README

  rudolfix

  committed
  0d5a894

  Copy full SHA for 0d5a894

Commits on Oct 25, 2022

• adds project structure example

  rudolfix

  committed
  20a9186

  Copy full SHA for 20a9186

• adds working with schemas doc

  rudolfix

  committed
  150da6d

  Copy full SHA for 150da6d

Commits on Oct 26, 2022

• first implementation of source and resource decorators, adds pipeline context

  rudolfix

  committed
  8282f30

  Copy full SHA for 8282f30

• extracts destination and pipeline common code

  rudolfix

  committed
  8ca3b9b

  Copy full SHA for 8ca3b9b

• adds ignored example secrets.toml

  rudolfix

  committed
  ce9c3b9

  Copy full SHA for ce9c3b9

Commits on Oct 27, 2022

• implements toml config provider, changes how embedded and config namespaces work, config initial values behaving like any other values + tests

  rudolfix

  committed
  c4f6c8c

  Copy full SHA for c4f6c8c

Commits on Oct 28, 2022

• ports pipeline v1 util methods

  rudolfix

  committed
  8b4d6fd

  Copy full SHA for 8b4d6fd

• renames modules holding client implementations in load

  rudolfix

  committed
  3fa7d21

  Copy full SHA for 3fa7d21

• removes pipeline v1

  rudolfix

  committed
  08c10db

  Copy full SHA for 08c10db

• moves pipeline v2 in

  rudolfix

  committed
  0290aa3

  Copy full SHA for 0290aa3

• fixes pipeline imports

  rudolfix

  committed
  d16b430

  Copy full SHA for d16b430

• fixes typing errors, adds overloads

  rudolfix

  committed
  b2a9bde

  Copy full SHA for b2a9bde

• moves source decorators to extract

  rudolfix

  committed
  3ce749e

  Copy full SHA for 3ce749e

Commits on Oct 29, 2022

• adds restore pipeline and better state management, with cli support

  rudolfix

  committed
  daba606

  Copy full SHA for daba606

• fixes dependent resources handling, partially adds missing exceptions

  rudolfix

  committed
  dab58bf

  Copy full SHA for dab58bf

• adds self importing via name or module name for destination reference

  rudolfix

  committed
  0a9691a

  Copy full SHA for 0a9691a

• fixes config exceptions names

  rudolfix

  committed
  1f7bf39

  Copy full SHA for 1f7bf39

Commits on Oct 31, 2022

• adds optional embedded namespaces extension in config resolve + tests

  rudolfix

  committed
  371a058

  Copy full SHA for 371a058

• refactors interface to select resources in source, adds missing exceptions and typings

  rudolfix

  committed
  bc61466

  Copy full SHA for bc61466

• changes the schema file pattern _schema. -> .schema.

  rudolfix

  committed
  161b8d6

  Copy full SHA for 161b8d6

• adds flag to wipe out loader storage before initializing

  rudolfix

  committed
  32d769b

  Copy full SHA for 32d769b

• various typing improvements

  rudolfix

  committed
  11a80d8

  Copy full SHA for 11a80d8