Ingest files from a directory

[cgardens]

Tell us about the problem you're trying to solve

• I have a directory on S3. New files are added to that directory on some cadence. Whenever a new file is added to the directory I want to sync that data to a destination. The names of the files are monotonically increasing and could be used as cursor fields.

Describe the solution you’d like

• This is either a new feature in the file source or a new connector altogether.

heads up sherifnada

requested by SG. tagging roshan

┆Issue is synchronized with this Asana task by Unito

[sherifnada]

@roshan would all the files have the same schema/contribute to the same stream?

[roshan]

Yes, they represent a group of new records into the same stream.

…

On Thu, Mar 25, 2021, 22:44 Sherif A. Nada ***@***.***> wrote: @roshan <https://github.com/roshan> would all the files have the same schema/contribute to the same stream? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2622 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABX3XZHST4NUEKRICHMEURTTFQNLZANCNFSM4Z2NZ7TQ> .

[roshan]

If you would like, it is possible to arrange it so that each prefix represents a stream. I am happy to work with whatever method you think is most general.

[sherifnada]

related:

#1874
#2512
https://airbytehq.slack.com/archives/C019WJFR1JT/p1616080620015200

[jrhizor]

• file name history in state or timestamp based "cursor" (timestamp would work for cloud storage, won't work for local disk if you're doing incremental updates)
• user defined mapping of regex -> jsonschema for each stream? default to object (no normalization)
• open questions around how this fits with the existing file connector (should it be in the same connector or separate?)

[harshithmullapudi]

wouldn't it be good to solve independently for sources like S3, Google Storage as the s3-csv tap

[harshithmullapudi]

We have a usecase for which we need to solve this for S3. Thinking of bringing that tap-s3-csv to airbyte any thoughts here

[Phlair]

Brainstorm & MVP definition:
https://docs.google.com/document/d/1YbY9Css9e_4jRiKkkB60CfrUNkdr1Sx72ElPmANy0gU/edit#

[cgardens]

@Phlair can we close this?

[Phlair]

There's still outstanding work to extend this across formats / storage providers but those are all separate issues so this can be closed.
For sake of creating a trail back to this issue:
#5116 https://github.com/airbytehq/airbyte-internal-issues/issues/188 https://github.com/airbytehq/airbyte-internal-issues/issues/187 #5111 #5110 #5109 https://github.com/airbytehq/airbyte-internal-issues/issues/186 https://github.com/airbytehq/airbyte-internal-issues/issues/185 https://github.com/airbytehq/airbyte-internal-issues/issues/184 #5105 #5103 #5102