Support loading environmental data from files and environment variables

[ramonacat]

Describe the problem you are trying to solve

Often it is needed to read tokens, or other static information and use it in pipelines. Currently there's no explicit support for such things (it's possible to do it with file/env connectors, but it's not a straightforward solution).

Describe the solution you'd like
Simple syntax that allows providing static file contents or environment variables to pipelines.

Notes
Current solution (working, but seems unnecessarily complicated):
pipes.yaml:

onramp:
  - id: metronome
    type: metronome
    config:
      interval: 1000
  - id: secret
    type: file
    codec: string
    config:
        source: "/etc/tremor/sosecret"
offramp:
  - id: stdout-output # The unique id for binding/mapping
    type: stdout # The unique type descriptor for the offramp ( stdout here )
    codec: json # The underlying data format expected for application payload data
    config:
      prefix: ">> " # A prefix for data emitted on standard output by this offramp
binding:
  - id: pipeline # The unique name of this binding template
    links:
      "/onramp/secret/{instance}/out":
        - "/pipeline/pipeline/{instance}/file"
      "/onramp/metronome/{instance}/out": # Connect the input to the pipeline
        - "/pipeline/pipeline/{instance}/in"
      "/pipeline/pipeline/{instance}/out": # Connect the pipeline to the output
        - "/offramp/stdout-output/{instance}/in"
mapping:
  /binding/pipeline/01:
    instance: "01"

pipeline.trickle:

define script secretiser
script
    use std;
    use tremor;

    match tremor::origin::scheme() of
        case "tremor-file" => let state = {"secret": event}, drop
        default => {"secret": state.secret, "metronome_timestamp": event.ingest_ns, "id": event.id}
    end;
end;

create script secretiser;
create stream file;

select event from file into secretiser;

select event from in into secretiser;
select event from secretiser into out;

[Licenser]

I think this is a duplicate of #1136 which details the problem with it and possible solutions

[ramonacat]

Nice, but #1136 is closed. I understand that this is not easy, but I still think we should look for a solution.

[Licenser]

It is closed because the env on-ramps is as the solution we landed on, in what way is that not working here?

[ramonacat]

We're using the file onramp here (which I assume works similarly to env here). It works, but it feels unnecessarily complex, I expect this would get even more messy with more than one file.
So, this issue is not about getting it to work (because there is a working solution), but more about developer experience.

[Licenser]

I see so it's not about can but about feel. Perhaps a library that includes a reader and enriched to be plugged in would help here? The config posted is 0.11 so with the yaml it's bound to be painful. But We have the option of predefining pipelines and connectors as libraries now so if this is a reoccurring topic that logic could be put into into one.

That way we avoid the salad of problems outlined in the original issue, and e avoid introducing a point solution for a single problem, we avoid extending the language, and it becomes a 3 or 4 liner. In addition it can be tested in user space and include in core later.

[Licenser]

Closing this as it'll be impossible with clustering