Add Tremor connector to read environment variables

[anuprout]

Describe the problem you are trying to solve

It will be nice to have functions to be used in Tremor scripts to get environment variables. It will be very much helpful in deploying the same script on multiple environments without templating it.

Describe the solution you'd like

Add functions such as :
get_env("variable_name")
get_env_default("variable_name","default_value")

[Licenser]

I've been thinking about this yesterday and I like the idea but think putting this in a function is very problematic. In general environment variables are useful so I will try to layout the issues this approach has and offer an alternative that avoids those problems to see if it would solve the issue.

The main two issues I see resulting from functions giving access to environment variables would be:

Problem 1: Testability

Pipelines and scripts are testable because their execution is deterministic. Given the same inputs, the same outputs will be produced - that goes as far as that we use a deterministic random number generator to even make the rand functions ensure determinism.

Introducing env::... will break this determinism, now the pipeline is dependant on the external state that is injected 'magically' at some point of the execution.

With testability it also makes debugging harder as, again the output of the pipeline no longer purely depends on its inputs.

This additional vector will also increase complexity to think off when designing a system, now it's scripts + inputs + environment.

This is the smaller of the two complications. Adding mocking for those functions, enabling the test framework to overwrite env settings, things like that all need to be done to make it testable - a lot of work but possible.

Problem 2: Clustering

This would completely break clustering. If a pipeline depends on the environment it is started in it can't be moved, it's no longer location independent. That is a core requirement for clustering.

In a clustered setup pipelines, need to be self-contained. Otherwise, problems could raise like (shortlist off the top of my head)

• missing variables
• different variables on different hosts
• conflicting variables between scripts (two or more scripts use the same variable)

This is a big one and pretty much renders this approach a no-go.

Solution: Connectors / linked ramps

We already have a mechanism to disintermediate this kind of need. Connectors / Linked ramps. The DNS connector is an example here, as well as the KV one.

In short - making this a connector would solve all the problems :D!

This is also more in line with the general system design and philosophy of tremor. Connectors are what is used to interact with the outside world - environment variables are the outside world, it makes sense to access them over a connector.

How this solves testing

Connectors are completely independent of ramps, and for testing can be stubbed out with mocks that just provide the same data without the need for the same logic. We do this already by using the file or blaster source when testing pipelines. So this just works out of the box 🚀

How this solves clustering

Connectors are not, and can not, be location independent, while it might not make it into the very first release of clustering but we know we'll need targeted connectors, moving them around isn't a reasonable solution - this problem comes up already when looking at HTTP server connectors; we can't just move the HTTP endpoint and expect everything to keep working - same goes for TCP or UDP, those need to already be bound. The same would apply to a connector for environment variables.