(#25) Create very basic implementation of Local Docker Compose #97

mdelapenya · 2019-09-11T06:57:06Z

What is this PR doing?

This is the very beginning of an initiative to port the Docker-Compose approach from the Java implementation to Golang.

It simply creates the Local Docker Compose approach (no socat container at this moment), which forces developers to have the docker-compose binary in PATH.

Following Java's approach, a LocalDockerCompose (LDC) object needs to be created with an array of paths to docker-compose.yml files, and an identifier (maybe this second field could be internally created).

It's possible to configure the LDC with a command (i.e. "up -d"), which will be executed after the method Invoke is called.

We have added a convenient Down() method, to be executed after tests run to destroy resources created by docker-compose.

Besides that, we are parsing Docker Compose YAML file to capture service names, and put into an array, so that it's possible to perform operations over the services in the compose file, accessing the compose.Services field.

Why is it important?

This will allow developers to configure more elaborated setups for testing, like an Elasticsearch + Kibana, or Metricbeat + MySQL, as an example, being those services coordinated and scheduled by Docker Compose under the hood, with no need to orchestrate them on the programatic side.

On the other hand, using declarative way of describing services is more familiar to developers (🤞), so the entry barrier could be lower.

How to test it?

I added a few unit tests, which verify that there are no errors after executing the compose, checking that environment variables are properly propagated from compose to the containers, and destroying it in a deferred manner.

Related Issues

Closes #25

mdelapenya · 2019-09-16T13:35:04Z

Hey @gianarb, any feedback with this one?

mdelapenya · 2019-09-18T23:34:15Z

Closing for travis rebuild

gianarb · 2019-09-23T13:34:32Z

Sweet! The direction is right! I am trying it just now!

gianarb · 2019-09-23T13:38:04Z

I am looking at how testcontainers-java does that.

https://www.testcontainers.org/modules/docker_compose/

Note that it is not necessary to define ports to be exposed in the YAML file; this would inhibit reuse/inclusion of the file in other contexts.

Instead, Testcontainers will spin up a small 'ambassador' container, which will proxy between the Compose-managed containers and ports that are accessible to your tests. This is done using a separate, minimal container that runs socat as a TCP proxy.

We should do the same, mainly because we can avoid the ports collision, but I am not sure about how LocalCompose in used in Java. Do developers tend to use a single docker-compose file for development and all tests or they write more of them (worst case one per test) using it as a specification language? @bsideup

mdelapenya · 2019-09-23T14:35:47Z

Yeah, this first proposal is trying to address the easy one: the local docker-compose approach, which is totally decoupled from the one with the minimal socat container.

As decoupled they are, I'd review them separately. Wdyt?

gianarb · 2019-09-29T22:06:11Z

Something I do not see from the examples is around how do you think we can get URLs to access containers deployed with docker-compose? Let's say we have a MySQL and we should like to get the service endpoint for it? @mdelapenya
Thanks!

mdelapenya · 2019-09-30T07:03:59Z

Ok, let me check out Java implementation for that. I agree that there should be a manner to access the container from the compose

Thanks!

mdelapenya · 2019-10-29T07:15:06Z

Hey @gianarb I don't think the local compose for Java supports accessing a service. I looked through the code and asked in their repo (testcontainers/testcontainers-java#1928) and did not find a manner to access the service 🤷‍♂

mdelapenya · 2019-10-29T11:34:03Z

I think this PR is ready for review

mdelapenya · 2019-10-29T17:09:18Z

Reopening for relaunching travis

mdelapenya · 2019-10-30T08:04:27Z

Tests are passing locally, closing and reopening again

mdelapenya · 2019-10-30T08:40:01Z

@gianarb not sure why, but Travis fails consistently with:

--- FAIL: TestLocalDockerComposeWithMultipleComposeFiles (2.35s)
    compose_test.go:114: {<nil> read |0: file already closed read |0: file already closed}
FAIL
FAIL	github.com/testcontainers/testcontainers-go	129.580s

although tests are passing locally. Can you double check please?

mdelapenya · 2019-11-12T17:06:35Z

Reopening and checking Travis 🤞

mdelapenya · 2019-11-14T00:31:09Z

@gianarb Any feedback about this one?

ClaytonNorthey92

I know it is a bit of work, but would it be possible to add a test that has more than one service in docker-compose and test the interaction between them?

for example, if we could use nginx or envoy to reverse proxy a request, for example, I might want to write to redis, and read that value from redis, so I might want a docker compose that I could test functionality that

write <key/value> --> nginx/envoy --> redis
read <key/value> --> nginx/envoy --> redis

I think this would be a common use case that would be a good use case for docker-compose to perform, thoughts?

ClaytonNorthey92 · 2019-11-15T19:05:41Z

utility.go

+var alfabet = []rune("123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz")
+
+// RandomString returns a random string
+func RandomString(length int) string {


would it be better to generate a UUID? rather than write our own RandomString function here? if it is just being used as an identifier, we just need to guarantee its uniqueness, no?

Totally agree, and I'm using it only for tests, which makes no sense to maintain new code.

Thanks for the feedback!

ClaytonNorthey92 · 2019-11-15T19:07:04Z

compose.go

+
+// LocalDockerCompose represents a Docker Compose execution using local binary
+// docker-compose or docker-compose.exe, depending on the underlying platform
+type LocalDockerCompose struct {


would it be possible to add a usage example to the README? I like that I can read through the tests for examples, but I think our README is lacking features in general

mdelapenya · 2019-11-18T09:02:32Z

I know it is a bit of work, but would it be possible to add a test that has more than one service in docker-compose and test the interaction between them?

for example, if we could use nginx or envoy to reverse proxy a request, for example, I might want to write to redis, and read that value from redis, so I might want a docker compose that I could test functionality that

write <key/value> --> nginx/envoy --> redis
read <key/value> --> nginx/envoy --> redis

I think this would be a common use case that would be a good use case for docker-compose to perform, thoughts?

Wouldn't it mean testing that docker-compose works? I agree in adding a new test with a combination of services that interact with each other (i.e. a service that populates a database), but testing the network and the dns discovery would be a responsibility for docker guys imho, which I'm sure it's something they are already doing and it's working 😃

ClaytonNorthey92 · 2019-11-18T16:56:28Z

I know it is a bit of work, but would it be possible to add a test that has more than one service in docker-compose and test the interaction between them?
for example, if we could use nginx or envoy to reverse proxy a request, for example, I might want to write to redis, and read that value from redis, so I might want a docker compose that I could test functionality that
write <key/value> --> nginx/envoy --> redis
read <key/value> --> nginx/envoy --> redis
I think this would be a common use case that would be a good use case for docker-compose to perform, thoughts?

Wouldn't it mean testing that docker-compose works? I agree in adding a new test with a combination of services that interact with each other (i.e. a service that populates a database), but testing the network and the dns discovery would be a responsibility for docker guys imho, which I'm sure it's something they are already doing and it's working 😃

ah yes, you're right! good point!

It is shared by invoke and down

It could disclose environment variables

Referenced from https://www.testcontainers.org/modules/docker_compose/

I'm not adding information about the environment to avoid disclosing sensible values, but it could be easily added

Read from: https://blog.kowalczyk.info/article/wOYk/advanced-command-execution-in-go-with-osexec.html

It was just a matter of setting the container name at docker-compose level, because we just want to access the container to read its environment variables and do some calculations with them.

mdelapenya · 2020-04-16T10:44:58Z

Hey @gianarb, I rebased the branch and updated the PR

gianarb

Thank you for your patent!! 😄

mdelapenya force-pushed the 25-docker-compose branch 2 times, most recently from b8622f4 to 38810f1 Compare September 11, 2019 07:22

mdelapenya closed this Sep 18, 2019

mdelapenya reopened this Sep 18, 2019

mdelapenya changed the title ~~[WIP] (#25) Create very basic implementation of Local Docker Compose~~ (#25) Create very basic implementation of Local Docker Compose Oct 29, 2019

mdelapenya closed this Oct 29, 2019

mdelapenya reopened this Oct 29, 2019

mdelapenya closed this Oct 30, 2019

mdelapenya reopened this Oct 30, 2019

mdelapenya closed this Oct 30, 2019

mdelapenya reopened this Oct 30, 2019

mdelapenya closed this Nov 12, 2019

mdelapenya reopened this Nov 12, 2019

ClaytonNorthey92 reviewed Nov 15, 2019

View reviewed changes

mdelapenya added 25 commits April 16, 2020 12:37

(#25) Extract default environment generation to a function

47362c1

(#25) Extract compose execution to a function

d9e1579

It is shared by invoke and down

(#25) Support passing multiple docker-compose files

2604294

(#25) Support checking for absent env vars in tests

4807a91

(#25) Make tests more readable

80f3758

(#25) Remove unneeded log

7614627

It could disclose environment variables

(#25) Use Google's UUID for generating compose's identifier

28b4ce4

(#25) Add local compose to the docs

866cd2a

Referenced from https://www.testcontainers.org/modules/docker_compose/

(#25) Rename simple compose test resource

26eee98

(#25) Add a test with a more complex scenario

8096d05

(#25) Fix path in test

efa9db2

(#25) Add info about the command and the arguments when compose fails

3c26102

I'm not adding information about the environment to avoid disclosing sensible values, but it could be easily added

(#25) Create defer functions the soonest

a07447a

(#25) Add comments and Go Examples for docs

a3e1f0b

(#25) Do not override ports on Travis CI

ef1c203

(#25) Support getting the services from the compose files

71af051

(#25) Execute go mod tidy

956a03b

(#25) Provide more information when failing a Compose test

446944d

(#25) Extract array population on exec errors

b587960

(#25) Enhance exec error

661033b

(#25) Optimise reading for docker-compose output

88efbd0

Read from: https://blog.kowalczyk.info/article/wOYk/advanced-command-execution-in-go-with-osexec.html

(#25) Do not assert the environment variables in the simple test

9570dd9

(#25) Fix tests reading environment variables from the compose file

7182ddc

It was just a matter of setting the container name at docker-compose level, because we just want to access the container to read its environment variables and do some calculations with them.

(#25) Add troubleshooting Travis guide

6830723

(#25) Run go mod tidy

03c0820

mdelapenya force-pushed the 25-docker-compose branch from ae22e2a to 03c0820 Compare April 16, 2020 10:42

gianarb approved these changes Apr 20, 2020

View reviewed changes

gianarb merged commit a911c8e into testcontainers:master Apr 20, 2020

mdelapenya deleted the 25-docker-compose branch April 20, 2020 12:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(#25) Create very basic implementation of Local Docker Compose #97

(#25) Create very basic implementation of Local Docker Compose #97

mdelapenya commented Sep 11, 2019 •

edited

Loading

mdelapenya commented Sep 16, 2019

mdelapenya commented Sep 18, 2019

gianarb commented Sep 23, 2019

gianarb commented Sep 23, 2019 •

edited

Loading

mdelapenya commented Sep 23, 2019

gianarb commented Sep 29, 2019 •

edited

Loading

mdelapenya commented Sep 30, 2019

mdelapenya commented Oct 29, 2019

mdelapenya commented Oct 29, 2019

mdelapenya commented Oct 29, 2019

mdelapenya commented Oct 30, 2019

mdelapenya commented Oct 30, 2019

mdelapenya commented Nov 12, 2019

mdelapenya commented Nov 14, 2019

ClaytonNorthey92 left a comment

ClaytonNorthey92 Nov 15, 2019

mdelapenya Nov 18, 2019

ClaytonNorthey92 Nov 15, 2019

mdelapenya commented Nov 18, 2019

ClaytonNorthey92 commented Nov 18, 2019

mdelapenya commented Apr 16, 2020

gianarb left a comment

(#25) Create very basic implementation of Local Docker Compose #97

(#25) Create very basic implementation of Local Docker Compose #97

Conversation

mdelapenya commented Sep 11, 2019 • edited Loading

What is this PR doing?

Why is it important?

How to test it?

Related Issues

mdelapenya commented Sep 16, 2019

mdelapenya commented Sep 18, 2019

gianarb commented Sep 23, 2019

gianarb commented Sep 23, 2019 • edited Loading

mdelapenya commented Sep 23, 2019

gianarb commented Sep 29, 2019 • edited Loading

mdelapenya commented Sep 30, 2019

mdelapenya commented Oct 29, 2019

mdelapenya commented Oct 29, 2019

mdelapenya commented Oct 29, 2019

mdelapenya commented Oct 30, 2019

mdelapenya commented Oct 30, 2019

mdelapenya commented Nov 12, 2019

mdelapenya commented Nov 14, 2019

ClaytonNorthey92 left a comment

Choose a reason for hiding this comment

ClaytonNorthey92 Nov 15, 2019

Choose a reason for hiding this comment

mdelapenya Nov 18, 2019

Choose a reason for hiding this comment

ClaytonNorthey92 Nov 15, 2019

Choose a reason for hiding this comment

mdelapenya commented Nov 18, 2019

ClaytonNorthey92 commented Nov 18, 2019

mdelapenya commented Apr 16, 2020

gianarb left a comment

Choose a reason for hiding this comment

mdelapenya commented Sep 11, 2019 •

edited

Loading

gianarb commented Sep 23, 2019 •

edited

Loading

gianarb commented Sep 29, 2019 •

edited

Loading