Stable gemset for integration testing

[marcotc]

This PR creates two distinct CircleCI pipelines for ddtrace:

• build-and-test: executes all tests using a stable gemset. Runs on every commit. Generates coverage report and prerelease artifacts.
• edge: executes all tests using the latest available gems. Runs once per day.

This should allow us to have a much more stable CI pipeline for daily development, while also being aware of any issues created by new external gem releases.

To accomplish this, all Gemfiles and lock files generated by Appraisals are now checked in the repository. Because Appraisal doesn't allow changing the destination directory of the generated gemfiles (I tried to modify the gem to support it, but ended up with mixed results), and it generates overlapping names (./gemfiles/contrib.gemfile for all versions of Ruby, for example), I've introduced some code in Appraisal and Rakefile to ensure that each gemset has a unique name.
This should have to effect on daily development workflow.

One unintended upside is that CI build steps are much faster: around 1 minute or less for CRuby variants.

[ivoanjo]

Looks good to me! Seems like a reasonable balance between getting advance warning about these upstream changes, and not hindering all contributors' work.

Would be great if we could do the same thing for the Gemfile as well. While there's no single Gemfile.lock that works for every single Ruby version we test with, we could as well check in Gemfile-ruby-2.0.0.lock, ... etc as well and make use of it. (Bundler allows the Gemfile name to be controlled with BUNDLE_GEMFILE).

Nevertheless, I consider the above as an extra that we can do at any later time, and thus not in scope of this PR.

I see to reason not to merge this in ASAP, so here, take my approval.

[ivoanjo]

Would it be possible to add an &ci_jobs id to the jobs above, and here just use jobs: *ci_jobs?

There's already quite a lot of copy/paste going on to maintain this list, would be great if we could avoid yet another thing that needs to be kept in sync.

[marcotc]

Unfortunately the value expected in the jobs: field is an array, and there's no way to merge arrays in YAML. In other words, we can't have ci_jobs with a list of jobs but also add a few more to it (coverage, lint, deploy release) 😢

[ivoanjo]

😭

[ivoanjo]

Might be worth documenting this -- e.g. in both sections, state "This is exactly the same thing as [below/above], but with the [foo/bar/baz] sections [added/removed]", so it's harder to forget to update them, and to confuse why they are different.

[ivoanjo]

It's a bit of a shame that Appraisal doesn't seem to support the new gems.rb name for Gemfiles. Because it ends in .rb, it's not a "magic" name anymore so we'd get syntax highlighting :|

[delner]

I think this is a good way to keep CI from being overwhelmed by public gem changes, so overall 👍 to strategy!

It does however add a crazy number of gemfiles.

• Is there any way to reduce/auto-generate at least some of these? I understand the point of locking is to commit some specific versions, but maybe we could keep just the lock files? (If that's possible?)
• Would it be possible to put gemfiles in subdirectories (jruby/9.2.0.0/) instead of just a prefix? The flat structure makes it a bit overwhelming to deal with almost 400 files in one directory.

Just some thoughts.

[delner]

As we discussed, it would be cool if there's some way to have this workflow report to Slack when the build finishes (success or failure) so we can keep an eye on it.

[marcotc]

@delner

Is there any way to reduce/auto-generate at least some of these? I understand the point of locking is to commit some specific versions, but maybe we could keep just the lock files? (If that's possible?)

I guess it would likely be _)technically possible to only keep the lockfiles. I'm not sure if it would play well with any changes to the Gemfile, when needed.

Would it be possible to put gemfiles in subdirectories (jruby/9.2.0.0/) instead of just a prefix? The flat structure makes it a bit overwhelming to deal with almost 400 files in one directory.

I tried this at first, but Appraisal has the path ./gemfiles/*.gemfile.* hard-coded. I created a fork of Appraisal to see if I could change it, but turns out it's too complicated to make it flexible enough for it being accepted upstream, and maintaining a fork of Appraisal doesn't seen too manageable.

Overall, I see the gemfiles folder similar to a vendor folder: you don't review everything that's in there, but you want to make sure that new changes after the initial commit are sane.

This is hopefully the last commit to have 400 files changed: all future changes should only affect a handful of lines across a subset of these files.

[marcotc]

Ruby 2.6 build is failing because of some misconfiguration bundle platform. I'll dig into it.

[ivoanjo]

Could you add some more details on this change? I'm not entirely sure why this needs to be removed.

[marcotc]

This needs to be removed because - gemfiles-2.1:/app/gemfiles gives each container its own gemfiles, because before we didn't check it into the repo, and each container generated its own set of gemfiles.

Now that it's checked in, we want gemfiles to be shared across all containers, which is done indirectly by - .:/app a few lines above.

[marcotc]

@delner I'm going to merge this with @ivoanjo's blessing, as this will stabilized our builds across the board.

This is not a perfect solution and I'm more than happy to make any changes we see fit to improve it going forward.

In it's current state, it addresses a longstanding needed we've had for CI stability.

I'll schedule the work on daily CI alerts and improvements to the top level Gemfile (which is shared by all versions).