Meeting: Test Tooling Breakout Session

[BethGriggs]

There were good and lengthy discussions about tooling for testing modules at the collaborator summit - I think those discussions will benefit from their own breakout meeting(s) (as discussed in #305).

Related issues:

• Workflow + Tooling for Package Maintainers #299
• Suggestion: CITGM for a module dependants - #84
• Demo of CITGM work/Module Insights - #179

Please indicate your interest in joining these discussions with a 👍 and I'll set up a Doodle to figure out a good time.

[lholmquist]

👍

[Trott]

Leaving a 👍here in addition to the reaction because I believe the reaction will truncate the list of folks if it gets too large. Sorry for the noise.

[BethGriggs]

Slightly late notice but I have created a Doodle to cover the next week (until Monday 27th) to gauge availability - https://doodle.com/poll/9dmaa52xyv66597g

I'll spin up a new Doodle for the following week if we cannot find any times that work this week.

[Trott]

/ping @boneskull

[Trott]

/ping @boneskull

Oops, sorry, this is "test tooling for package maintenance" and not "tooling for packages that do testing". Ignore my ping.

[boneskull]

i failed to ignore this

[BethGriggs]

I've updated the doodle to cover the first two weeks of February - hopefully, that'll give us more ability to find a good time for everyone - https://doodle.com/poll/9dmaa52xyv66597g

(Apologies to those that completed the previous one)

[mhdawson]

I added my availability but since I'm out next week don't wait for me.

[BethGriggs]

Thursday 13th February 5PM - 6PM UTC looks good for those that have completed the poll. I think we should start with that time and iterate from there. I'll work on adding to the Node.js calendar so that we can use the Zoom account.

So far it looks like it'll be @dominykas, @mhdawson, @andrewhughes101, and myself. Ping @nodejs/citgm in case they're interested.

[wesleytodd]

I will be there! Also thought I think some collaboration between us and @nodejs/tooling is a good idea, so @boneskull if you want to attend you are more then welcome!

[bcoe]

👋 anything you need from me, not sure if this is test tooling in general, or specifically CITGM stuff you're discussing?

[BethGriggs]

@bcoe, this will be more around discussing use cases/requirements for tooling to support the package maintenance efforts, rather than test tooling in general.

The types of discussion I anticipate will happen:

• How can we test across multiple Node.js versions?
• How can a module author test the health of their module tree?
• How can a module author test whether their changes are going to impact modules that depend on them?
• Can we leverage CITGM for this? Or should we prototype something else?

I hope that clarifies the scope a bit!

[BethGriggs]

Because the Website Redesign is at the same time as this call, we'll be unable to use the Community Zoom account. I've scheduled a meeting under my personal account but that means we will not be able to stream. It may be possible to record the meeting, and we'll aim to take some good minutes/notes and to populate issues on the package maintenance board.

Zoom - https://us04web.zoom.us/j/157787412

[BethGriggs]

Summary of discussions on 2020-02-13

How can a module author test the health of their module tree?

• For module authors testing the health of their parents/tree is less important.
• There is merit in CITGM having the ability to test a modules dependency tree for Node.js.

How can a module author test whether their changes are going to impact modules that depend on them?

• Can we leverage CITGM for this?
  • CITGM is resource-intensive.
• A suggestion of a service like GreenKeeper in reverse direction - figure out dependents on npm, sort/filter them, and then automatically run tests in that projects CI.
• Differentiate between what the cause of the failures - were the module's tests passing before?
• Subscription/opt-in service for the modules?
• How do we manage forks?
• What if the module has an LTS policy?

Tooling prototype

1. Get a list of the modules that depend on a given module.
   • via npm?
     • check whether there is an npm API to find out the list of depended on
     • open RFC for deps vs dev deps
   • via GitHub API?
2. Apply filters to those modules.
   • filter by stability, downloads, etc.?
3. Running tests.
   • Open PR to update the package.json to updated module version, which will run CI in the modules own CI setup.
   • Could be a draft pull request that is then updated to be non-draft when the module version is released.

Actions

• Adding this to the package maintenance meeting agenda to provide a summary.
• Work to start on the prototype/call for volunteers.
  • @andrewhughes101 may have some time to help out
• We'll spin-off breakout meetings as required.

[BethGriggs]

Uploaded the recording to YouTube - https://www.youtube.com/watch?v=F3WDWokqrhU

[thescientist13]

Regarding this point, and not sure if it came up during the call

How can we test across multiple Node.js versions?

Can GitHub usage be assumed for the most part, or at least the representative majority of package authors? If so, would GitHub Actions be applicable to this item? Their workflows support the ability to define a build matrix, so that that this exact use case could be implemented.

ex.

strategy:
  matrix:
    node: [8, 10, 12]
steps:
  - uses: actions/setup-node@v1
    with:
      node-version: ${{ matrix.node }}

Just a thought and something I would be happy to help prototype / work on if desired.

[ljharb]

@thescientist13 every CI allows testing on multiple node versions and some kind of a build matrix; github actions isn't unique here.

[thescientist13]

Cool, yeah, I figured as much, but wasn't sure if the topic itself (build matrix) had come up specifically as a suggestion / answer here. I'm sure it's about the same API from provider to provider, so probably not a lot of effort to maintain some examples for popular services (CircleCI, Travis, GitHub).

Not sure about all the policies on when / what and should / can be endorsed outside of the NodeJS ecosystem but I think I just defaulted to GitHub since it would likely be the past of least resistance for the majority of authors and since NodeJS itself is using GitHub (already). I have no issue suggesting other services if that's appropriate here and working on examples for those as well.

[dominykas]

I'm sure it's about the same API from provider to provider, so probably not a lot of effort to maintain some examples for popular services

@thescientist13 there's some ongoing discussion to provide an "official" and "auto-updating" set of matrices for common services in #280.

[thescientist13]

Thanks! Let me know if you need any help with anything. 👍

[BethGriggs]

I think this can be closed as the meeting happened - #327 will cover the first prototype of the tool that was discussed.