Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLI feature set #129

Closed
kytrinyx opened this issue Mar 30, 2017 · 34 comments · Fixed by exercism/cli#410
Closed

CLI feature set #129

kytrinyx opened this issue Mar 30, 2017 · 34 comments · Fixed by exercism/cli#410

Comments

@kytrinyx
Copy link
Member

kytrinyx commented Mar 30, 2017

I have been pondering rethinking the feature set for the client(s) for a while. The current feature set grew organically, and has some rough edges and surprising corners.

I think it would be interesting to take a step back and think through what the feature set should be, given what we now know about the app.

We have a number of open issues that are about tweaking the feature set:

And another issue about feature parity:

And finally, we have some concerns that need to be taken into account:

What would be the consistent set of interactions?

/cc @exercism/cli-committers @holandes22

@ErikSchierboom
Copy link
Member

So basically a wishlist of which features we would like the CLI/GUI to have? Would you like us to respond in this issue, or in the mentioned issues?

@kytrinyx
Copy link
Member Author

So basically a wishlist of which features we would like the CLI/GUI to have?

Yepp. And what it look like to be the best possible (most consistent, least surprising) user experience.

Would you like us to respond in this issue, or in the mentioned issues?

Let's discuss it here.

@ErikSchierboom
Copy link
Member

Okay. Let me kick-off things then.

CLI

In general, I'd like the CLI to help me with:

  • Fetching an exercise
  • Submitting an exercise
  • Showing progress

Fetching an exercise

  • Fetch the next exercise for a specific language track
  • Fetch all exercises for a specific language track
  • Re-fetch an already fetched exercise for a specific language track
  • Re-fetch all fetched exercises for a specific language track

Submitting an exercise

  • Submit an exercise for a specific language track using a single source file
  • Submit an exercise for a specific language track using multiple source files
  • Submit an exercise for a specific language track using a directory
  • Submit all exercises for a specific language track using a directory

Showing progress

  • List number of (un)completed exercises for a specific language track
  • List number of (un)completed exercises for all language tracks
  • List number of (un)completed language tracks
  • Request diff for an exercise for a specific language track (comparing local file with remote file)

GUI

For the GUI, its most important tasks to me are:

  • Displaying solutions
  • Displaying comments
  • Displaying language track documentation
  • Displaying general documentation

Displaying solutions

  • View a submitted solution
  • View the iterations of a solution
  • Compare iterations of a solution with a diff
  • View your submitted solutions for a specific language track
  • View your submitted solutions for all language tracks
  • View your submitted solutions for a specific exercise (same exercise in multiple languages)
  • View all submitted solutions for a specific exercise on a specific language track
  • View all submitted solutions of a specific user for a specific language track
  • View all submitted solutions of a specific user for all language tracks

The above items will (of course) have to take into account what solutions you are allowed to see.

Displaying comments

  • View a submitted comment
  • View all submitted comments for a specific solution
  • View all submitted comments for a specific language track
  • View all submitted comments for a specific user for a specific language track
  • View all submitted comments for a specific user for all language tracks
  • Submit a new comment
  • Editing an existing comment

Displaying language track documentation

  • Display about documentation
  • Display installation documentation (how to run install)
  • Display tests documentation (how to run tests)
  • Display learning resources
  • Display contributing guide
  • View all available exercises for the language track
  • Link to progress page for the language track
  • Link to submitted exercises for the language track
  • Link to submitted comments for the language track

Displaying general documentation

  • Getting started documentation for the website
  • Getting started documentation for the CLI
  • Overview of website (what can you do where)

In general, this is mostly how things already work, with some more polishing. One particular aspect I would really like to see improved, is the discovery of data. At the moment, it is not doable to quickly show all comments I made. In general, there should be overview pages for exercises and comments, with several filtering options to drill-down.

@Tonkpils
Copy link

This is a list of current, relevant, commands for the exercism CLI:

COMMANDS:
     download, dl  Downloads a solution given the ID of the latest iteration.
     fetch, f      Fetches the next unsubmitted problem in each track.
     list, li      Lists the available problems for a language track, given its ID.
     open, op      Opens exercism.io to your most recent iteration of a problem given the track ID and problem slug.
     restore, r    Downloads the most recent iteration for each of your solutions on exercism.io.
     skip          Skips a problem given a track ID and problem slug.
     status, st    Fetches information about your progress with a given language track.
     submit, s     Submits a new iteration to a problem on exercism.io.
     tracks, t     Lists the available language tracks.

From the CLI standpoint, the features you've requested are there we just need to polish them and their outputs. Also, it would help a lot to have the command names match what they do. We could also take this time to work on a v2 of the CLI so it's a bit easier to iterate over.

@kytrinyx
Copy link
Member Author

kytrinyx commented Apr 17, 2017

One particular aspect I would really like to see improved, is the discovery of data.

Yes, me too. I think that we should consider what it would mean for someone to do all discovery via the CLI rather than through the website (we should make both possible, of course).

For the GUI, its most important tasks to me are:

@ErikSchierboom Do you mean the website, or the GUI which is the equivalent of the CLI (on the user's local machine)?

We could also take this time to work on a v2 of the CLI so it's a bit easier to iterate over.

I would love that, and would be happy to implement a v3 of the API (all in http://exercism.io/api/v3, rather than having two different domains that we talk to). Also I'd love to use spf13/cobra for that. I've come to like it better than the one we're using.

I'd like to have a --debug flag on every command, which would output the HTTP headers that it sends and receives, what endpoints it talks to, and full paths of anything files that it does anything with.

We might consider whether there's a level between --debug and normal. --verbose. (If so, what's the difference?)

We might also want to consider whether a --force flag makes sense on each of the commands.

Another potential thing to think about is providing an --interactive which would do extra hand-holding.

I'd like all of the user-facing text (error messages, confirmation messages, notices) to come from the API so that we can easily change these without having to release a new client. This will also let us do A/B testing on the copy.

I think that we should consistently require people to specify which track they're acting on (except when saying which tracks are available), and only act on one track at a time. We're only going to get more and more tracks, and more and more exercises, so if we act on all the tracks, that's guaranteed to develop performance problems.

When delivering files (either from the CLI/GUI to the API or from the API to the CLI/GUI) I think we should do chunked transfers of zip files. That would avoid our current problems with encoding (can't send binary in a JSON file) as well as problems with performance when acting on multiple exercises in a single track (we're only ever going to get more exercises).

For error messages, we should always tell people what is wrong and how to fix it. Since the how to fix it will depend on whether it's the CLI or the GUI, we will need to account for having the main text in the API, and provide some way to interpolate known values on the client side.

@kytrinyx
Copy link
Member Author

kytrinyx commented Apr 17, 2017

Some thoughts about fetching exercises, based on the above thoughts, as well as comments from @petertseng in exercism/cli#369

If we can infer what the person wants to fetch based on the current directory, then we should do so.

If we can't then we should provide an error message and suggest how to fix it (either by changing directory, or specifying flags).

If we make it so that people take an explicit action in order to start a track, then for many people fetch will be on the only track that they've started.

To do this, we would need to store the fact that they've explicitly chosen to start a track independently of them submitting their first exercise (because they might not submit an exercise, even though we know they've chosen to do that).

We'd need to store this on the server side of things, because people use multiple computers (e.g. work and personal).

In other words, by making exercism fetch behave in an ideal fashion, we probably need to work out how people start a track.

This also makes sense, because we support having global track-level files (one-time setup stuff for files that they would need to use across all the exercises, such as a testing library or package config).

This in turn also opens up the possibility of having a track-level tutorial, interactively walking people through the step-by-step, hand-hold-y version of how to do TDD etc.

Fetch the next exercise for a specific language track

Assuming that someone has started just one track, say Haskell, then we can assume that the CLI knows this. exercism fetch would fetch the next exercise.

We should consider whether we prefer options or flags for fetching a specific exercise. E.g.

# which is preferable?
exercism fetch --exercise leap # shortflag -e
exercism fetch leap

Re-fetch an already fetched exercise for a specific language track

Use the --force flag with the above command. Or perhaps consider --update (per Insti's suggestion).

Fetch all exercises for a specific language track

exercism fetch --all # shortflag -a

Re-fetch all fetched exercises for a specific language track

This is an interesting one. What is the ideal name to distinguish between all and the ones I've previously fetched? My first thought is --fetched but that might actually look weird. Maybe --again or --submitted? But submitted means that we wouldn't give you the next unsubmitted one.

exercism fetch --fetched

And do we store this on the server side so that people could fetch everything they've already fetched on a different computer?

I would posit that for consistency we should use --force if we want to overwrite existing files.

If we want to overwrite, then use --force with any of the above.

A related question is how we let people fetch their solution. We can infer the exercise from the directory, or accept a flag or both, but it gets weird:

exercism fetch --solution

The default behavior for fetch is the next available exercise, whereas fetching your solution needs a specific exercise. For that reason I would suggest that we make it so people have to specify the exercise in the same way as when they're fetching just the exercise.

And related to that, how do people specify that they want all the submitted solutions in a track?

exercism fetch --submitted --solution

The other question about solutions is whether or not we submit the test file as well. When people set up the track locally, we can put the pattern in the config file that the CLI reads, and always submit it.

On the website we could collapse the file in the UI so that it's minimized by default. This would remove the need to have versioning. People submit whatever version they want. Or we could use a different type of versioning that only the CLI cares about, in order to warn people that a new version of the exercise is available, and would they like to fetch it and update their existing solution.

@kytrinyx
Copy link
Member Author

Another flag to consider is --interactive which would be like --force but it would interactively ask the user before overwriting a file.

@kytrinyx
Copy link
Member Author

Thinking more about this, I'd really like to consider the implication of versioning the exercises in such a way that only the app and the client know the version, but that it doesn't affect the actual test and solution files.

We could store which version of a test suite someone has last submitted on the backend, which would let us tell the user about updated test suites either in the app or via the client.

There are some interesting questions around diffing and overwriting new versions of test suites in exercism/cli#362

@petertseng
Copy link
Member

I'm pretty lazy and like to type less, so it is good if arguments can be inferred, and if there is no chance of positional ambiguity then positional args are good.

Looks like these are the main things I use the CLI for:

  • fetching exercise files
    • if a new file exists that I don't have, write it (current behaviour: already does)
    • if a file differs from what I have locally, show diff (current behaviour: nothing) On download, optionally overwrite existing files cli#362 (comment)
      • it would be really helpful if the stub file were excluded from this, but that means something will need to know what a stub file is.
    • exercism f <exercise-slug> - track is inferred from my current directory (current behaviour: the argument in the one-arg form is the track name and fetches the next exercise) wishlist: exercism fetch $PROBLEM cli#369
    • exercism f - track and exercise inferred from current directory (current behaviour: fetch the next exercise in all tracks)
  • submitting
  • download someone's submission. current exercism dl behaviour suits me fine.

To me, it's not going to be a big deal if any of these features don't make it in. I would edit them into my local copy or write wrapper scripts that do it.

@petertseng
Copy link
Member

petertseng commented May 4, 2017

Hi. I've just started using another feature of the CLI:

@kytrinyx
Copy link
Member Author

@petertseng How often do you fetch specific slugs? When you do, where did you get the slug name? (list from the CLI? memory? The website?)

Would you be OK with exercism f -s hello-world as a shortcut for --slug and not having a positional argument?

I think we can probably do positional arguments, but it's going to be tricky because we have both tracks and slugs. If we always infer the track, though, and error if we can't, then maybe we could use the positional argument for the slug without too much confusion.

@petertseng
Copy link
Member

How often do you fetch specific slugs?

Every time I fetch (I can't put a specific frequency on how often I fetch, but I think the relative frequency is what's important to the question anyway). I know that I will never use the exercism fetch <language> form since I would rather look at the list and pick what exercise I do next, rather than either have to submit or exercism skip an exercise I don't want.

When you do, where did you get the slug name? (list from the CLI? memory? The website?)

I used to use the website, but these days I prefer the colour-coded (above) exercism l (aliased list to l instead of li) so that it's easier to tell what I haven't fetched yet (can't get that from website)

Would you be OK with exercism f -s hello-world as a shortcut for --slug and not having a positional argument?

As long as the API allows requesting a specific track+exercise, I imagine I will always say "yes, OK", because if the CLI's arguments do not exactly match my preferences I will simply modify my copy of the CLI until they do, or make a wrapper script that matches.

So, exercism f -s hello-world actually doesn't match my preferences, but it doesn't matter that it doesn't match, so I'm still saying OK.

@kytrinyx
Copy link
Member Author

if the CLI's arguments do not exactly match my preferences I will simply modify my copy of the CLI until they do, or make a wrapper script that matches.

:-) I think it's worth figuring out how people currently use the CLI, because I imagine that a lot of people would have your same preferences, but not be able to modify the CLI

@petertseng
Copy link
Member

petertseng commented May 19, 2017

It's a tough question because I can't really be sure how widespread my preferences are. When I was younger I used to overestimate, but it's possible I've overcompensated for that by now and am now underestimating.

My reasoning is: Because for my uses a single argument to exercism fetch always unambiguously specifies the exercise and not the track, I don't want to have to add -e or -s or anything.

This does get tricky with the two-argument form (both exercise and track are specified). It's actually not useful for me because I know I will always cd into the track directory before fetching (allowing the track to be inferred from my current directory).

But suppose someone opens the exercise files in an IDE and only opens a terminal to fetch. That person would probably not want to cd into the track directory and would instead want to be able to specify the track.

Then the interesting question is should that be specified as exercism f <track> <exercise> (matches the directory structure!) exercism f -t <track> -e <exercise> (flags mean you can specify them in any order), etc... and whether a choice made for the two-argument form feels inconsistent with whatever choice is made for the one-argument form.

(I know, there was talk about inferring the track by keeping something server-side, but I think at some point someone will ask for a way to specify both track and exercise in one command?)

@kytrinyx
Copy link
Member Author

I think that for the track it makes sense to infer it if we can, and then allow the use of a flag (-t and --track)to override or to specify in the cases where it can't be inferred.

@kytrinyx kytrinyx changed the title CLI/GUI feature set CLI feature set May 20, 2017
@kytrinyx
Copy link
Member Author

kytrinyx commented Jun 30, 2017

As part of the new redesign, a lot will change on the website, and we'd like to tweak the CLI to feel more streamlined as well.

One big change that we want to make is to use the terminology download for download an exercise.

So exercism download clock would download the Clock exercise in the current track (optional --track flag to specify if ambiguous), and if you have submitted a solution, it would also pull down your most recent iteration of it. If there are unsubmitted changes, then it would not overwrite the files. It might show you a diff, it might interactively ask you if you want to overwrite it, or it might just bail and ask you to call the command again with the --force flag.

This raises the question what do we call it when you're downloading someone else's solution to play with it locally?

@petertseng
Copy link
Member

It would be possible to overload the term.

exercism download clock is sufficiently distinctive from exercism download fb1310e51c6f4ee9b21c51a4dc328087 that it is possible to determine for a given argument whether it designates an exercise or a submitted solution, right?

It would be possible, but I haven't expressed an opinion on whether it would be a good idea.

@kytrinyx
Copy link
Member Author

Hm. Yeah, it would be possible. My gut feeling about it is that I'd rather not, but I don't know that I can make a cogent argument about why we shouldn't overload it.

Maybe if we always passed a URL if we meant someone else's solution it might feel different. Not sure.

@wobh
Copy link

wobh commented Jun 30, 2017

I'd like to be able to submit from a path that's a symbolic link to the configured exercism path. Nice-to-have.

@kytrinyx
Copy link
Member Author

@wobh Would you open an issue in the CLI repository about the symlink thing and describe your use case? I think that it shouldn't be hard to implement.

@kytrinyx
Copy link
Member Author

kytrinyx commented Jul 29, 2017

Ok, everyone. I've gone through the discussion in detail, and the various linked issues, and I've tried to come up with a suggested user interface for a small initial feature set for the Nextercism launch.

Proposal: Nextercism CLI

For the beta launch I suggest focusing on a minimal feature set:

  • configure
  • download exercise
  • submit exercise

Before the public launch, we should flesh this out to include debug, upgrade, and open, though I think that will not be much work, since these will probably be identical to (or very similar to) the existing commands.

Aliases and Shortflags

All CLI commands should have a single-letter alias, and flags should have a long and short version. By default, the short version is the first letter of the long version.

We may choose troubleshoot rather than debug in order to avoid the collision with download.

Response Headers

All responses from the Nextercism API will provide:

  • X-Exercism-CLI- containing the version of the latest CLI release
  • X-Exercism-CLI-Settings - containing the version of the configuration settings

TODO: these are tentative header values, we need to discuss/confirm.

This will enable the CLI to update settings and prompt the user to upgrade the CLI.

Authentication

Authorization: Bearer $TOKEN

Authentication happens via an HTTP request header. This header is sent on every request from the client.

Commands

Configure (shortname: c)

$ exercism configure
$ exercism configure --token=abc123

The configure command will call an API endpoint that provides values for all user-facing copy (confirmation messages, error messages, etc). It will also provide a list of API endpoints to use. Lastly it can take flags for user-provided values, as well as to override settings that are useful during development (e.g. the base API hostname).

This will allow us to update the copy without releasing a new version of the CLI, as well as let us do A/B testing on copy based on flags in the database, and more easily deprecate old API endpoints or move the API should we need to.

The CLI will ship with defaults (e.g. write default values to the config, which will then be overwritten after the first call is made).

Download (shortname: d)

$ exercism download $EXERCISE

Users will no longer be able to download exercises that are not yet available to them on the site, however typically more than a few exercises will be available to them at once, so this should be a good experience.

The API might return a 403, meaning that you can't access the exercise.

The track can almost always be inferred based on context. Most people only work on one track. At other times we can determine the track based on the current location on the file system. If we are unable to disambiguate, we ask for confirmation. People can specify the --track flag if necessary.

If you download an exercise and there are already files there, we will either ask you if you want to overwrite the file (each file individually) or write them with a separate suffix.

TODO: we need to decide whether to ask about overwriting, writing with a suffix.

If you run the download command with --force, it will automatically overwrite any existing files.

You can download someone else's solution to something given a valid UUID.

$ exercism download --uuid $UUID
$ exercism download --url $URL

This downloads it to a separate directory tree (TBD). E.g.

./$EXERCISM_DIR/users/:username/:track/:exercise

Submit (shortname: s)

$ exercism submit
$ exercism submit $DIRECTORY
$ exercism submit $FILE1 $FILE2

Calling submit with no arguments will try to submit the current directory. If the current directory is not an identifiable exercise, we can either ask interactively what track (if multiple) and what exercise. Or we can error out and ask them to specify the path to the directory to submit.

@kytrinyx
Copy link
Member Author

/cc @nywilken @nilbus @exercism/cli-committers in case you hadn't seen this thread. If you have time to read through the proposal for the new CLI for nextercism I'd love your feedback.

@ErikSchierboom
Copy link
Member

The track can almost always be inferred based on context. Most people only work on one track. At other times we can determine the track based on the current location on the file system. If we are unable to disambiguate, we ask for confirmation.

Calling submit with no arguments will try to submit the current directory. If the current directory is not an identifiable exercise, we can either ask interactively what track (if multiple) and what exercise.

I love both of these things. They will remove some of the barrier and learning curve there is to the CLI.

@kytrinyx
Copy link
Member Author

@ErikSchierboom Yeah, there have been some great ideas in this and earlier threads about simplifying. Another thing that we're going to do with the new one is ask for specific things interactively if they're not set. Ideally the new Exercism user will only need to know about download and submit, until they get comfortable enough to explore further.

@nywilken
Copy link

@kytrinyx I'm still working my way through the discussion points to get a sense of what/why certain things are being proposed.

But I do have some initial thoughts.

Response Headers

Great idea, this is great for informing about updates, depreciations, and general support.

Download

You can download someone else's solution to something given a valid UUID.
...
This downloads it to a separate directory tree

I have some hesitation to downloading into a separate directory. The hesitations being:
⁃ File reachability: A user will have to change directories to get to the file, diff it,or even edit it.
⁃ Responsibility for the cleanup: The user may anticipate a clean command.

Has the option of downloading another users' solution alongside the users' implementation with a file name like leap.username.ext been considered? I think, it would improve file reachability and possibly allow for the reuse of the suffix feature.

@Tonkpils
Copy link

Has the option of downloading another users' solution alongside the users' implementation with a file name like leap.username.ext been considered? I think, it would improve file reachability and possibly allow for the reuse of the suffix feature.

I like this idea, but I'm wondering if there's any potential issues with tracks/solutions that contain multiple files? (e.g a solution whose entire directory was submitted)

@kytrinyx
Copy link
Member Author

kytrinyx commented Jul 31, 2017

I have some hesitation to downloading into a separate directory.

The most important concern for me right now is that you should be able to commit everything for a track to a git repository. Other people's solutions shouldn't go inside that.

We would tell you where it got downloaded to so that you can change directory.

The path that I'm thinking of using is $EXERCISM_WORKSPACE/users/:username/:track/:exercise/* where * is all the files, recursively.

@NobbZ
Copy link
Member

NobbZ commented Aug 1, 2017

$EXERCISM_WORKSPACE/users/:username/:track/:exercise/

I liked the SHA/UUID beeing part of the path in the current version, it allowed me to compare different iterations locally side by side. I think we should keep a way to distinguish different iterations after downloading them.

@kytrinyx
Copy link
Member Author

kytrinyx commented Aug 1, 2017

I think we should keep a way to distinguish different iterations after downloading them.

How would you feel about numeric suffixes, so if you're looking at several iterations of Clock in Go by the user alice, it would be:

$EXERCISM_WORKSPACE/users/alice/go/clock
$EXERCISM_WORKSPACE/users/alice/go/clock-2
$EXERCISM_WORKSPACE/users/alice/go/clock-3

@NobbZ
Copy link
Member

NobbZ commented Aug 2, 2017

Assuming that a user will be able to delete iterations as he is now, I do see some ambiguities in this approach:

  • Alice submits, Bob downloads, Alice deletes and submits then, Bob pulls again. How will that get numbered?
  • Alice submits 2 times, Bob pulls second, Alice deletes first. Bob sees an iteration number that does not exist anymore.
  • Same as before, but Bob pulls both iterations. His iteration 1 is different to that one on the web page

I think there are more cases to consider which makes this a not-so-good-plan. I really do prefer to have the ID in the path. It also has the advantage that I can reconstruct a URL easily from the path (assuming that we will keep the currently used /:uuid route)

@kytrinyx
Copy link
Member Author

kytrinyx commented Aug 2, 2017

@NobbZ I need to think about this a bit more.

I really do prefer to have the ID in the path.

Do you prefer it for other things than reconstructing the URL? My plan for the exercism open command is to extend it so that it can take a path:

$ exercism open . # or exercism open path/to/exercise

We will be keeping some hidden metadata in the exercise directory that will let the CLI reconstruct the exercise.

My thinking with the sequential suffix on the directory was not to match the actual iteration number, but rather as a serial number indicating how many copies of the exercise you have locally.

@NobbZ
Copy link
Member

NobbZ commented Aug 3, 2017

Using it just as a local counter, unrelated to the actual iteration does make sense for me.

Also providing an option to open (or at least falling back to generating a link) from the CLI is a nice thing.

@kytrinyx
Copy link
Member Author

kytrinyx commented Aug 7, 2017

I've opened issues for the nextercism reimplementation (https://github.com/exercism/cli/issues?q=is%3Aissue+is%3Aopen+label%3Anextercism) and I've started implementing the new client.

The base branch for the new client is in exercism/cli#410, and I have a number of open pull requests to that branch https://github.com/exercism/cli/pulls?q=is%3Aopen+is%3Apr+label%3Anextercism

@kytrinyx kytrinyx closed this as completed Aug 7, 2017
@kytrinyx
Copy link
Member Author

kytrinyx commented Aug 7, 2017

Thanks, everyone, for helping hash this out! 🌻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants