Before describing UI design in Next Generation Shell, let’s clarify which problems we are trying to solve.

Problems

Warming exercise: CLI is an interface that focuses on a single line, the Command Line. Stop for a moment to ponder that.

Unusable Output and Missing Semantics

Treating the output of a program as if it was printed on paper and providing zero interactivity with the output. The output is only intended for human consumption, exactly like in teleprinter below.

Command-line interfaces emerged in the mid-1960s (source: Wikipedia). I find it hard to believe that we can’t do better today, 60 years later.

Just pipe the output to get what you want

The UI should support the user in what they are doing, not to force the user to program to overcome the UI limitations.

The shell is not supposed to get into semantics

The fun part in refuting this claim is that many shells do that already and one of the most valued features is command line completion, which requires semantic understanding of inputs of commands. Outputs of the same commands on the other hand are “shell is not supposed to do that” for some reason.

The more semantically aware the software is the more productive it makes the user. Shells are largely failing here.

Chains of Commands

The user is trying to accomplish a task. For that the user issues commands. Ignoring the semantic grouping of the commands and relation between these commands, which is happening today, is unproductive.

For example, command line completion doesn’t use anything from the output as if ls *.txt command followed by vi command have nothing to do with each other so that vi TAB is completed from the list of all files, not the ones that are on the screen.

This is direct consequence of the previous point. You can’t follow what’s the relation between commands if you don’t “understand” the output of the first one.

Summary: The Shell Experience

Imagine yourself using an online shopping website where the list of products is not interactive.

You copy product ID from the list and paste it in a text input field at the bottom (along with a bit of typing) to get a description. You paste it again (along with another bit of typing) to add to a basket and so on.

Also, there is no concept of a basket, it’s a list of products with zero semantics (you just happen to buy each one of the products in the list later) and you need a bit of code to maintain that list. You want to know the total cost? A bit more code.

Solution Status

Fueled by frustration with the CLI paradigm (while using bash), the work on NGS started in 2013.

Most of the work on NGS up until now was focused on the programming language. The reasons are:

1. Ops people were lacking a programming language designed for them specifically. I have found bash and Python to be inadequate. Fixed. We now use NGS at work for many use cases for which we were previously using bash and Python. There is a room for improvement of course but the language is very usable.
2. It would be weird that after using the programming language in the shell, to fix the UI of the said shell one needs to learn/switch to another programming language. The UI in NGS is being written in NGS, unlike many other shells.

Solution Plan

Most of the work on NGS from now on will be on the UI.

The UI design in NGS is based on a few insights about current situation vs how the things “should be” in order for the person in front of the computer to be effective. Design points follow. They are all using the same sample imaginary interactive session.

aws ec2 describe-instances

Output Interactivity

The output above is interactive. The user right-clicks on “DB”.

After the click (or keyboard navigation + space), the UI responds with two things:

Record of what Happened

This is important for reproducibility and for the operator to know what exactly happened. Web UIs just throw away this information. In our circumstances, “Hold a sec. What did I just click?” is unacceptable. Convenient UI instead of typing doesn’t have to mean non-reproducibility.

Just below the table appears a message roughly like this:

“You have selected a an instance with tag Name having the value DB. Instance ID was i-456″.

To assist reproducibility (described later), behind the message there is an object describing the semantics of what happened.

Menu

The following menu appears:

Assuming it’s a dev environment where the team is testing that without the DB, the web server returns well-formed error messages. The user selects “Shut Down”.

UI responds with something like:

“You have selected the Shut Down command to perform on the selected EC2 instance(s)”.

Commands Construction

Instead of the usual composition of a new command on the command line by typing some text and then hunting and copy-pasting ids from the output to be command line arguments, NGS constructs the next command based on the interaction with the menu above.

The following command appears and runs:

aws ec2 stop-instances --instance-ids i-456

What’s important for reproducibility (described later) is that behind the command there is an object describing the semantics. This allows constructing new similar command should the selection above change (without parsing the command).

Repeating theme here is: everything is structured data, “text” is last resort as it’s notoriously hard to work with (in our context).

Chains of Commands

There will be a way to select several commands from the Timeline/History to group them together into a “script” (aka “chain”) which can be named, parametrized, replayed, and modified using UI.

Following our example, we can group the commands above and name them “shut down by name”.

To parametrize the group, in the message “You have selected a an instance with tag Name having the value DB. Instance ID was i-456″ select DB and choose “parametrize” from a menu. The value of the Name tag becomes the parameter of the script: when you run this script you pass which Name tag you are looking for.

Chains can be serialized and therefore saved in history and shared between users.

Reproducibility

User interaction is recorded. Behind messages of “You have selected …” there is a data structure which describes what happened. This means that the system knows how to replay what the user did. Several interesting caveats here. We’ll get into just one.

Correctness

Imagine we are now running the above script but somebody added “qa” environment in the same account and region. Now we have two instances named DB. Assume they have additional tag “Env” with different values.

When replaying the script, we know that during recording there was exactly one instance that matched the selection. Now there are two. At this point NGS must stop executing the script and ask what to do. It could look like the following:

“Selecting an EC2 instance with tag Name having the value DB failed. During previous runs the selection was matching exactly one instance. Now it matches 2 EC2 instances”

The user responds with “Modify selection” (from a menu which also had “Stop”, “convert to Assertion”, etc). UI shows the selection.

The user updates the selection using a menu which has roughly the following options:

• Use all instances matching the selection
• Modify selection (must result single EC2 instance)
  • Add tag Env with the value dev (guessed from previous runs of the selection)
  • Add tag Env with a value from a new parameter
  • Add “pick the oldest instance” filter
  • Add “pick the newest instance” filter
  • Add custom condition

Summarizing reproducibility

In other shells reproducibility comes in the form of scripts. In NGS, it’s in the form of recorded interactions (code scripts are also possible of course). Recorded interactions can later be adjusted for new circumstances in the UI, as opposed to forcing the user to modify code.

Processes, Generalized

In the past, interacting with the shell meant mostly managing the local machine. Then ssh-ing to other machines. Today, at least in my environment, the shell is used to interact with the cloud using its CLI a lot.

When using cloud, it makes sense that UI-wise, processes running in the cloud will have similar representation to processes running on local machine. Non-exhaustive list of fields showing commonalities between local and cloud processes:

• user who started the process
• time of start
• time of finish
• output
• errors
• progress bar / ETA (when possible)
• result: success / failure
• sub-processes (when applicable)

An example of process running in the cloud: AWS CodePipeline execution, with AWS CodeBuild runs and CloudFormation deploys as sub-processes.

A process can be triggered by something the user or their colleagues do, like git commit/push, not necessarily a command that the user issued to start the process directly.

Objects of Interest

When a process is started indirectly by the user or someone else, in many cases it should be displayed in UI because it’s important to the user. I really had a hard time imagining a situation when you commit and push the code and you don’t want to see the triggered build/deploy process. Can you come up with such scenario?

While in other shells (and Web UIs) you will have to hunt (or write a script) for observing these processes, NGS has a concept of Objects of Interest.

When something is marked as Object of Interest, NGS will show activity (when there is any) automatically on the Timeline. The set of OOIs by default will have any pipelines triggered by user’s commits among other things.

When such automatic display of OOI happens, it is accompanied by phrases such as:

• “You are seeing this CodePipeline execution because you have triggered the execution”
• “You are seeing this CodePipeline execution because your commits went into this execution”
• “You are seeing this CodePipeline execution because you deployed the CodePipeline”
• “You are seeing this CodePipeline execution because you manually selected this CodePipeline”
• “You are seeing this CodePipeline execution because you manualy selected to watch GitHub user 10x-coder, which triggered the execution”

Additionally, there must be a visible option to reconfigure OOIs, especially option to disable the specific type of OOIs which is now showing on the screen.

Exercise to the reader: how can you implement something like that in other shells: showing an output of a processes that you didn’t directly run?

That’s it for now. Hope you enjoyed. Let me know what you think.

To be Added

Leaving a note for myself mostly with topics to add to this post.

• Tools with better semantic understanding are more powerful, debunk “this tools are limiting”.
• Web vs CLI – having the good parts of both in NGS UI
• Auto-investigation of errors – perform necessary digging automatically to find the error.
• Running processes stick to top instead of scrolling out of visible area.