Next Generation Shell

You deserve a better-than-telegraph shell

AWS CLI with NGS instead of jq

I’ve been using jq to process the output of AWS CLI for years. I suspect many others are doing the same. Let’s take a closer look at jq and its alternatives.

jq

The Good

Why jq is so good? Because it’s domain specific; think sed, grep, and awk mixed together and not for text but for JSON. jq scripts are short and expressive. Doing the same in Python (for example) would be way more verbose.

The Bad

The strength is the weakness. For domain specific languages, it’s doing anything outside of the intended domain. For jq, the weakness presents itself as an uphill battle for any non tiny piece of code and a syntax that some would consider unfriendly.

Alternatives

If there is good and bad and tradeoffs were made, it makes sense to look into alternatives. You might discover other tradeoffs that make more sense for your tasks and personal background.

JMESPath

It stands out because it is built into AWS CLI. Otherwise, I find it less convenient and less powerful than jq. My anecdotal impression is that many others prefer jq over JMESPath too.

Way More Alternatives

I’ve made a big list of JSON tools for CLI few years ago. Not sure why but apparently none of them is close in popularity to jq.

Next Generation Shell

I am mostly using Next Generation Shell (NGS) now to work with AWS CLI. It made different tradeoffs than jq, which I prefer. Here is how NGS compares to jq and others.

  1. Fully fledged programming language
    • NGS is more verbose than jq.
    • NGS is more readable than jq.
    • You can write “real programs” in NGS without feeling awkward constraints when trying to stretch jq out of its intended use case.
  2. Data manipulation is one of the intended use cases of NGS so it’s easy. NGS in this regard is probably somewhat “closer” to jq than other programming languages.
  3. NGS is particularly ergonomic when working with AWS CLI output because it “understands” its output to some extent. No more .Reservations[].Instances . In NGS, aws ec2 describe-instances returns the array of instances at the top level. Why? Because it’s ergonomic.

NGS

NGS was born out of frustration with bash, Python, absence of programming language for DevOps, and the outdated CLI paradigm. More in why NGS was born.

Running External Programs in NGS

This is a tiny background for understanding the examples below.

There are several syntactical constructs for running external programs, with different semantics. The interesting one for our use case is the double-backtick syntax with associated run-and-parse-the-output semantics:

``my_program arg1 arg2 ..``

The expression above evaluates to a data structure: the parsed output. When the command is aws (instead of my_program), NGS uses several heuristics from its stdlib to transform the output so it’s more comfortable less horrible to work with.

AWS CLI with NGS

Names of EC2 Instances

jq

aws ec2 describe-instances --filters "Name=tag:Name,Values=INFRA*" | jq -r '.Reservations[].Instances[].Tags[] | select(.Key=="Name").Value'

Source: https://www.reddit.com/r/aws/comments/raew9u/aws_cli_use_jq_with_tag_name/

NGS

ngs -pl '``aws ec2 describe-instances --filters "Name=tag:Name,Values=INFRA*"``.Tags.Name'

There are quite a few things going on here. Let’s go over.

The ``aws ...`` expression returns array of instances. No .Reservations and no .Instances nonsense thanks to the standard library in NGS. Isn’t that cool when your programming language supports you with what you are trying to achieve?

.Tags of an instance would return the tags of that instance.

.Tags of an array of instances, like in our example above, evaluates to array of tags. Each element in the resulting array is tags coming from a particular instance.

I was not able to follow the genius idea from AWS to represent key-value pairs as an array so NGS converts that atrocity to a Hash. Hence, .Tags of an instance returns a Hash on which one can do .Name for example. .Name on an array of tags evaluates to array of the .Name of each element.

Note that jq has similar functionality to the above (converting array to object) using from_entries and friends but it’s not automatic.

The result of ngs -pl 'THIS_EXPRESSION' is an array. Each element is the value of the Name tag of an instance.

-pl causes each element to be printed on its own line

List EC2 Instances with Name Tag that has Specific Prefix

Let’s assume that like me, due to shitty AWS CLI UX (specifically syntax in this case), you didn’t make friends with how you query resources with specific tag and you filter the resources by tags on the client (unless it’s a very big list in which case you go to documentation each time).

jq

aws ec2 describe-instances | jq '.Reservations[].Instances | map(select(.Tags | from_entries | .Name | startswith("o")))'

Based on: https://alexwlchan.net/2023/working-with-jq-and-tags-in-the-aws-cli/

NGS

ngs -pj '``aws ec2 describe-instances``.filter({"Tags": {"Name": /^o/}})'

.filter() takes a pattern. The pattern is recursive. Each element in the pattern can be a predicate – a code that checks for a match so that you are not limited to only supported patterns. Example: my_instances.filter({"Tags": {"Name": my_checker}}), where my_checker function can do arbitrary checks on the Name tag value.

Filtering – Side by Side

Assuming the tags would be straightened by jq, I would like to focus on comparing the filtering alone.

# jq
map(select(.Tags.Name | startswith("o")))

# NGS
.filter({"Tags": {"Name": /^o/}})'

Have to admit, map(select()) instead of just .filter() is breaking my head a bit but that’s what the docs say.

ECS Environment Variables

The problem: run an app for local development using environment variables from a container running in ECS. Following is an excerpt of the code that solving the problem. Imagining bash + jq or pure jq solution is left to the reader.

F task_def() {
	cluster = ``log: aws ecs list-clusters``.filter(Ifx("/${ENV.OUR_APP_ENV}-Cluster")).the_one()
	tasks = ``log: aws ecs list-tasks --cluster ${cluster}``
	tasks = ``log: aws ecs describe-tasks --tasks $*{tasks} --cluster ${cluster}``.tasks
	task_def_arn = tasks.taskDefinitionArn.Set().the_one()  # Not handling transition when tasks definitions are of different versions
	task_def = `log: aws ecs describe-task-definition --task-definition ${task_def_arn}`.decode_json().taskDefinition  # bug in ``...``
}

F container_def(name) {
	task_def().containerDefinitions.the_one({'name': name})::{
		A.environment .= Hash('name', 'value')
	}
}

F app_local() {
	e = ENV + container_def('app').environment + {'AWS_SDK_LOAD_CONFIG': '1'}
	$(cd:'app' log: npx tsc)
	$(top_level:: log: env:e node app/src/server/index.js)	
}

Notes:

F is a function definition.

When a function is called, the value of last expression in the body of F is returned as result.

ENV – local environment variables

Ifx("blah") – infix match, like not anchored /blah/ would do.

.the_one() returns the only element in an array. Throws exception of there isn’t exactly one element.

.the_one(pattern) returns the only element in an array that matches the pattern. Throws exception of there isn’t exactly one element.

data::{ some code } returns the data. Typically the code would modify the data, referenced as A inside the curly braces.

A.environment .= Hash('name', 'value') – converts A.environment from array where each element is of the form {"name": ..., "value": ...} to a Hash (key-value dictionary).

$* stands for several arguments when calling external program. In NGS, $var never expands to several arguments (hello $bash, sorry "$bash")

log: causes the command to be logged before execution

cd: changes to the given directory before running the command

env: supplies the environment variables to the command

Downsides of NGS

For the sake of completeness and to avoid the usual comments, I’ll state the obvious here.

  • NGS is way less popular than jq now. That means way less resources and fewer ways to get help. Does not balance out but: you are welcome to contact me either through GitHub issues (I get notified) or in Discord.
  • The burden of learning a new language is there. My biased opinion: worth it. For me personally, the unbearable situation justified creating a programming language and that’s way more effort. As DevOps, many of our tasks involve light (and not so light) scripting and data manipulation. NGS was specifically designed for that.

Happy scripting!

Ilya Sher

←
→

Leave a comment