&pattern
Display only lines which match the pattern; lines which do not
match the pattern are not displayed. If pattern is empty (if
you type & immediately followed by ENTER), any filtering is
turned off, and all lines are displayed. While filtering is in
effect, an ampersand is displayed at the beginning of the
prompt, as a reminder that some lines in the file may be hidden.
$ less --version
less 487 (GNU regular expressions)
Copyright (C) 1984-2016 Mark NudelmanYeah.
I was viewing some files and thinking "it'd be really cool if less(1) had some way of filtering to just lines matching a specific pattern".
RTFM'd and TIL'd.
Mind that at that point I'd used less for ... oh, at least 30 years.
I work in ops, and I don’t even have that dedication. My hat is off to them.
man -Tps bash | ps2pdf - bash.pdf
Better yet, 'bash unofficial strict mode':
set -euo pipefail
IFS=$'\n\t'Over may carrier I've seen many shell scripts:
The most common case is no `-e` and no explicit error/status checks for most commands. It's unreliable and such scripts is one of reasons why shell got it's bad reputation (and criticized by coders using languages where an unhanded exception would terminate a program).
In my own scripts I often use `-e` (but not always as it is a tradeoff). If you unfamiliar with shell `-e` also could cause problems. But `-e` is not magic which makes a script better, one have to know how it works.
The option author of this article advocates (no `-e` but explicitly check every command which could fail) is the least common IMHO. Unless the scripts calls just 1-2 commands it's tedious and the resulting script is harder to read. May be you can find such style in some opensource projects but I've never seen such scripts at work.
Then again, so does enforcing passing shellcheck in CI.
I mean, viewing Python strictly as a scripting language? I am honestly lost for words. There are many huge and major applications and web sites written in Python, without people regretting it after the fact. And yet here you are dismissing it out of hand without a single argument.
Further down is a comment about that: https://news.ycombinator.com/item?id=42664939
Yeah, I have these same response patterns. Shell works really well for some use cases. I generally don’t respond to the comments that list the various “footguns” of shell, or that complain about security holes, etc. My use cases are not sensitive to these concerns, and even besides this, I find the concerns overstated.
But Python is not designed to only be a scripting language:
> What is Python?
> Python is an interpreted, interactive, object-oriented programming language. It incorporates modules, exceptions, dynamic typing, very high level dynamic data types, and classes. It supports multiple programming paradigms beyond object-oriented programming, such as procedural and functional programming. Python combines remarkable power with very clear syntax. It has interfaces to many system calls and libraries, as well as to various window systems, and is extensible in C or C++. It is also usable as an extension language for applications that need a programmable interface. Finally, Python is portable: it runs on many Unix variants including Linux and macOS, and on Windows.
That said, Sonnet 3.5 had gotten me much further in bash than was possible before - and it's all really maintainable too. I highly suggest consulting with Sonnet on your longer scripts, even just asking it "what would you suggest to improve".
It is 2025. bash is present almost everywhere. If, by chance, your script is run somewhere that doesn't have bash then guess what?
Your POSIX script probably won't work anyway. It will be a different system, with different utilities and conventions.
Line count is not a good reason to choose or avoid bash. Bash is quite reliably the lowest common denominator.
I dare say it is even the best choice when your goal is accomplished by just running a bunch of commands.
All that said, bash is gross. I wish we had something as pervasive that wasn't so yucky.
With xargs or GNU parallel, you can have multi-processing, too. Combining with curl or ffmpeg, you can literally build a production-grade web scraper or video transcoding pipeline in a couple of minutes and a few dozen lines of code.
Totally agree with that - I maintain a big txt file too.
Maybe this bash compendium is a bit similar:
• <https://learnxinyminutes.com/>
And some code search engines:
• Debian: <https://codesearch.debian.net/>
• Python: <https://www.programcreek.com/python/>
• Linux: <https://livegrep.com/search/linux>
• GitHub: <https://grep.app/>
• HTML, JS and CSS: <https://publicwww.com/>
One thing that has bitten me in the past is that, if you declare your associative arrays within a function, that associative array is ALWAYS global. Even if you declare it with `local -A` it will still be global. Despite that, you cannot pass an associative array to a function by value. I say "by value" because while you can't call `foo ${associative_array}` and pick it up in foo with `local arg=$1, you can pass it "by reference" with `foo associative_array` and pick it up in foo with `local -n arg=$1`, but if you give the passed in dereferenced variable a name that is already used in the global scope, it will blow up, eg `local -n associative_array=$1`.
As a general rule for myself when writing bash, if I think one of my colleagues who has an passable knowledge of bash will need to get man pages out to figure out what my code is doing, the bash foo is too strong and it needs to be dumbed down or re-written in another language. Associative arrays almost always hit this bar.
#!/bin/bash
declare -A map1=([x]=2)
echo "1. Global scope map1[x]: ${map1[x]}"
func1() {
echo " * Enter func1"
local -A map1
map1[x]=3
echo " Local scope map1[x]: ${map1[x]}"
}
func1
echo "2. Global scope map1[x]: ${map1[x]}"
1. Global scope map1[x]: 2
* Enter func1
Local scope map1[x]: 3
2. Global scope map1[x]: 2 #!/bin/bash
declare -A map1=([x]=2)
echo "1. Global scope map1[x]: ${map1[x]}"
func1() {
echo " * Enter func1"
local -A map1
map1[x]=3
echo " Local scope map1[x]: ${map1[x]}"
func2
}
func2() {
echo " * Enter func2"
echo " Local scope map1[x]: ${map1[x]}"
}
func1
func2
echo "2. Global scope map1[x]: ${map1[x]}"
UPDATE: I did a bit of exploration and it turns out ANY variable declared `local` is in the scope of a function lower down in the call stack. But if you declare a variable as `local` in a called function that shadows the name of a variable in a callee function, it will shadow the callee's name and reset the variable back to the vallee's value when the function returns. I have been writing bash for years and did not realise this is the case. It is even described in the man page: When local is used within a function, it causes the variable name to have a visible scope restricted to that function and its children.
Thank you. You have taught me two things today. One is a bash feature I did not know existed. The second is a new reason to avoid writing complex bash.
Otherwise wouldn't be getting the full shell experience.
- some of the things that make shell scripting terrible can't be fixed in the shell itself, and need changes to the entire ecosystem of console applications. e.g. it would be awesome if every Unix utility output structured data like JSON which could be parsed/filtered, instead of the soup of plaintext that has to be memorized and manipulated with awk, but that almost certainly won't happen. There's a bunch of backward-compatibility requirements like VT-100 and terminal escape sequences limiting the scope of potential improvements as well
- there's a great deal of overlap between "people who do a lot of shell scripting" and "people who are suspicious of New Stuff and reluctant to try it"
I see this argument a lot, and I think it has a ton of overlap with the struggles from devs trying to grok RDBMS I see as a DBRE.
Most (?) people working with web apps have become accustomed to JSON, and fully embrace its nesting capabilities. It’s remarkably convenient to be able to deeply nest attributes. RDBMS, of course, are historically flat. SQL99 added fixed-size, single-depth arrays, and SQL2003 expanded that to include arbitrary nesting and size; SQL2017 added JSON. Still, the traditional (and IMO, correct) way to use RDBMS is to treat data as having relationships to other data, and to structure it accordingly. It’s challenging to do, especially when the DB providers have native JSON types available, but the reasons why you should are numerous (referential integrity, size efficiency, performance…).
Unix tooling is designed with plaintext output in mind because it’s simple, every other tool in the ecosystem understands it, and authors can rest assured that future tooling in the same ecosystem will also understand it. It’s a standard.
JSON is of course also a standard, but I would argue that on a pure CLI basis, the tooling supporting JSON as a first-class citizen (jq) is far more abstruse than, say, sed or awk. To be fair, a lot of that is probably due to the former’s functional programming paradigm, which is foreign to many.
Personally, I’m a fan of plaintext, flat output simply because it makes it extremely easy to parse with existing tooling. I don’t want to have to fire up Python to do some simple data manipulation, I want to pipe output.
A hypothetical Unix equivalent doesn't need to be JSON, I just brought that up as an example. But any structured data would be an improvement over the situation now, and as the PS example shows, with the appropriate ecosystem tooling you can do all the data manipulation over pipes.
I'm a huge fan of `fish` for personal scripting and interactive use: https://fishshell.com/
Obviously you'll still have `bash` installed somewhere on your computer, but at least you don't have to look at it as much.
Why hasn't using a rock as a tool evolved? Exactly, because it's god-awful and got superseded, use a real hammer or a cordless screwdriver instead.
Just like the post about improvements to C on the front page today, that list can be made infinite. Language designers have already learned from those mistakes and created zig, rust, go or even c++, that fixes all of them and more. Fixing all the flaws will turn it into a different language anyhow. There is only that much you can polish a turd.
I’d guess there’s a solution to almost any set of priorities you have for shell scripting.
The domain space is extremely challenging: by default, executing any program on behalf of the caller using arbitrary text inputs, and interpreted.
All modern shells allow calling of literally any binary using the #!/usr/bin/env randombinary incantation.
Upshot: bash has its place, and while some of that place is unearned inertia, much of it is earned and chosen often by experienced technologists.
In my case, if I’m doing a lot of gluing together of text outputs from binaries, bash is my go-to tool. It’s extremely fast and expressive, and roughly 1/4 the length of say python and 1/8 the length of say go.
If I’m doing a lot of logic on text: python. Deploying lots of places/different architectures: go
I use this as my main shell on Windows, and as a supplementary on Mac and Linux.
It wouldn't be a worthy bash feature if after learning about it you wouldn't spend a few days figuring out why the damn thing doesn't work the way it works in any other language.
Debian variants tend to link dash, which self consciously limits itself to posix compatibility.
A lot of the sins of shell are due to it's primary focus as an interactive language. Many of those features that make it so nice interactively really hurt it as a scripting language.
# Write a shell script:
* Heavy lifting done by powerful tool (sort, grep, curl, git, sed, find, …)
* Script will glue diverse tools
* Workflow resembles a pipeline
* Steps can be interactively developed as shell commands
* Portability
* Avoid dependency hell
* One-off job
# Avoid shell scripting:
* Difficult to see the preceding patterns
* Hot loops
* Complex arithmetic / data structures / parameters / error handling
* Mostly binary data
* Large code body (> 500 LoC)
* Need a user interface
A need for associative arrays (implemented in Bash as via hashmaps) moves the task to the second category (avoid shell scripting).
So this has inspired me to Ask HN, I’m getting ready to post it with a reference to this discussion, but thought I’d start here: does anyone know of a good resource for learning Python (or Go, Perl…any good small project/scripting/solo hacking languages) that’s tailored for folks who already have a good foundation in shell scripting, markup, config/infra automation, build tools, etc.? I’m open to books, web-based tutorials, apps, or just good examples for studying.
I’m open to the notion that I simply haven’t put in the work, and powered through the doldrums of novice tutorials far enough to get to where the meaty stuff is. Worst case, I’m a big fan of taking courses at the local community college for my own edification, absent any specific degree program or professional certification. It would still necessitate a lot of remedial/redundant steps, but I can always help others through those chapters while filling in any of my own gaps. Day-to-day, I generally work alone, but I find such things are easier with others to help motivate and focus my efforts. Perhaps I have answered my own question, but even still, I appreciate any advice/recommendations folks might have.
My guess is that you can easily learn the syntax and that you have the logical and analytical skills, but the part you have to learn is how to THINK about programming and how to DESIGN a program. If you have or haven't taking coding classes, I think reviewing topics like data structures, algorithms, encapsulation/object oriented programming, functional programming, etc. is the way to learn how to think about general programming. I don't think the language matters, but there might be resources about these topics in the language you're interested in.
An example of what you DON'T want is something like Effective Go (not just because it's out-of-date now): https://go.dev/doc/effective_go. This page can give you a really good base of information about what Go is, but I think you'll get much more bang for your buck with a resource that is more about the WHYs of programming rather than the WHATs.
If anyone else is interested, this thread from 2020 is where I am starting, it seems to align with my own quest pretty well: https://news.ycombinator.com/item?id=22932794
Learning Go by Jon Bodner is a good choice. It seems to assume that Go is the reader's second (or tenth) language.
- Between using Bash's functions and breaking large scripts into multiple, separate scripts, one can keep things reasonably tidy. Also, functions and scripts can be combined in all the ways (e.g., piping and process substition) that other programs can.
- If I run into a case where Bash is a poor fit for a job, I ask myself, "Self, what program would make this easy to do in a shell script?" If I can respond with a good answer, I write that, then continue with the shell scripting. If not, I write in something else (what the kids would call a "proper" language).
Until it does, and major POSIX shs have shipped with it for a decade, then the feature will actually exist in a way that the average shell coder cares about. You're better off just shipping Rust/Go/etc binaries and leave sh for your simple glue tasks.
Even I've eventually switched to this, and I've written 10k+ long Bash scripts that followed best practices and passed shellcheck's more paranoid optional tests.
Use the right language for the right job.
I think it's worth picking at this a bit. At least IME, a fairly small fraction of the Shell I write needs to run literally (any|every)where.
I don't mean to suggest there aren't people/projects for whom the opposite is true--just that it's worth chunking those cases and thinking about them differently.
It obviously isn't a global solution, but in the Nix ecosystem we have decent idioms for asserting control over the shell and tools any given script runs with.
Reliable environment provisioning can spare hundreds of lines of code real-world Shell tends to accumulate to sanity-check the runtime environment and adapt to the presence/absence of various utilities. It also enables us to use newer shells/features (and shell libraries) with confidence.
It's enough leverage that I only give it up when I must.
Bash itself isn't a very big language, so I wouldn't call it "impossible to memorize".
<https://web.archive.org/web/20240810094701/https://meta.ath0...>
$ bash --version
GNU bash, version 5.2.32(1)-release (aarch64-apple-darwin23.4.0)
$ declare -A aaa; aaa[a]=a; aaa[b]=bb; for i in ${!aaa[@]}; do echo "$i --> ${aaa[$i]}"; done
b --> bb
a --> a
Seems to suggest it's indeed a hashmap in implementation, but I can't be bothered to look any closer.
IIRC, the latest bash addresses all of these, but that doesn't help too much if you are stuck with an older, stable OS, e.g., RHEL 7 or 8; even 9 likely has a few remaining.
These leaks become an issue if you have a long running bash script with frequent adds and updates. The update leak can be mitigated somewhat by calling unset, e.g., unset a[b], before updating, but only partially (again, apologies, no notes, just the memory of the discovery and the need to drop bash).
I'd agree with the idea that bash wasn't the best choice for this purpose in the first place, but there was history and other technical debt involved.
- my site builder (for evalapply.org): inject metadata into page templates. e.g. https://github.com/adityaathalye/shite/blob/b4163b566f0708fd...
- oxo game (tic-tac-toe): reverse index lookup table for board positions: https://github.com/adityaathalye/oxo/blob/7681e75edaeec5aa1f...
- personal machine setup: associate name of installed application to its apt source name, so we can check for the app, and then install package https://github.com/adityaathalye/bash-toolkit/blob/f856edd30...
[1] I'd say "hashmap" is acceptable, colloquially. However, I don't think Bash computes hashes of the keys.
(edit: fix formatting snafu)
I found out about hashmaps in Bash a year ago[1], and it came as a surprise. This morning, I came across dynamic shell variables and the indirection syntax[2]. Each time I wrote about them and learned more than I would have if I had just passively grokked the manual.
Having to implement hash-tables, while still keeping the appearances of everything being a string is the anti-pattern known as "string programming" (i.e. when application develops a convention about storing type / structure information inside unstructured data, often strings).
I would love for there to be a different system shell with a small number of built-in types that include some type for constructing complex types. There are many languages that have that kind of type system: Erlang, for example. But, extending Unix Shell in this way is not going to accomplish this.
That ship has sailed. These were introduced in Bash 4, released in 2009, and Bash is already on version 5.
This policy comes from a six hour debugging session involving (somewhere) a script that manipulated a binary file - bash can't store zero bytes in variables and zsh can, but it's not like it'll tell you that it's pointlessly truncating your data and I never thought to look. So now every step in that script gets converted back and forth through xxd, and I don't trust Bash anymore.
For example: `${!myvar[@]}` would list all of the keys.
I've written about associative arrays in Bash a bit here: https://nickjanetakis.com/blog/associative-arrays-in-bash-ak...
In a related question I said something about switching to a language like Python when a script started to get "complicated"
Then the interviewer explained how his favorite language was bash, how he uses it for everything...
I did not get the job. Ironically my next job I did a bunch of Perl to Bash conversion.
[0]: https://hyperpolyglot.org/unix-shells#associative-arrays
> A: You use the command `declare -A HASHMAP_NAME`
This is why I think Bash is a horrible language
(I feel obliged to add that when you feel the need to add type annotations a bit later is when to abandon python)
Now to add hashtables to that.
A hashmap is a very specific structure that uses a hashing algorithm to normalize the key sizes, and then usually constructs a tree-like structure for fast key lookups. Anything that doesn't use such an implementation under the hood should not be called a hashmap, whether you care or not.
For reference the main implementations of associative array are alists/plists (linear time), balanced binary trees, b-trees (logarithmic time) and hash maps (amortised constant time).