Pipeline and redirections in Bash

This post summarizes all types of pipelines, redirections in bash.

Basic

• Continuous pipeline

command1 | command2: standard output of command1 is redirected as standard input of command2. Standard error of the command1 is redirected to /dev/stderr as without the pipeline.

command1 &| command2: shorthand for command1 2>&1 | command2

• Redirections

Some default file descriptors: 0 (/dev/stdin), 1 (/dev/stdout), 2 (/dev/stdin)
To redirect standard input command < file (or command 0< file)
To redirect output command 1>file1 2>file2 ( > is referred as 1>)

To append output command 1>>file1 2>>file2.

To redirect or append both standard output and standard error, use &> or >&, and &>> or >>&.

• Process Substitution is a handy syntax to couple with redirection: mapfile -t my_list 0< <(echo *.sh; echo *.txt)  or wc -l 0< <(sort -u myfile.txt).
• Here string is useful to put a variable to standard input
  wc -l 0<<< "$content" is equivalent as wc -l 0< <(echo "$content").
• Here document is used when you want to inline the standard input

myvar=world
cat <<MY_EOF
echo hello
echo $myvar
MY_EOF

#result
hello
world

myvar=world
cat <<"MY_EOF"
echo hello
echo $myvar
"MY_EOF"

#result
hello
$myvar

Use <<- operator to strip all leading tab characters (not spaces)

• &fd is useful to reference a file descriptor. For example, 2>&1 is to redirect file descriptor 2 to file description 1.

Advance

This section may be hard to be read and applied if you do not understand file descriptor (number) in bash correctly. I suggest you read through this post before continue reading.

• Redirect output support noclobber option overwrite via >| operator

$ touch dump.txt
$ set -o noclobber
$ echo hello >dump.txt
bash: dump.txt: cannot overwrite existing file
$ echo hello >|dump.txt

• exec command is useful to manipulate file descriptors. In exec --help

exec: exec [-cl] [-a name] [command [arguments ...]] [redirection ...]
Replace the shell with the given command.

Execute COMMAND, replacing this shell with the specified program.
ARGUMENTS become the arguments to COMMAND.  If COMMAND is not specified,
any redirections take effect in the current shell.

Options:
-a name pass NAME as the zeroth argument to COMMAND
-c execute COMMAND with an empty environment
-l place a dash in the zeroth argument to COMMAND
If the command cannot be executed, a non-interactive shell exits, unless
the shell option `execfail' is set.

Exit Status:
Returns success unless COMMAND is not found or a redirection error occurs.

Note the bold text in the above quote. If a redirection appears in a command, it takes effect in that command only. Except in the exec command without COMMAND argument, the redirection takes effect in the current shell.

N> filename, N> &M, N< filename, N<& M are used to duplicate a file descriptor of file filename or denoted by the file descriptor M, and new duplicated file descriptor is denoted by N. For e.g. exec 3>&1 means: make a new file descriptor of /dev/stdout (denoted by the file descriptor 1), and assign to file descriptor 3.

N<> filename is to open file filename to reading and writing.

>&- and <&- are used to close an output/input file descriptor.

N>&M- and N<&M- are used to move file descriptor denoted by M to be denoted by N. N>&M- is equivalent to N>&M M>&-.

• {varname}>&N: auto increasing the file descriptor number from 10, assign to the variable named varname. Duplicate the file descriptor denoted by file descriptor number N then assign to file descriptor number $varname.

$ echo hello,world {tmp}>&1
$ echo $tmp
10
$ exec {tmp}>&2
$ echo $tmp
11

Note that named descriptor has an effect outside the command even though the command is not exec.