This post summarize all types of pipelines, redirections in bash


Basic

command1 | command2: standard output of command1 is redirected as standard input of command2. Standard error of the command1 is redirected to /dev/stderr as without the pipeline.

command1 &| command2: shorthand for command1 2>&1 | command2

Some default file descriptors: 0 (/dev/stdin), 1 (/dev/stdout), 2 (/dev/stdin)
To redirect standard input command < file (or command 0< file)
To redirect output command 1>file1 2>file2 ( > is referred as 1>)

To append output command 1>>file1 2>>file2.

To redirect or append both standard output and standard error, use &> or >&, and &>> or >>&.

  • Process Substitution is a handy syntax to couple with redirection: mapfile -t my_list 0< <(echo *.sh; echo *.txt)  or wc -l 0< <(sort -u myfile.txt).
  • Here string is useful to put a variable to standard input
    wc -l 0<<< "$content" is equivalent as wc -l 0< <(echo "$content").
  • Here document is used when you want to inline the standard input
myvar=world
cat <<MY_EOF
echo hello
echo $myvar
MY_EOF

#result
hello
world
myvar=world
cat <<"MY_EOF"
echo hello
echo $myvar
"MY_EOF"

#result
hello
$myvar

Use <<- operator to strip all leading tab characters (not spaces)

  • &fd is useful to reference a file descriptor. For example, 2>&1 is to redirect file descriptor 2 to file description 1.

Advance

This section may be hard to be read and applied if you do not understand file descriptor (number) in bash correctly. I suggest you read through this post before continue reading.

  • Redirect output support noclobber option overwrite via >| operator
$ touch dump.txt
$ set -o noclobber
$ echo hello >dump.txt
bash: dump.txt: cannot overwrite existing file
$ echo hello >|dump.txt
  • exec command is useful to manipulate file descriptor. In exec --help
exec: exec [-cl] [-a name] [command [arguments ...]] [redirection ...]
Replace the shell with the given command.

Execute COMMAND, replacing this shell with the specified program.
ARGUMENTS become the arguments to COMMAND.  If COMMAND is not specified,
any redirections take effect in the current shell
.

Options:
-a name pass NAME as the zeroth argument to COMMAND
-c execute COMMAND with an empty environment
-l place a dash in the zeroth argument to COMMAND
If the command cannot be executed, a non-interactive shell exits, unless
the shell option `execfail' is set.

Exit Status:
Returns success unless COMMAND is not found or a redirection error occurs.

Note the bold text in the above quote. If a redirection appears in a command, it takes effect in that command only. Except in the exec command without COMMAND argument, the redirection takes effect in the current shell.

N> filename, N> &M, N< filename, N<& M are used to duplicate a file descriptor of file filename or denoted by file descriptor M, and new duplicated file descriptor is denoted by N. For e.g. exec 3>&1 means: make a new file descriptor of /dev/stdout (denoted by file descriptor 1), and assign to file descriptor 3.

N<> filename is to open file filename to reading and writing.

>&- and <&- are used to close a output/input file descriptor.

N>&M- and N<&M- are used to move file descriptor denoted by M to be denoted by N. N>&M- is equivalent to N>&M M>&-.

  • {varname}>&N: auto increasing the file descriptor number from 10, assign to variable named varname. Duplicate the file descriptor denoted by file descriptor number N then assign to file descriptor number $varname.
$ echo hello,world {tmp}>&1
$ echo $tmp
10
$ exec {tmp}>&2
$ echo $tmp
11

Note that named descriptor has effect outside the command even though the command is not exec.