Shell commands generally have input and output. By default, input comes from the standard input device (stdin), and the result is displayed on the standard output device (stdout). Typically:

  • Standard input device: the keyboard
  • Standard output device: the terminal, i.e., the display

When a Shell command runs, it opens three files:

  • Standard input file (stdin): file descriptor 0, reads data from stdin
  • Standard output file (stdout): file descriptor 1, writes data to stdout
  • Standard error file (stderr): file descriptor 2, writes error information to the stderr stream

But input and output can also come from other places — this is input/output redirection.

1. Output Redirection

1.1 Standard output redirection

Write a command’s standard output to a file:

command 1> log_a.txt # writes directly, overwriting the previous file
command 1>> log_b.txt # appends to the end of the previous file

Note that there’s no space between 1 and > or >> after command. If written like below, 1 becomes an argument to command instead:

command 1 > log_a.txt

Since standard output is the default, 1 can also be omitted:

command > log_a.txt
command >> log_a.txt

1.2 Standard error redirection

Standard error’s file descriptor is 2:

command 2>log_a.txt
command 2>>log_a.txt

The 2 in standard error redirection must be placed right before > or >>, with no space in between.

1.3 Redirecting both at once

command >log.txt 2>&1 # send both standard error and standard output to the same file
command 1>log_a.txt 2>log_b.txt # send them to two different files

Merging output files:

n >& m # merges output file m and n.

1.4 The special /dev/null

/dev/null is a special file: anything written to it is discarded, and if you try to read from it, you get nothing back. (It’s like a black hole.) But /dev/null is very useful — redirecting a command’s output to it has the effect of suppressing output.

command > /dev/null
command 2> /dev/null
command > /dev/null 2>&1

2. Input Redirection

This can be understood as the opposite of output redirection: input comes from a file.

2.1 Input redirection

Standard input is the keyboard, but it can also come from a file:

shell_command < filea.txt # uses the contents of filea.txt as standard input

2.2 Merging input files

n <& m # merges input file m and n.

2.3 Here documents

This means entering multiple lines of text right in the command line:

command << delimiter
document
delimiter

This passes the content between the two delimiters (document) as input to command:

  • The closing delimiter must start at column 1 — no characters before or after it, including spaces or tab indentation.
  • Whitespace around the opening delimiter is ignored.

Here documents can also be used inside scripts:

#!/bin/bash
cat << EOF
This is a simple lookup program for good (and bad) restaurants
in Cape Town.
EOF

3. Input and Output Redirection Together

Handling both input and output at the same time:

command < file1 >file2 # uses file1 as standard input, file2 as standard output

4. Pipe Commands

4.1 The | command

The pipe command (|) turns one command’s output into another command’s input.

A common case is filtering a directory (such as .) for files containing a specific name (such as y):

ls . | grep y

Here, the pipe command | is used to turn ls .’s output into grep’s input. Pipe commands can be chained, as in ls . | grep y | grep -v x: filtering for every file that contains y but not x.

A pipe command runs in a new thread — for example, grep y doesn’t run in the same thread as ls ..

4.2 Operating on output with xargs

A pipe command’s output doesn’t necessarily match what the next command expects as input, so sometimes it needs to be processed before being passed along — this is where the xargs command comes in. xargs (extended arguments) can do:

  1. Argument processing
  2. Reading information from a file as input

xargs needs a command after it to process further input; if not specified, it defaults to echo. Although xargs’s input can contain spaces and newlines, after being processed by echo, they all become spaces.

ls . | ls -l

This errors out because ls -l’s input isn’t valid — it needs to be converted with the xargs command:

ls . | xargs ls -l

Common options are as follows; to illustrate, some example files are set up here.

➜ cat x.txt
shell_00.md
shell_01.md
shell_02.md
shell_03.md
shell_04.md
shell_05.md

  • -a xx: gets input from a file

    ➜ xargs -a x.txt file
shell_00.md: Unicode text, UTF-8 text
shell_01.md: Unicode text, UTF-8 text
shell_02.md: Unicode text, UTF-8 text
shell_03.md: Unicode text, UTF-8 text
shell_04.md: Unicode text, UTF-8 text
shell_05.md: Unicode text, UTF-8 text

  • -p: asks the user once before running each set of arguments — enter y + Enter or yes + Enter to confirm, or just Enter alone to skip:

    ➜ xargs -a x.txt -p file
file shell_00.md shell_01.md shell_02.md shell_03.md shell_04.md shell_05.md?...yes
shell_00.md: Unicode text, UTF-8 text
shell_01.md: Unicode text, UTF-8 text
shell_02.md: Unicode text, UTF-8 text
shell_03.md: Unicode text, UTF-8 text
shell_04.md: Unicode text, UTF-8 text
shell_05.md: Unicode text, UTF-8 text

  • -n num: the number of arguments used per invocation of the command; by default all of them are used, and if there are more arguments than this number, it runs in multiple batches.

    ➜ xargs -a x.txt -p -n 2 file
file shell_00.md shell_01.md?...yes
shell_00.md: Unicode text, UTF-8 text
shell_01.md: Unicode text, UTF-8 text
file shell_02.md shell_03.md?...yes
shell_02.md: Unicode text, UTF-8 text
shell_03.md: Unicode text, UTF-8 text
file shell_04.md shell_05.md?...yes
shell_04.md: Unicode text, UTF-8 text
shell_05.md: Unicode text, UTF-8 text

  • -t: prints the actual command being run before running it

  • -i or -I: assigns each item from xargs, generally line by line, to {}, which can be used in the command

  • -r (no-run-if-empty): stops xargs from running at all if its input is empty

  • -L or -l num: reads num lines at a time from standard input to pass to the following command

    ➜ xargs -a x.txt -t -L 2 file
file shell_00.md shell_01.md
shell_00.md: Unicode text, UTF-8 text
shell_01.md: Unicode text, UTF-8 text
file shell_02.md shell_03.md
shell_02.md: Unicode text, UTF-8 text
shell_03.md: Unicode text, UTF-8 text
file shell_04.md shell_05.md
shell_04.md: Unicode text, UTF-8 text
shell_05.md: Unicode text, UTF-8 text

    This needs to be distinguished from -n above: this one refers to the number of input lines, while -n refers to the number of output arguments.

  • -d delimiter: the delimiter, by default a space, used to split xargs’s input

    ➜ echo "ambmcmdm" | xargs -dm
a b c d

  • -e or -E flag: e stands for “end” — flag must be a whitespace-separated marker, and xargs stops as soon as it encounters this marker

  • -s num: the maximum number of characters in the command line, i.e., the maximum length of the command line following xargs

  • -x (exit): used together with -s

  • -P: changes the maximum number of processes; the default is 1, and 0 means as many as possible.

4.3 The tee command

tee reads data from standard input and also writes its content out to a file. Common options:

  • -a or --append: appends to the end of an existing file instead of overwriting it.
  • -i or --ignore-interrupts: ignores interrupt signals.

5. Summary

This article covered input/output redirection in Shell, along with using pipe commands to feed one command’s output as another command’s input, and introduced the xargs and tee commands.