Pipes and Redirection

Table of Contents

1. Three Streams, Again
2. Sending Output to a File: > and >>
3. Redirecting Errors: 2>
4. Reading Input from a File: <
5. Pipes: Connecting Commands with |
6. The Toolbox: grep, wc, sort, uniq, cut
7. Building a Pipeline Step by Step
8. Seeing and Saving at the Same Time: tee
9. Try It Yourself

So far, every command has printed its results straight to the screen. This chapter is about changing where that output goes — into a file, or into another command. It's the idea that makes the command line far more powerful than it looks.

cd ~/cli-practice

1. Three Streams, Again

In Input/Output you learned that every process starts with three standard streams:

A program like ls doesn't know or care where its output ends up. It just writes to stdout. It's the shell that decides where stdout is connected — and you can tell the shell to connect it somewhere else. That's all redirection is.

Why are there two output streams? So that normal results and error messages can be separated. On the screen they look the same, but as you'll see, you can send them to different places.

2. Sending Output to a File: > and >>

> sends a command's stdout into a file instead of the screen:

ls -l notes > listing.txt

Nothing appears on the screen — the output went into listing.txt. Check it:

cat listing.txt

If the file doesn't exist, > creates it. If it does exist, > replaces everything in it, with no warning:

echo "first" > test.txt
echo "second" > test.txt
cat test.txt

second

first is gone. To add to the end of a file instead, use >>:

echo "third" >> test.txt
echo "fourth" >> test.txt
cat test.txt

second
third
fourth

An easy way to remember: > is one arrow — start fresh; >> is two arrows — keep adding.

This works with any command that prints something:

date > timestamp.txt
history > my-commands.txt
seq 1 100 > hundred.txt

Classic mistake: never redirect a command's output into the same file it's reading. sort names.txt > names.txt doesn't sort the file — the shell empties names.txt before sort starts, so sort reads an empty file and you lose everything. Write to a new file instead, then rename it with mv.

3. Redirecting Errors: 2>

Try listing one folder that exists and one that doesn't, sending the output to a file:

ls notes nothere > out.txt

ls: nothere: No such file or directory

The error still appears on the screen, and out.txt only contains the successful part. That's because > only redirects stdout. The error went to stderr, which is still connected to the screen.

To redirect stderr, put its number, 2, in front of the arrow:

ls notes nothere > out.txt 2> errors.txt

Now nothing appears on screen: results are in out.txt, errors are in errors.txt. This is genuinely useful — for example, running a long build and saving its errors to read later.

Two more forms you'll see in tutorials and scripts:

command > all.txt 2>&1   # stdout AND stderr into the same file
command 2> /dev/null     # throw the errors away

2>&1 means "send stream 2 to the same place as stream 1". And /dev/null is a special file that swallows everything written to it — a black hole for output you don't want. Use it carefully: hiding errors also hides the clues you'd need when something goes wrong.

4. Reading Input from a File: <

The opposite of > is <, which feeds a file into a command's stdin:

wc -l < hundred.txt

100

You won't use < nearly as often as the others, because most commands can open a file themselves if you just pass its name. Compare:

wc -l hundred.txt     # 100 hundred.txt
wc -l < hundred.txt   # 100

The second version prints only the number, because wc never learns the file's name — it just receives the contents on stdin.

5. Pipes: Connecting Commands with |

Here's the big idea. The pipe | connects the stdout of one command directly to the stdin of the next:

ls -l /usr/bin | less

ls lists hundreds of programs in /usr/bin. Instead of flooding the screen, its output flows straight into less, where you can scroll through it. (Press q to quit.)

You can chain as many commands as you like:

command1 | command2 | command3

Each command does one small job, and the pipe passes the results along — like workers on an assembly line, or water flowing through pipes, which is where the name comes from. This is one of the oldest design ideas in Unix: small tools that each do one thing well, and can be combined.

No temporary files are created. The data flows from one process to the next in memory, and all the commands in a pipeline actually run at the same time — exactly the kind of concurrent processes you met in How Computers Work.

6. The Toolbox: grep, wc, sort, uniq, cut

Pipes become powerful once you know a handful of filter commands — programs that read lines, do something to them, and print the result.

Let's make a small web-server log to practice on. printf prints each quoted string on its own line:

printf '%s\n' \
  "GET /home 200" \
  "GET /about 200" \
  "POST /login 401" \
  "GET /home 200" \
  "GET /products 500" \
  "POST /login 200" \
  "GET /home 200" \
  "GET /products 500" \
  "GET /contact 404" \
  "GET /about 200" > notes/access.log

The \ at the end of each line just means "this command continues on the next line". You can copy the whole block and paste it at once.

Each line is a request: the method, the page, and the status code (200 is OK, 404 is not found, 500 is a server error).

grep — keep only matching lines

grep "500" notes/access.log

GET /products 500
GET /products 500

Useful options:

grep -v " 200" notes/access.log   # every request that wasn't OK
grep -c "POST" notes/access.log   # how many POST requests: 2

wc — count

wc (word count) counts lines, words, and characters. You'll mostly use -l for lines:

wc -l notes/access.log

10 notes/access.log

sort — put lines in order

sort notes/access.log        # alphabetical
sort -r notes/access.log     # reversed
seq 1 20 | sort -n -r        # numbers, largest first

-n sorts numerically. Without it, sort compares text, so 10 comes before 9 (because 1 comes before 9).

uniq — remove repeated lines

uniq collapses repeated lines into one. Add -c to count how many times each one appeared. There's a catch: it only notices duplicates that are next to each other, so you almost always sort first.

cut — pick out a column

cut extracts part of each line. -d " " says the columns are separated by spaces, and -f 2 picks the second one:

cut -d " " -f 2 notes/access.log

/home
/about
/login
...

7. Building a Pipeline Step by Step

Now let's answer a real question: which pages are visited most?

The trick to writing pipelines is to build them one step at a time, running the command after each new | to see what comes out.

Step 1 — get just the page names:

cut -d " " -f 2 notes/access.log

Step 2 — sort them, so identical pages sit together:

cut -d " " -f 2 notes/access.log | sort

Step 3 — count each group:

cut -d " " -f 2 notes/access.log | sort | uniq -c

Step 4 — sort by that count, biggest first:

cut -d " " -f 2 notes/access.log | sort | uniq -c | sort -rn

   3 /home
   2 /products
   2 /login
   2 /about
   1 /contact

Step 5 — keep only the top three:

cut -d " " -f 2 notes/access.log | sort | uniq -c | sort -rn | head -n 3

Five small commands, none of which knows anything about web logs, combine into a little analytics tool. Change -f 2 to -f 3 and the same pipeline counts status codes instead. Point it at a real log with a million lines and it still finishes in a second or two.

A few more pipelines you'll actually use:

history | grep "git"           # find that git command you ran yesterday
ls -l | grep ".txt"            # only .txt entries in a long listing
cat notes/*.txt | wc -l        # total lines across all your notes

About cat file | grep: you'll often see cat access.log | grep 500. It works, but grep 500 access.log does the same thing with one less process. Either is fine while you're learning.

8. Seeing and Saving at the Same Time: tee

With >, output goes into a file and you don't see it. Sometimes you want both. tee reads stdin, writes it to a file, and passes it along to stdout — like a T-shaped pipe fitting:

grep -v " 200" notes/access.log | tee notes/problems.txt

The problem lines appear on screen and are saved to problems.txt. Add -a to append instead of replace, just like >>.

9. Try It Yourself

1. Save the output of ls -la ~ into ~/cli-practice/home-listing.txt. Then append today's date to the end of the same file with >>. Check it with cat.
2. Run ls notes nothere and send the normal output to ok.txt and the error to err.txt. Look inside both.
3. Using grep, print every request in notes/access.log that was a GET. Then count them with -c (you should get 8).
4. Build a pipeline that counts how many requests got each status code. Build it one step at a time.
5. Create a file of names with duplicates using printf, for example printf '%s\n' Dara Sokha Dara Vanna Sokha Dara > names.txt. Find which name appears most often.
6. Use history | grep to find every cd command you've run in this series.

You can now combine commands into tools of your own. Next, let's put grep to work on whole folders and learn how to find any file, anywhere.