The command line as a power tool: streams, pipes, curl and calm stack-trace reading

The mental model: small programs, text streams, exit codes

The Unix philosophy, from the people who built Unix at Bell Labs, comes down to three ideas. Write programs that do one thing well. Write programs that work together. Make text the universal interface. grep only filters lines. sort only sorts. wc only counts. None of them knows the others exist, yet joined with a pipe they answer questions no single tool was designed for.

Every process gets three standard streams when it starts:

• stdin (file descriptor 0): input, from the keyboard or from the previous command in a pipe.
• stdout (1): the program's actual output, the data.
• stderr (2): diagnostics, warnings and errors, kept separate so they do not pollute the data.

And every process ends with an exit code: 0 means success, anything from 1 to 255 means some kind of failure. The shell keeps the last one in $?. Exit codes are how scripts, CI pipelines, Docker health checks and Kubernetes decide whether something worked, so they matter as much as the output.

You can watch all three from Python:

import subprocess
import sys

child = r'''
import sys
print("order 101 processed")                                     # stdout: the data
print("warning: GSTIN missing for order 102", file=sys.stderr)   # stderr: diagnostics
sys.exit(3)                                                      # exit code: the verdict
'''
r = subprocess.run([sys.executable, "-c", child], capture_output=True, text=True)
print("stdout:   ", repr(r.stdout))
print("stderr:   ", repr(r.stderr))
print("exit code:", r.returncode)

# An uncaught exception prints a traceback to STDERR and exits with code 1.
r = subprocess.run([sys.executable, "-c", "import json; json.loads('{bad')"],
                   capture_output=True, text=True)
print("exit code:", r.returncode)
print("stdout was empty:", r.stdout == "")
print("last line of stderr:", r.stderr.strip().splitlines()[-1])

stdout:    'order 101 processed\n'
stderr:    'warning: GSTIN missing for order 102\n'
exit code: 3
exit code: 1
stdout was empty: True
last line of stderr: json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Pipes and redirection

A pipe | connects one program's stdout to the next program's stdin. Redirection sends a stream to a file instead of the terminal:

command > out.txt        # stdout to a file (overwrite)
command >> out.txt       # stdout to a file (append)
command 2> err.txt       # stderr to a file
command > all.txt 2>&1   # both: send stdout to the file, then point stderr where stdout goes
command 2>/dev/null      # throw stderr away
command < input.txt      # stdin from a file

Order matters in 2>&1: it means "make stderr a copy of wherever stdout points *right now*". So cmd > all.txt 2>&1 captures both, while cmd 2>&1 > all.txt sends only stdout to the file. Two operators use exit codes directly: a && b runs b only if a succeeded, and a || b runs b only if a failed.

$ ls app.log nope.log > out.txt 2> err.txt
$ cat out.txt
app.log
$ cat err.txt
ls: cannot access 'nope.log': No such file or directory

Searching and shaping text: grep, sort, uniq, awk, sed

Create this small log so you can follow along (the <<'EOF' form, a heredoc, feeds the lines to cat on stdin):

cat > app.log <<'EOF'
2026-10-12T09:14:02 INFO  POST /orders 201 182ms user=u_1042
2026-10-12T09:14:05 INFO  GET /orders/881 200 35ms user=u_1042
2026-10-12T09:14:09 ERROR POST /payments 502 3012ms user=u_2210 upstream=razorpay
2026-10-12T09:15:11 INFO  GET /products 200 48ms user=u_0007
2026-10-12T09:15:12 WARN  GET /products 429 2ms user=u_0007
2026-10-12T09:16:40 ERROR POST /payments 502 3008ms user=u_1042 upstream=razorpay
2026-10-12T09:17:03 INFO  GET /orders/882 404 12ms user=u_3301
2026-10-12T09:18:27 ERROR POST /payments 504 10001ms user=u_2210 upstream=razorpay
2026-10-12T09:19:55 INFO  POST /orders 201 167ms user=u_3301
2026-10-12T09:21:30 ERROR GET /invoices/77 500 91ms user=u_0007
EOF

grep prints the lines that match a pattern. -c counts, -i ignores case, -v inverts, -E enables extended regular expressions, -o prints only the matched part, -r searches a directory tree, and -n shows line numbers.

$ grep -c ERROR app.log
4
$ grep -E ' 5[0-9]{2} ' app.log | wc -l
4

Now the real power: a question answered by a pipeline. "Which failures are most common?"

$ grep ERROR app.log | awk '{print $3, $4, $5}' | sort | uniq -c | sort -rn
      2 POST /payments 502
      1 POST /payments 504
      1 GET /invoices/77 500

Read it left to right: keep error lines, print fields 3 to 5 (method, path, status), sort so identical lines sit together, count adjacent duplicates with uniq -c (which only works on sorted input), then sort numerically in reverse. Each stage is simple and you can check it by running the pipeline up to that point.

awk splits each line into whitespace-separated fields $1, $2 and so on, and can filter on them. "Which requests took over a second?"

$ awk '$6+0 > 1000 {print $1, $4, $6}' app.log
2026-10-12T09:14:09 /payments 3012ms
2026-10-12T09:16:40 /payments 3008ms
2026-10-12T09:18:27 /payments 10001ms

($6+0 turns 3012ms into the number 3012.) sed edits streams; its most common use is substitution, for example masking user ids before you paste a log into a ticket or an AI tool:

$ sed 's/user=u_[0-9]*/user=REDACTED/' app.log | head -2
2026-10-12T09:14:02 INFO  POST /orders 201 182ms user=REDACTED
2026-10-12T09:14:05 INFO  GET /orders/881 200 35ms user=REDACTED

find locates files by name, age or size: find . -name "*.log" -mmin -60 lists log files modified in the last hour. On a live server, tail -f app.log | grep ERROR follows a log as it grows.

Exit codes again: grep returns 0 if it found a match, 1 if it found none, and 2 if something went wrong. Scripts rely on that difference.

$ grep TIMEOUT app.log; echo "exit code: $?"
exit code: 1
$ grep ERROR nope.log; echo "exit code: $?"
grep: nope.log: No such file or directory
exit code: 2

In a pipeline, $? is the exit code of the last command only, so failing_cmd | sort looks successful. Shell scripts should start with set -euo pipefail: stop on errors, treat unset variables as errors, and make a pipeline fail if any stage fails.

JSON on the command line: jq

APIs speak JSON, and jq is grep, awk and sed for it. Save the example response from GitHub's "Get a repository" documentation as repo.json (trimmed):

{"full_name": "octocat/Hello-World", "stargazers_count": 80, "language": null,
 "license": {"key": "mit", "spdx_id": "MIT"}, "open_issues_count": 0,
 "topics": ["octocat", "atom", "electron", "api"]}

$ jq -r '.license.spdx_id' repo.json
MIT
$ jq '{name: .full_name, stars: .stargazers_count, language}' repo.json
{
  "name": "octocat/Hello-World",
  "stars": 80,
  "language": null
}
$ jq -r '.topics[]' repo.json | sort
api
atom
electron
octocat

.a.b walks into objects, .[] iterates an array, {...} builds a new object, and -r prints raw strings without quotes so the output can be piped on.

Talking to HTTP with curl

curl sends an HTTP request and prints the response. It is the fastest way to answer "is it my code or the API?". Practise offline first: in one terminal run python3 -m http.server 8000 inside a folder containing a file health.json, then in another:

$ curl -i http://localhost:8000/health.json
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.13.16
Date: Tue, 13 Oct 2026 20:54:26 GMT
Content-type: application/json
Content-Length: 17
Last-Modified: Tue, 13 Oct 2026 20:54:24 GMT

{"status": "ok"}

$ curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" http://localhost:8000/missing.json
404 0.001787s

$ curl -sS --fail http://localhost:8000/missing.json; echo "exit code: $?"
curl: (22) The requested URL returned error: 404
exit code: 22

$ curl -sS http://localhost:9999/; echo "exit code: $?"
curl: (7) Failed to connect to localhost port 9999 after 0 ms: Couldn't connect to server
exit code: 7

The flags worth memorising: -i include response headers, -s silent (no progress bar), -S still show errors, -o write the body somewhere, -w print details such as the status code and timing, -H add a request header, -X set the method, -d send a body, -L follow redirects, -v show everything including the request. Note that by default curl exits 0 even for a 404 or 500: the HTTP exchange itself succeeded. Add --fail in scripts so HTTP errors become exit code 22. Exit code 7 (could not connect) is a completely different problem from a 404: the first means nothing is listening, the second means a server answered "no such thing".

Now a real public API. GitHub's REST API allows 60 unauthenticated requests an hour and documents its headers:

curl -sS -i \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2026-03-10" \
  https://api.github.com/repos/fastapi/fastapi | head -25

In the headers, find the status line (HTTP/2 200, or HTTP/1.1 200 depending on your curl build), content-type: application/json; charset=utf-8, and the rate-limit headers x-ratelimit-limit, x-ratelimit-remaining and x-ratelimit-reset. Run it a few times and watch remaining fall. Then pipe the body through jq, and try a repository that does not exist with -o /dev/null -w "%{http_code}\n" to see the 404. Sending JSON looks like curl -X POST -H "Content-Type: application/json" -d '{"title": "test"}' URL; you will do exactly that against your own API in the next lesson.

Environment variables

Every process inherits a set of key-value strings from its parent: the environment. Configuration and secrets reach your Python code this way, which is why code reads os.environ instead of hard-coding keys.

$ export APP_ENV=staging          # set for this shell and every program it starts
$ python3 -c 'import os; print(os.environ.get("APP_ENV"), os.environ.get("DATABASE_URL", "<not set>"))'
staging <not set>
$ # VAR=value before a command sets it for that one command only
$ DATABASE_URL=sqlite:///dev.db python3 -c 'import os; print(os.environ.get("DATABASE_URL", "<not set>"))'
sqlite:///dev.db

env lists everything; echo "$PATH" shows where the shell looks for programs, and which python3 tells you which one wins, which answers half of all "but I installed it!" mysteries. A .env file is not read automatically by anything: your code or a library such as python-dotenv must load it. And remember that a key typed into export lands in your shell history file.

Reading a Python stack trace without panicking

A traceback is a list of function calls that were in progress when the exception happened, printed oldest first. That ordering means the most important line is the last one, and the frame that matters most is usually the lowest one in your own code. Here is a real one (paths shortened):

Traceback (most recent call last):
  File "/home/asha/shop/report.py", line 23, in <module>
    main()
    ~~~~^^
  File "/home/asha/shop/report.py", line 18, in main
    order = parse_order(line)
  File "/home/asha/shop/report.py", line 5, in parse_order
    return json.loads(raw)
           ~~~~~~~~~~^^^^^
  File "/usr/lib/python3.13/json/__init__.py", line 352, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.13/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/json/decoder.py", line 361, in raw_decode
    obj, end = self.scan_once(s, idx)
               ~~~~~~~~~~~~~~^^^^^^^^
json.decoder.JSONDecodeError: Illegal trailing comma before end of object: line 1 column 47 (char 46)

The method, every time:

1. Read the last line. Exception type and message: the input was not valid JSON, with a trailing comma at character 46.
2. Walk up to your frame. Skip frames in /usr/lib/python3.x/ or site-packages/; they are the standard library and installed packages, and they are almost never the bug. The lowest frame in your project is parse_order, line 5.
3. Ask what value reached that line. The library did its job; the question is why raw contained bad JSON. Going up one more frame, main line 18 shows where the data came from.
4. Reproduce small. Call parse_order with just that one line in a REPL, or put breakpoint() above line 5 and inspect raw.

Two more things to recognise. The ~~~^^^ markers (Python 3.11 and later) point to the exact expression that failed within a line. And if you see The above exception was the direct cause of the following exception or During handling of the above exception, another exception occurred, there are several tracebacks chained together: the first one printed is usually the root cause, and the later ones are what your code did about it.

You can apply the same method in code, which is handy for logging:

import json
import sysconfig
import traceback

STDLIB = sysconfig.get_paths()["stdlib"]

def parse_order(raw: str) -> dict:
    return json.loads(raw)

def load_orders(lines: list[str]) -> list[dict]:
    return [parse_order(line) for line in lines]

try:
    load_orders(['{"id": 1}', '{"id": 2,}'])
except json.JSONDecodeError as exc:
    frames = traceback.extract_tb(exc.__traceback__)
    print("Last line:", type(exc).__name__, "-", exc)
    for f in frames:
        owner = "library" if f.filename.startswith(STDLIB) else "YOURS"
        print(f"  [{owner:7}] {f.name:<12} line {f.lineno}: {f.line}")
    mine = [f for f in frames if not f.filename.startswith(STDLIB)]
    print("Start reading at:", mine[-1].name, "line", mine[-1].lineno)

Last line: JSONDecodeError - Illegal trailing comma before end of object: line 1 column 9 (char 8)
  [YOURS  ] <module>     line 14: load_orders(['{"id": 1}', '{"id": 2,}'])
  [YOURS  ] load_orders  line 11: return [parse_order(line) for line in lines]
  [YOURS  ] parse_order  line 8: return json.loads(raw)
  [library] loads        line 352: return _default_decoder.decode(s)
  [library] decode       line 345: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  [library] raw_decode   line 361: obj, end = self.scan_once(s, idx)
Start reading at: parse_order line 8

Windows users

Use WSL (Windows Subsystem for Linux): run wsl --install in PowerShell as administrator, restart, and you get a real Ubuntu shell where every command in this course works. Git Bash, installed with Git for Windows, is a lighter option that covers Git and the basic tools (grep, sed, awk, curl), though you may need to install jq separately. Avoid mixing: in older Windows PowerShell, curl is an alias for a different command, Invoke-WebRequest, with different flags; type curl.exe if you must stay in PowerShell.

How this shows up in FDE work

A logistics client in Chennai reports that their WhatsApp order-status agent "randomly fails". You SSH into the VM. grep -c ERROR on today's log shows 312 errors; the awk | sort | uniq -c | sort -rn pipeline shows 300 of them are 502 from one upstream tracking API. curl -sS -o /dev/null -w "%{http_code} %{time_total}s\n" against that API from the same VM returns 502 after ten seconds, so it is not your agent. env | grep TRACKING reveals the service is still pointed at the vendor's old sandbox URL. Twenty minutes, no code changes, and a precise message to the client's ops team. That is the job.