curl and wget Basics: HTTP Communication from the Command Line

curl and wget Basics: HTTP Communication from the Command Line
Reading time: About 12 min Difficulty: Beginner

What You'll Learn

  • The role difference between curl and wget so you stop guessing which to use
  • Core commands for downloading files, checking headers, and calling APIs
  • Why beginners hit "nothing was saved," "redirects don't work," and "garbled characters" — and how to fix each
  • Copy-paste templates for inspecting HTTP status codes and posting JSON

Quick Summary

  • One-shot file download → wget URL
  • Hit an API, inspect headers, or POST → curl URL
  • Want to follow redirects → curl -L URL
  • Download was interrupted → wget -c URL

Environment

  • OS: Ubuntu / typical Linux
  • curl is usually preinstalled. Install wget via sudo apt install wget
  • We focus on HTTP/HTTPS (curl also supports ftp, sftp, smtp, etc.)

1. curl vs wget: Different Roles

Lina: Senpai, both curl and wget "fetch data from a URL," right? I always hesitate over which one to use...
Linny-senpai: Great question. They actually have different design goals, and once you know that, picking one becomes easy. In one line: wget is a downloader, curl is a multi-purpose HTTP client.
Lina: Downloader vs client...?
Linny-senpai: wget's default action is "given a URL, save the file." curl's default is "given a URL, print the response to the screen." That single difference cascades into every other choice.

At a glance

Aspect curl wget
Default behavior Prints to stdout Saves as a file
HTTP methods GET/POST/PUT/DELETE/... Mostly GET
Follow redirects Need -L flag Automatic
Recursive download Weak Strong (-r)
Resume -C - -c
Best for APIs, debugging, POST Mirroring sites, large DL

Rule of thumb: APIs, headers, or POST involved → curl. Just grabbing a file → wget.

2. curl Basics: Print Before You Save

2-1. Hit a URL and Print the Response

$ curl https://example.com
<!doctype html>
<html>
<head>
    <title>Example Domain</title>
...
Lina: Whoa, the entire HTML just flooded my screen.
Linny-senpai: That's curl's default behavior — dump everything to stdout. To save to a file, add -o (specify name) or -O (use the URL's filename).

2-2. Save With a Custom Name: -o

$ curl -o page.html https://example.com

-o stands for output. Lowercase o takes a filename as its argument.

2-3. Save Using the URL's Filename: -O

$ curl -O https://example.com/sample.tar.gz

Uppercase O saves using the URL's trailing filename (sample.tar.gz here).

Mixing up -o and -O ends in tears

  • -O requires the URL to end with a real filename
  • curl -O https://example.com/ (trailing slash) fails — no filename to use
  • When in doubt, use -o name and be explicit

2-4. Show Download Progress

$ curl -O --progress-bar https://example.com/big.iso

--progress-bar gives a clean progress bar — handy for long transfers.

3. The #1 Beginner Trap: Redirects and -L

Lina: Senpai, I ran curl on https://github.com/torvalds/linux and the response looks wrong...
Linny-senpai: Classic. GitHub URLs and short URLs often involve HTTP redirects, and curl does not follow them by default. So instead of the redirected page, you get the "moved here" placeholder.
Lina: It doesn't follow them? wget did, though.
Linny-senpai: Right — that's a key difference. With curl you have to add -L explicitly to make it follow redirects.

3-1. Follow Redirects With -L

# BAD: only the redirect placeholder is returned
$ curl https://github.com/torvalds/linux

# GOOD: fetch the final destination
$ curl -L https://github.com/torvalds/linux

-L stands for Location (the HTTP Location header).

Memorize the pattern

  • When hitting external sites with curl, always add -L to avoid surprises
  • File downloads often require -L too (GitHub Releases, S3 redirects, etc.)
# Standard safe pattern
$ curl -LO https://github.com/some/repo/releases/download/v1.0/binary.tar.gz

Want to check "is this link alive?" or "where does this redirect to?" without downloading the body? Use -I.

$ curl -I https://example.com
HTTP/2 200
content-type: text/html; charset=UTF-8
content-length: 1256
date: Sun, 26 May 2026 09:00:00 GMT
server: ECS
Lina: So HTTP/2 200 means "success," right?
Linny-senpai: Exactly. The number is the HTTP status code. Memorize these and you cover 90% of real-world cases.

HTTP status cheat sheet

Code Meaning Examples
2xx Success 200 OK, 204 No Content
3xx Redirect 301 (permanent), 302 (temp)
4xx Client-side error 404 (not found), 401/403
5xx Server-side error 500 (bug), 503 (overload)

4-1. Trace Where a Redirect Leads

$ curl -ILs https://bit.ly/3xxxxx | grep -i location

-L follows redirects, -s silences progress, -I shows headers only. The location: lines reveal each hop.

5. POST and JSON: Talking to APIs

Lina: I want to call a REST API with curl. How do I send JSON?
Linny-senpai: API testing is where curl shines. Use -X for the HTTP method, -H for headers, and -d for the body. Remember those three flags and you're done.

5-1. GET With Query Parameters

$ curl "https://api.example.com/users?id=42"

Always quote URLs containing ? and & — otherwise the shell interprets & as background-execution.

5-2. POST JSON

$ curl -X POST https://api.example.com/users \
    -H "Content-Type: application/json" \
    -d '{"name":"lina","role":"beginner"}'

The three-flag combo

  • -X POST: explicit HTTP method (optional with -d, but explicit is safer)
  • -H "Content-Type: application/json": declare "this is JSON"
  • -d '...': the request body. Use single quotes so the inner " don't need escaping

5-3. Bearer Token Authentication

$ curl https://api.example.com/me \
    -H "Authorization: Bearer YOUR_TOKEN_HERE"

Keep tokens out of your shell history

# BAD: visible in `history` and `ps`
$ curl -H "Authorization: Bearer abc123..." ...

# GOOD: pass via environment variable
$ export API_TOKEN=abc123...
$ curl -H "Authorization: Bearer $API_TOKEN" ...

Tokens visible in history or ps are an incident waiting to happen. Use env vars or ~/.netrc.

5-4. Basic Auth

$ curl -u username:password https://example.com/private

-u accepts user:pass. HTTPS only.

6. wget's Strength: Reliable Downloads

6-1. Basic: Save as a File

$ wget https://example.com/sample.tar.gz
Lina: wget saves the file with no flags at all. Easy.
Linny-senpai: Right — wget saves by default. And unlike curl, it follows redirects automatically. For simple downloads, wget is harder to mess up.

6-2. Save With a Custom Name: -O

$ wget -O custom.tar.gz https://example.com/sample.tar.gz

-O means the opposite in curl and wget

  • curl's -O: save using the URL's filename
  • wget's -O: specify the filename (equivalent to curl's -o)

Everyone who uses both gets bitten by this at least once.

6-3. Resume an Interrupted Download: -c

$ wget -c https://example.com/big.iso

-c stands for continue. If your connection dropped halfway, this picks up from where it stopped instead of restarting.

6-4. Retries and Timeouts

$ wget --tries=5 --timeout=30 https://example.com/file.zip

Survives flaky networks by retrying.

7. Recursive Download wget -r (Handle With Care)

$ wget -r -l 2 -np https://example.com/docs/
  • -r: recurse through links
  • -l 2: cap depth at 2 levels
  • -np: don't ascend to parent directories

8. Encoding and Line-Ending Traps

Lina: The HTML I fetched is full of &lt; and weird characters. I can't read it.
Linny-senpai: Browsers auto-decode, but curl/wget hand you the raw bytes. For non-UTF-8 pages, run the output through iconv. For structured data like JSON, pipe through a parser like jq.
Lina: Also, line endings sometimes look off...
Linny-senpai: Files from Windows servers often use \r\n (CRLF). On Linux, strip them with dos2unix or tr -d '\r'.
# Convert Shift_JIS to UTF-8 while saving
$ curl -s https://example.com/sjis.html | iconv -f SHIFT_JIS -t UTF-8 > page.html

# Strip CRLF to LF
$ curl -s https://windows-server.example/data.csv | tr -d '\r' > data.csv

9. Common Beginner Pitfalls

9-1. curl URL Saved Nothing

Cause: curl prints to stdout by default. To save, use -O or -o.

$ curl -O https://example.com/file.zip
$ curl -o my.zip https://example.com/file.zip

9-2. Got the Redirect Page Instead of the Content

Cause: forgot -L.

$ curl -L https://github.com/...

9-3. Query String Triggers bash: command not found

Cause: shell interpreted & as the background-execution operator.

# BAD
$ curl https://api.example.com/search?q=linux&page=2

# GOOD
$ curl "https://api.example.com/search?q=linux&page=2"

9-4. SSL Certificate Error

Cause: self-signed cert, expired cert, internal CA, etc.

# With verification (recommended)
$ curl https://internal.example.com

# Skip verification (emergencies only — never in production)
$ curl -k https://internal.example.com

-k disables verification, opening the door to man-in-the-middle attacks. For production, fix the certificate chain instead.

9-5. Proxy Environment Can't Connect

$ export http_proxy=http://proxy.example.com:8080
$ export https_proxy=http://proxy.example.com:8080
$ curl https://example.com

In corporate environments, set http_proxy / https_proxy.

10. Mini Exercises

Lina: Theory's in! Let me try this in the terminal.
Linny-senpai: Three exercises. https://httpbin.org is a safe sandbox for HTTP testing — hit it freely.

Exercise 1: Check the HTTP status code and content-type of https://httpbin.org/get.

Show hint

There's a flag that fetches only headers.

Show answer 
$ curl -I https://httpbin.org/get
HTTP/2 200
date: ...
content-type: application/json
...

Exercise 2: POST the JSON {"hello":"world"} to https://httpbin.org/post.

Show hint

The -X POST / -H / -d three-flag combo.

Show answer 
$ curl -X POST https://httpbin.org/post \
    -H "Content-Type: application/json" \
    -d '{"hello":"world"}'

You'll see "json": {"hello": "world"} echoed back in the response.

Exercise 3: Follow the redirect chain at https://httpbin.org/redirect/3 and fetch the final page.

Show hint

You need the flag that follows redirects.

Show answer 
$ curl -L https://httpbin.org/redirect/3

After three hops it lands on /get and returns that response.

11. Copy-Paste Templates

curl templates

# Print to screen (debug)
curl URL

# Save (using URL's filename)
curl -O URL

# Save (custom name)
curl -o name URL

# Follow redirects and save (GitHub Releases, etc.)
curl -LO URL

# Headers only
curl -I URL

# Get just the status code
curl -s -o /dev/null -w "%{http_code}\n" URL

# POST JSON
curl -X POST URL \
    -H "Content-Type: application/json" \
    -d '{"key":"value"}'

# Bearer token
curl URL -H "Authorization: Bearer $API_TOKEN"

# Basic auth
curl -u user:pass URL

# With progress bar
curl -O --progress-bar URL

wget templates

# Basic download
wget URL

# Save with custom name
wget -O custom.name URL

# Resume an interrupted download
wget -c URL

# Retries and timeout
wget --tries=5 --timeout=30 URL

# Quiet download (minimal log)
wget -q URL

# Recursive (mind the manners)
wget -r -l 2 -np --wait=2 https://example.com/docs/

Summary: What to Read Next