Host header required?
yes! many webservers serve multiple websites, and the Host header is
how the server can tell which domain/subdomain you're requesting.
example: Host: mail.google.com
User-Agent header required?
the only required header is the Host header.
User-Agent
is almost
always set, though, and servers sometimes use it to decide which version of the site to serve you.
Cookie!
your browser saves any cookies the server set and will send them back to the server on every request it makes.
Referer!
when someone clicks a link, by default browsers will set the
Referer header to the address of the site they were on
when they clicked the link.
(and yes, Referer is misspelled :) )
Accept-Encoding!
Accept-Encoding: gzip requests a compressed response. This
saves on bandwidth, so browsers will basically always set this header.
Accept-Language!
Accept-Language: es-ES
requests a response in Spanish. As with Accept-Encoding,
there's no guarantee that you'll actually get a response in Spanish,
but some websites will respect this header!
Authorization
header!
the Authorization
header lets you send a username & password. They're base64
encoded but not encrypted, so it's important to use TLS.
Example: Authorization: Basic OTA0MmYyNzExM...
some APIs instead need you to put the API key somewhere else, like in the request body
wget has a --continue flag that lets you
continue a download you interrupted. Which header does it use to do
that?
Range
Range lets you request a specific part of a document, like "everything after 20392383 bytes" for large download.
Content-Type
Content-Type: application/json.
If you don't set Content-Type, often the server won't know how to interpret your request.
If-None-Match or
If-Modified-Since
the server will return 304 Not Modified if no update is needed. Your browser sets If-None-Match to the last value of the ETag response header.