using wget on sites with basic http authentication:

wget -b -r -np --http-user=<user> --http-pass=<pass> http://.....

Only show the http headers

wget -S -O /dev/null --header="Host: www.compute.info" http://10.2.39.39/Download.xml

fetch contents to stdout

wget -qO- ip.compute.info

Test mod_deflate

wget -S -O /dev/null --header="Accept-Encoding: gzip" "http://man.compute.info/home"

Recursive download
wget -r -np http://linux.dell.com/repo/hardware/OMSA_6.1/pe1950/rh40_64/srvadmin/