As you can probably tell from my blog, I am constantly learning about new technology products. When I decide that I want to play with a new operating system release, or test out a new piece of software, I will typically retrieve the latest stable version of the software. When operating system ISO images are involved, this typically requires me to download several gigabytes of data prior to beginning my testing.
Periodically transfers will fail, leaving me with a chunk of the original file. To ensure that I’m not wasting time and bandwidth retrieving data that has already been downloaded, I have come to rely on the wget “–continue” option. When this option is present, wget will check to see if the downloaded file exists, and will proceed to grab the remainder of the file using the existing file’s size as an offset to the retrieval method:
$ ls -la Fedora-11-x86_64-DVD.iso
-rw-r--r-- 1 matty matty 543656488 2009-07-28 10:34 Fedora-11-x86_64-DVD.iso
$ wget --continue
http://www.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/11/Fedora/x86_64/iso/Fedora-11-x86_64-DVD.iso**
--2009-07-28 10:41:33--
http://www.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/11/Fedora/x86_64/iso/Fedora-11-x86_64-DVD.iso
Resolving www.gtlib.gatech.edu... 128.61.111.9, 128.61.111.10,
128.61.111.11, ...
Connecting to www.gtlib.gatech.edu|128.61.111.9|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location:
ftp://ftp.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/11/Fedora/x86_64/iso/Fedora-11-x86_64-DVD.iso
[following]
--2009-07-28 10:41:34--
ftp://ftp.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/11/Fedora/x86_64/iso/Fedora-11-x86_64-DVD.iso
=> `Fedora-11-x86_64-DVD.iso'
Resolving ftp.gtlib.gatech.edu... 128.61.111.10, 128.61.111.11,
128.61.111.9, ...
Connecting to ftp.gtlib.gatech.edu|128.61.111.10|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD
/pub/fedora.redhat/linux/releases/11/Fedora/x86_64/iso ... done.
==> SIZE Fedora-11-x86_64-DVD.iso ... 4268124160
==> PASV ... done. ==> REST 543656488 ... done.
==> RETR Fedora-11-x86_64-DVD.iso ... done.
Length: 4268124160 (4.0G), 3724467672 (3.5G) remaining
12% [+++++++ ] 549,004,000 1.82M/s
In the example above we can see that we have retrieved approximately 540MB of the Fedora 11 image, and wget uses this to begin the transfer approximately 540MB into the file. Wget and curl are truly awesome!