For occasional downloads, it is usually sufficient to manually download the files you need to your computer and then upload them to your server via FTP, SFTP, or SCP. However, when you need to move large files or multiple files, these methods become cumbersome.
Using a Linux command line tool, such as wget, you can download files directly to your server, bypassing your local machine completely. This saves bandwidth, time, and ultimately, money.
In its most basic form, the wget command string only involves two components: the command itself and the URL. For example, if you wanted to download a single video file, the command string would look like this:
By default, wget will seek out the URL and download the file to the user’s current directory. The output will look like this:
–2010-08-12 11:52:10– http://www.domain-name.tld/videos/january/scene1.mp4
Resolving www.domain-name.tld… 184.108.40.206
Connecting to www.domain-name.tld 220.127.116.11|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 115025 (112K) [application/mp4]
Saving to: ‘scene1.mp4’
100%[=======================================>] 115,025 134K/s in 0.8s
2010-08-12 11:52:11 (134 KB/s) – `scene1.mp4′ saved [115025/115025]
It will actively show you the download progress in the form of a percentage and with the arrow moving in between the brackets.
Simple single-file downloads are only the beginning. You can use wget to download multiple files and even crawl through directories recursively. For example, the “-r” flag turns on recursive retrieving, and the “-o” flag, followed by a file name, creates a log file. Therefore, to download the file and all files in the directories, while saving download activity to a log file, enter the following command:
wget -r -o logfile http://www.domain-name.tld/videos/january/
Wget can also perform more advanced pre-download functions, such as loading cookies, which may be required by some websites. Wget is free and open source software, available for most Linux distributions. For complete documentation, type “man wget” from the command line.