wget is a command-line utility that allows you to download files from the internet. It supports HTTP, HTTPS, and FTP protocols and is widely used for downloading files, mirroring websites, or even for batch downloading from servers. The name “wget” stands for “World Wide Web get.”
With wget, you can:
- Download files recursively (entire directories)
- Download files in the background
- Download files using FTP or HTTP protocols
- Continue downloads that were interrupted
- Limit download speeds or retry failed downloads
Checking if wget is Installed
Before using wget, it’s important to verify whether it’s installed on your system. To check if wget is installed, open a terminal and type:
$ wget –version
If wget is installed, you’ll see its version information. If not, the terminal will show an error like “command not found” or something similar, depending on the system.
If wget is not installed, you can install it easily on most Linux distributions. Here are installation instructions for two popular Linux systems:
Red Hat-based Linux (RHEL, CentOS, Fedora)
To install wget on a Red Hat-based distribution, use the following command:
# yum install wget # For older versions (CentOS 7/RHEL 7 and below)# dnf install wget # For newer versions (CentOS 8/Fedora)
Solaris
To install wget on Solaris, you can use pkgutil if it’s available:
# pkgutil -i wget
Alternatively, if pkgutil is not available, you can download the source code from the official GNU repository and compile it manually:
# cd /tmp# wget http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz# tar -xvzf wget-latest.tar.gz# cd wget-*# ./configure# make# make install
Basic Usage
# wget –help
– simple usage
# wget http://www.google.com
-r means recursively
-m mirror the source directory
-nc or –no-clobber is to skip downloads that would overwrite existing files
-np or –no-parent is to stop Wget from ascending into a parent directory
-k means convert links. So links on the webpage will be localhost instead of example.com/bla
-p means get all webpage resources so obtain images and javascript files to make website work properly.
-N is to retrieve timestamps so if local files are newer than files on remote website skip them.
-e robots=off is a flag option it needs to be there for the robots=off to work. means ignore directive of the robots file. the robots file tells a spider or script to ignore all files listed in the robots file. this means that wget will also ignore all files in that file.
–accept jpg,gif,bmp (or you can inversely use –reject)
–level=0 dictates the depth of the directories you’d like to search and download, a 0 means no depth limit, recursively download everything
Commonly Used Cases
Basic File Download
To download a single file from the internet, use:
$ wget http://example.com/file.zip
This command will download the file.zip from the specified URL.
Typical output:
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2Saving to: `strx25-0.9.2.1.tar.bz2.1’31% [=================> 1,213,592 68.2K/s eta 34sDownload completed:
And:
$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2Saving to: `strx25-0.9.2.1.tar.bz2’100%[======================>] 3,852,374 76.8K/s in 55s 2009-09-25 11:15:30 (68.7 KB/s) – `strx25-0.9.2.1.tar.bz2′ saved [3852374/3852374]
Download in Background
If you want to download a file in the background, you can use the -b (background) option:
$ wget -b http://example.com/largefile.zip
The download will continue in the background, and you can check the progress by viewing the wget-log file in the same directory.
For example:
$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2Continuing in background, pid 1984.Output will be written to `wget-log’.
It will initiate the download and gives back the shell prompt to you. You can always check the status of the download using tail -f as shown below.
$ tail -f wget-logSaving to: `strx25-0.9.2.1.tar.bz2.4′ 0K ………. ………. ………. ………. 1% 65.5K 57s 50K ………. ………. ………. ………. 2% 85.9K 49s 100K ………. ………. ………. ………. 3% 83.3K 47s 150K ………. ………. ………. ………. 5% 86.6K 45s 200K ………. ………. ………. ………. 6% 33.9K 56s 250K ………. ………. ………. ………. 7% 182M 46s 300K ………. ………. ………. ………. 9% 57.9K 47s
Download Multiple Files
You can download multiple files at once by listing them in a text file and using the -i option:
- Create a text file (urls.txt) with the URLs of the files you want to download.
http://example.com/file1.ziphttp://example.com/file2.ziphttp://example.com/file3.zip
- Run the command:
$ wget -i urls.txt
Resuming a Download
To resume a download that was interrupted, use the -c (continue) option:
$ wget -c http://example.com/largefile.zip
This will resume the download from where it left off.
Limiting Download Speed
If you want to limit the download speed to avoid overloading the network, you can use the –limit-rate option:
$ wget –limit-rate=200k http://example.com/largefile.zip
This command limits the download speed to 200 KB/s.
Advanced Use Cases
Downloading an Entire Website (Recursive Download)
You can download an entire website using the -r (recursive) option. This will download the HTML pages, images, and other linked resources from the site:
$ wget -r http://example.com
For more control, you can specify the depth of the recursion with the -l (level) option. For example, to download up to 3 levels deep:
$ wget -r -l 3 http://example.com
Mirror a Website
To create a mirror of a website for offline browsing, you can use the following options:
- -m (mirror) enables recursion, time-stamping, and preserving file structure.
- -np (no parent) ensures you don’t ascend to parent directories.
$ wget -m -np http://example.com
This will download the entire website and its contents recursively.
Download Files with Specific Extensions
To download only certain types of files, use the -A option followed by the file extension:
$ wget -r -A jpg,png,gif http://example.com/images/
This will download only .jpg, .png, and .gif images from the specified directory.
Download with Authentication
If you need to download a file from a server that requires HTTP authentication, you can use the –user and –password options:
$ wget –user=username –password=password http://example.com/protectedfile.zip
Limiting the Number of Retries
To limit the number of retries when a download fails, use the –tries option:
$ wget –tries=5 http://example.com/file.zip
This will attempt to download the file 5 times before giving up.
Using Proxies
If your environment has / requires the use of a proxy service, then it’s always best to set the following environment variables before executing the wget command:
$ export http_proxy=proxy.server.address:port$ export https_proxy=proxy.server.address:port Eg:$ export http_proxy=10.0.0.99:3128$ export https_proxy=10.0.0.99:3128
If proxy authentication is required (Note that if the password has an ‘@’ it will break this usage)
$ export http_proxy=http://username:password@proxy.server.address:port/
$ export https_proxy=http://username:password@proxy.server.address:port/
$ wget http://example.com/file.zip
Else specify username in the wget command instead:
$ export http_proxy=http://proxy.server.address:port/
$ export https_proxy=http:/proxy.server.address:port/
$ wget –proxy-user=userid –proxy-password=password http://example.com/file.zip
Summary of Common wget Options
|
Option |
Description |
|
-r |
Recursive download |
|
-l <level> |
Set the recursion level |
|
-np |
No parent directory traversal |
|
-A <filetypes> |
Accept only specified file types |
|
-c |
Continue an interrupted download |
|
-b |
Download in the background |
|
–limit-rate=<speed> |
Limit download speed |
|
–tries=<number> |
Set the number of retries |
|
–user=<username> |
Specify username for HTTP authentication |
|
–password=<password> |
Specify password for HTTP authentication |
|
-i <file> |
Read URLs from a file |
|
-m |
Mirror the website (recursive, timestamping) |
Conclusion
wget is a powerful and versatile tool for downloading files from the internet. It can handle everything from simple single file downloads to complex recursive website mirroring and handling of large datasets. This cheat sheet should help you understand both basic and advanced uses of the tool.
Check out our other Cheat Sheets and Blogs and if you would like us to write a cheat sheet for you, for FREE, (and we find it suitable) Contact Us.