Tips and tricks for curl and wget
![curling wave in ocean](/sysadmin/sites/default/files/styles/full/public/2020-06/wave-1913559_1280.jpg?itok=0ViZRuW-)
photo by pixabay
The *nix commands curl
and wget
are useful for accessing URLs without resorting to a browser. Both commands allow you to transfer data from a network server, with curl
being the more robust of the two. You could use either of them to automate downloads from various servers.
The curl command
As mentioned, the curl
command allows you to transfer data from a network server, but it also enables you to move data to a network server. In addition to HTTP, you can use other protocols, including HTTPS, FTP, POP3, SMTP, and Telnet. Administrators commonly rely on curl
to interact with APIs using the DELETE, GET, POST, and PUT methods, as explained here.
The syntax for curl
is fairly straight-forward at first glance. Here is an example:
$ curl http://www.example.com/help.txt
curl Options
You can supply various options to your command syntax:
curl [options] [url]
It is the options which make curl
so robust. The following are some of the available options used with curl
and examples of their use.
-a, --append
When uploading a file, this option allows you to append to the target file instead of overwriting it (FTP, SFTP).
$ curl --append file.txt ftp://ftp.example.com/file.txt
--connect-timeout
The --connect-timeout
option sets the maximum time in seconds that curl
can use to make its connection to the remote server. This option is handy to prevent the connection from terminating too quickly, and to minimize the amount of time you want the command to attempt the connection.
$ curl --connect-timeout 600 http://www.example.com/
--dns-servers
This option allows you to list DNS servers curl
should use instead of the system default. This list can be handy when troubleshooting DNS issues or if you need to resolve an address against a specific nameserver.
$ curl --dns-servers 8.8.8.8 http://www.example.com/
--http3
You can specifically tell curl to use the http3 protocol to connect to the host and port provided with a https URL. --http2 and --http1.1 function in the same way and can be used to verify a webserver.
$ curl --http3 http://www.example.com:8080/
--output
If you need to retrieve a file from a remote server via a URL, --output
is an easy way to save the file locally.
$ curl http://www.example.com/help.txt --output file.txt
--progress-bar
This option displays the progress of the file transfer when combined with the --output
option.
$ curl --progress-bar http://www.example.com/help.txt --output file.txt
--sslv2
Like with HTTP, you can specifically tell curl to use a specific SSL option for the command to connect to and in this case we are specifying version 2. --ssl specifies SSL needs to be used and --sslv3 specifies SSL version 3. Note: sslv2 and sslv3 are considered legacy by the maintainer though still available.
$ curl --sslv2 https://www.example.com/
--verbose
The --verbose
option with curl
is useful for debugging and displaying what is going on during the call to the URL.
$ curl --verbose http://www.example.com
The wget command
Unlike curl
, the wget
command is solely for the retrieval of information from a remote server. By default, the information received is saved with the same name as in the provided URL.
Here is an example of the basic wget
syntax:
$ wget http://www.example.com/help.txt
wget Options
Like curl
, you can supply various options to your wget
command syntax:
wget [option] [url]
--dns-servers=ADDRESSES
You can specify one or more specific DNS servers to use when utilizing wget
to access a remote server. The syntax differs, however, if the option and nameserver addresses are joined with an =.
$ wget --dns-servers=8.8.8.8 http://www.example.com
-O
To save a file with a new name when using wget
, utilize the --output-document
option, or more simply -O
.
$ wget http://www.example.com/help.txt -O file.txt
--progress=type
With wget
, you can supply a type (dot or bar) to determine the ASCII visual of the progress bar. If a type is not specified, it will default to dot.
$ wget --progress=dot http://www.example.com
Wrap up
The curl
and wget
commands can be very useful when added to scripts to automatically download RPM packages or other files. This post only touches some of the most common features of what these commands can do. Check the related man pages for a complete list of options available for both curl
and wget
.
[ Want to test your sysadmin skills? Take a skills assessment today. ]
![Author’s photo](/sysadmin/sites/default/files/styles/user_picture_square/public/pictures/2019-09/Amy_Headshot.jpeg?itok=CokBs3qX)
Amy Marrich
Amy Marrich is a Principal Technical Marketing Manager at Red Hat. More about me