Wget Spider

Wednesday, January 19th, 2011 by Anoop Manoj (See all posts by Anoop Manoj)

The wget command is one of the powerful commands of the Linux operating environment. wget command is not just used as a downloading tool, but it can also be used for getting more information about a remote server. You can use the following command from your Linux shell to know the basic webserver details of a remote domain:

Command: wget -S –spider example.com

-S swtich will help to get the information of the headers sent by the webserver.

–spider – if you are enabling the spider mode, the page will not be downloaded to your local machine and it will checked from its remote location itself.

Example:

[root@twenty ~]# wget -S –spider example.com
Spider mode enabled. Check if remote file exists.
–2011-01-17 12:00:32–  http://example.com/
Resolving example.com… IP address
Connecting to example.com|IP address|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.1 301 Moved Permanently
Date: Mon, 17 Jan 2011 06:21:42 GMT
Server: Apache/2.2.16 (EL)
Location: http://www.example.com/
Connection: close
Content-Type: text/html; charset=iso-8859-1
Location: http://www.example.com/ [following]
Spider mode enabled. Check if remote file exists.
–2011-01-17 12:00:34–  http://www.example.com/
Resolving www.example.com… IP address
Connecting to www.example.com|IP address|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.1 200 OK
Date: Mon, 17 Jan 2011 06:21:43 GMT
Server: Apache/2.2.16 (EL)
X-Powered-By: PHP/5.2.14
Connection: close
Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled — not retrieving.

Now you will able to check the basic information of your favorite domain. Try it out!