There is a lot of valuable information that can be obtained from a target's website. Most corporate websites list their executive team, public figures, and members from recruiting and human resource contacts. These can become targets for other search efforts and social engineering attacks.

More valuable information can be obtained by looking at what other companies are listed as partners, current job postings, business information, and security policies. Reconnaissance on a high-valued partner can be as important as the primary target, because partners may provide a new source for obtaining intelligence. An example is compromising a contracted resource that manages the helpdesk at a target's headquarters.

The Robots.txt file is publicly available and found on websites that gives instructions to web robots (also known as search engine spiders), about what is and not visible using the Robots Exclusion Protocol. The Disallow: / statement tells a browser not to visit a source; however, a Disallow can be ignored by giving a researcher intelligence on what a target hopes to not disclose to the public.

To view the Robots.txt file, find the Robots.txt file in the root directory of a target website. For example, adding the Robots.txt file to Facebook would look as shown in the following screenshot:

There are archived versions of most public websites available on sources such as the WayBack Machine at archive.org. Interesting information can be found in an older version of a target's website, such as outdated organizational charts, phone numbers, customer intelligence, systems information listed in fields, such as view source or /robots.txt, older business partnerships, vulnerabilities fixed in later versions, and other useful data, the target doesn't want on the current website version. It is important to understand that the publicly available information is hard to remove completely, making historical sources a valuable place for Reconnaissance research.

To access the WayBack Machine, open up the web browser and navigate to http://archive.org, you will see the Internet Archive WayBack Machine in the middle of the page, as shown in the following screenshot:

Type the URL you would like to browse and see if any archives have been captured. A history of the archive can be viewed here, as shown in the following screenshot:

As a Penetration Tester, this is a valuable tool, because it doesn't leave evidence of Reconnaissance on your target. In fact, your target is never even touched using this tool. All the information has been archived online in the Wayback Machine. The next two screenshots show www.lancope.com in 2002 compared to 2013:

RIR is an organization that manages the allocation and registration of IP resources within a particular region of the world. There are five major RIRs: the USA, Canada, and parts of the Caribbean region can be found at www.arin.net. You can gather information on a target such as Lancope, as seen in the following screenshot:

The EDGAR database contains registration statements, periodic reports, and other forms of information on companies since 1994. Companies in the United States of America are required by law to file, and all information is publicly available. The following two screenshots show public documents found while searching Lancope:

Social media is everywhere, and in most cases, publicly accessible. Most people have a Facebook, LinkedIn, blogs, or other forms of cloud accounts containing valuable information. This information can be used as a means of social engineering intelligence from a target's current or previous staff. An example is searching Glassdoor.com to identify previous employees that are disgruntled, based on feedback.

There are many people finding web resources such as Maltego (found in Kali Linux) that can comb popular social media, public records, and job recruiting websites to fingerprint an individual based on limited information, such as a first and last name. A researcher could gather information such as everywhere an individual has lived, done business, people with which they socialize, special interests, favorite sport teams, and other useful data for future research and social engineering attacks.

Most people are naturally trusting and assume information posted on public sources is real. To test this concept, the writers of this book created a fake person through social media and pretended to be a new hire for a target company. The fake person would become friends with associates of our target, post fake holiday cards that are linked to a BeEF system designed to compromise vulnerable Internet browsers (using BeEF is covered later in this book), and captured sensitive information from compromised systems. We were able to map out the entire organization, obtain network information, and even had hardware shipped to us without an internal e-mail or phone number. Our fake person, Emily Williams isn't real, yet received job offers, was provided inside information, and access to events hosted by the target. Information is power, and people will give it to a requester who seems like they can be trusted.

More information on this project can be found at: http://www.thesecurityblogger.com/?p=1903

Job postings contain a wealth of knowledge about a target's environment. Job listings can provide details on what type of systems are installed, who manages them, how large the staff is, and the staff's skill level. Human Resource representatives are typically eager to share information with a potential new hire, which can be used as an avenue to inside information. An example is targeting a job posting for a Oracle developer to understand the hardware, version of Oracle, names of existing and previous administrators, existing operation issues, security gaps, and methods to access such as asking "can administrators work from home, and how do they access the systems?"

Another avenue to review is a job's expected salary, benefits, and turnover rate on popular job boards. These trends may uncover new vectors for attack. Glassdoor.com is an example of a popular source for this type of data.

The investment in cyber security for a target can typically be determined based on the level of physical security. One would assume a building with fences and armed guards would have a higher investment in cyber security than a target located within a public building. Online mapping sources such as Google maps can help identify where physical security is implemented, and trends on how people move to and from the target. Other areas of interest are identifying where a Penetration Tester could camp out to scan for wireless networks, and possible methods to bypass access controls, such as attire and badges used to obtain physical access.

Shodan is a search engine that can identify a specific device, such as computer, router, server, using a variety of filters, such as metadata from system banners. For example, you can search for a specific system, such as a Cisco 3850, running a version of software such as IOS Version 15.0(1)EX.

The following example is a use case searching for any SCADA system with public Internet access, which in theory isn't supposed to exist however, Shodan can show this is not necessarily true. SCADA systems control things like power management and water treatment, so identifying public accessible systems is extremely bad!

Google hacking is the most common form of search engine Reconnaissance of web applications. Google hacking uses advanced operations in the Google search engine to locate specific strings of text within search results. Search filters can zero in on specific versions of vulnerable web applications such as Powered by Apache in the intitle:"index of" operator or identify log files such as ws_ftp.log, containing sensitive IP information. The following few screenshots demonstrate using a Google search for Linksys to find publicly available Linksys cameras. The first screenshot shows the search command followed by some example results from issuing the search. The last screenshot shows a camera feed that could be found using this technique.

Some example search queries are as follows:

For more information on Google hacking, check out a very good book titled Google Hacking for Penetration Testers by Johnny Long, as well as his website at http://johnny.ihackstuff.com.

The Google Hacking Database (GHDB) created by Johnny Long of Hackers For Charity (http://www.hackersforcharity.org/), is the definitive source for Google search queries. Searches for usernames, passwords, vulnerable systems, and exploits have been captured and categorized by Google hacking aficionados. The aficionados who have categorized the Google searches are affectingly known as Google dorks.

To access the GHDB, navigate to http://www.exploit-db.com/google-dorks/. You will see the latest GHDB searches listed on the web page. You can click on any of the search queries yourself.

You will find different categories of searches at the bottom of the page that have been saved. In the following example, we scroll to the category Vulnerable Files and select the query ionCube Loader Wizard.

We can select the search query, and it will bring us to Google, performing the same search query.

The preceding example shows Google has found a few results. The ionCube Loader is apparently not configured or misconfigured. The ionCube Loader is actually a great piece of software that protects software written in PHP from being viewed or changed from unlicensed computers. However, in this case, administrators left the default wizard running without any configuration.

When we click on the first link, we get the home screen to configure the software.

The GHDB essentially turns Google into a limited web application scanner for a Penetration Tester. In this case, good software that can increase security can now potentially be used against a web server by an attacker.

Many people do not understand the true purpose of researching the network of a target prior to launching an attack. Amateur Penetration Testers understand the need to pick a target before they can perform a Penetration Test. After all, a Penetration Tester needs someplace at which to point their arsenal of tools. Many amateurs will run Nmap, ping sweeps, or other noisy tools to determine what targets are available disrupting the environment, which later yields poor results.

Network Reconnaissance is about selecting a target. A seasoned network security professional will tell you good Reconnaissance is about selecting a quality target, spending the majority of their time watching, rather than acting. The first step of every Penetration Test is accurately finding and selecting quality targets.

The following are the best available tools in Kali for web application Reconnaissance. Other tools may be available for web applications or different target types however, the focus of this chapter is enabling a reader for evaluating web application-based targets.

Nmap stands for Network Mapper, and is used to scan hosts and services on a network. Nmap has advanced features that can detect different applications running on systems as well as services and OS fingerprinting features. It is one of the most widely used network scanners making it very effective, but also very detectable. We recommend using Nmap in very specific situations to avoid triggering a target's defense systems.

For more information on how to use Nmap, see http://nmap.org/.

Additionally, Kali comes loaded with Zenmap. Zenmap gives Nmap a graphical user interface (GUI) to run commands. Although there are many purists who will tell you the command-line version is the best version because of its speed and flexibility, Zenmap has come a long way and has incorporated most of the Nmap features. Zenmap also offers exclusive features not offered in Nmap, such as developing graphical representations of a scan, which can be used later by other reporting systems.

To open Zenmap, go to the Backtrack menu. Navigate to Information Mapping | DNS Analysis, and launch Zenmap.

You will notice under the Profile menu that there are several options to determine what type of scan you would like to run, as shown in the following screenshot:

The first step is creating a new profile. A profile in Zenmap allows a Penetration Tester to create what type of scan to execute and what different options to include. Navigate to the Profile menu and select New Profile or Command to create a new profile, as shown in the following screenshot:

When you select New Profile or Command, the profile editor will launch. You will need to give your profile a descriptive name. For example, you can call the profile My First Scan or anything else you would like.

Optionally, you can give the profile a description. During your course of using Zenmap you will probably create many profiles and make multiple scans. A natural reflex may be to delete profiles post execution. Here is a word of advice: profiles don't take any space and come handy when you want to recreate something. We recommend being extremely descriptive in profile names and come up with a standard naming method. I start all my profile description with the date, time, description of my location, my target network scan location, and customer name.

When you completed your description, click on the Scan tab. In the Targets section, you will add what hosts or networks you would like to scan. This field can take a range of IP addresses (10.0.1.1-255) or it can take a network in CIDR format (10.0.1.0/24).

You can see option -A is selected by default to enable aggressive scanning. Aggressive scanning will enable OS detection (-O), version detection (-sV), script scanning (-sC) and traceroute (--traceroute). Essentially, aggressive scanning allows a user to turn on multiple flags without the need of having to remember them.

Aggressive scan is considered intrusive, meaning it will be detected by most security devices. An aggressive scan may go unnoticed if your target is an extremely specific host, but regardless of the situation, it's recommended you have the permission to scan before using this or scanning option. As a reminder, completing the ACK in the three-way handshake with an unauthorized system is considered illegal by the US standards.

We can use the information we received from our DNS Reconnaissance exercise to target a very specific host. Before we do that, let's set a few common options first.

Click on the Ping tab. Select the -Pn flag option so Nmap will not ping the host first. When this flag is not set, Nmap will ping your target hosts and networks. Default settings only perform scans on hosts that are considered alive or reachable. The -Pn flag tells Nmap to scan a host even without a ping response. Although this makes the scan considerably more lengthy, the –Pn flag allows Nmap to avoid a common problem of not receiving a ping response when the ping requests are blocked by security defenses.

Save changes made by clicking on the Save Changes button in the lower-right hand corner. Once saved, select the Scan button on the top-right side of the screen to start the scan. Notice your options and target that you configured in the profile editor are listed.

The network Topology tab will give you a quick look at how your scan on the target network was completed, and if you had to cross any routers. In this example, you see the scan stayed local to the network.

The Hosts tab will give a list of the hosts discovered.

When a host is selected, Zenmap will display a detailed list of the hosts, their operating systems, and common services. In the following screenshot, you can see one of our hosts is a satellite DVR/receiver combo.

If you look at the scan window, you will not only see what ports are open on specific hosts, but also what applications are running on those hosts. Notice that Nmap can determine things, such as a server is running IIS 5.0 as a web server over port 80. The scan results will yield the IP address of the server, the operating system the server is running, as well as the web applications running on the host. Penetration Testers will find these results valuable when they are searching for exploits against this host.

.

It is now possible for you to concentrate your efforts on the target's running web services or port 80, because it is open.

Zenmap is the best way to get output from Nmap scans. Zenmap offers a rich graphical user interface that displays scans that can be exported into different formats, such as text or Microsoft Excel.

Although there are many ways to get outputs from Nmap (for example, the authors in this book prefer the command-line techniques) we have included this technique because it is constantly referenced in many web penetration standards and is a common way for people to use it.

In addition, several places in GUI for Zenmap allow the user to export graphics and certain parts of the report in CSV files or image files. These exports are extremely valuable when creating reports.