Reconnaissance

The term Reconnaissance by definition comes from the military warfare strategy of exploring beyond the area occupied by friendly forces to gain information about the enemy for future analysis or attack. Reconnaissance of computer systems is similar in nature, meaning typically a Penetration Tester or hacker will attempt to learn as much as possible about a target's environment and system traits prior to launching an attack. This is also known as establishing a Footprint of a target. Reconnaissance is typically passive in nature and in many cases not illegal (however, we are not lawyers and cannot offer legal advice) to perform as long as you don't complete a three-way handshake with an unauthorized system.

Examples of Reconnaissance include anything from researching a target on public sources such as Google, monitoring employee activity to learn operation patterns, and scanning networks or systems to gather information, such as manufacture type, operating system, and open communication ports. The more information that can be gathered about a target brings a better chance of identifying the easiest and fastest method to achieve a penetration goal, as well as best method to avoid existing security. Also, alerting a target will most likely cause certain attack avenues to close as a reaction to preparing for an attack. Kali's official slogan says this best:

"The quieter you become, the more you are able to hear"

Reconnaissance objectives

• Target background: What is the focus of the target's business?
• Target's associates: Who are the business partners, vendors, and customers?
• Target's investment in security: Are security policies advertised? What is the potential investment security, and user security awareness?
• Target's business and security policies: How does the business operate? Where are the potential weaknesses in operation?
• Target's people: What type of people work there? How can they become your asset for the attack?
• Define targets: What are the lowest hanging fruit targets? What should be avoided?
• Target's network: How do the people and devices communicate on  the network?
• Target's defenses: What type of security is in place? Where is it located?
• Target's technologies: What technologies are used for e-mail, network traffic, storing information, authentication, and so on? Are they vulnerable?

Kali Linux contains an extensive catalog of tools titled Information Gathering specified for Reconnaissance efforts. It could fill a separate book to cover all tools and methods offered for Information Gathering. This chapter will focus on various web application Reconnaissance topics and relate the best tools found on the Internet as well as that offered by Kali Linux.

Initial research

Reconnaissance should begin with learning as much as possible about people and business associated with the target.

As a Penetration Tester, you need to know your target. If your target happens to be a website, you should look at all aspects of that website. It will give you a better understanding of how the site is maintained and run. Great Reconnaissance returns more possible vulnerabilities.

It is scary how much information is available on public sources. We have found the unimaginable, such as classified documents, passwords, vulnerability reports, undesirable photography, and access to security cameras. Many Penetration Testing project objectives start with leveraging information off public sources. Here are some starting points for gathering information from public sources.

Company website

There is a lot of valuable information that can be obtained from a target's website. Most corporate websites list their executive team, public figures, and members from recruiting and human resource contacts. These can become targets for other search efforts and social engineering attacks.

The Robots.txt file is publicly available and found on websites that gives instructions to web robots (also known as search engine spiders), about what is and not visible using the Robots Exclusion Protocol. The Disallow: / statement tells a browser not to visit a source; however, a Disallow can be ignored by giving a researcher intelligence on what a target hopes to not disclose to the public.

 

Web history sources

There are archived versions of most public websites available on sources such as the WayBack Machine at archive.org. Interesting information can be found in an older version of a target's website, such as outdated organizational charts, phone numbers, customer intelligence, systems information listed in fields, such as view source or /robots.txt, older business partnerships, vulnerabilities fixed in later versions, and other useful data, the target doesn't want on the current website version. It is important to understand that the publicly available information is hard to remove completely, making historical sources a valuable place for Reconnaissance research. To access the WayBack Machine, open up the web browser and navigate to  http://archive.org, you will see the Internet Archive WayBack Machine in the middle of the page

As a Penetration Tester, this is a valuable tool, because it doesn't leave evidence of Reconnaissance on your target. In fact, your target is never even touched using this tool. All the information has been archived online in the Wayback Machine.

Social media resources

Social media is everywhere, and in most cases, publicly accessible. Most people have a Facebook, LinkedIn, blogs, or other forms of cloud accounts containing valuable information. This information can be used as a means of social engineering intelligence from a target's current or previous staff. An example is searching Glassdoor.com to identify previous employees that are disgruntled, based on feedback.

There are many people finding web resources such as Maltego (found in Kali Linux) that can comb popular social media, public records, and job recruiting websites to fingerprint an individual based on limited information, such as a first and last name. A researcher could gather information such as everywhere an individual has lived, done business, people with which they socialize, special interests, favorite sport teams, and other useful data for future research and social engineering attacks.

Trust

Most people are naturally trusting and assume information posted on public sources is real. To test this concept, the writers of this book created a fake person through social media and pretended to be a new hire for a target company. The fake person would become friends with associates of our target, post fake holiday cards that are linked to a BeEF system designed to compromise vulnerable Internet browsers (using BeEF is covered later in this book), and captured sensitive information from compromised systems. We were able to map out the entire organization, obtain network information, and even had hardware shipped to us without an internal e-mail or phone number.

Our fake person, Emily Williams isn't real, yet received job offers, was provided inside information, and access to events hosted by the target. Information is power, and people will give it to a requester who seems like they can be trusted. More information on this project can be found at: http://www.thesecurityblogger.com/?p=1903

Job postings

Job postings contain a wealth of knowledge about a target's environment. Job listings can provide details on what type of systems are installed, who manages them, how large the staff is, and the staff's skill level. Human Resource representatives are typically eager to share information with a potential new hire, which can be used as an avenue to inside information. An example is targeting a job posting for a Oracle developer to understand the hardware, version of Oracle, names of existing and previous administrators, existing operation issues, security gaps, and methods to access such as asking "can administrators work from home, and how do they access the systems?"

Glassdoor.com is an example of a popular source for this type of data.

Location

The investment in cyber security for a target can typically be determined based on the level of physical security. One would assume a building with fences and armed guards would have a higher investment in cyber security than a target located within a public building. Online mapping sources such as Google maps can help identify where physical security is implemented, and trends on how people move to and from the target. Other areas of interest are identifying where a Penetration Tester could camp out to scan for wireless networks, and possible methods to bypass access controls, such as attire and badges used to obtain physical access.

Shodan

Shodan is a search engine that can identify a specific device, such as computer, router, server, using a variety of filters, such as metadata from system banners. For example, you can search for a specific system, such as a Cisco 3850, running a version of software such as IOS Version 15.0(1)EX.

http://www.shodanhq.com/

Google hacking

Google hacking is the most common form of search engine Reconnaissance of web applications. Google hacking uses advanced operations in the Google search engine to locate specific strings of text within search results. Search filters can zero in on specific versions of vulnerable web applications such as Powered by Apache in the intitle:"index of" operator or identify log files such as ws_ftp.log, containing sensitive IP information. The following few screenshots demonstrate using a Google search for Linksys to find publicly available Linksys cameras. The first screenshot shows the search command followed by some example results from issuing the search.

The last screenshot shows a camera feed that could be found using this technique.

 

Some example search queries are as follows:

• Identifies sensitive documents: intext: classified top secret
• Identifies Linksys Camera Management GUIs (caution: you may not like what you find): inurl:main.cgi
• Identifies Nessus reports to find vulnerable systems: inurl:NESSUSXXXXXXXX

For more information on Google hacking, check out a very good book titled Google Hacking for Penetration Testers by Johnny Long, as well as his website at http://johnny.ihackstuff.com.

Google Hacking Database

The Google Hacking Database (GHDB) created by Johnny Long of Hackers For Charity (http://www.hackersforcharity.org/), is the definitive source for Google search queries. Searches for usernames, passwords, vulnerable systems, and exploits have been captured and categorized by Google hacking aficionados. The aficionados who have categorized the Google searches are affectingly known as Google dorks. To access the GHDB, navigate to http://www.exploit-db.com/google-dorks/. You will see the latest GHDB searches listed on the web page. You can click on any of the search queries yourself.

Key difference between a Penetration Test and Vulnerability Assessment

Penetration Testing is the method of attacking system vulnerabilities in a similar way to real malicious attackers.

Typically, Penetration Testing services are requested when a system or network has exhausted investments in security and clients are seeking to verify if all avenues of security have been covered. Penetration Testing can be Black, White, or Gray box depending on the scope of work agreed upon.

The key difference between a Penetration Test and Vulnerability Assessment is that a Penetration Test will act upon vulnerabilities found and verify if they are real reducing the list of confirmed risk associated with a target.

A Vulnerability Assessment of a target could change to a Penetration Test once the asset owner has authorized the service provider to execute attacks against the vulnerabilities identified in a target.

Typically, Penetration Testing services have a higher cost associated since the services require more expensive resources, tools, and time to successfully complete assignments. One popular misconception is that a Penetration Testing service enhances IT security since services have a higher cost associated than other security services:

• Penetration Testing does not make IT networks more secure, since services evaluate existing security! A customer should not consider a Penetration Test if there is a belief the target is not completely secure.

• Penetration Testing can cause a negative impact to systems: It's critical to have authorization in writing from the proper authorities before starting a Penetration Test of an asset owned by another party. Not having proper authorization could be seen as illegal hacking by authorities. Authorization should include who is liable for any damages caused during a penetration exercise as well as who should be contacted to avoid future negative impacts once a system is damaged. Best practice is alerting the customers of all the potential risks associated with each method used to compromise a target prior to executing the attack to level set expectations. This is also one of the reasons we recommend targeted Penetration Testing with a small scope. It is easier to be much more methodical in your approach. As a common best practice, we receive confirmation, which is a worst case scenario, that a system can be restored by a customer using backups or some other disaster recovery method.

Penetration Testing deliverable expectations should be well defined while agreeing on a scope of work. The most common methods by which hackers obtain information about targets is through social engineering via attacking people rather than systems. Examples are interviewing for a position within the organization and walking out a week later with sensitive data offered without resistance. This type of deliverable may not be acceptable if a customer is interested in knowing how vulnerable their web applications are to remote attack. It is also important to have a defined end-goal so that all parties understand when the penetration services are considered concluded.

Usually, an agreed-upon deliverable serves this purpose.

A Penetration Testing engagement's success for a service provider is based on profitability of time and services used to deliver the Penetration Testing engagement. A more efficient and accurate process means better results for less services used. The higher the quality of the deliverables, the closer the service can meet customer expectation, resulting in a better reputation and more future business. For these reasons, it's important to develop a methodology for executing Penetration Testing services as well as for how to report what is found.

Calculating risk

It is important to understand how to calculate risk associated with vulnerabilities found, so that a decision can be made on how to react. Most customers look to the CISSP triangle of CIA when determining the impact of risk.

CIA is the confidentiality, integrity, and availability of a particular system or application.

When determining the impact of risk, customers must look at each component individually as well as the vulnerability in its entirety to gain a true perspective of the risk and determine the likelihood of impact.

It is up to the customer to decide if the risk associated to vulnerability found justifies or outweighs the cost of controls required to reduce the risk to an acceptable level. A customer may not be able to spend a million dollars on remediating a threat that compromises guest printers; however, they will be very willing to spend twice as much on protecting systems with the company's confidential data.

The Certified Information Systems Security Professional (CISSP) curriculum lists formulas for calculating risk as follow.

A Single Loss Expectancy (SLE) is the cost of a single loss to an Asset Value (AV).

Exposure Factor (EF) is the impact the loss of the asset will have to an organization such as loss of revenue due to an Internet-facing server shutting down. Customers  should calculate the SLE of an asset when evaluating security investments to help identify the level of funding that should be assigned for controls. If a SLE would cause a million dollars of damage to the company, it would make sense to consider that in the budget.

The Single Loss Expectancy formula:

SLE = AV * EF

The next important formula is identifying how often the SLE could occur. If an SLE worth a million dollars could happen once in a million years, such as a meteor falling out of the sky, it may not be worth investing millions in a protection dome around your headquarters. In contrast, if a fire could cause a million dollars worth of damage and is expected every couple of years, it would be wise to invest in a fire prevention system. The number of times an asset is lost is called the Annual Rate of Occurrence (ARO).

The Annualized Loss Expectancy (ALE) is an expression of annual anticipated loss due to risk. For example, a meteor falling has a very low annualized expectancy (once in a million years), while a fire is a lot more likely and should be calculated in future investments for protecting a building.

Annualized Loss Expectancy formula:

ALE = SLE * ARO

The final and important question to answer is the risk associated with an asset used to figure out the investment for controls. This can determine if and how much the customer should invest into remediating vulnerability found in a asset.

Risk formula:

Risk = Asset Value * Threat * Vulnerability * Impact

It is common for customers not to have values for variables in Risk Management formulas. These formulas serve as guidance systems, to help the customer better understand how they should invest in security. In my previous examples, using the formulas with estimated values for a meteor shower and fire in a building, should help explain with estimated dollar value why a fire prevention system is a better investment than metal dome protecting from falling objects.

Penetration Testing methodology

There are logical steps recommended for performing a Penetration Test. The first step is identifying the project's starting status. The most common terminology defining the starting state is Black box testing, White box testing, or a blend between White and Black box testing known as Gray box testing.

Black box assumes the Penetration Tester has no prior knowledge of the target network, company processes, or services it provides. Starting a Black box project requires a lot of reconnaissance and, typically, is a longer engagement based on the concept that real-world attackers can spend long durations of time studying targets before launching attacks.

As a security professional, we find Black box testing presents some problems when scoping a Penetration Test. Depending on the system and your familiarity with the environment, it can be difficult to estimate how long the reconnaissance phase will last. This usually presents a billing problem. Customers, in most cases, are not willing to write a blank cheque for you to spend unlimited time and resources on the reconnaissance phase; however, if you do not spend the time needed then your Penetration Test is over before it began. It is also unrealistic because a motivated attacker will not necessarily have the same scoping and billing restrictions as a professional Penetration Tester. That is why we recommend Gray box over Black box testing.

White box is when a Penetration Tester has intimate knowledge about the system. The goals of the Penetration Test are clearly defined and the outcome of the report from the test is usually expected. The tester has been provided with details on the target such as network information, type of systems, company processes, and services. White box testing typically is focused on a particular business objective such as meeting a compliance need, rather than generic assessment, and could be a shorter engagement depending on how the target space is limited. White box assignments could reduce information gathering efforts, such as reconnaissance services, equaling less cost for Penetration Testing services.

Gray box testing falls in between Black and White box testing. It is when the client or system owner agrees that some unknown information will eventually be discovered during a Reconnaissance phase, but allows the Penetration Tester to skip this part. The Penetration Tester is provided some basic details of the target; however, internal workings and some other privileged information is still kept from the Penetration Tester.

Real attackers tend to have some information about a target prior to engaging the target. Most attackers do not choose random targets. They are motivated and have usually interacted in some way with their target before attempting an attack. Gray box is an attractive choice approach for many security professionals conducting Penetration Tests because it mimics real-world approaches used by attackers and focuses on vulnerabilities rather than reconnaissance.

Kicking off a Penetration Testing service engagement should include an information gathering session used to document the target environment and define the boundaries of the assignment to avoid unnecessary reconnaissance services or attacking systems that are out of scope. A well-defined scope of work will save a service provider from scope creep (defined as uncontrolled changes or continuous growth in a project's scope), operate within the expected timeframe and help provide more accurate deliverable upon concluding services.

Real attackers do not have boundaries such as time, funding, ethics, or tools meaning that limiting a Penetration Testing scope may not represent a real-world scenario.

In contrast to a limited scope, having an unlimited scope may never evaluate critical vulnerabilities if a Penetration Test is concluded prior to attacking desired systems. For example, a Penetration Tester may capture user credentials to critical systems and conclude with accessing those systems without testing how vulnerable those systems are to network-based attacks. It's also important to include who is aware of the Penetration Test as a part of the scope. Real attackers may strike at anytime and probably when people are least expecting it.

Some fundamentals for developing a scope of work for a Penetration Test are as follows:

• Definition of Target System(s)
• Timeframe of Work Performed
• How Targets Are Evaluated
• Tools and software
• Notified Parties
• Initial Access Level
• Definition of Target Space
• Identification of Critical Operation Areas
• Definition of the Flag
• Deliverable
• Remediation expectations

Web application Penetration Testing concepts

A web application is any application that uses a web browser as a client. This can be a simple message board or a very complex spreadsheet. Web applications are popular based on ease of access to services and centralized management of a system used by multiple parties. Requirements for accessing a web application can follow industry web browser client standards simplifying expectations from both the service providers as well as the hosts accessing the application.

Web applications are the most widely used type of applications within any organization. They are the standard for most Internet-based applications. If you look at smartphones and tablets, you will find that most applications on these devices are also web applications. This has created a new and large target-rich surface for security professionals as well as attackers exploiting those systems.

Penetration Testing web applications can vary in scope since there is a vast number of system types and business use cases for web application services. The core web application tiers which are hosting servers, accessing devices, and data depository should be tested along with communication between the tiers during a web application Penetration Testing exercise.

An example for developing a scope for a web application Penetration Test is testing a Linux server hosting applications for mobile devices.

The scope of work at a minimum should include:        

Evaluating the Linux server include:

operating system

• network configuration
• applications hosted from the server
• how systems and users authenticate
• client devices accessing the server and communication between all three tiers
• Additional areas of evaluation that could be included in the scope of work are how devices are obtained by employees
• how devices are used outside of accessing the application, the surrounding network(s)
• maintenance of the systems, 
• and the users of the systems

Penetration Testing and Setup

Difference between security audit, network or risk assessment, and Penetration Test

Many organizations offer security services and use terms such as security audit, network or risk assessment, and Penetration Test with overlapping meanings.

By definition, an audit is a measurable technical assessment of a system(s) or application(s).

Security assessments are evaluations of risk, meaning services used to identify vulnerabilities in systems, applications, and processes.

Penetration Testing goes beyond an assessment by evaluating identified vulnerabilities to verify if the vulnerability is real or a false positive. For example, an audit or an assessment may utilize scanning tools that provide a few hundred possible vulnerabilities on multiple systems. A Penetration Test would attempt to attack those vulnerabilities in the same manner as a malicious hacker to verify which vulnerabilities are genuine reducing the real list of system vulnerabilities to a handful of security weaknesses.

The most effective Penetration Tests are the ones that target a very specific system with a very specific goal. Quality over quantity is the true test of a successful Penetration Test. Enumerating a single system during a targeted attack reveals more about system security and response time to handle incidents than wide spectrum attack. By carefully choosing valuable targets, a Penetration Tester can determine the entire security infrastructure and associated risk for a valuable asset.

Penetration Testing evaluates the effectiveness of existing security. If a customer does not have strong security then they will receive little value from Penetration Testing services. As a consultant, it is recommended that Penetration Testing services are offered as a means to verify security for existing systems once a customer believes they have exhausted all efforts to secure those systems and are ready to evaluate if there are any existing gaps in securing those systems.