Crawlers and spiders

Crawlers and spiders are used for mapping web applications, automating the task of cataloging all the content and functionality. The tool automatically crawls the application by following all the links it finds, submitting forms, analyzing the responses for new content, and repeating this process until it covers the whole application.

There are standalone crawlers and spiders such as Scrapy (http://scrapy.org), which are written in Python or command-line tools such as HT track (http://www.httrack.com). We have crawlers and spiders integrated with the proxies such as Burp and ZAP that will benefit from the content that has passed through the proxy to enrich knowledge about the app.

One good example on why this is valuable is when the application is heavy on JavaScript. Traditional crawlers won't interpret JS, but the browsers will. So, the proxy will see it and add it to the crawler catalog. We'll see Scrapy in more detail later.