Crawls external links as well.

Thank you for making your code available. I've tried two websites so far (one is mine). The first website I tried has links to books on Amazon. Those links are higher up in the page, so they get crawled first, and the script starts going on a marathon crawling every page on Amazon.com.

I haven't looked too deep in the class yet, but wondered if you had any suggestion.

Thank you!