This is purely asthetic. I may implement a robots.txt parser for this in the future if it doesn't spend too much time parsing through to follow robots.txt rules. This also isn't technically a crawler as it only checks if the remote server is online
It doesn't quite work yet, but it should soon
The link tracking is very basic, but it doesn't need to be any more complicated than it already is
Mostly the same as http://lob.li/e76 without API stuff