This thing is just a big pain and isn't really needed, so I'm removing it.
Really the options for getting the title or even the description involved downloading the whole file.
Using a sorted set makes it easier to sort from highest to lowest on the stats page.
Minor change how tracking is done on the index and link checking has been updated.
expire() and setex() store as a countdown from now and doesn't persist it between server reboots and repopulation.
This is fixed by setting the keys to expire at a time in the future
This is purely asthetic. I may implement a robots.txt parser for this in the future if it doesn't spend
too much time parsing through to follow robots.txt rules. This also isn't technically a crawler
as it only checks if the remote server is online
