expire() and setex() store as a countdown from now and doesn't persist it between server reboots and repopulation.
This is fixed by setting the keys to expire at a time in the future
Since I will be using Redis from multiple files, the connection to the server should be in the same file as the other DB connection.
This icon looks a little bit better I think.
This is purely asthetic. I may implement a robots.txt parser for this in the future if it doesn't spend
too much time parsing through to follow robots.txt rules. This also isn't technically a crawler
as it only checks if the remote server is online
