mirror of
https://github.com/gamaio/lobli.git
synced 2024-12-23 04:02:40 +00:00
4a66e9aa3b
This is purely asthetic. I may implement a robots.txt parser for this in the future if it doesn't spend too much time parsing through to follow robots.txt rules. This also isn't technically a crawler as it only checks if the remote server is online |
||
---|---|---|
.. | ||
Bootstrap | ||
CSS | ||
HTML | ||
img | ||
PHP |