mirror of
https://github.com/gamaio/lobli.git
synced 2025-12-13 07:38:27 +00:00
This is purely asthetic. I may implement a robots.txt parser for this in the future if it doesn't spend
too much time parsing through to follow robots.txt rules. This also isn't technically a crawler
as it only checks if the remote server is online
2.8 KiB
2.8 KiB