robot exclusion downloads

RoboGen ROBOTS.TXT Editor

1.52 download

RoboGen is a visual editor for Robot Exclusion Files. It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which ...

Website Extractor

10.520 download

... your specific requirements, allowing for the inclusion or exclusion of various elements as per your needs (like ... downloaded. This makes the Extractor an efficient search robot, optimized for quick navigation across hyperlinks and downloading ...