Plukkie web crawler/robot/spider

Project:
Indexing worldwide for new to release worldwide search engine (botje.com).

Indexing The Netherlands, Belgium, France, United Kingdom to update Dutch (botje.nl)/Belgium (botje.be)/France (botje.fr)/United Kingdom (botje.co.uk) search engine.

If you don't want search engine users can search some or all pages then create or add to a robots.txt file placed in the root web directory from your website a rule for example disallow all pages:

User-agent: plukkie
Disallow: /

It's possible to set manually a minimum crawl delay between two visits to the same IP address.
When no value is set, a default value of 5 is used, how higher this value, how more delay between each visit.
Don't put this value to high otherwise it takes very long before all web pages are indexed and updated.

User-agent: plukkie
Crawl-delay: 10

It can take some days before crawler plukkie take over the changes in the robots.txt file.

If you think crawler plukkie has a bug in filtering or visiting URLs or reading robots.txt files please email info (at) ykoon (dot) nl so we can investigate the reported bug and find a solution.

Thanks the crawler team.

Last updated June 2012