Plukkie web crawler/robot/spider
Indexing worldwide for new to release worldwide search engine (botje.com).
Indexing The Netherlands, Belgium, France, United Kingdom to update Dutch (botje.nl)/Belgium (botje.be)/France (botje.fr)/United Kingdom (botje.co.uk) search engine.
If you don't want search engine users can search some or all pages then create or add to a robots.txt file placed in the root web directory from your website a rule for example disallow all pages:
It's possible to set manually a minimum crawl delay between two visits to the same IP address.
When no value is set, a default value of 5 is used, how higher this value, how more delay between each visit.
Don't put this value to high otherwise it takes very long before all web pages are indexed and updated.
It can take some days before crawler plukkie take over the changes in the robots.txt file.
If you think crawler plukkie has a bug in filtering or visiting URLs or reading robots.txt files please email info (at) ykoon (dot) nl so we can investigate the reported bug and find a solution.
Thanks the crawler team.
Last updated June 2012