SharePoint 2019 Search: Crawl failed

When experiencing the error:

Item not crawled due to one of the following reasons: Preventive crawl rule; Specified content source hops/depth exceeded; URL has query string parameter; Required protocol handler not found; Preventive robots directive. ( This item was deleted because it was excluded by a crawl rule. )

Make sure to add a robots.txt to each WebApp’s Virtual Directory.

E.g. the default path to one one the WebApps:


robots.txt content:

User-agent: MS Search 6.0 Robot


No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *