Earlier this week, Google removed its Robots.txt FAQ help document from its search developer documentation. When asked, John Mueller from Google replied to Alexis Rylko saying, "We update the documentation from time to time. Feel free to submit feedback if you feel something's missing. Robots.txt is definitely still a thing."
The Robots FAQ document lived over here: developers.google.com/search/docs/crawling-indexing/robots/robots-faq
That now redirects to the main Google robots.txt help page.
What did the Robots FAQ page say, well the Wayback Machine has a copy, so I will archive it here:
(Q) Does my website need a robots.txt file?
(A) No. When Googlebot visits a website, we first ask for permission to crawl by attempting to retrieve the robots.txt file. A website without a robots.txt file, robots meta tag, or X-Robots-Tag HTTP headers will generally be crawled and indexed normally.
(Q) Which method should I use to block crawlers?
(A) It depends. In short, there are good reasons to use each of these methods:
- robots.txt: Use it if crawling of your content is causing issues on your server. For example, you may want to disallow crawling of infinite calendar scripts. Don't use the robots.txt to block private content (use server-side authentication instead), or handle canonicalization. To make sure that a URL is not indexed, use the robots meta tag or X-Robots-Tag HTTP header instead.
- robots meta tag: Use it if you need to control how an individual HTML page is shown in...
Read Full Story: https://news.google.com/rss/articles/CBMiQmh0dHBzOi8vd3d3LnNlcm91bmR0YWJsZS5jb20vZ29vZ2xlLXJvYm90cy1mYXFzLXJlbW92ZWQtMzY0MzYuaHRtbNIBAA?oc=5
Your content is great. However, if any of the content contained herein violates any rights of yours, including those of copyright, please contact us immediately by e-mail at media[@]kissrpr.com.