.Google.com's John Mueller addressed a concern concerning why Google.com indexes pages that are actually disallowed coming from crawling by robots.txt as well as why the it's safe to ignore the related Explore Console documents regarding those crawls.Bot Website Traffic To Inquiry Criterion URLs.The person talking to the inquiry chronicled that robots were producing hyperlinks to non-existent inquiry criterion Links (? q= xyz) to web pages along with noindex meta tags that are additionally blocked out in robots.txt. What urged the concern is actually that Google.com is actually crawling the hyperlinks to those web pages, acquiring shut out through robots.txt (without seeing a noindex robotics meta tag) at that point getting shown up in Google.com Search Console as "Indexed, though shut out by robots.txt.".The person inquired the complying with concern:." Yet here's the huge question: why would Google index webpages when they can not even view the web content? What's the perk during that?".Google.com's John Mueller confirmed that if they can not creep the page they can not view the noindex meta tag. He also makes a fascinating reference of the website: hunt operator, encouraging to overlook the results given that the "ordinary" individuals will not view those results.He wrote:." Yes, you are actually correct: if our team can't crawl the web page, we can't see the noindex. That mentioned, if our team can not crawl the web pages, then there's not a whole lot for our company to mark. Therefore while you could view some of those webpages with a targeted web site:- inquiry, the normal consumer will not find all of them, so I would not fuss over it. Noindex is actually also fine (without robots.txt disallow), it simply means the Links will find yourself being actually crept (as well as find yourself in the Browse Console file for crawled/not indexed-- neither of these standings induce issues to the rest of the internet site). The essential part is that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limitations in using the Internet site: search evolved hunt driver for diagnostic explanations. One of those reasons is since it is actually certainly not attached to the regular search index, it's a different point completely.Google.com's John Mueller discussed the website hunt operator in 2021:." The quick solution is actually that an internet site: question is certainly not implied to become total, nor used for diagnostics objectives.A site inquiry is a particular type of hunt that restricts the results to a particular web site. It is actually basically only words internet site, a bowel, and after that the site's domain.This question limits the end results to a particular website. It's certainly not suggested to be a complete assortment of all the web pages coming from that internet site.".2. Noindex tag without utilizing a robots.txt is alright for these kinds of situations where a crawler is linking to non-existent webpages that are actually getting uncovered by Googlebot.3. URLs with the noindex tag will definitely generate a "crawled/not listed" item in Explore Console and that those won't possess a damaging result on the remainder of the web site.Review the question and also address on LinkedIn:.Why would Google.com index pages when they can't also view the content?Featured Photo by Shutterstock/Krakenimages. com.