.Google's John Mueller addressed an inquiry about why Google.com marks web pages that are disallowed from creeping by robots.txt and why the it is actually safe to dismiss the relevant Look Console records concerning those crawls.Bot Traffic To Question Specification URLs.The individual talking to the question recorded that crawlers were generating web links to non-existent concern specification Links (? q= xyz) to web pages with noindex meta tags that are actually additionally blocked in robots.txt. What caused the inquiry is that Google is creeping the web links to those webpages, receiving blocked by robots.txt (without seeing a noindex robots meta tag) then acquiring reported in Google Look Console as "Indexed, though shut out through robots.txt.".The individual asked the following inquiry:." But right here's the big question: why will Google mark web pages when they can not even see the material? What's the perk during that?".Google's John Mueller confirmed that if they can not crawl the web page they can not find the noindex meta tag. He likewise creates an interesting reference of the site: hunt driver, urging to neglect the outcomes since the "normal" consumers won't see those results.He composed:." Yes, you are actually right: if our team can not creep the page, our team can not observe the noindex. That stated, if our experts can't crawl the web pages, then there's not a lot for our team to mark. So while you could observe several of those pages along with a targeted website:- concern, the typical customer will not observe all of them, so I definitely would not fuss over it. Noindex is actually additionally fine (without robots.txt disallow), it only suggests the URLs will certainly wind up being crept (and find yourself in the Look Console document for crawled/not listed-- neither of these conditions induce problems to the remainder of the site). The fundamental part is actually that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the limitations being used the Website: hunt progressed search driver for diagnostic factors. Some of those factors is since it is actually not connected to the normal hunt index, it's a distinct thing altogether.Google.com's John Mueller talked about the internet site search driver in 2021:." The quick answer is actually that a web site: concern is actually certainly not indicated to be comprehensive, neither used for diagnostics objectives.A site inquiry is a specific sort of search that limits the results to a certain internet site. It is actually primarily only the word web site, a colon, and after that the site's domain.This question confines the results to a details internet site. It's not meant to be an extensive compilation of all the web pages coming from that website.".2. Noindex tag without utilizing a robots.txt is great for these type of circumstances where a crawler is connecting to non-existent web pages that are receiving found through Googlebot.3. Links with the noindex tag are going to generate a "crawled/not catalogued" entry in Search Console and that those will not possess a negative effect on the remainder of the site.Go through the concern as well as respond to on LinkedIn:.Why would Google.com mark pages when they can't even view the information?Featured Image by Shutterstock/Krakenimages. com.