This is a longstanding UX issue with unlisted URL’s. I’ve encountered this in my own products. Folks don’t know that sharing a link means it definitely will get indexed by Google.
Unfortunately this can’t be fixed via robots.txt — Google won’t proactively crawl the URL, but it does still index pages it finds from external links.
A `noindex` meta tag, however, does prevent pages from being indexed, so that feels like the appropriate solution here.
Not quite. People shared these links but there was no indication that the links were accessable without the url. Google docs has a similar system where you can make a link to your doc and only people with that link can see it. But in this case the convo was made publicly findable just by making the link
It's up to them to tell Google to not index those pages, either via <meta> or X-Robots-Tag http header. Once accidentally indexed they have to tell Google via search console to reindex them to actually get it out of the index.
Somehow this keeps happening, followed by a rediscovery of this ancient old knowledge.
While you're absolutely right, the pronoun "they" is unnecessarily load-bearing here. Most people who are sharing links have no control over the HTML code or HTTP headers, nor are they aware of the Google Search Console.
And you're talking about relieving a symptom on a well-behaved and well-documented public service. Getting indexed by Google indicates that a link has already been crawled, indexed, and scraped by a lot of services, most of which cannot be seen or controlled by ordinary mortals.
The operator of the links has to do this. Not the person sharing the link.
It's the job of the companies to make sure their pages don't get publicly indexed. It's basic web dev stuff that has worked like that for decades now.
"The operator of the links"? Not the person sharing the link? The pages of the companies?
What do you think I just explained?
Yes, Google Docs (and Sheets and Drive and Photos) can share via Google users (secure, granular) or "Public Link". The reason that you don't see results for these services in Google Search is because Google de-indexes them automatically if they are found by their crawler!
Everyone else crawling the web and social media? They can also find Public Links shared by Google users, or MS 365 users, or Dropbox or whatever.
And it's a feature, not a bug. Public links are public. When Claude or Gemini or Grok provide you a public link, it's public. The URL is like a "password" in that its UUID will work for anyone who clicks on it. That's simply the nature of public links.
If you wanna share securely then you get an Enterprise-class service, or you tell your LLM to generate a Word doc file, you stash it in your secure cloud storage, and you grant access to other authenticated users, and you don't ever create a public link.
This is a longstanding UX issue with unlisted URL’s. I’ve encountered this in my own products. Folks don’t know that sharing a link means it definitely will get indexed by Google.
Unfortunately this can’t be fixed via robots.txt — Google won’t proactively crawl the URL, but it does still index pages it finds from external links.
A `noindex` meta tag, however, does prevent pages from being indexed, so that feels like the appropriate solution here.
you can also do it via http response header, feels cleaner to me
Am I misunderstanding something, or isn’t this a non-issue, as people shared those links publicly for google to find them?
Not quite. People shared these links but there was no indication that the links were accessable without the url. Google docs has a similar system where you can make a link to your doc and only people with that link can see it. But in this case the convo was made publicly findable just by making the link
It's up to them to tell Google to not index those pages, either via <meta> or X-Robots-Tag http header. Once accidentally indexed they have to tell Google via search console to reindex them to actually get it out of the index.
Somehow this keeps happening, followed by a rediscovery of this ancient old knowledge.
While you're absolutely right, the pronoun "they" is unnecessarily load-bearing here. Most people who are sharing links have no control over the HTML code or HTTP headers, nor are they aware of the Google Search Console.
And you're talking about relieving a symptom on a well-behaved and well-documented public service. Getting indexed by Google indicates that a link has already been crawled, indexed, and scraped by a lot of services, most of which cannot be seen or controlled by ordinary mortals.
The operator of the links has to do this. Not the person sharing the link. It's the job of the companies to make sure their pages don't get publicly indexed. It's basic web dev stuff that has worked like that for decades now.
"The operator of the links"? Not the person sharing the link? The pages of the companies?
What do you think I just explained?
Yes, Google Docs (and Sheets and Drive and Photos) can share via Google users (secure, granular) or "Public Link". The reason that you don't see results for these services in Google Search is because Google de-indexes them automatically if they are found by their crawler!
Everyone else crawling the web and social media? They can also find Public Links shared by Google users, or MS 365 users, or Dropbox or whatever.
And it's a feature, not a bug. Public links are public. When Claude or Gemini or Grok provide you a public link, it's public. The URL is like a "password" in that its UUID will work for anyone who clicks on it. That's simply the nature of public links.
If you wanna share securely then you get an Enterprise-class service, or you tell your LLM to generate a Word doc file, you stash it in your secure cloud storage, and you grant access to other authenticated users, and you don't ever create a public link.
You can edit the headline to remove “may”:
site:claude.ai/public/artifacts