I have this situation in Google Search Console:
Under
Page indexing > Crawled - currently not indexed:
https://<my website URL>/forums/<private forum name>/<private topic name>/
This was last crawled 2 days ago.
- Normal users ("Registered") have a No access Forum Permission associated with that forum.
- Guests (Google should be a Guest) have a No access Forum Permission associated with that forum.
"Crawled" means that a Google bot was able to get to that page.
How is this possible?
I believe the forum is embedded onto the page. It's possible that the page was crawled, but none of the content of the forum was shown.
If you navigate to the link for a private forum in an incognito browser you'll see the experience.
I think the question is how Google knew about the link, perhaps there's an issue where private forums are still being added to the sitemap? The would seem like a bug IMO.
I think the question is how Google knew about the link, perhaps there's an issue where private forums are still being added to the sitemap? The would seem like a bug IMO.
Thanks Vanessa, that's exactly the question I have.
I have already tested the link with a private browsing session and, of course, the site asks you to log in to view it, which is the expected behaviour.
The problem is, Google shouldn't even have known what the URL of that topic was!
Google indexes something when it can view it.
Do some tests to see how the private topics are public. sitemap? Check it.
Do some work ......
@dimalifragis Thx I already checked the sitemaps. None of the private forums or topics are in there.
@fawp Then this is a mystery ....
Do you have a robots.txt file on the site? If do have excluded the forum/s in there?
Example:
User-agent: * Disallow: /community/name-of-private_forum1/ Disallow: /community/name-of-private_forum2/
Do you have a robots.txt file on the site? If do have excluded the forum/s in there?
Example:
User-agent: * Disallow: /community/name-of-private_forum1/ Disallow: /community/name-of-private_forum2/
Thanks Verek, I do have a robots.txt file but I don't have the private forum names in there.
Why should I?
@fawp Nope. Why put private (sensitive) urls in the public robots.txt ?
(my avatar AGAIN changed .... Why this happens ????)
Well, most respectable crawlers would obey the directives in robots.
You would in any event need to deploy stronger security against all the other riffraff...
@verek I agree that crawlers respect robots.txt but in the case of private forums this should not be needed.
After all, the content is not accessible if you are not logged in and have the right permissions!
Have you also set those private forums to "no index" in the wpForo Forums Settings dashboard?
Do you mean this one?
@tutrix thanks for that, but, let me ask you, why should I do that?
Google is not able to access the private forums because it does not have a "private" account. How is it able to read their titles?
EDIT: isn't this also going to put the URL of the private forums in robots.txt?
Settings there in no way affect what is in robots.txt
Sure those forums have restricted access settings but I think wpForo might be leaking the topics of those forums into the sitemap or (RSS) but search engines cannot index them 'cos they cannot read them. Perhaps by listing these forums into the no-index list prevents them from leaking.
I have always put my private forums into there and have not noticed any content from those forums in the Google console
Settings there in no way affect what is in robots.txt
Sure those forums have restricted access settings but I think wpForo might be leaking the topics of those forums into the sitemap (RSS) but search engines cannot index them 'cos they cannot read them. Perhaps by listing these forums into the no-index list prevents them from leaking.
I have always put my private forums into there and have not noticed any content from those forums in the Google console
Thanks for the explanation Verek.
Although in my case I don't see any of the private URLs in my sitemaps, I have updated that setting as you and Tutrix suggested, in case there is a bug somewhere.
Thanks again guys, I will keep an eye on this.