Notifications
Clear all

[Closed] Why is Google indexing private forum content?

17 Posts
5 Users
8 Likes
435 Views
Posts: 201
 fawp
Topic starter
(@fawp)
Reputable Member
Joined: 5 years ago

I have this situation in Google Search Console:

 

Under

Page indexing > Crawled - currently not indexed:

https://<my website URL>/forums/<private forum name>/<private topic name>/

 

This was last crawled 2 days ago.

 

  • Normal users ("Registered") have a No access Forum Permission associated with that forum.
  • Guests (Google should be a Guest) have a No access Forum Permission associated with that forum.

 

"Crawled" means that a Google bot was able to get to that page.

How is this possible?

16 Replies
Posts: 94
(@vanessa)
Estimable Member
Joined: 4 months ago

I believe the forum is embedded onto the page. It's possible that the page was crawled, but none of the content of the forum was shown.

If you navigate to the link for a private forum in an incognito browser you'll see the experience.

I think the question is how Google knew about the link, perhaps there's an issue where private forums are still being added to the sitemap? The would seem like a bug IMO.

Posts: 201
 fawp
Topic starter
(@fawp)
Reputable Member
Joined: 5 years ago

Posted by: @vanessa

I think the question is how Google knew about the link, perhaps there's an issue where private forums are still being added to the sitemap? The would seem like a bug IMO.

Thanks Vanessa, that's exactly the question I have.

I have already tested the link with a private browsing session and, of course, the site asks you to log in to view it, which is the expected behaviour.

 

The problem is, Google shouldn't even have known what the URL of that topic was!

dimalifragis
Posts: 2539
(@dimalifragis)
Famed Member
Joined: 4 years ago

Google indexes something when it can view it.

Do some tests to see how the private topics are public. sitemap? Check it.

Do some work ......

2 Replies
 fawp
(@fawp)
Joined: 5 years ago

Reputable Member
Posts: 201

@dimalifragis Thx I already checked the sitemaps. None of the private forums or topics are in there.

dimalifragis
(@dimalifragis)
Joined: 4 years ago

Famed Member
Posts: 2539

@fawp Then this is a mystery ....

VereK
Posts: 482
(@verek)
Honorable Member
Joined: 6 years ago

Do you have a robots.txt file on the site? If do have excluded the forum/s in there?

Example:

User-agent: *
Disallow: /community/name-of-private_forum1/
Disallow: /community/name-of-private_forum2/
10 Replies
 fawp
(@fawp)
Joined: 5 years ago

Reputable Member
Posts: 201

Posted by: @verek

Do you have a robots.txt file on the site? If do have excluded the forum/s in there?

Example:

User-agent: *
Disallow: /community/name-of-private_forum1/
Disallow: /community/name-of-private_forum2/

Thanks Verek, I do have a robots.txt file but I don't have the private forum names in there.

 

Why should I?

 

 

dimalifragis
(@dimalifragis)
Joined: 4 years ago

Famed Member
Posts: 2539

@fawp Nope. Why put private (sensitive) urls in the public robots.txt ?

 

(my avatar AGAIN changed .... Why this happens ????)

VereK
(@verek)
Joined: 6 years ago

Honorable Member
Posts: 482

@fawp 

Well, most respectable crawlers would obey the directives in robots.

You would in any event need to deploy stronger security against all the other riffraff...

 fawp
(@fawp)
Joined: 5 years ago

Reputable Member
Posts: 201

@verek I agree that crawlers respect robots.txt but in the case of private forums this should not be needed.

After all, the content is not accessible if you are not logged in and have the right permissions!

VereK
(@verek)
Joined: 6 years ago

Honorable Member
Posts: 482

@fawp 

Have you also set those private forums to "no index" in the wpForo Forums Settings dashboard? 

 fawp
(@fawp)
Joined: 5 years ago

Reputable Member
Posts: 201

Posted by: @verek

@fawp 

Have you also set those private forums to "no index" in the wpForo Forums Settings dashboard? 

Do you mean this one?

 

Tutrix
(@tutrix)
Joined: 4 years ago

Noble Member
Posts: 1186

@fawp 

Dashboard > wpForo > Settings > wpForo Seo

 fawp
(@fawp)
Joined: 5 years ago

Reputable Member
Posts: 201

@tutrix thanks for that, but, let me ask you, why should I do that?

Google is not able to access the private forums because it does not have a "private" account. How is it able to read their titles?

 

EDIT: isn't this also going to put the URL of the private forums in robots.txt?

VereK
(@verek)
Joined: 6 years ago

Honorable Member
Posts: 482

@fawp 

Settings there in no way affect what is in robots.txt

Sure those forums have restricted access settings but I think wpForo might be leaking the topics of those forums into the sitemap or (RSS) but search engines cannot index them 'cos they cannot read them. Perhaps by listing these forums into the no-index list prevents them from leaking.

I have always put my private forums into there and have not noticed any content from those forums in the Google console

 fawp
(@fawp)
Joined: 5 years ago

Reputable Member
Posts: 201

Posted by: @verek

@fawp 

Settings there in no way affect what is in robots.txt

Sure those forums have restricted access settings but I think wpForo might be leaking the topics of those forums into the sitemap (RSS) but search engines cannot index them 'cos they cannot read them. Perhaps by listing these forums into the no-index list prevents them from leaking.

I have always put my private forums into there and have not noticed any content from those forums in the Google console

Thanks for the explanation Verek.

 

Although in my case I don't see any of the private URLs in my sitemaps, I have updated that setting as you and Tutrix suggested, in case there is a bug somewhere.

 

Thanks again guys, I will keep an eye on this.