Notifications
Clear all

wpForo 1.x.x Robots TXT and Google Search Console

7 Posts
3 Users
5 Likes
420 Views
Macky
Posts: 977
Topic starter
(@percysgrowroom)
Noble Member
Joined: 4 years ago

I need to reduce the amount of pages google crawls when it comes to my site.Β 

I would like to restrict the crawl to the sections with most information, to make sure google crawl the best stuff.Β 

I can do this in the tool section of the forum, or through robots txt, but would this show as an error on my google search console? Or would it not show at all?Β 

Also, does anyone have an example of what robots txt they use? Any advice on this would be massively appreciated.

6 Replies
alexas
Posts: 34
(@alexas)
Eminent Member
Joined: 5 years ago

I have a standard robot, I tried everything and there was even a robot 25 lines each. The load is minimal. Average attendance of 400 people per day. You should have caching enabled in the forum settings and no plugins are needed except for kaptha bank. I hope I didn't hurt you!

User-agent: *
Disallow: /wp-admin/
Disallow: /r/
Disallow: /wp-json/
Reply
2 Replies
dimalifragis
(@dimalifragis)
Joined: 3 years ago

Noble Member
Posts: 1828
Posted by: @alexas

I have a standard robot, I tried everything and there was even a robot 25 lines each. The load is minimal. Average attendance of 400 people per day. You should have caching enabled in the forum settings and no plugins are needed except for kaptha bank. I hope I didn't hurt you!

User-agent: *
Disallow: /wp-admin/
Disallow: /r/
Disallow: /wp-json/

wp-json is already excluded i think.

$this->send_header(Β 'X-Robots-Tag',Β 'noindex'Β );

Β Also this should be WP robots.txt

Allow: /wp-admin/admin-ajax.php

Reply
Macky
(@percysgrowroom)
Joined: 4 years ago

Noble Member
Posts: 977

@dimalifragis Thanks bro, ill add that πŸ™‚Β 

Β 

Reply
Macky
Posts: 977
Topic starter
(@percysgrowroom)
Noble Member
Joined: 4 years ago

This is how mine look:

User-Agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
Disallow: /forum/profile/

Sitemap: https://percysgrowroom.com/forum/sitemap.xml
Sitemap: https://percysgrowroom.com/sitemap_index.xml

I added the profile bit last night, and just added that line told me too, is this ok do you think?

Thanks for the input guys, I really appreciate the helpΒ 

Reply
dimalifragis
Posts: 1828
(@dimalifragis)
Noble Member
Joined: 3 years ago

@percysgrowroom Seems right, still i'm no expert in all that.

Reply
1 Reply
Macky
(@percysgrowroom)
Joined: 4 years ago

Noble Member
Posts: 977

@dimalifragis More of an expert than me mate lol, thanks for your help

Reply