Notifications
Clear all

wpForo 1.x.x [Closed] Robots TXT and Google Search Console

7 Posts
3 Users
5 Reactions
894 Views
Posts: 1159
Topic starter
(@percysgrowroom)
Noble Member
Joined: 5 years ago

I need to reduce the amount of pages google crawls when it comes to my site.Β 

I would like to restrict the crawl to the sections with most information, to make sure google crawl the best stuff.Β 

I can do this in the tool section of the forum, or through robots txt, but would this show as an error on my google search console? Or would it not show at all?Β 

Also, does anyone have an example of what robots txt they use? Any advice on this would be massively appreciated.

6 Replies
alexas
Posts: 34
(@alexas)
Eminent Member
Joined: 7 years ago

I have a standard robot, I tried everything and there was even a robot 25 lines each. The load is minimal. Average attendance of 400 people per day. You should have caching enabled in the forum settings and no plugins are needed except for kaptha bank. I hope I didn't hurt you!

User-agent: *
Disallow: /wp-admin/
Disallow: /r/
Disallow: /wp-json/
2 Replies
dimalifragis
(@dimalifragis)
Joined: 4 years ago

Famed Member
Posts: 2612
Posted by: @alexas

I have a standard robot, I tried everything and there was even a robot 25 lines each. The load is minimal. Average attendance of 400 people per day. You should have caching enabled in the forum settings and no plugins are needed except for kaptha bank. I hope I didn't hurt you!

User-agent: *
Disallow: /wp-admin/
Disallow: /r/
Disallow: /wp-json/

wp-json is already excluded i think.

$this->send_header(Β 'X-Robots-Tag',Β 'noindex'Β );

Β Also this should be WP robots.txt

Allow: /wp-admin/admin-ajax.php

(@percysgrowroom)
Joined: 5 years ago

Noble Member
Posts: 1159

@dimalifragis Thanks bro, ill add that πŸ™‚Β 

Β 

Posts: 1159
Topic starter
(@percysgrowroom)
Noble Member
Joined: 5 years ago

This is how mine look:

User-Agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
Disallow: /forum/profile/

Sitemap: https://percysgrowroom.com/forum/sitemap.xml
Sitemap: https://percysgrowroom.com/sitemap_index.xml

I added the profile bit last night, and just added that line told me too, is this ok do you think?

Thanks for the input guys, I really appreciate the helpΒ 

dimalifragis
Posts: 2612
(@dimalifragis)
Famed Member
Joined: 4 years ago

@percysgrowroom Seems right, still i'm no expert in all that.

1 Reply
(@percysgrowroom)
Joined: 5 years ago

Noble Member
Posts: 1159

@dimalifragis More of an expert than me mate lol, thanks for your help