Robots TXT and Google Search Console - How-to and Troubleshooting - wpForo 2.0

RE: Robots TXT and Google Search Console

Macky — Wed, 17 Mar 2021 15:00:37 +0000

@dimalifragis More of an expert than me mate lol, thanks for your help

RE: Robots TXT and Google Search Console

dimalifragis — Wed, 17 Mar 2021 13:11:41 +0000

@percysgrowroom Seems right, still i'm no expert in all that.

RE: Robots TXT and Google Search Console

Macky — Wed, 17 Mar 2021 13:06:13 +0000

This is how mine look:

User-Agent: *
Allow: /wp-admin/admin-ajax.php
Allow: /wp-content/uploads/
Disallow: /wp-content/plugins/
Disallow: /wp-admin/
Disallow: /readme.html
Disallow: /refer/
Disallow: /forum/profile/

Sitemap: https://percysgrowroom.com/forum/sitemap.xml
Sitemap: https://percysgrowroom.com/sitemap_index.xml

I added the profile bit last night, and just added that line told me too, is this ok do you think?

Thanks for the input guys, I really appreciate the help

RE: Robots TXT and Google Search Console

Macky — Wed, 17 Mar 2021 13:04:44 +0000

@dimalifragis Thanks bro, ill add that :)

RE: Robots TXT and Google Search Console

dimalifragis — Wed, 17 Mar 2021 12:29:26 +0000

Posted by: @alexas

I have a standard robot, I tried everything and there was even a robot 25 lines each. The load is minimal. Average attendance of 400 people per day. You should have caching enabled in the forum settings and no plugins are needed except for kaptha bank. I hope I didn't hurt you!
User-agent: *
Disallow: /wp-admin/
Disallow: /r/
Disallow: /wp-json/

wp-json is already excluded i think.

$this->send_header( 'X-Robots-Tag', 'noindex' );

Also this should be WP robots.txt

Allow: /wp-admin/admin-ajax.php

RE: Robots TXT and Google Search Console

alexas — Wed, 17 Mar 2021 09:56:43 +0000

I have a standard robot, I tried everything and there was even a robot 25 lines each. The load is minimal. Average attendance of 400 people per day. You should have caching enabled in the forum settings and no plugins are needed except for kaptha bank. I hope I didn't hurt you!

User-agent: *
Disallow: /wp-admin/
Disallow: /r/
Disallow: /wp-json/

Robots TXT and Google Search Console

Macky — Tue, 16 Mar 2021 23:28:32 +0000

I need to reduce the amount of pages google crawls when it comes to my site.

I would like to restrict the crawl to the sections with most information, to make sure google crawl the best stuff.

I can do this in the tool section of the forum, or through robots txt, but would this show as an error on my google search console? Or would it not show at all?

Also, does anyone have an example of what robots txt they use? Any advice on this would be massively appreciated.