Did you know that WordPress will create a robots.txt file for you – well, sorta…
If you don’t create an actual robots.txt file, WordPress will create a virtual one for you, meaning that if yoursite.com/robots.txt is requested, WordPress will serve up robots.txt page, even though it doesn’t actually exist on your server.
Now that we’ve got that covered. How can you edit your virtual robots file? Easy
add_filter( 'robots_txt', 'robots_mod', 10, 2 );
function robots_mod( $output, $public ) {
$output .= "Disallow: /topsecret/\n";
return $output;
}
You need to append n at the end of the Disallow line, otherwise the next plugin changing robots.txt will append it’s output right to your line.
Thanks!
Awesome! Exactly what I needed. I find it a bit irritating that this isn’t documented on wordpress.org. (If it is, I couldn’t find it.)
please do u think that i should create a real robot.txt file when i use the wordpress xml sitemap plugin? caz i`m using this plugin and when i go to the webmaster crawler i find that one url is blocked, so what do u think?
Still not quite right. The newline command is “\n” Try this:
add_filter( ‘robots_txt’, ‘robots_mod’, 10, 2 );
function robots_mod( $output, $public ) {
$output .= “Disallow: /topsecret/” . “\n”;
return $output;
}
Nice tip though – thanks.
“Now that we’ve got that covered. How can you edit your virtual robots file? Easy”
It would be a better post if you’d actually say what file you’re adding this to.
You can just ask. A lot of the posts on this site were imported when I closed down a separate snippets site I used to maintain.
Any snippet on this site can either be added to your theme’s
functions.phpfile (easiest, but not the best), or added to its own php file and uploaded towp-content/mu-plugins/(preferred).Hi
I just launched my website and saw that Google webmaster tools is giving me 332 warnings saying my URL’s are blocked by Robots.txt. There is no robots.txt file in my wordpress directory which means virtual robots.txt is used. Here is URL of my image where I have attached snapshot of Google webmaster tools.
Few important pages of my website like the Homepage itself is not being indexed by Google. Please can someone suggest how do I enable the search engine to index my website including the Home page of the website. Right now only few of the pages of website are indexed by search engine.
ANy help will be appreciated. Thank you in advance.
http://www-scf.usc.edu/~yashdosh/important.png this is the URL where snapshot of Google webmasters tool is.
HI all,
I have used seo Yoast in my site. It creates robot.txt by default. I want to remove all kinds of robot.txt. Because it is creating the problem to google crawl. Can you please tell me any solution.