Change the Virtual Robots.txt File

Did you know that WordPress will create a robots.txt file for you – well, sorta…

If you don’t create an actual robots.txt file, WordPress will create a virtual one for you, meaning that if yoursite.com/robots.txt is requested, WordPress will serve up robots.txt page, even though it doesn’t actually exist on your server.

Now that we’ve got that covered. How can you edit your virtual robots file? Easy

add_filter( 'robots_txt', 'robots_mod', 10, 2 );
function robots_mod( $output, $public ) {
	$output .= "Disallow: /topsecret/\n";
	return $output;
}

10 thoughts on “Change the Virtual Robots.txt File”

  1. ypx

    You need to append n at the end of the Disallow line, otherwise the next plugin changing robots.txt will append it’s output right to your line.

  2. adam

    please do u think that i should create a real robot.txt file when i use the wordpress xml sitemap plugin? caz i`m using this plugin and when i go to the webmaster crawler i find that one url is blocked, so what do u think?

  3. Dave Steer

    Still not quite right. The newline command is “\n” Try this:

    add_filter( ‘robots_txt’, ‘robots_mod’, 10, 2 );
    function robots_mod( $output, $public ) {
    $output .= “Disallow: /topsecret/” . “\n”;
    return $output;
    }

    Nice tip though – thanks.

  4. Johnny

    “Now that we’ve got that covered. How can you edit your virtual robots file? Easy”

    It would be a better post if you’d actually say what file you’re adding this to.

    1. Kailey Lampert

      You can just ask. A lot of the posts on this site were imported when I closed down a separate snippets site I used to maintain.

      Any snippet on this site can either be added to your theme’s functions.php file (easiest, but not the best), or added to its own php file and uploaded to wp-content/mu-plugins/ (preferred).

  5. Yash Doshi

    Hi

    I just launched my website and saw that Google webmaster tools is giving me 332 warnings saying my URL’s are blocked by Robots.txt. There is no robots.txt file in my wordpress directory which means virtual robots.txt is used. Here is URL of my image where I have attached snapshot of Google webmaster tools.

    Few important pages of my website like the Homepage itself is not being indexed by Google. Please can someone suggest how do I enable the search engine to index my website including the Home page of the website. Right now only few of the pages of website are indexed by search engine.

    ANy help will be appreciated. Thank you in advance.

  6. Sehrish

    HI all,
    I have used seo Yoast in my site. It creates robot.txt by default. I want to remove all kinds of robot.txt. Because it is creating the problem to google crawl. Can you please tell me any solution.

Leave a Reply to Kailey Lampert Cancel reply

Your email address will not be published. Required fields are marked *