Dynamic Robots.txt in ASP MVC

I recently had the need to switch a robots.txt file dynamically, depending on whether a site was deployed to live or beta servers. The solution was very simple, ditch the real file and replace it with a controller action. This is a brief note on how I did this. I don’t pretend this is an ideal solution, it was something I knocked together before I went out for dinner.

The solution I went for was in three parts: a web.config transform, routing and the controller itself.

The web config transform for my live site looks like this:

    
          <?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="SiteStatus" value="live" xdt:Transform="SetAttributes" xdt:Locator="Match(key)"/>
    </appSettings>
</configuration>

view raw Web.live.config This Gist brought to you by GitHub.

The route is equally simple:

    
          routes.MapRoute(
    "Robots.txt",
    "Robots.txt",
    new 
    { 
        controller = "Robots", 
        action = "RobotsText" 
    }
);

          view raw
          routing.cs
          This Gist brought to you by GitHub.

And finally the actual controller:

    
      
        
        
          public class RobotsController : Controller
{
    public FileContentResult RobotsText()
    {
        var content = "User-agent: *" + Environment.NewLine;
 
        if (string.Equals(ConfigurationManager.AppSettings["SiteStatus"], "live", StringComparison.InvariantCultureIgnoreCase))
        {
            content += "Disallow: /elmah.axd" + Environment.NewLine;
            content += "Sitemap: http://www.jacquelinewhite.co.uk/sitemap.xml";
        }
        else
        {
            content += "Disallow: /" + Environment.NewLine; 
        }
 
        return File(
                Encoding.UTF8.GetBytes(content),
                "text/plain"); 
    }
}

        
      

  

          view raw
          RobotsController.cs
          This Gist brought to you by GitHub.
        

And there we have it, my robots.txt file now is customised on a per deployment basis.

2 comments Submit a comment

Marie Sligh commented on 2012/03/02 at 16:21


Thank you! Just what I was looking for!
david kavanagh commented on 2013/03/07 at 11:16


Sweet! .. what about an iis redirect rule ? this might work too ..maybe more easily ?

Ie conditionally serve a robots.txt file depending on the domain of the request.

The Old Sewing Factory

Leave a Reply Cancel reply

Recent Posts

Categories