Dynamic Robots.txt file for ASP.NET MVC Sites

Changing the contents of a robots.txt file when a site is moved from staging to a live environment is quite a manual and somewhat cumbersome process. I sometimes have the fear of forgetting to replace the "Disallow: /" line with correct indexable entries required for the website.

To give me one less thing to remember during my pre-live deployments, all my current and upcoming ASP.NET MVC sites will use a dynamic robots.txt file containing different entries depending on whether a site is in stage or live. In order to do this, we need to let the ASP.NET MVC application serve up the robots.txt file. This can be done by the following:

  • Create a new controller called "SEOController"
  • Add a new FileContentResult action called "Robots".
  • Add a new route.
  • Modify web.config.

SEOController

I have created a new controller that renders our "Robots" action as a plain text file. As you can see, my controller is not a type of "ActionResult" but a "FileContentResult". The great thing about "FileContentResult" is that it allows us to return bytes from the controller.

In this example, I am converting bytes from a string using Encoding.UTF8.GetBytes() method. Ideal for what we need to generate a robots.txt file.

[Route("robots.txt")]
[Route("Robots.txt")]
public FileContentResult Robots()
{
    StringBuilder robotsEntries = new StringBuilder();
    robotsEntries.AppendLine("User-agent: *");

    //If the website is in debug mode, then set the robots.txt file to not index the site.
    if (System.Web.HttpContext.Current.IsDebuggingEnabled)
    {
        robotsEntries.AppendLine("Disallow: /");
    }
    else
    {
        robotsEntries.AppendLine("Disallow: /Error");
        robotsEntries.AppendLine("Disallow: /resources");
        robotsEntries.AppendLine("Sitemap: http://www.surinderbhomra.com/sitemap.xml");
    }

    return File(Encoding.UTF8.GetBytes(robotsEntries.ToString()), "text/plain");
}

RouteConfig

Since I add my routing at controller level, I add the "MapMvcAttributeRoutes" method to the RouteConfig.cs file. But if you prefer to add your routes directly here, then this method can be removed.

public class RouteConfig
{
    public static void RegisterRoutes(RouteCollection routes)
    {
        routes.IgnoreRoute("{resource}.axd/{*pathInfo}");

        routes.MapMvcAttributeRoutes(); //Add this line!

        routes.MapRoute(
            name: "Default",
            url: "{controller}/{action}/{id}",
            defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
        );
    }
}

Web.config

Add a "runAllManagedModulesForAllRequests" attribute to the modules tag in the web.config file to allow our robot.txt text file to be rendered by ASP.NET.

<system.webserver>
  <modules runallmanagedmodulesforallrequests="true"></modules>
</system.webserver>

Caching Static Files Through Web.config

When running my website through Google Page Insights, one of things I didn't do was cache static content, such as CSS, JavaScript and site images. Since I am on a shared hosting plan, I didn't think it was possible to have the option to cache a specific directory without direct IIS access.

Normally, when working on client sites hosted on a dedicated server, I set the cache header within "HTTP Response Headers" area in IIS. But all this actually does is generate a web.config file within the directory you wish to cache:

<!--?xml version="1.0" encoding="UTF-8"?-->
<configuration>
    <system.webServer>
        <httpProtocol>
            <customHeaders>
                <add name="Cache-Control" value="public, max-age=604800" />
            </customHeaders>
        </httpProtocol>
    </system.webServer>
</configuration>

So if you too are on shared hosting, add a web.config file with similar settings. In this case, I have cached my files for a week.

You can also set the cache settings in your main web.config file by wrapping a location path around the <system.webServer> node:

<location path="resources">
  <system.webServer>
        <httpProtocol>
            <customHeaders>
                <add name="Cache-Control" value="public, max-age=604800" />
            </customHeaders>
        </httpProtocol>
    </system.webServer>
 </location>

Render Partial View As A String

One of the many nice things of using ASP.NET MVC Razor is that you have full control over how you segregate your HTML markup when building a page through rendering PartialViews. Since becoming an avid MVC developer, I am increasingly noticing how easy it is to make nice neat reusable code, whether it is used server or client-side.

Just today, I found something really useful that is a truly defines this, where markup within PartialViews can be output to a page as string:

/// <summary>
/// Controller extension class that adds controller methods
/// to render a partial view and return the result as string.
///
/// Based on http://craftycodeblog.com/2010/05/15/asp-net-mvc-render-partial-view-to-string/
/// </summary>
public static class ControllerExtension
{
 
  /// <summary>
  /// Renders a (partial) view to string.
  /// </summary>
  /// <param name="controller">Controller to extend</param>
  /// <param name="viewName">(Partial) view to render</param>
  /// <returns>Rendered (partial) view as string</returns>
  public static string RenderPartialViewToString(this Controller controller, string viewName)
  {
    return controller.RenderPartialViewToString(viewName, null);
  }
 
  /// <summary>
  /// Renders a (partial) view to string.
  /// </summary>
  /// <param name="controller">Controller to extend</param>
  /// <param name="viewName">(Partial) view to render</param>
  /// <param name="model">Model</param>
  /// <returns>Rendered (partial) view as string</returns>
  public static string RenderPartialViewToString(this Controller controller, string viewName, object model)
  {
    if (string.IsNullOrEmpty(viewName))
      viewName = controller.ControllerContext.RouteData.GetRequiredString("action");
 
      controller.ViewData.Model = model;
 
      using (var sw = new StringWriter())
      {
        var viewResult = ViewEngines.Engines.FindPartialView(controller.ControllerContext, viewName);
        var viewContext = new ViewContext(controller.ControllerContext, viewResult.View, controller.ViewData, controller.TempData, sw);
        viewResult.View.Render(viewContext, sw);
 
        return sw.GetStringBuilder().ToString();
      }
    } 
}

I can't take credit for this code. But here is the guy who can: Jan Jonas.

Being able to output PartialViews as a string is actually quite handy, since you could have a paginated news listings page that displays the first page of articles server-side and any additional pages could be loaded in via jQuery Ajax. Each article item would be a PartialView so you could serve the same markup client-side. My code below probably explains things a little better:

Article Listing View

This page will list all my News Articles. As you can see, I am using an "ArticleListItem" as my PartialView.

@model List<Article>

@if (Model.Any())
{
    <div class="article-list">
    @foreach (var a in Model.Select((value, index) => new { value, index }))
    {
        Html.RenderPartial("/Views/Article/_ArticleListItem.cshtml", new ArticleListItemView { Article = a.value, CssClass = ArticleHtmlHelper.GetItemCssClass((a.index + 1)), IsFullWidth = false});
    }
    </div>
}
else
{
    <div>
        No articles could be returned.
    </div>
}

Article List Item PartialView

My PartialView has quite a bit going on to determine how the markup should be rendered and it's definitely something I wouldn't want to have to duplicate elsewhere just to load in client-side. Nice!

@model Site.Web.Models.Views.ArticleListItemView
@{
    string fullWidthClass = String.Empty;

    if (Model.IsFullWidth)
    {
        fullWidthClass = "full-width";
    }
}
<div class="article-summary @Model.CssClass @fullWidthClass">
    <a href="@Model.Article.PageUrl" class="img">
        @if (Model.CssClass == "large")
        {
        <img src="@Model.Article.Images.ImageCollection[1].Url" />
        }
        else
        {
        <img src="@Model.Article.Images.ImageCollection[0].Url" />
        }
    </a>
    @if (Model.Article.Category != null)
    {
    <span class="cat">@Model.Article.Category.Name</span>
    }
    @if (Model.Article.ReadTime != null)
    {
    <span class="time">@String.Format("{0} read", Model.Article.ReadTime)</span>
    }
    <h2 class="@Model.CssClass"><a href="@Model.Article.PageUrl">@Model.Article.Title</a></h2>
    @if (Model.Article.Author != null)
    {
    <a href="@Model.Article.Author.PageUrl.Url" class="author">
        <img src="@Model.Article.Author.Images.ImageCollection[0].Url" />
        <span>@String.Concat(Model.Article.Author.FirstName, " ", Model.Article.Author.LastName)</span>
    </a>
    }
</div>

GetArticleItems() Controller

This is where the RenderPartialViewToString() method shines! This controller is called within my jQuery Ajax function to get the next page of news articles. I am then calling my "ArticleListItem" PartialView to return the HTML markup as a string through my client-side call.

[HttpPost]
public JsonResult GetArticleItems(DBContext ctx, int pageNo, int pageSize, string categoryId)
{
    ApiDocumentInfo docInfo = DocumentHelper.SearchDocuments(ctx, true, "article", "category", categoryId, pageSize, pageNo, "articles", "date desc");

    List<Article> articles = docInfo.Documents.Select(doc => doc.ToArticle(ctx)).ToList();

    StringBuilder articleHtml = new StringBuilder();

    if (articles.Any())
    {
        for (int a = 0; a < articles.Count; a++)
            articleHtml.Append(this.RenderPartialViewToString("_ArticleListItem", new ArticleListItemView { Article = articles[a], CssClass = ArticleHtmlHelper.GetItemCssClass((a + 1)), IsFullWidth = false } ));
    }

    return Json(articleHtml.ToString());
}

XML Parsing Error In A MVC Razor View

If you set a Controller's response type to "text/xml", you may encounter an: "XML Parsing Error: XML or text declaration not at start of entity". Your View may look something like this:

@{
    Layout = null;
}

<?xml version="1.0" encoding="utf-8" ?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @if (Model.Any())
    {
        foreach (SitemapNode node in Model)
        {
            <url>
                <loc>@node.Location</loc>
                <lastmod>@node.LastModified</lastmod>
                <changefreq>monthly</changefreq>
            </url>
        }
    }
</urlset>

In this case, I was creating a sitemap for one of my websites. So I created a Controller and View as I normally would do. However, when generating an XML output, you'll have to do something a little different in MVC:

@{
    Layout = null;
}<?xml version="1.0" encoding="utf-8" ?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    @if (Model.Any())
    {
        foreach (SitemapNode node in Model)
        {
            <url>
                <loc>@node.Location</loc>
                <lastmod>@node.LastModified</lastmod>
                <changefreq>monthly</changefreq>
            </url>
        }
    }
</urlset>

Can you see what is the difference? You'd be forgiven for not seeing it. But if you look a little closer, you'll see that I pushed up my XML declaration right up next to where I set the Layout block. This is because Razor outputs extra lines within its markup.

So when I left an empty line after my Layout block (as seen my my first code example), this gets rendered as an empty line when you run the page which would not be valid XML.

Update - 28/08/2014

Just found an even better way to get around the same issue from reading Joe Raczkowski blog. All that needs to be done is place the main XML declaration at the top of the page inside the @{} braces:

@{
Layout = null;
Response.ContentType = "text/xml";
Response.Write("<?xml version=\"1.0\" encoding=\"UTF-8\" ?>");
}

MVC Custom Extension Methods

I've been toying around with MVC for quite some time now. Initially, I couldn't imagine breaking away from the safety-net that is ASP.NET Web Forms. What many developers are not aware of when moving over to the ASP.NET MVC framework is that you have to write everything from scratch. You will not have the comfort of dragging and dropping event driven controls in a GUI-centric way.

But nowadays, I am itching to build new sites in MVC. I like to be in control of the whole page lifecycle and the mark-up that is generated.

Since there are no pre-built resusable controls, I decided start developing my own library of extensions that I could use in future MVC projects I work on. Ranging from pagination to tag clouds.

Creating custom extensions is really easy. I started off by creating a Category Navigation that returns a IHtmlString (HTML-encoded string that should not be encoded again).

public static IHtmlString CategoryNavigation(this WebViewPage wvp)
{
    StringBuilder navBuilder = new StringBuilder();

    List<CustomCategory> categories = CustomCategoryLogic.GetCategories();

    if (categories.Count > 0)
    {
        navBuilder.Append("<ul class=\"nav\">");
        navBuilder.Append("<li><a href=\"/\">Home</a></li>");

        foreach (CustomCategory cc in categories)
            navBuilder.AppendFormat("<li><a href=\"/{0}\">{1}</a></li>", cc.Slug, cc.Name);

        navBuilder.Append("</ul>");
    }

    return MvcHtmlString.Create(navBuilder.ToString());
}

To display my category navigation in one of my Views, I just need to write:

@this.CategoryNavigation()

How easy is that!?

Easy Way to Generate Strongly-Typed C# Classes From JSON

In nearly all my previous projects I've worked on required some kind of manipulation with reading some form of JSON data.

To allow better integration into my application, I always create strongly-typed classes that mirror the structure of the incoming data feed. This in turn allows me to serialize the JSON and select specific properties I require. But sometimes, its quite difficult and confusing to the exact class structure right, especially when the JSON contains lots of fields of data.

When I find this is the case, I use json2csharp.com. By simply pasting the contents of your JSON, the site does all the hard work and creates all the strongly-typed classes for you.

Twitter OAuth Authentication Without Third Party Libraries

Ever since Twitter ditched version 1 of their API to version 1.1, an additional hurdle created when attempting to get any data from Twitter. Authentication (using OAuth) is now required on all API request endpoints. I can see why Twitter decided to go down this route but it does add a little headache when carrying out the most simplest requests.

In all the sites I have worked on, I've always relied on third party libraries to help me interact with Twitter, such as Twitterizer (no v1.1 support) and TweetSharp. But due to the increased complexity on some of the sites I've been working on, third party libraries don't seem to cut the mustard any more. A more scalable solution is required...

I came across a really simple and expandable class library called OAuthTwitterWrapper that stemmed from a StackOverflow question I found. Out-of-the-box, this library contains calls required to retrieve a user's timeline and return searches which is great to get you up and running quickly.

The OAuthTwitterWrapper provided me a really good basis to add further Twitter features, for example a list of users favourite tweets.

If you don't plan on doing anything too complex when interacting with the Twitter API, third party libraries such as TweetSharp will meet your everyday needs. However, if you want more control over how you make requests, the OAuthTwitterWrapper provides a good foundation.

Focusing Control On Postback Using JavaScript

Postbacks can be annoying. Especially when you have a long page of content with a form at the bottom and on button click causes the page to skip back to the top of the page.

For a user who is not used to a site, this can be incredibly annoying and disorientating simply due to the fact the page has moved to a different position. This isn't so much of an issue if you happen to have a webpage that is low on content and a form will always be in clear view.

But there is quite a nice easy way to get back to a specific area of a page by using the following line of JavaScript will be added in the page after a postback has occurred.

if (Page.IsPostBack)
{
    ScriptManager.RegisterStartupScript(this, typeof(string), "PostbackFocus", "window.location.hash = 'feedback-form'", true);
}

By placing this code in your Page_Load method, the user will be taken to an anchor point placed within the page. In this case, a <div> with an ID attribute of "feedback-form".

A Hidden User Control Is A Running User Control

I've been a .NET Developer for around 6 years and it still amazes me how I can overlook something that I never really questioned. For example, when a user control is hidden, I always assumed that all the code it contained would never run since until it was made visible.

However, after being told by one of my work colleagues that in fact a hidden user control will always run, it will just simply is hidden by the client. After searching the web for a definitive answer, I found a StackOverflow post that fully backed up his theory: http://stackoverflow.com/questions/12143693/hiding-user-controls-using-code-behind-does-internal-code-still-run-ie-am-i.

As the StackOverflow post suggests, the most performance efficient way to show/hide a user control is by dynamically loading it in when required.

if (jobs.Count > 0)
{
     MyPlaceholder.Controls.Add(Page.LoadControl("/Controls/Listings/JobsList.ascx"));
}

As you can see in my example above, I'm loading my user control to a place holder in the page.

Optimising Image Quality In System.Drawing

Sometimes optimising images to have an adequate image to file size ratio can be difficult when dynamically generating images using “System.Drawing”.

Over the years, I have worked on quite a few different projects around the use of “System.Drawing” and recently I found a flexible way of being able to have control over the image quality and file size.

Here’s a snippet of of code from my Generic Handler (.ashx) file:

public void ProcessRequest(HttpContext context)
{
    context.Response.ContentType = "image/jpeg";

    //Create a new Bitmap
    Bitmap oBitmap = new Bitmap(800, 800, PixelFormat.Format24bppRgb);

    //Load Background Graphic from Image
    Graphics oGraphics = Graphics.FromImage(oBitmap);

    #region Your Image Code
     
    //Insert your code here.

    #endregion

    #region Stage 1: Image Quality Options
    
    oGraphics.InterpolationMode = InterpolationMode.HighQualityBicubic;
    oGraphics.SmoothingMode = SmoothingMode.HighQuality;
    oGraphics.PixelOffsetMode = PixelOffsetMode.HighQuality;
    oGraphics.CompositingQuality = CompositingQuality.HighQuality;
    
    #endregion

    //Clear graphic resources
    oGraphics.Dispose();

    #region Stage 2: Image Quality Options

    //Output image
    ImageCodecInfo[] Info = System.Drawing.Imaging.ImageCodecInfo.GetImageEncoders();
    EncoderParameters Params = new System.Drawing.Imaging.EncoderParameters(1);
    Params.Param[0] = new EncoderParameter(Encoder.Quality, 100L); //Set image quality
    context.Response.ContentType = Info[1].MimeType;
    oBitmap.Save(context.Response.OutputStream, Info[1], Params);

    #endregion
}

System.Drawing.Graphics Optimisations

In all my “System.Drawing” development projects, I’ve always set the graphic object quality options (lines 19-22) and found that it never really worked for me. The reason for this is because I always placed these settings after creating my System.Drawing.Graphics object prior to my custom code. So the image was getting optimised before any of my functionality had taken place. Rubbish!

The key is to set all your System.Drawing.Graphics object settings just before you dispose of it. Makes sense doesn’t it? Don’t know how I made such a noob mistake.

By default, .NET uses the web safe options when converting Bitmap to an image and setting those four properties will have a big affect on how your image looks.

Compression Level

This is the good bit!

.NET gives you the ability to carry out further tweaks on how your image will be rendered by allowing us to set the compression level. The compression level can be tweaked by modifying the value passed to the “EncoderParameter” constructor (line: 34).

For another example of how this can be used, take a look at the following MSDN article: http://msdn.microsoft.com/en-us/library/bb882583.aspx