Azure Function: 404 Page Checker

Sometimes the simplest piece of development can be the most rewarding and I think my Azure Function that checks for broken links on a nightly basis is one of those things. The Azure Function reads from a list of links from a database table and carries out a check to determine if a 200 response is returned. If not, the link will be logged and sent to a user by email using the Sendgrid API.

Scenario

I was working on a project that takes a list of products from an API and stores them in a Hubspot HubDB table. This table contained all product information and the expected URL to a page. All the CMS pages had to be created manually and assigned the URL as stored in the table, which in turn would allow the page to be populated with product data.

As you can expect, the disadvantage of manually created pages is that a URL change in the HubDB table will result in a broken page. Not ideal! In this case, the likelihood of a URL being changed is rare. All I needed was a checker to ensure I was made aware on the odd occasion where a link to the product page could be broken.

I won't go into any further detail but rest assured, there was an entirely legitimate reason for this approach in the grand scheme of the project.

Azure Function

I have modified my original code purely for simplification.

using System;
using System.Collections.Generic;
using System.Net;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
using SendGrid;
using SendGrid.Helpers.Mail;

namespace ProductsSyncApp
{
  public static class ProductLinkChecker
  {
    [FunctionName("ProductLinkChecker")]
    public static void Run([TimerTrigger("%ProductLinkCheckerCronTime%"
      #if DEBUG
      , RunOnStartup=true
      #endif
      )]TimerInfo myTimer, ILogger log)
    {
      log.LogInformation($"Product Link Checker started at: {DateTime.Now:G}");

      #region Iterate through all product links and output the ones that return 404.

      List<string> brokenProductLinks = new List<string>();

      foreach (string link in GetProductLinks())
      {
        if (!IsEndpointAvailable(link))
          brokenProductLinks.Add(link);
      }

      #endregion

      #region Send Email

      if (brokenProductLinks.Count > 0)
        SendEmail(Environment.GetEnvironmentVariable("Sendgrid.FromEmailAddress"), Environment.GetEnvironmentVariable("Sendgrid.ToAddress"), "www.contoso.com - Broken Link Report", EmailBody(brokenProductLinks));

      #endregion

      log.LogInformation($"Product Link Checker ended at: {DateTime.Now:G}");
    }

    /// <summary>
    /// Get list of a product links.
    /// This would come from a datasource somewhere containing a list of correctly expected URL's.
    /// </summary>
    /// <returns></returns>
    private static List<string> GetProductLinks()
    {
      return new List<string>
      {
        "https://www.contoso.com/product/brokenlink1",
        "https://www.contoso.com/product/brokenlink2",
        "https://www.contoso.com/product/brokenlink3",
      };
    }

    /// <summary>
    /// Checks if a URL endpoint is available.
    /// </summary>
    /// <param name="url"></param>
    /// <returns></returns>
    private static bool IsEndpointAvailable(string url)
    {
      try
      {
        HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);

        using HttpWebResponse response = (HttpWebResponse)request.GetResponse();

        if (response.StatusCode == HttpStatusCode.OK)
          return true;

        return false;
      }
      catch
      {
        return false;
      }
    }

    /// <summary>
    /// Create the email body.
    /// </summary>
    /// <param name="brokenLinks"></param>
    /// <returns></returns>
    private static string EmailBody(List<string> brokenLinks)
    {
      StringBuilder body = new StringBuilder();

      body.Append("<p>To whom it may concern,</p>");
      body.Append("<p>The following product URL's are broken:");

      body.Append("<ul>");

      foreach (string link in brokenLinks)
        body.Append($"<li>{link}</li>");

      body.Append("</ul>");

      body.Append("<p>Many thanks.</p>");

      return body.ToString();
    }

    /// <summary>
    /// Send email through SendGrid.
    /// </summary>
    /// <param name="fromAddress"></param>
    /// <param name="toAddress"></param>
    /// <param name="subject"></param>
    /// <param name="body"></param>
    /// <returns></returns>
    private static Response SendEmail(string fromAddress, string toAddress, string subject, string body)
    {
      SendGridClient client = new SendGridClient(Environment.GetEnvironmentVariable("SendGrid.ApiKey"));

      SendGridMessage sendGridMessage = new SendGridMessage
      {
        From = new EmailAddress(fromAddress, "Product Link Report"),
      };

      sendGridMessage.AddTo(toAddress);
      sendGridMessage.SetSubject(subject);
      sendGridMessage.AddContent("text/html", body);

      return Task.Run(() => client.SendEmailAsync(sendGridMessage)).Result;
    }
  }
}

Here's a rundown on what is happening:

  1. A list of links is returned from the GetProductLinks() method. This will contain a list of correct links that should be accessible on the website.
  2. Loop through all the links and carry out a check against the IsEndpointAvailable() method. This method carries out a simple check to see if the link returns a 200 response. If not, it'll be marked as broken.
  3. Add any link marked as broken to the brokenProductLinks collection.
  4. If there are broken links, send an email handled by SendGrid.

As you can see, the code itself is very simple and the only thing that needs to be customised for your use is the GetProductLinks method, which will need to output a list of expected links that a site should contain for cross-referencing.

Email Send Out

When using Azure functions, you can't use the standard .NET approach to send emails and Microsoft recommends that an authenticated SMTP relay service that reduces the likelihood of email providers rejecting the message. More insight into this can be found in the following StackOverflow post - Not able to connect to smtp from Azure Cloud Service.

When it comes to SMTP relay services, SendGrid comes up favourably and being someone who uses it in their current workplace, it was my natural inclination to make use of it in my Azure Function. Plus, they've made things easy by providing a Nuget package to allow direct access to their Web API v3 endpoints.

Hubspot CMS for Marketers Certified

Since around September last year, I've been involved in a lot of Hubspot projects at my place of work - Syndicut. It's the latest edition to the numerous other platforms that are offered to clients.

The approach to developing websites in Hubspot is not something I'm used to coming from a programming background where you build everything custom using some form of server-side language. But I was surprised by what you can achieve within the platform.

Having spent months building sites using the Hubspot Markup Language (HUBL), utilising a lot of the powerful marketing features and using the API to build a custom .NET Hubspot Connector, I thought it was time to attempt a certification focusing on the CMS aspect of Hubspot.

There are two CMS certifications:

  1. Hubspot CMS for Marketers
  2. Hubspot CMS for Developers

I decided to tackle the "CMS for Marketers" certification first as this mostly covers the theory aspect on how you use Hubspot to create a user-friendly, high-performing website and leveraging that with Hubspot CRM. These are the areas you can get quite shielded from if you're purely just developing in pages and modules. I thought it would be beneficial to expose myself from a marketing standpoint to get an insight into how my development forms part of the bigger picture.

I'm happy to report I am now Hubspot CMS for Marketers certified.

Hubspot CMS for Marketers Certification

UniFi: Restrict Network Device Access On A Guest Network

On my UniFi Dream Machine, I have set up a guest wireless network for those who come to my house and need to use the Internet. I've done this across all routers I've ever purchased, as I prefer to use the main non-guest wireless access point (WAP) just for me as I have a very secure password that I rather not share with anyone.

It only occurred to me a few days ago that my reason for having a guest WAP is flawed. After all, the only difference between the personal and guest WAP's is a throw-away password I change regularly. There is no beneficial security in that. It is time to make good use of UniFi’s Guest Control settings and prevent access to internal network devices. I have a very simple network setup and the only two network devices I want to block access to is my Synology NAS and IP Security Camera.

UniFi’s Guest Control settings do a lot of the grunt work out the box and is pretty effortless to set up. Within the UniFi controller (based on my own UniFi Dream Machine), the following options are available to you:

  1. Guest Network: Create a new wireless network with its own SSID and password.
  2. Guest User Group: Set download/upload bandwidth limitations that can be attached to the Guest Network.
  3. Guest Portal: A custom interface can be created where a guest will be served a webpage to enter a password to access the wireless network - much like what you'd experience when using the internet at an airport or hotel. UniFi gives you enough creative control to make the portal interface look very professional. You  can expire the connection by a set number of hours.
  4. Guest Control: Limit access to devices within the local network via IP address.

I don't see the need to enable all guest features the UniFi controller offers and the only two that are of interest to me is setting up a guest network and restricting access (options 1 and 4). This is a straight-forward process that will only take a few minutes.

Guest Network

A new wireless network will need to be created and be marked as a guest network. To do this, we need to set the following:

  • Name/SSID: MyGuestNetwork
  • Enable this wireless network: Yes
  • Security: WPA Personal. Add a password
  • Guest Policy: Yes

All other Advanced Options can be left as they are.

UniFi Controller - Guest Network Access Point

Guest Control

To make devices unavailable over your newly create guest network, you can simply add IPV4 hostname or subnet within the "Post Authorisation Restrictions" section. I've added the IP to my Synology NAS - 172.16.1.101.

UniFi Controller - Guest Control

If all has gone to plan when connecting to the guest WAP you will not be able to access any network connected devices.

UniFi: Unable To Access Synology On Local Network

Investing in a UniFi Dream Machine has been one of the wisest things I've done last year when it comes to relatively expensive purchases. It truly has been worth every penny for its reliability, security and rock-solid connection - something that is very much needed when working from home full-time.

The Dream Machine has been very low maintenance and I just leave it to do its thing apart from carrying out some minor configuration tweaks to aid my network. The only area that I did encounter problems was accessing the Synology Disk Station Manager (DSM) web interface. I could access Synology if I used the local IP address instead of the "myusername.synology.me" domain. Generally, this would be an ok solution, but not the right one for two reasons:

  1. Using a local IP address would restrict connection to my Synology if I was working outside from another location. This was quite the deal-breaker as I do have a bunch of Synology apps installed on my Mac, such as Synology Drive that carries out backups and folder synchronisation.
  2. I kept on getting a security warning in my browser when accessing DSM regarding the validity of my SSL certificate, which is to be expected as I force all connections to be carried out over SSL.

To my befuddlement, I had no issue accessing the data in my Synology by mapping them as network drives from my computer.

There was an issue with my local network as I was able to access the Synology DSM web interface externally. From perusing the UniFi community forum, there have been quite a few cases where users have reported the same thing and the common phrase that came popping up in all the posts was: Broken Hairpin NAT. What is a Hairpin NAT?

A Hairpin NAT allows you to run a server (in this case a NAS) inside your network but connect to it as if you were outside your network. For example via a web address, "myusername.synology.me" that will resolve to the internal IP of the server.

What I needed to do was to run an internal DNS server and a local entry for "myusername.synology.me" and point that to the internal IP address of the NAS. What was probably happening is that my computer/device was trying to make a connection past the firewall and then back in again to access the NAS. Not the most efficient way to make a connection for obvious reasons and in some cases may not work. A loopback would resolve this.

A clever user posted a solution to the issue on the UniFi forum that is very easy to follow and worked like a charm - Loopback/DNS Synology DiskStation.

I have also saved a screenshot of the solution for posterity.