Azure, Azure DevOps, Azure Pipelines, C# tip

Adding Environment Variables to an ARM template for Azure App Services

I like to have one click deployments for my web code – first to create my infrastructure, and then deploy the website.

But one of the challenges I’ve had in the past is that I’ve needed to make sure my resources (e.g. databases, key vaults etc) have unique names. ARM templates helpfully provide resource functions to generate unique names, but I won’t know what these names are until the infrastructure has been created.

So does that mean I need to first deploy infrastructure, and then introduce a manual step of updating my environment-specific configuration files?  Fortunately not – I can populate environment variables in my website with the names of resources, and in my code I just refer to those variables.

The default ARM template for Azure App Services is really useful, but by default it doesn’t have a section for environment variables to be used by the app service, and I find this is usually one of the first things I do when I’m creating an app service. I’ve needed to do this recently, and I decided to write about how I did that.

When you’re creating an Azure App Service in an ARM template, you probably have something that looks similar to the code below – a resource section to create the hosting plan (i.e. the server farm), and a resource section to create the app service to host your site.

"resources": [
  {
    "apiVersion": "2015-08-01",
    "name": "[parameters('hostingPlanName')]",
    "type": "Microsoft.Web/serverfarms",
    "location": "[resourceGroup().location]",
    "tags": {
      "displayName": "HostingPlan"
    },
    "sku": {
      "name": "F1",
      "capacity": "1"
    },
    "properties": {
      "name": "[parameters('hostingPlanName')]"
    }
  },
  {
    "apiVersion": "2015-08-01",
    "name": "[variables('webSiteName')]",
    "type": "Microsoft.Web/sites",
    "location": "[resourceGroup().location]",
    "tags": {
      "[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "Resource",
      "displayName": "Website"
    },
    "dependsOn": [
      "[resourceId('Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]"
    ],
    "properties": {
      "name": "[variables('webSiteName')]",
      "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]"
    }
  }

Environment variables can be created at the same time as the app service by adding another resource to the ARM template.

I like to do this through Visual Studio 2019 – as shown below, I open up my ARM project in VS2019, and right click on my template (named WebSite.json) to open the context menu. In the context menu, there’s an item “Show Outline”:

arm outline

When I click on “Show Outline”, this opens up a new window called “JSON Outline” which shows me what’s in my ARM template – helpfully split into parameters, variables, outputs and resources. As you can see in the image below, I have two resources in my template – the hosting plan and the website.

arm json outline

So if I want to add a new resource, I can right click on the appropriate node, and select “Add New Resource” (as shown below).

It’s important to note that to add environment variables to a website, we have to add the resource to the existing website, so the environment variables will appear as a child object of the website.

arm json add resource

When you click on “Add New Resource”, a window will open that looks like the one below. To add environment variables to the app service, find the “Application Settings for Web Apps” resource (which is highlighted below), and choose a name for the resource – I’ve chosen to call mine “appsettings”.

application settings for web apps

Click on “Add”, and now you’ll see that in your ARM template, your website resource will have an “appsettings” child. This the place where your site’s environment variables will live.

Of course these aren’t the only environment variables in your app service – these are just the ones you’re choosing to add during your infrastructure deployment process.

added appsettings

And this new “appsettings” child object corresponds to the code highlighted in red below:

{
      "apiVersion": "2015-08-01",
      "name": "[variables('webSiteName')]",
      "type": "Microsoft.Web/sites",
      "location": "[resourceGroup().location]",
      "tags": {
        "[concat('hidden-related:', resourceGroup().id, '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]": "Resource",
        "displayName": "Website"
      },
      "dependsOn": [
        "[resourceId('Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]"
      ],
      "properties": {
        "name": "[variables('webSiteName')]",
        "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]"
      },
      "resources": [
        {
          "name": "appsettings",
          "type": "config",
          "apiVersion": "2015-08-01",
          "dependsOn": [
            "[resourceId('Microsoft.Web/sites', variables('webSiteName'))]"
          ],
          "tags": {
            "displayName": "appsettings"
          },
          "properties": {
            "key1": "value1",
            "key2": "value2"
          }
        }
      ]
    }

This new resource adds a couple of environment variables by default as examples – “key1” and “key2“.

Obviously these are just placeholders, so let’s consider a real example – what about if I want to deploy the same infrastructure across numerous environments, i.e. use the same ARM template, but parameterise it so that one environment is called “integration”, one is called “demo”, and others could be “staging” or “production”?

Parameterising our ARM template to accept an environment name is straightforward – we just need to add a parameter to our template, as shown below:

"parameters": {
  "environmentName": {
    "type": "string",
    "minLength": 1
  }
}

And now we can update the environment variable properties to have a key called “ASPNETCORE_ENVIRONMENT”, with the value set by the process that calls the ARM template.

"resources": [
        {
          "name": "appsettings",
          "type": "config",
          "apiVersion": "2015-08-01",
          "dependsOn": [
            "[resourceId('Microsoft.Web/sites', variables('webSiteName'))]"
          ],
          "tags": {
            "displayName": "appsettings"
          },
          "properties": {
            "ASPNETCORE_ENVIRONMENT": "[parameters('environmentName')]"
          }
        }
      ]

I usually would call my ARM template using the “Azure Resource Group” deployment task in Azure Pipelines, and I can just specify the value of this variable at each deployment stage as follows:

-environmentName "integration"

Or

-environmentName "demonstration"

After I create an Azure Pipeline to deploy this infrastructure, I can use the Azure Portal to look at the environment variables for my website by clicking on the app service, and looking in the “Configuration” item, as shown in the image below. This shows that there is now an environment variable called “ASPNETCORE_ENVIRONMENT”.

env variables in web app

And I can access the environment variable’s parameter in C# using the code below:

Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")

This is a tremendously powerful technique when creating infrastructure.

Another example is one I mentioned in the introduction – consider when you’re creating a website that uses something like Azure Key Vault to hold connection strings. As I’ve talked about in my last post, you might want the name of the Azure Key Vault to be automatically created during the pipeline with a unique name.

But if I don’t know the name of the key vault when I’m developing my code, how can I refer to it from my code? A common approach is to specify resource names in a configuration file, but if we can’t be sure of the name of the resource until after it has been created, this means we’d have to modify the config file after deployment.

Instead, my approach is:

  • Refer to an environment variable called something like “AZURE_KEY_VAULT” in my code
  • Create the Azure Key vault with a unique name using an ARM template
  • Add an environment variable section which depends on the successful creation of the Azure Key Vault, and update the “AZURE_KEY_VAULT” environment variable with the unique name of the Azure Key Vault.

Now as a developer, I never need to care about what any environment’s key vault name is, and I can happy tear any environment down without worrying about having to manually update or references to resources which might have changed name when I spin it back up – I just need to refer to the name of the environment variable, without ever having to worry about its contents.

The code below shows what my appsettings resource looks like for this scenario:

"resources": [
        {
          "name": "appsettings",
          "type": "config",
          "apiVersion": "2015-08-01",
          "dependsOn": [
            "[resourceId('Microsoft.Web/sites', variables('webSiteName'))]",
            "[resourceId('Microsoft.KeyVault/vaults', concat('myKeyVault-', uniqueString(resourceGroup().id)))]"
          ],
          "tags": {
            "displayName": "appsettings"
          },
          "properties": {
            "AZURE_KEY_VAULT": "[concat('myKeyVault-', uniqueString(resourceGroup().id))]"
          }
        }
      ]

Wrapping up

This has been quite a long post about quite a simple concept – automatically create environment variables in your website for all the Azure resources it uses, and then automatically populate the values of those variables while those resources are being created. But it’s a worthwhile technique to employ – I’ve found this makes spinning up and tearing down environments in one step much easier, rather than having a multi-step process of creating infrastructure, noting the name of resources created, and then updating and deploying XML/JSON configuration files.

.net core, Azure, Azure Application Insights, C# tip, Instrumentation, MVC

Healthcheck endpoints in C# in MVC projects using ASP.NET Core, and writing results to Azure Application Insights

Every developer wants to build a system that never breaks, but in reality things go wrong. The best systems are built to expect that and handle problems, that rather than just silently failing.

Maybe your database becomes unavailable (e.g. runs out of hard disk space) and your failover doesn’t work – or maybe a third party web service that you depend on stops working.

Sometimes your application can be programmed to recover from things going wrong – here’s my post on The Polly Project to find out more about one way of doing that – but when there’s a catastrophic failure that you can’t recover from, you want to be alerted as soon as it happens, rather than hear from a customer.

And it’s kind to provide a way for your customers to find out about the health of your system. As an example, just check out the monitoring hub below from Postcodes.io – this is a great example of being transparent about key system metrics like service status, availability, performance, and latency.

postcode

MVC projects in ASP.NET Core have a built in feature to provide information on the health of your website. It’s really simple to add it to your site, and this instrumentation comes packaged as part of the default ASP.NET Core toolkit. There are also some neat extensions available on NuGet to format the data as JSON, add a nice dashboard for these healthchecks, and finally to push the outputs to Azure Application Insights. As I’ve been implementing this recently, I wanted to share with the community how I’ve done it.

Scott Hanselman has blogged about this previously, but there have been some updates since he wrote about this which I’ve included in my post.

Returning system health from an ASP.NET Core v2.2 website

Before I start – I’ve uploaded all the code to GitHub here so you can pull the project and try yourself. You’ll obviously need to update subscription keys, instrumentation keys and connection strings for databases etc.

Edit your MVC site’s Startup.cs file and add the line below to the ConfigureServices method:

services.AddHealthChecks();

And then add the line of code below to the Configure method.

app.UseHealthChecks("/healthcheck");

That’s it. Now your website has a URL available to tell whether it’s healthy or not. When I browse to my local test site at the URL below…

http://localhost:59658/healthcheck

..my site returns the word “Healthy”. (obviously your local test site’s URL will have a different port number, but you get the idea)

So this is useful, but it’s very basic. Can we amp this up a bit – let’s say want to see a JSON representation of this? Or what about our database status? Well fortunately, there’s a great series of libraries from Xabaril (available on GitHub here) which massively extend the core healthcheck functions.

Returning system health as JSON

First, install the AspNetCoreHealthChecks.UI NuGet package.

Install-Package AspNetCore.HealthChecks.UI

Now I can change the code in my StartUp.cs file’s Configure method to specify some more options.

The code below changes the response output to be JSON format, rather than just the single word “Healthy”.

app.UseHealthChecks("/healthcheck", new HealthCheckOptions
    {
        Predicate = _ => true,
        ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
    });

And as you can see in the image below, when I browse to the healthcheck endpoint I configured as “/healthcheck”, it’s now returning JSON:

healthcheck basic json

What about checking the health of other system components, like URIs, SQL Server or Redis?

Xabaril has got you covered here as well. For these three types of things, I just install the NuGet packages with the commands below:

Install-Package AspNetCore.HealthChecks.Uris
Install-Package AspNetCore.HealthChecks.Redis
Install-Package AspNetCore.HealthChecks.SqlServer

Check out the project’s ReadMe file for a full list of what’s available.

Then change the code in the ConfigureServices method in the project’s Startup.cs file.

services.AddHealthChecks()
        .AddSqlServer(connectionString: Configuration.GetConnectionString("SqlServerDatabase"),
                  healthQuery: "SELECT 1;",
                  name: "Sql Server", 
                  failureStatus: HealthStatus.Degraded)
        .AddRedis(redisConnectionString: Configuration.GetConnectionString("RedisCache"),
                        name: "Redis", 
                        failureStatus: HealthStatus.Degraded)
        .AddUrlGroup(new Uri("https://localhost:59658/Home/Index"),
                        name: "Base URL",
                        failureStatus: HealthStatus.Degraded);

Obviously in the example above, I have my connection strings stored in my appsettings.json file.

When I browse to the healthcheck endpoint now, I get much a richer JSON output.

health json

Can this information be displayed in a more friendly dashboard?

We don’t need to just show JSON or text output – Xabaril allows the creation of a clear and simple dashboard to display the health checks in a user friendly form. I updated my code in the StartUp.cs file – first of all, my ConfigureServices method now has the code below:

services.AddHealthChecks()
        .AddSqlServer(connectionString: Configuration.GetConnectionString("SqlServerDatabase"),
                  healthQuery"SELECT 1;",
                  name"Sql Server", 
                  failureStatusHealthStatus.Degraded)
        .AddRedis(redisConnectionString: Configuration.GetConnectionString("RedisCache"),
                        name"Redis", 
                        failureStatusHealthStatus.Degraded)
        .AddUrlGroup(new Uri("https://localhost:59658/Home/Index"),
                        name"Base URL",
                        failureStatusHealthStatus.Degraded);
        
services.AddHealthChecksUI(setupSettings: setup =>
{
    setup.AddHealthCheckEndpoint("Basic healthcheck", "https://localhost:59658/healthcheck");
});

And my Configure method also has the code below.

app.UseHealthChecks("/healthcheck"new HealthCheckOptions
    {
        Predicate = _ => true,
        ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
    });
 
app.UseHealthChecksUI();

Now I can browse to a new endpoint which presents the dashboard below:

http://localhost:59658/healthchecks-ui#/healthchecks

health default ui
And if you don’t like the default CSS, you can configure it to use your own. Xabaril has an example of a css file to include here, and I altered my Configure method to the code below which uses this CSS file.

app.UseHealthChecks("/healthcheck"new HealthCheckOptions
    {
        Predicate = _ => true,
        ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
    })
    .UseHealthChecksUI(setup =>
    {
        setup.AddCustomStylesheet(@"wwwroot\css\dotnet.css");
    });
 
app.UseHealthChecksUI();

And now the website is styled slightly differently, as you can see in the image below.

health styled ui

What happens when a system component fails?

Let’s break something. I’ve turned off SQL Server, and a few seconds later the UI automatically refreshes to show the overall system health status has changed – as you can see, the SQL Server check has been changed to a status of “Degraded”.

health degrades

And this same error appears in the JSON message.

health degraded json

Can I monitor these endpoints in Azure Application Insights?

Sure – but first make sure your project is configured to use Application Insights.

If you’re not familiar with Application Insights and .NET Core applications, check out some more information here.

If it’s not set up already, you can add the Application Insights Telemetry by right clicking on your project in the Solution Explorer window of VS2019, selecting “Add” from the context menu, and choosing “Application Insights Telemetry…”. This will take you through the wizard to configure your site to use Application Insights.

aitel

Once that’s done, I changed the code in my Startup.cs file’s ConfigureServices method to explicitly push to Application Insights, as shown in the snippet below:

services.AddHealthChecks()
        .AddSqlServer(connectionString: Configuration.GetConnectionString("SqlServerDatabase"),
                  healthQuery"SELECT 1;",
                  name"Sql Server", 
                  failureStatusHealthStatus.Degraded)
        .AddRedis(redisConnectionString: Configuration.GetConnectionString("RedisCache"),
                        name"Redis", 
                        failureStatusHealthStatus.Degraded)
        .AddUrlGroup(new Uri("https://localhost:44398/Home/Index"),
                        name"Base URL",
                        failureStatusHealthStatus.Degraded)
        .AddApplicationInsightsPublisher();
        
services.AddHealthChecksUI(setupSettingssetup =>
{
    setup.AddHealthCheckEndpoint("Basic healthcheck""https://localhost:44398/healthcheck");
});

Now I’m able to view these results in the Application Insights – the way I did this was:

  • First browse to portal.azure.com and click on the “Application Insights” resource which has been created for your web application (it’ll probably be top of the recently created resources).
  • Once that Application Insights blade opens, click on the “Metrics” menu item (highlighted in the image below):

app insights metrics

When the chart windows opens – it’ll look like the image below – click on the “Metric Namespace” dropdown and select the “azure.applicationinsights” value (highlighted below).

app insights custom metric

Once you’ve selected the namespace to plot, choose the specific metric from that namespace. I find that the “AspNetCoreHealthCheckStatus” metric is most useful to me (as shown below).

app insights status

And finally I also choose to display the “Min” value of the status (as shown below), so if anything goes wrong the value plotted will be zero.

app insights aggregation

After this, you’ll have a graph displaying availaility information for your web application. As you can see in the graph below, it’s pretty clear when I turned on my SQL Server instance again so the application health went from a overall health status of ‘Degraded’ to ‘Healthy’.

application insights

Wrapping up

I’ve covered a lot of ground in this post – from .NET Core 2.2’s built in HealthCheck extensions, building on that to use community content to check other site resources like SQL Server and Redis, adding a helpful dashboard, and finally pushing results to Azure Application Insights. I’ve also created a bootstrapper project on GitHub to help anyone else interested in getting started with this – I hope it helps you.

 

.net core, Azure, C# tip, Clean Code, Dependency Injection, Inversion of Control, MVC

Azure Cache as a session data store in C# and MVC

Using the HTTP Session is one of those things that provokes…opinions. A lot of developers think that sessions are evil. And whereas I understand some of the reasons why that’s a common viewpoint – I’ve had my share of problems with code that uses sessions in the past – I’d have to qualify any criticism by saying the problems were more about how I was using the technique, rather than any inherent problem with sessions as a concept.

For example, some instances where using an in-memory session store can cause problems are:

  • If you chuck lots of website data into an in-memory session, you’ll quickly eat up lots of RAM on your web server. This might eventually cause performance problems.
  • Sessions are often short lived – often around 20 minutes – leading to a poor user experience after a period of inactivity (like being unexpectedly logged out).
  • Also in a load balanced environment, users might experience issues – if their first request leads to a session being created on one web server, and then their next request is routed to a different (less-busy) web server, then it won’t know anything about their previous session. Apparently you can work around this with using sticky sessions…and my own experiences with the sticky sessions approach are best described as “mixed”. But YMMV.
  • If you’re not using SSL/TLS, your session might be vulnerable to the Man-in-the-Middle attack vector. But the easy answer to this is….use SSL.

Anyway – I think most people would agree with the basic need for one web page to access data entered on another web page. However, if you have high throughput, a need for a large session store, or a load balanced environment, then the out-of-the-box HTTP Session object might not be for you. But that doesn’t mean you can’t use sessions at all.

‘Azure Cache for Redis’ to the rescue

So even though my application isn’t in a load-balanced environment right now, I’d still like to make sure it’s easy to port to one in the future. So I’ve been looking for alternatives to using the Session object:

  • I could use Cookies, but I can’t store very much information in them.
  • I could use a SQL database, but this seems heavyweight for my need for shortlived session-based information.
  • Something like a NoSQL store like Redis would suit very well – it’s super fast with low-latency, high-throughput performance.

I’ve written about using Redis as a fast-access data store a long time ago, but that post is out of date now and worth updating as there’s now a built-in Azure option – Azure Cache for Redis.

Spinning up Azure Cache for Redis

Check out the official docs for how to create a cache in Azure – it’s clearly described here with lots of screenshots to guide you through.

But how can I using Azure Cache for Redis in a website?

I don’t really like the ASP.NET implementation from the official documentation. It works, but there’s a lot of code in the controller’s action, and I’d like a cleaner solution. Ideally I’d like to inject an interface into my controller as a dependency, and use ASP.NET Core’s service container to instantiate the dependency on demand.

I found this really useful post from Simon Holman, and he also has created a super helpful example on GitHub. I tested this with an MVC project in .NET Core v2.2, and the implementation is very simple (check out Simon’s source code for exactly where to put these snippets).

  • Update the Startup.cs file’s ConfigureServices method after putting the connection string into your appsettings.json file:
services.AddDistributedRedisCache(options =>
{
    options.Configuration = Configuration.GetConnectionString("RedisCache");
});
 
services.AddSession();
  • Update the Startup.cs file’s Configure method:
app.UseSession();
  • Here’s how to set data into the cache:
var sessionstartTime = DateTime.Now.ToLongTimeString();
HttpContext.Session.SetString("mysessiondata"sessionstartTime);
  • …get data from the cache:
var sessionstartTime = HttpContext.Session.GetString("mysessiondata");
  • …and remove data from the cache:
HttpContext.Session.Remove("mysessiondata");

Some things to bear in mind about Azure Cache for Redis

I wanted to dive into the details of Azure Cache a bit more, just to understand what’s actually going on beneath the hood when we’re reading from, and writing to, the session object.

You can see what session keys are saved in your Redis cache using the Azure portal

redis console arrow

Once you’re in the console, you run the command “scan 0 count 100 match *” to see up to the first 100 keys in your Redis cache.

redis console

From the screenshot above, I can see that I’ve got 16 sessions open.

The Guid in the Key is actually your Session’s *private* “_sessionKey”

In the image above, you can see a bunch of GUID objects which are the keys of the items in my Redis cache. And if you look at the 16th item in the list above, you can see that it corresponds to the private “_sessionKey” value, which is held in my HttpContext.Session object (compare with the VS2019 watch window below).

redis session

So this information is interesting…but I’m not sure how useful it is. Since that property is private, you can’t access it (well you can, but not easily, you have to use reflection). But it might be helpful to know this at debug time.

Browsers behave differently when in incognito mode

I thought I’d try the application with my browser in incognito mode. And every time I hit refresh on the browser when I was in incognito or private browsing mode, a new session key was created on the server – which meant it wasn’t able to obtain data from the session previously created in the same browser instance.

You can see the number of keys has hugely increased in the image below, corresponding to the number of times I hit refresh:

private window redis

But at least I can detect when the session isn’t available through the HttpContext.Session.IsAvailable property – when a session is available, the image below is what I can see in the session using a watch in the VS2019 debugger:

session available

And when a session isn’t available (such as when my browser is in incognito mode), this is what I see:

session unavailable

So at least I can programmatically distinguish between when the session can work for the user and when it can’t.

In summary, this behaviour had a couple of implications for me:

  • Session persistence didn’t work in incognito/private windows – values weren’t persistent in the same session across pages.
  • Hitting refresh a bunch of times in incognito will create lots of orphan session objects in your server, which might have security/availability implications for your application, especially if your sessions are large and fill up available memory.

Clearing down sessions was harder than I thought

HttpContext.Session.Clear() emptied my session, but didn’t delete the key from the server, as I could still see it in the Redis console.

In fact, the only way I was able to remove sessions held in Redis was to get right into the guts of the StackExchange.Redis package using the code below. Since I knew the exact session that I wanted to delete had the key “57154387-d8b7-c361-a174-9d27b6c6caae“:

var connectionMultiplexer = StackExchange.Redis.ConnectionMultiplexer.Connect(Configuration.GetConnectionString("RedisCache"));
 
connectionMultiplexer.GetDatabase(connectionMultiplexer.GetDatabase().Database).KeyDelete("57154387-d8b7-c361-a174-9d27b6c6caae");

But this is only useful if you can get the exact session key that you want to delete, and that isn’t particularly easy. You could use reflection to get that private value like in the code below, but I get why that’s not something you might want to do.

var _sessionKey = typeof(DistributedSession)
                .GetField("_sessionKey"BindingFlags.NonPublic | BindingFlags.Instance)
                .GetValue(HttpContext.Session);

Wrapping up

I’ve talked a little bit about sessions in this post – they’re not a magic hammer, but also aren’t inherently a bad tool – maybe consider an alternative to in-memory sessions if you have large session objects, or are working in a load balanced environment. Azure Cache for Redis might be one of those alternatives. I’ve found it to be interesting and useful, and relatively easy to set up as an alternative to an in-memory session, but there are a few quirks – sessions may not work the way you expect them to for users who are incognito/using private browsing, and it’s hard to completely delete a session once it has been created.

.net core, C# tip, MVC

Adding middleware to your .NET Core MVC pipeline to prettify HTML output with AngleSharp

I was speaking to a friend of mine recently about development and server side generated HTML, and they said that one thing they would love to do is improve how HTML code is when it’s rendered. Often when they look at the HTML source of a page, the indentation is completely wrong, and there are huge amounts of whitespace and unexpected newlines.

And I agreed – I’ve seen that too. Sometimes I’ve been trying to debug an issue in the rendered output HTML, and one of the first things I do format and indent the HTML code so I can read and understand it. And why not – if my C# classes aren’t indented logically, I’d find it basically unreadable. Why should my HTML be any different?

So it occurred to me that I might be able to find a way to write some middleware for my .NET Core MVC website that formats and indents rendered HTML for me by default.

This post is just a fun little experiment for me – I don’t know if the code is performant, or if it scales. Certainly on a production site I might want to minimise the amount of whitespace in my HTML to improve download speeds rather than just change the formatting.

Formatting and Indenting HTML

I’ve seen a few posts asking how to do this with HtmlAgilityPack – but even though HtmlAgilityPack is amazing, it won’t format HTML.

I’ve also seen people recommend a .NET wrapper for the Tidy library, but I’m going to use AngleSharp. AngleSharp is a .NET library that allows us to parse HTML, and contains a super useful formatter called PrettyMarkupFormatter.

var parser = new AngleSharp.Html.Parser.HtmlParser();
var document = parser.ParseDocument("<html><body>Hello, world</body></html>");
 
var sw = new StringWriter();
document.ToHtml(swnew AngleSharp.Html.PrettyMarkupFormatter());
 
var indentedHtml = sw.ToString();

And I can encapsulate this in a function as below:

private static string PrettifyHtml(string newContent)
{
    var parser = new AngleSharp.Html.Parser.HtmlParser();
    var document = parser.ParseDocument(newContent);
 
    var sw = new StringWriter();
    document.ToHtml(swnew AngleSharp.Html.PrettyMarkupFormatter());
    return sw.ToString();
}

Adding middleware to modify the HTML output

There’s lots of information on writing ASP.NET Core middleware here and I can build on this and the AngleSharp code to re-format the rendered HTML. The code below allows me to:

  • Check I’m in my development environment,
  • Read the rendered HTML from the response,
  • Correct the indentation using AngleSharp and the new PrettifyHtml method, and
  • Write the formatted HTML back to the Response.
if (env.IsDevelopment())
{
    app.Use(async (contextnext=>
    {
        var body = context.Response.Body;
 
        using (var updatedBody = new MemoryStream())
        {
            context.Response.Body = updatedBody;
 
            await next();
 
            context.Response.Body = body;
 
            updatedBody.Seek(0SeekOrigin.Begin);
            var newContent = new StreamReader(updatedBody).ReadToEnd();
 
            await context.Response.WriteAsync(PrettifyHtml(newContent));
        }
    });
}

And now the HTML generated by my MVC application is formatted and indented correctly.

Wrapping up

This post is really just a proof of concept and for fun – I’ve restricted the effect to my development environment in case it doesn’t scale well. But hopefully this is useful to anyone trying to format HTML, or intercept an HTML response to modify it.


About me: I regularly post about Microsoft technologies and .NET – if you’re interested, please follow me on Twitter, or have a look at my previous posts here. Thanks!

Continue reading

.net, C# tip

Instantiating a C# object from a string using Activator.CreateInstance in .NET

Recently I hit an interesting programming challenge – I need to write a library which can instantiate and use a C# class object from a second C# assembly.

Sounds simple enough…but the catch is that I’m only given some string information about the class at runtime, such as the class name, its namespace, and what assembly it belongs to.

Fortunately this is possible using the Activator.CreateInstance method in C#. First I need to format the namespace, class name and assembly name in a special way – as an assembly qualified name.

Let’s look at an example – the second assembly is called “MyTestProject” and the object I need to instantiate from my library looks like the one below.

namespace SampleProject.Domain
{
    public class MyNewTestClass
    {
        public int Id { getset; }
 
        public string Name { getset; }
 
        public string DoSpecialThing()
        {
            return "My name is MyNewTestClass";
        }
    }
}

This leads to the assembly qualified name:

"SampleProject.Domain.MyNewTestClass, MyTestProject"

Note that the format here is along the lines of:

"{namespace}.{class name}, "{assembly name}"

Another way of finding this assembly qualified name is to run the code below:

Console.WriteLine(typeof(MyNewTestClass).AssemblyQualifiedName);

This will output something like:

SampleProject.Domain.MyNewTestClass, MyTestProject, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null

There’s extra information about Version, Culture and PublicKeyToken – you might need to use this if you’re targetting different versions of a library, but in my simple example I don’t need this so I won’t elaborate further here.

Now I have the qualified name of the class, I can instantiate the object in my library using the Activator.CreateInstance, as shown below:

const string objectToInstantiate = "SampleProject.Domain.MyNewTestClass, MyTestProject";
 
var objectType = Type.GetType(objectToInstantiate);

var instantiatedObject = Activator.CreateInstance(objectType);

But it’s a bit difficult to do anything useful with the instantiated object – at runtime we obviously know it has the type of Object, and I suppose we could call the ToString() method, but for a new class that’s of limited use. How can we access properties and methods in the instantiated class?

One way is to use the dynamic keyword to manipulate the new object

We could use the dynamic keyword with the instantiated object, and set/get methods dynamically, like in the code below:

const string objectToInstantiate = "SampleProject.Domain.MyNewTestClass, MyTestProject";
 
var objectType = Type.GetType(objectToInstantiate);

dynamic instantiatedObject = Activator.CreateInstance(objectTypeas ITestClass;
 
// set a property value
instantiatedObject.Name = "Test Name";
 
// get a property value
string name = instantiatedObject.Name;
 
// call a method - this outputs "My name is MyNewTestClass"
Console.Write(instantiatedObject.DoSpecialThing());

Another way to manipulate the instantiated object is through using a shared interface

We could make the original object implement an interface shared across all projects. If I add the interface below to my project…

namespace SampleProject.Domain
{
    public interface ITestClass
    {
        int Id { getset; }
        string Name { getset; }
        string DoSpecialThing();
    }
}

…then our original object could implement this interface:

namespace SampleProject.Domain
{
    public class MyNewTestClass : ITestClass
    {
        public int Id { getset; }
 
        public string Name { getset; }
 
        public string DoSpecialThing()
        {
            return "My name is MyNewTestClass";
        }
    }
}

So if we happen to know at design time that the object we want to instantiate implements the ITestClass interface, we can access methods exposed by that interface – there’s no need to use the dynamic keyword now.

const string objectToInstantiate = "SampleProject.Domain.MyNewTestClass, MyTestProject";
 
var objectType = Type.GetType(objectToInstantiate);

var instantiatedObject = Activator.CreateInstance(objectTypeas ITestClass;
 
// set a property value
instantiatedObject.Name = "Test Name";
 
// get a property value
var name = instantiatedObject.Name;
 
// call a method - this outputs "My name is MyNewTestClass"
Console.Write(instantiatedObject.DoSpecialThing());

And of course if I have another domain object which implements the same interface but has different behaviour, like the one below…

namespace SampleProject.Domain
{
    public class DifferentTestClass : ITestClass
    {
        public int Id { getset; }
 
        public string Name { getset; }
 
        public string DoSpecialThing()
        {
            return "This is a different special thing";
        }
    }
}

..then I can use similar code to instantiate and manipulate the object – I just need to use the different object’s assembly qualified name:

const string objectToInstantiate = "SampleProject.Domain.DifferentTestClass, MyTestProject";
 
var objectType = Type.GetType(objectToInstantiate);

var instantiatedObject = Activator.CreateInstance(objectTypeas ITestClass;
 
// set a property value
instantiatedObject.Name = "Other Test Name";
 
// get a property value
string name = instantiatedObject.Name;
 
// call a method - but this now outputs "This is a different special thing"
Console.Write(instantiatedObject.DoSpecialThing());

Hopefully this helps anyone else facing a similar challenge – it’s worth bearing in mind that reflection is very powerful, but also can be a bit slower than other techniques.

.net, C# tip

An extension method for .NET enumerations that uses the DescriptionAttribute

Sometimes little improvements make a big difference to my day to day programming. Like when I discovered the String.IsNullOrEmpty method (I think it came into .NET Framework way back in v2.0) – when I was testing for non-empty strings , I was able to use that super useful language feature without having to remember to write two comparisons (like myString != null && myString != string.Empty). I’ve probably used that bit of syntactical candy in every project I’ve worked on since then.

I work with enumerations all the time too. And a lot of the time, I need a text representation of the enumeration values which is a bit more complex than the value itself. This is where I find the DescriptionAttribute so useful – it’s a place supplied natively by .NET where I can add that text representation.

I’ve provided a simple example of this kind of enumeration with descriptions for each item below – maybe for a production application I’d localise the text, but you get the idea.

public enum Priority
{
    [Description("Highest Priority")]
    Top,
    [Description("MIiddle Priority")]
    Medium,
    [Description("Lowest Priority")]
    Low
}

But I’ve always felt I’d love to have native access to a method that would give me the description. Something like:

Priority.Medium.ToDescription();

Obviously it’s really easy to write an extension method like this, but every time I need it on a new project for a new client, I have to google for how to use types and reflection to access the attribute method, and then write that extension method. Then I think about the other extension methods that I might like to write (what about ToDisplayName(), that might be handy…), and how to make it generic so I can extend it later, and what about error handling…

…and anyway, this process always includes the thought, “I’ve lost count how many times I’ve done this for my .NET enumerations, why don’t I write this down somewhere so I can re-use it?

So I’ve written it down below.

public static class EnumerationExtensions
{
    public static string ToDescription(this Enum enumeration)
    {
        var attribute = GetText<DescriptionAttribute>(enumeration);
 
        return attribute.Description;
    }
 
    public static T GetText<T>(Enum enumeration) where T : Attribute
    {
        var type = enumeration.GetType();
        
        var memberInfo = type.GetMember(enumeration.ToString());
 
        if (!memberInfo.Any())
            throw new ArgumentException($"No public members for the argument '{enumeration}'.");
 
        var attributes = memberInfo[0].GetCustomAttributes(typeof(T), false);
 
        if (attributes == null || attributes.Length != 1)
            throw new ArgumentException($"Can't find an attribute matching '{typeof(T).Name}' for the argument '{enumeration}'");
 
        return attributes.Single() as T;
    }
}

I’ve split it into two methods, so if I want to create a ‘ToDisplayName()’ extension later, it’s really easy for me to do that. I might include it in a NuGet package so it’s even easier for me to re-use later. I’ll update this post if I do.

Wrapping up

This was a quick blog post covering a simple area, about something that bugs me – having to rewrite a simple function each time I start a new project. I can imagine a bunch of reasons why this isn’t a native extension in .NET Framework/Core – and my implementation is an opinionated extension. I’m sure there’d be a lot of disagreement about what good practice is. Anyway, this class works for me – I hope it’s useful to you also.


About me: I regularly post about Microsoft technologies and .NET – if you’re interested, please follow me on Twitter, or have a look at my previous posts here. Thanks!

.net, C# tip

Correctly reading encoded text with the StreamReader in .NET

So here’s a problem…

I’ve got a list of my team members in a text file which I need to parse and process in .NET. The file is pretty simple – it’s called MyTeamNames.txt and it contains the following names:

  • Adèle
  • José
  • Chloë
  • Auróra

I created the text file on Windows 10 machine and used Notepad. I saved the file with the default ANSI encoding.

ansi save

I’m going to read names from this text file using a .NET Framework StreamReader – there’s a simple and clear example on the docs.microsoft.com site describing how to do this. So I’ve written a spike of code to use a StreamReader – I’ve more or less copied directly from the link above – and it looks like this:

using System;
using System.IO;
 
namespace ConsoleApp
{
    internal static class Program
    {
        private static void Main()
        {
            try
            {
                const string myTeamNamesFile = @"C:\Users\jeremy.lindsay\Desktop\MyTeamNames.txt";
 
                // Open the text file using a stream reader.
                using (var streamReader = new StreamReader(myTeamNamesFile))
                {
                    // Read the stream to a string, and write the string to the console.
                    var line = streamReader.ReadToEnd();
                    Console.WriteLine(line);
                }
            }
            catch (Exception e)
            {
                Console.WriteLine("The file could not be read:");
                Console.WriteLine(e.Message);
            }
        }
    }
}

But when I run the code, there’s a problem. I expected to see my team’s names written to the console – but instead all those names now have question marks scattered throughout them, as shown in the image below.

wrongnames

What’s gone wrong?

The StreamReader object and original text file need to have compatible encoding types

It’s pretty obvious that the question marks relate to the non-ASCII characters, and each name on my list have either an accent or a grave, or an umlaut/diaeresis.

The problem in my .NET Framework console application is that my StreamReader is assuming that my file is encoded one way when it’s actually encoded in another way, and it doesn’t know what to do with some characters, so it uses a question mark as a default.

Can you detect the file’s encoding type with .NET?

Big thanks to Erich Brunner for pointing out a new bit of information to me about the default encoding type – I’ve updated this post to reflect his helpful steer.

It turns out detecting the file’s encoding type is quite a difficult thing to do in .NET. But as I mentioned earlier, I know I saved this file with the encoding type ANSI selected from the dropdown list in Notepad.

Interestingly, this doesn’t mean that I’ve saved the file as ANSI – this is a misnomer. From the MSDN glossary:

“The term “ANSI” as used to signify Windows code pages is a historical reference, but is nowadays a misnomer that continues to persist in the Windows community. The source of this comes from the fact that the Windows code page 1252 was originally based on an ANSI draft—which became International Organization for Standardization (ISO) Standard 8859-1. “ANSI applications” are usually a reference to non-Unicode or code page–based applications.”

There are a couple of different options open to me here:

Change the file’s encoding type and save it with a specified encoding – e.g. UTF-8, or Unicode.

This is very straightforward – I’ve chosen to just select the UTF-8 option from the dropdown list in NotePad’s ‘Save As…’ dialog box.

save as utf-8

This time when I run the code above, the names are displayed correctly on the console, as shown below.

correct display

Alternatively, try using the StreamReader overload to specify the encoding type.

I can use an overload where I specify the encoding type, which comes from System.Text.Encoding.

var streamReader = new StreamReader(myTeamNamesFile, encodingType)

But what values do these encoding types resolve to? Well that depends on whether I use the .NET Framework or .NET Core. I’ve listed the values below, and notice that the Encoding.Default is different depending on whether you use the .NET Framework or .NET Core.

I’ve highlighted the values for “Encoding.Default“, because this is a special case.

System.Encoding value .NET Framework Encoding Header Name .NET Core Encoding Header Name
Encoding.Default Windows-1252 (on my machine) utf-8
Encoding.ASCII us-ascii us-ascii
Encoding.BigEndianUnicode utf-16BE utf-16BE
Encoding.UTF32 utf-32 utf-32
Encoding.UTF7 utf-7 utf-7
Encoding.UTF8 utf-8 utf-8
Encoding.Unicode utf-16 utf-16

So let’s say I use Encoding.Default in my .NET Framework console application, as shown in the code snippet below.

var streamReader = new StreamReader(myTeamNamesFile, Encoding.Default)

And now my names are now correctly rendered in the Console, as shown in the image below. This makes sense – the text file with my team names was saved with “ANSI” encoding, which we know actually corresponds Windows Code Page 1252. The default encoding on my own Windows machine turns out to also be the 1252 encoding (as highlighted above in red), so I’m instructing my StreamReader to use the 1252 encoding when reading a file which has been encoded as “ANSI” (also known as 1252). They match up, and the text displays correctly.

correct display

Problem solved, right? Well, no, not really.

Microsoft actually do not recommend using Encoding.Default. From docs.microsoft.com:

“Different computers can use different encodings as the default, and the default encoding can change on a single computer. If you use the Default encoding to encode and decode data streamed between computers or retrieved at different times on the same computer, it may translate that data incorrectly. In addition, the encoding returned by the Default property uses best-fit fallback to map unsupported characters to characters supported by the code page. For these reasons, using the default encoding is not recommended.”

If I target .NET Core instead of .NET Framework in my console application – with exactly the same code – I’m back to displaying question marks in my console text.

wrongnames

So even though telling the StreamReader in my .NET Framework console application to use Encoding.Default seems to work, it’s a case of it only working on my machine – it might not work on someone else’s machine. It certainly doesn’t work in .NET Core.

So it seems to me that saving my original text file as UTF-8 or Unicode is a better option.

And as a final reason to save the text file to UTF-8 or Unicode, let’s say I add a new team member, called Łukasz. If I try to save my file with ANSI encoding type, I get this warning:

unicode warning

If I press on and save the file as ANSI, the text for “Łukasz” is changed to “Lukasz” (note the change in the first character). But if I save the file as UTF-8 or Unicode, the name stays the same, including the initial “Ł”.

Wrapping up

It’s pretty common to be asked to read and process text files with non-ASCII characters, and even though .NET provides some really helpful APIs to assist with this, compatibility issues can still occur. This is a complex area, with variations across the .NET Framework and .NET Core runtimes.

I think the best solution is to change the encoding type of the original document to be Unicode or UTF-8, rather than ANSI (more correctly, Windows Code Page 1252). Telling the StreamReader to use Encoding.Default also worked for me, but it might not work on someone else’s machine with different defaults, leading to incorrect translations.


About me: I regularly post about Microsoft technologies and .NET – if you’re interested, please follow me on Twitter, or have a look at my previous posts here. Thanks!