.net, C# tip, IIS, MVC, Non-functional Requirements, Performance, Web Development

More performance tips for .NET websites which access data

I recently wrote about improving the performance of a website that accesses a SQL Server database using Entity Framework, and I wanted to follow up with a few more thoughts on optimising performance in an MVC website written in .NET. I’m coming towards the end of a project now where my team built an MVC 5 site, and accessed a database using Entity Framework. The engineers were all scarred survivors from previous projects pretty experienced, so we were able to implement a lot of non-functional improvements during sprints as we went along. As our site was data driven, looking at that part was obviously important, but it wasn’t the only thing we looked at. I’ve listed a few of the other things we did during the project – some of these were one off settings, and others were things we checked for regularly to make sure problems weren’t creeping in.

Compress, compress, compress

GZip your content! This makes a huge difference to your page size, and therefore to the time it takes to render your page. I’ve written about how to do this for a .NET site and test that it’s working here. Do it once at the start of your project, and you can forget about it after that (except occasionally when you should check to make sure someone hasn’t switched it off!)

Check your SQL queries, tune them, and look out for N+1 problems

As you might have guessed from one of my previous posts, we were very aware of how a few poorly tuned queries or some rogue N+1 problems could make a site grind to a halt once there were more than a few users. We tested with sample data which was the “correct size” – meaning that it was comparable with the projected size of the production database. This gave us a lot of confidence that the indexes we created in our database were relevant, and that our automated integration tests would highlight real N+1 problems. If we didn’t have “real sized data” – as often happens where a development database just has a few sample rows – then you can’t expect to discover real performance issues early.

Aside: Real sized data doesn’t have to mean real data – anonymised/fictitious data is just as good for performance analysis (and obviously way better from a security perspective).

Use MiniProfiler to find other ADO.NET bottlenecks

Just use it. Seriously, it’s so easy, read about it here. There’s even a nuget repository to make it even easier to include in your project. It automatically profiles ADO.NET calls, and allows you to profile individual parts of your application with a couple of simple lines of code (though I prefer to use this during debugging, rather than pushing those profile customisations into the codebase). It’s great for identifying slow parts of the site, and particularly good at identifying repeated queries (which is a giveaway symptom of the N+1 problem).

Reduce page bloat by optimising your images

We didn’t have many images in the site – but they were still worth checking. We used the Firefox Web Developer Toolbar plugin, and the “View Document Size” item from the “Information” menu. This gave us a detailed breakdown of all the images on the page being tested – and highlighted a couple of SVGs which had crept in unexpectedly. These were big files, and appeared in the site’s header, so every page would have been affected. They didn’t need to be SVGs, and it was a quick fix to change it to a GIF which made every page served a lot smaller.

For PNGs, you can use the PNGOut utility to optimise images – and you can convert GIFs to PNG as well using this tool.

For JPEGs, read about progressive rendering here. This is something where your mileage may vary – I’ll probably write more about how to do this in Windows at a future time.

Minifying CSS and JavaScript

The Web Developer Toolbar saved us in another way – it identified a few JavaScript and CSS files issues. We were using the built in Bundling feature of MVC to combine and minify our included scripts – I’ve written about how to do this here – and initially it looked like everything had worked. However, when we looked at the document size using the Web Developer Toolbar, we saw that some documents weren’t being minified. I wrote about the issue and solution here, but the main point was that the Bundling feature was failing silently, causing the overall page size to increase very significantly. So remember to check that bundling/minifying is actually working – just because you have it enabled doesn’t mean it’s being done correctly!

Remember to put CSS at the top of your page, and include JavaScript files at the bottom.

Check for duplicated scripts and remove them

We switched off bundling and minification to see all the scripts being downloaded and noticed that we had a couple of separate entries for the JQuery library, and also for some JQuery-UI files. These were big files and downloading them once is painful enough, never mind unnecessarily doing it again everytime. It’s really worth checking to make sure you’re not doing this – not just for performance reasons, but if you find this is happening it’s also a sign that there’s maybe an underlying problem in your codebase. Finding it early gives you a chance to fix this.

Do you really need that 3rd party script?

We worked hard to make sure that we weren’t including libraries just for the sake of it. There might be some cool UI feature which is super simple to implement by just including that 3rd party library…but every one of those 3rd party libraries includes page size. Be smart about what you include.

Tools like JQuery UI even allow you to customise your script to be exactly as big or small as you need it to be.

Is your backup schedule causing your site to slow down?

I witnessed this on a previous project – one of our team had scheduled the daily database backup to happen after we went home…leading to some of our users in the world in a later time zone to see a performance deterioration for about half an hour at the same time every day. Rescheduling the daily backup to later in the day caused us no problems and removed a significant problem for our users.

Is someone else’s backup schedule causing your site to slow down?

There’s a corollary to the previous point – if you’re seeing a mysterious performance deterioration at the same time every day and you’re absolutely sure it’s not something that you or your users are doing, check if your site is on shared hosting. When I contacted our hosts and requested that our company VMs were moved onto a different SAN, it miraculously cleared up a long-standing performance issue.

Summary

There’s a few tips here which really helped us keep our pages feeling fast to our users (and some other tips that I’ve picked up over the years). We didn’t do all of this at the end of the project, this was something we focussed on all the way through. It’s really important to make sure you’re checking these things during sprints – and part of your Definition of Done if possible.

.net, C# tip, Non-functional Requirements, Performance

Performance tips for database access and Entity Framework

One of the most common ‘gotchas’ in a development project is to forget about performance until there’s a problem. I’ve often heard people quote Knuth saying “premature optimisation is the root of all evil” – hinting that right now is too early to think about performance tuning.

Of course performance tuning and improvement is put off, and put off, and put off some more…until there’s a performance test in pre-production and everything fails. (That’s if you’re lucky – at least you’ve caught it before it goes to production. A lot of the time that’s the first place the issue is spotted).

I believe in making it work first before you make it work fast – but within that statement, there’s an implication that “working” and “working fast” are both necessary. Making it just work isn’t enough. And Knuth is being quoted out of context – the full quote is “We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.” (my emphasis). That’s small efficiencies, not big ones. He also says “In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal and I believe the same viewpoint should prevail in software engineering“. 12%!!

I’d like to share 3 tips that I’ve used to make a huge difference to the performance of a .NET application using Entity Framework. I’ve often heard people criticise Entity Framework as slow, but I’m staying out of the pointless endless religious arguments about whether it is or isn’t. All I can say is that from my experience, the performance bottleneck has never been Entity Framework’s fault – it’s either somewhere else, or the way that Entity Framework has been used.

Missing Indices

This is nothing to do with Entity Framework – this is a change to the database, not the .NET code. Entity Framework generates SQL behind the scenes and sends this to the database for execution, and it has no idea whether this SQL is going to perform a hugely expensive full table scan, or whether it’s going to use indices cleverly to prevent having to search every row in the database.

For me, this is the first port of call when someone says an application accessing a database is slow. SQL Server has some great tools to help with this – you can use SQL Profiler to record a trace file of all the SQL queries hitting a database over a period of time, and then use this trace file in Database Engine Tuning Advisor to identify what indices that the engine thinks will make the biggest difference to your application.

I’ve seen amazing improvements result from this technique – 97% improvements are not uncommon. Again, it’s not really an Entity Framework tip, but it’s worth checking.

The “Select N+1” Problem

So again, not really an Entity Framework problem…yes, there’s a bit of a theme emerging here! This is something that’s common to a lot of ORMs.

Basically I think of the problem as being a side effect of “lazy loading”. For example, say your application queries a database about cars. Cars are represented by a “Car” POCO object, which contains a list of child objects of POCO type “Wheel”.

From your application, you might query by primary key for a car with registration plate “ABC 123”, which (hopefully) returns one object as the result. Then you call the “Wheels” method to get information about the car’s wheels.

If your database is logically normalised, you’ve probably made at least two queries here – the original one to get the car, and then another to get information about the wheels. If you then call a property from the “Wheel” object which makes up the list, you’ll probably make another database query to get that information.

This is actually a massive advantage of ORMs – you as a developer don’t have to do extra work to load in information about child objects, and the query only happens when the application asks for information about that object. It’s all abstracted away from you, and it’s called lazy-loading.

There’s nothing wrong or evil with lazy-loading. Like any tool, it has a place and there are opportunities to mis-use it. Where I’ve seen it misused most is in the scenario where a developer:

  • returns an object from an Entity Framework call;
  • closes the session (i.e. connection to the database);
  • looks in the parent object for a child object, and gets an exception saying the session is closed;

The developer then does one of two things:

  • The developer moves all the logic into the method where the session is open because lazy loading fixes all their problems. This leads to a big mess of code. At some point – always – this code is copied and pasted, usually into a loop, leading to loads and loads of database queries. Because SQL Server is brilliant, it’s probably done all of these queries in a few seconds, and no-one really notices until it’s deployed to production and hundreds of users try to do this all at once and the site collapses. (Ok this is over dramatic – your performance testing events will catch this. Because of course you’re doing performance testing before going to production, aren’t you. Aren’t you?)
  • The better developer realises that moving all the code into one method is a bad idea, and even though lazy loading allows you to do this, it’s mis-using the technique. They read a few blogs, discover this thing called eager loading and write code like this:
var car = (from c in context.Cars.Include("Wheel")
            where c.RegistrationPlate == "ABC 123"
            select c).FirstOrDefault<Car>();

Entity Framework is smart enough to recognise what’s going on here – instead of doing a dumb query on the Car table, it joins to the Wheel table and sends one query out to get everything it needs for the Car and the Wheels.

So this is good – but in my career, almost every application has a much more complex relationship between object and database entities than just one simple parent and child. This leads to much more complex chains of queries.

One technique I’ve used successfully is to create a database view which includes everything needed for the application business method. I like using views because it gives me much more granualar control over exactly what the joins are between tables, and also what fields are returned from the database. It also simplifies the Entity Framework code. But the biggest advantage is that the view becomes an interface – a contract really – between the database and the code. So if you have a DB expert who tells you “Look, your performance issues are down to how your database is designed – I can fix this, but if I do it’ll probably break your application“, you’ll be able to respond “Well, we query the database through a view, so as long as you’re able to create a view that has the same columns and output, you can change the database without affecting us.

Of course if you’re using a database view, that means you won’t be able to update objects using Entity Framework because a view is read-only…which kind of defeats the purpose of using an ORM. However, if you have someone demanding a fix for a slow site, it’s a lot less intrusive to create and index a view than it is to re-engineer the application.

Note: I’m not advocating this as a magic bullet – it’s just a technique that sometimes has its place.

AsNoTracking

This is an Entity Framework setting. If you’re using views – or you know that your Entity Framework call won’t need to update the database – you can get an extra performance boost by using the AsNoTracking keyword.

var cars = context.Cars.AsNoTracking().Where(c => c.Color == "Red");

This will give you a performance boost if you’re returning large volumes of data, but less so for smaller volumes. Your mileage may vary – but remember you need to be sure you aren’t updating the context to use this.

Summary

  • Ignore the wisdom of the newsgroup posts that say “Entity Framework’s just slow, nothing you can do”;
  • Instead, run SQL Server profiler on the database, and put the resulting trace file through SQL Server’s Database Engine Tuning Adviser to find indices that will improve the slowest queries;
  • Analyse the code to identify the “Select N+1” problem – there almost always is one of these in the code somewhere. If you want to find it, turn off lazy loading and run your tests.
  • If you’re returning large volumes of data into a read-only list, see if you can use AsNoTracking to squeeze a bit more performance from your application.

 

 

Non-functional Requirements, Security, Web Development

Use https://securityheaders.io to check your site’s header security in an instant

A while back I posted an article on how to improve the security of your site by configuring headers in IIS.

I thought I’d follow up on this with a quick post about a fantastic utility online – https://securityheaders.io/.

Plug your website URL into this site, and get a report immediately about how good your site headers are, and what you can do to tighten things up. The report is understandable, and every bit of information – whether that’s missing headers, or headers configured insecurely – will have a link to the site creator’s blog explaining what this means in great detail.

Sadly my blog – which is all managed by WordPress.com – comes out with an E rating. How embarrassing…one day I will find the time to host all this on my own domain.

Final hint – as you might expect, if you put https://securityheaders.io into the site, you’ll see what an A+ report looks like!

IIS, Non-functional Requirements

Some useful headers to add in IIS to improve security.

We use the OWASP ZAP tool to do some quick penetration testing on our site. This is a great free tool, and can be used as part of your continuous integration suite.

One of the things it looks for is whether your web application has some useful security related HTTP Headers. OWASP has a good list here and there’s 3 that I think are particularly important for you to configure in IIS.

You can look at headers for a site using http://cyh.herokuapp.com/cyh. This is a really excellent site and application – it highlights headers that it recognises as correctly configured, as well as warning about those which it might be configured wrongly or just plain missing. And even better than that, it recommends what the header should be – very nice constructive criticism!

The suggestions below obviously aren’t comprehensive – just part of what you could/should be doing.

Help Protect against Clickjacking

X-Frame-Options: deny

This makes sure you’re not accidentally rendering content inside a frame that you don’t intend to.

Help Protect against Cross Site Scripting (XSS)

X-XSS-Protection: 1; mode=block

Modern web browsers have some XSS protection built in by default, but having this header on your site is a good belt-and-braces approach to making sure it’s active (in case it has been disabled for some reason).

Help Protect against Drive-by-Downloads

X-Content-Type-Options: nosniff

This makes sure that IE and Chrome won’t look at some content and try to “sniff” the mime-type, which could cause content to be treated as an executable.

Final note – a colleague of mine who’s another Technical Architect suggested this – a way of using Powershell to add these headers to your IIS instance, so this can be part of your continous deployment practice. This way you’ll not forget those headers when you set up a new environment!

.net, C# tip, MVC, Non-functional Requirements, Performance

Performance and MVC tip – Optimize your View Engines

A very quick tip to improve your MVC site’s performance…

If your application only uses one ViewEngine (e.g. Razor), then there’s no point in making MVC iterate through all its registered view engines every time to find a match, when you already know it only needs one.

So make sure you’re only using the Razor view engine by adding the code below to the Application_Start method in your Global.asax.cs file:

ViewEngines.Engines.Clear();
ViewEngines.Engines.Add(new RazorViewEngine());
This should shave a bit of time from your page loads – more in the link here.
Accessibility, Non-functional Requirements

Accessibility and images – is it ever ok to not specify alternative text?

It’s good practice to specify alternative text for images using the “alt” attribute – although it’s unfortunately common to see images without it.

To answer the question in the title – you must always specify alternative text for images.

But as with anything on the web, it’s possible to find conflicting arguments and information about this practice – so to try to improve the credibility of the recommendations in this post, I frequently refer back to W3C.org.

Almost every image on a web page should have some text placed in the “alt” tag (or the long desc) – and this is a Priority 1 item on the W3C checklist.

Examples:
1. Images used as illustrative content, e.g.

<img src="me.jpg" alt="The article's author" />

2. Images used for spacers and bullets – yes, these are mandatory! Even though the image doesn’t add any real content, you can still specify alternative text, e.g.

<img src="bullet.gif" alt="* " />

3. Images used as links – there are a couple of different ways of doing this, which must be handled in different ways. if you provide no link text and the only content of the <a> link is the image, use the “alt” tag to specify a text equivalent, e.g.

<a href="home.html"><img src="home.gif" alt="Home page"/></a>

In the case where both an image and text are specified as the content of a link, repeating the anchor’s text in the “alt” attribute is unnecessary – W3C mandate using a space in this instance, e.g.

<a href="home.html"><img src="home.gif" alt=" "/>Home</a>

So the “alt” tag still has some text, even though it’s a space – you mustn’t omit the “alt” tag.

What about background images in CSS?

I haven’t been able to find information on W3C about how alternative text for background images should be treated. Christian Heilmann argues that images in CSS should be purely aesthetic, and therefore don’t need alternative text. I definitely agree with his argument – your page still needs needs to make sense with CSS switched off.

But what if you are supporting a site where someone has put background images with semantic value into the CSS? Well, you could change the code to bring this data into an <img> tag, and specify an “alt” value in your HTML rather than CSS. But sometimes this might not be possible – in this case, the Yahoo Developer Network recommends using ARIA-roles, which enables screen-readers to recognise your ARIA enhanced element as an image.

<div role="img" aria-label="The article's author">

I hope this article helps you improve the accessibility of your site.

Accessibility, Non-functional Requirements

Accessibility – specifying the language of your page

I’m going to write a few posts on specific things you can do to improve the accessibility of a webpage.

You can identify the primary natural language of a document by making a simple change to your HTML element:

<HTML lang="en">

That’s it – now your site complies with Checkpoint 4.3 of the W3C Accessibility checklist by identifying the primary language of a document.

This very simple tip improves the accessibility of your page – it’s presently Priority 3, so if only mandatory if you’re aiming at AAA compliance. But it’s good practice, a small change and low risk – why not add it to your master page?