I wrote a post recently introducing some of the work I’ve been doing with Azure DevOps, and ensuring my websites, tests, infrastructure and processes are stored as code in GitHub. This post stretches a bit beyond that introduction, and is about creating a multi-stage pipeline in YAML. I’m going to include lots of screenshots to help you follow along with what I’ve done.

I’m going to try to describe how to do small tasks in each of my next few posts – trying to do everything at once would just be overwhelming.

A multi-stage pipeline

In the last post I’ve described a sample scenario – a client who wants a demonstration of the work I’ve done for them website, and also values stability and system quality. I’ve suggested the architecture below:

reference-architecture-diagram-2

In order to implement this, I can imagine 8 stages in my pipeline:

  1. Build and Test the website pushed to source code;
  2. Build the Integration Environment;
  3. Deploy the website to the Integration Environment;
  4. Build the Test Environment;
  5. Deploy the website to the Test Environment;
  6. Build the Demo Environment;
  7. Deploy the website to the Demo Environment.

And as the 8th stage, I’d like a system quality test (perhaps vulnerability scanning or automated page accessibility checking) to complete after the website is deployed to the Integration Environment.

Let’s see how we can create logical stages in the Azure Build Pipeline to manage each of these operations.

Creating the YAML skeleton in Azure DevOps

As I said earlier, I’m going to build this architecture a bit at a time. In this post, I’m only going to cover creating a skeleton of my build pipelines – creating a series of logically named stages with dependencies.

In the following posts I’ll start populating these stages with tasks like creating resource groups, building infrastructure, running tests and deploying websites to the infrastructure.

First let’s create the source code repository

I’ve created a publically available GitHub repository called EverythingAsCode which is where I’ll store all of my code.

I could have done this in Azure DevOps also, but I’ve chosen work with GitHub this time.

Create a new project in Azure DevOps

The next step for me to log into Azure DevOps and create a project. This is very straightforward – the normal pre-requisite is to set up an account and organisation, and log on to https://dev.azure.com/.

When I’ve logged in, I can see a “Create project” button at the top right of the screen, as shown below.

create project

This opens up a dialog window where I can specify the project name and its visibility.

create devops project

After clicking on “Create”, after a short while the project is created and I can see an empty template.

empty project

Follow the steps in the wizard to create, save and run a single stage Build Pipeline

I click on the Pipelines option in the left navigation menu and see the screen below:

first build pipeline

Remember from the first post in this series that I’m using the multi-stage pipeline preview feature – it’s very simple to switch this on and there’s instructions on how to do this in the first part.

multistagepipelines

I click on “Create Pipeline”, and I’m taken into a wizard to guide me through the process. The first step I’m asked to complete is to choose where my source code lives – I select GitHub because that’s where my “EverythingAsCode” repo lives, but there’s a good selection of alternative options as shown below:

where is your code

When I select the “GitHub” option, the next page shows me my GitHub repositories (I’m logged into GitHub so DevOps knows which ones belong to me). I select the “EverythingAsCode” repo.

select a repository

After selecting this repo, I’m redirected to GitHub where I’m asked to install the Azure Pipelines app for my repo – I scroll to the bottom and hit “Approve and Install”.

azurepipelines

After approving the install, I’m redirected back to my DevOps organisation to carry on with the Wizard.

configure your pipeline

I selected “Starter pipeline”, and I’m taken to the screen shown below – DevOps is showing me a YAML editor, with some simple steps that write text to the standard output.

review your yaml

I hit the “Save and run” button, and then I’m told that this YAML file will be saved in the root of my repository – this is great. My first pipeline is being saved as code in GitHub.

save and run

After I hit “Save and Run”, the code is pushed to my GitHub repo, and the sample pipeline starts running. After clicking on “Pipelines” in the left hand nav and selecting my pipeline, I can see the progress (as shown below).

first run

And if I click on the running job, I’m taken to another screen where I can see the individual steps within the job running. You can see in the screen below how the script has written “Hello, world” to the standard output.

build output

That’s my first fully coded pipeline completed.

github with pipeline

And finally, when I browse to the repository in GitHub, I can see that the YAML file has been saved as code in my source code repository, as shown above.

So that’s single stage – what about multi-stage?

Let’s edit our sample pipeline through the Azure DevOps YAML editor.

First I click on Pipelines in the left hand navigation menu, and see a list of pipelines for the project.

pipelines

The area shown below in the red box above is a clickable area, so I click on this to see the pipeline detail, as shown below. On this detail page, there’s an “Edit” button in the top right (highlighted in a red box in the image below).

edit pipelines

After clicking on “Edit”, I’m taken to a screen which has a rich text editor that allows me to modify my YAML file.

edit yaml

So the starter pipeline was useful to test things were working, but it’s not really want I want. I pasted in the code shown below. This has just three stages right now (build the website, build the infrastructure, and deploy the website to the infrastructure), each of which presently just echo text to standard output.

trigger:
- master

stages:
- stage: build
  displayName: 'Build and test the website'
  jobs:
  - job: run_build
    pool:
      vmImage: 'Ubuntu 16.04'
    steps:
    - script: echo Build
      displayName: 'First step in building the website skeleton'

- stage: build_integration
  displayName: 'Build the integration environment'
  dependsOn: build
  jobs:
  - job: create_infrastructure
    pool:
      vmImage: 'Ubuntu 16.04'
    steps:
    - script: echo Build Integration Infrastructure
      displayName: 'First step in building the integration infrastructure'

- stage: deploy_to_integration
  displayName: 'Deploy the website to the integration environment'
  dependsOn: build_integration
  jobs:
  - job: deploy_artefacts_to_integration
    pool:
      vmImage: 'Ubuntu 16.04'
    steps:
    - script: echo Deploy Website to Integration
      displayName: 'First step in deploying the website to integration'

And when I save and run this, I can see my pipeline has changed in the screen below – I now have three distinct stages in my pipeline, and I can populate each of these with tasks like creating resource groups and deployiing infrastructure.

building with 3 stages

Let’s take this further – and in addition to creating stages for my infrastructure build and release, earlier I also mentioned I’d like stages for the Test and Demo environments also.

I’ve added code into the azure-pipelines.yml file which is available in GitHub – rather than posting pages and pages of code, I recommend you check it out over at GitHub.

When I run my updated code through Azure DevOps, I can now see many more stages in my pipeline, as shown below.

lots more stages

So far it’s a very linear series of stages. But we’ve only done 7 of our 8 stages – what about the 8th, the one where I wanted to run my system quality tests? I want that to depend on the third stage (where I deploy my website to the integration infrastructure) but I don’t want any other stage to depend on this 8th stage (yet, anyway).

It turns out this is quite easy – I can just add in another stage to the YAML, specify what stage it depends on (highlighted in red in the code sample below), and make sure that no other stage depends on this one.

- stage: run_system_quality_tests
  displayName: 'Run the system quality tests'
  dependsOn: deploy_to_integration
  jobs:
  - job: run_non_functional_tests
    pool:
      vmImage: 'Ubuntu 16.04'
    steps:
    - script: echo Run system quality tests
      displayName: 'Running the system quality tests'

And now my pipeline has two stages after deploying the website to integration, with one being an end point to that fork, and the other continuing on to build and deploy to the Test and Demo instances.

added system quality step

Wrapping up

This post has been about creating multi-stage YAML for an Azure Build Pipeline. Next time I’ll write about how to populate these stages with useful Azure tasks.

6 thoughts on “Everything as Code with Azure DevOps Pipelines: C#, ARM, and YAML: Part #2 – multi-stage builds in YAML

  1. Great post. I’ve also been messing with the new multi-stage pipelines. Here is what I’ve found:

    1. You don’t seem to need to add `dependsOn: build_integration` as it does this automatically. It assumes that one stage leads to the next. I think you only need `dependsOn` if you have a tree like structure.
    2. Documentation is woeful. I wish I’d seen your post a week ago.
    3. I had the added complexity of trying to run a matrix build on Windows, Mac and Linux to publish a NuGet package. For this there are special `deployment` jobs which are different from standard jobs in that they have an environment to deploy to and a deployment strategy.

    You can see my azure-pipelines.yml file here:

    https://github.com/RehanSaeed/Schema.NET/blob/master/azure-pipelines.yml

    1. Hey, thank you for the comment! That’s useful information. I totally agree, the documentation needs to get better – I burned a lot of my 1800 free minutes trying to work out how to do it 🙂

  2. My team is in the process of migrating to exclusively use YAML pipelines. We got tired of the UI changing on us so frequently.
    Given the volume of jobs/projects that we have, we’ve found the template features to be incredibly beneficial.

    Here’s my process for creating one from scratch:
    1) create a typical pipeline using the GUI
    2) In the GUI pipeline editor, click the View Yaml button (this gets us the basic syntax quickly)
    3) Go through the YAML and replace hardcode values with variables (this lets you abstract out what’s really different between pipelines, and can set defaults as feasible)
    4) Convert that to a template file and store in a central location
    5) Create a basic YAML file that calls into the template and passes the variables as needed.

    Once #5 is done, you can then reuse the file you created for that in every other similar pipeline without having to recreate the whole thing.

    1. I agree with that approach, I’m doing that too (I’ve written about that process in Part #3 of this series).

      I’m really looking forward to being able to manually approve progress through stages, as I need this before I can introduce it to my production pipelines.

Comments are closed.