I wrote a post recently introducing some of the work I’ve been doing with Azure DevOps, and ensuring my websites, tests, infrastructure and processes are stored as code in GitHub. This post stretches a bit beyond that introduction, and is about creating a multi-stage pipeline in YAML. I’m going to include lots of screenshots to help you follow along with what I’ve done.
I’m going to try to describe how to do small tasks in each of my next few posts – trying to do everything at once would just be overwhelming.
A multi-stage pipeline
In the last post I’ve described a sample scenario – a client who wants a demonstration of the work I’ve done for them website, and also values stability and system quality. I’ve suggested the architecture below:
In order to implement this, I can imagine 8 stages in my pipeline:
- Build and Test the website pushed to source code;
- Build the Integration Environment;
- Deploy the website to the Integration Environment;
- Build the Test Environment;
- Deploy the website to the Test Environment;
- Build the Demo Environment;
- Deploy the website to the Demo Environment.
And as the 8th stage, I’d like a system quality test (perhaps vulnerability scanning or automated page accessibility checking) to complete after the website is deployed to the Integration Environment.
Let’s see how we can create logical stages in the Azure Build Pipeline to manage each of these operations.
Creating the YAML skeleton in Azure DevOps
As I said earlier, I’m going to build this architecture a bit at a time. In this post, I’m only going to cover creating a skeleton of my build pipelines – creating a series of logically named stages with dependencies.
In the following posts I’ll start populating these stages with tasks like creating resource groups, building infrastructure, running tests and deploying websites to the infrastructure.
First let’s create the source code repository
I’ve created a publically available GitHub repository called EverythingAsCode which is where I’ll store all of my code.
I could have done this in Azure DevOps also, but I’ve chosen work with GitHub this time.
Create a new project in Azure DevOps
The next step for me to log into Azure DevOps and create a project. This is very straightforward – the normal pre-requisite is to set up an account and organisation, and log on to https://dev.azure.com/.
When I’ve logged in, I can see a “Create project” button at the top right of the screen, as shown below.
This opens up a dialog window where I can specify the project name and its visibility.
After clicking on “Create”, after a short while the project is created and I can see an empty template.
Follow the steps in the wizard to create, save and run a single stage Build Pipeline
I click on the Pipelines option in the left navigation menu and see the screen below:
Remember from the first post in this series that I’m using the multi-stage pipeline preview feature – it’s very simple to switch this on and there’s instructions on how to do this in the first part.
I click on “Create Pipeline”, and I’m taken into a wizard to guide me through the process. The first step I’m asked to complete is to choose where my source code lives – I select GitHub because that’s where my “EverythingAsCode” repo lives, but there’s a good selection of alternative options as shown below:
When I select the “GitHub” option, the next page shows me my GitHub repositories (I’m logged into GitHub so DevOps knows which ones belong to me). I select the “EverythingAsCode” repo.
After selecting this repo, I’m redirected to GitHub where I’m asked to install the Azure Pipelines app for my repo – I scroll to the bottom and hit “Approve and Install”.
After approving the install, I’m redirected back to my DevOps organisation to carry on with the Wizard.
I selected “Starter pipeline”, and I’m taken to the screen shown below – DevOps is showing me a YAML editor, with some simple steps that write text to the standard output.
I hit the “Save and run” button, and then I’m told that this YAML file will be saved in the root of my repository – this is great. My first pipeline is being saved as code in GitHub.
After I hit “Save and Run”, the code is pushed to my GitHub repo, and the sample pipeline starts running. After clicking on “Pipelines” in the left hand nav and selecting my pipeline, I can see the progress (as shown below).
And if I click on the running job, I’m taken to another screen where I can see the individual steps within the job running. You can see in the screen below how the script has written “Hello, world” to the standard output.
That’s my first fully coded pipeline completed.
And finally, when I browse to the repository in GitHub, I can see that the YAML file has been saved as code in my source code repository, as shown above.
So that’s single stage – what about multi-stage?
Let’s edit our sample pipeline through the Azure DevOps YAML editor.
First I click on Pipelines in the left hand navigation menu, and see a list of pipelines for the project.
The area shown below in the red box above is a clickable area, so I click on this to see the pipeline detail, as shown below. On this detail page, there’s an “Edit” button in the top right (highlighted in a red box in the image below).
After clicking on “Edit”, I’m taken to a screen which has a rich text editor that allows me to modify my YAML file.
So the starter pipeline was useful to test things were working, but it’s not really want I want. I pasted in the code shown below. This has just three stages right now (build the website, build the infrastructure, and deploy the website to the infrastructure), each of which presently just echo text to standard output.
trigger: - master stages: - stage: build displayName: 'Build and test the website' jobs: - job: run_build pool: vmImage: 'Ubuntu 16.04' steps: - script: echo Build displayName: 'First step in building the website skeleton' - stage: build_integration displayName: 'Build the integration environment' dependsOn: build jobs: - job: create_infrastructure pool: vmImage: 'Ubuntu 16.04' steps: - script: echo Build Integration Infrastructure displayName: 'First step in building the integration infrastructure' - stage: deploy_to_integration displayName: 'Deploy the website to the integration environment' dependsOn: build_integration jobs: - job: deploy_artefacts_to_integration pool: vmImage: 'Ubuntu 16.04' steps: - script: echo Deploy Website to Integration displayName: 'First step in deploying the website to integration'
And when I save and run this, I can see my pipeline has changed in the screen below – I now have three distinct stages in my pipeline, and I can populate each of these with tasks like creating resource groups and deployiing infrastructure.
Let’s take this further – and in addition to creating stages for my infrastructure build and release, earlier I also mentioned I’d like stages for the Test and Demo environments also.
I’ve added code into the azure-pipelines.yml file which is available in GitHub – rather than posting pages and pages of code, I recommend you check it out over at GitHub.
When I run my updated code through Azure DevOps, I can now see many more stages in my pipeline, as shown below.
So far it’s a very linear series of stages. But we’ve only done 7 of our 8 stages – what about the 8th, the one where I wanted to run my system quality tests? I want that to depend on the third stage (where I deploy my website to the integration infrastructure) but I don’t want any other stage to depend on this 8th stage (yet, anyway).
It turns out this is quite easy – I can just add in another stage to the YAML, specify what stage it depends on (highlighted in red in the code sample below), and make sure that no other stage depends on this one.
- stage: run_system_quality_tests displayName: 'Run the system quality tests' dependsOn: deploy_to_integration jobs: - job: run_non_functional_tests pool: vmImage: 'Ubuntu 16.04' steps: - script: echo Run system quality tests displayName: 'Running the system quality tests'
And now my pipeline has two stages after deploying the website to integration, with one being an end point to that fork, and the other continuing on to build and deploy to the Test and Demo instances.
This post has been about creating multi-stage YAML for an Azure Build Pipeline. Next time I’ll write about how to populate these stages with useful Azure tasks.