Create a scalable private deployment pool

Introduction

Sometimes you may find yourself in a scenario where you need a private deployment pool. For whatever reason, the Microsoft Hosted agent, cannot work for your organization. Notable examples of this could be requirements for specific software on the agent, or perhaps network restrictions. This post covers how to deploy private Azure Pipeline Agents 🚀. In addition, it covers how to keep the agents up-to-date and scale the agents when needed.

Using container instances

It’s possible to run an agent in a docker container. Container Instances are relatively cheap (compared to Virtual Machines) and are easier to create and destroy ♻️. When running Azure agents, keeping the agents up-to-date, and easy to manage is key in the devops process. After all, if the agents are offline, you cannot deploy to your environment.

The prep work 📝

Microsoft supplies all the needed documentation 👍, and they do it way better than I could ever write down in this blog, therefore I won’t try to replace the documentation, but rather give you a way to efficiently deploy according to the best practices. For this we will need the following components.

• A Devops Subscription (and an Azure subscription)
• A Microsoft Hosted Pipeline to deploy the private agent
• An Azure Container Registry

Start out by creating a variable group that we will use to deploy the bicep file. I like to name my variable groups as follows:

• subscription-{subname}: for all shared variables that are needed ot deploy, such as service connections
• {reponame}: for all variables, used in ARM templates, or environment variables for programs
• {reponame}-secrets: for all secrets

The variable groups can be found under Pipelines -> Library. Refer to the screenshot below.

Deploying the agents 🏎️

After creating the needed variable groups add the variables., azpPool, azpUrl, containerAmount, deploymentPipelineName, imageBuildTag, imageName, imageRegistry, subnetAddressPrefix, vnetAddressPrefix and lastly, the secret in the secret variable group: azpToken. The bicep file contains a description for each of these and how to get these variables.

If you have saved the Personal Access Token (PAT) anywhere else than the secret variable, it should be considered compromised and revoked!

Please know that this token is very powerful, if the token is read, saved or shared with anything other than the secret in the variable group, it should be considered compromised and revoked! It’s easy to create a new token, it’s not easy to redeploy your entire Azure environment if the token is misused!

Now copy the example repo from here https://github.com/yetanothercloudblog/private-deployment-agents , set up the Azure Pipeline with the existing YAML file. You can adjust as needed for your organization. It should automatically create the Azure Agents in a subscription and resource group of your choice.

Set up and validate automated updates 🕐

Docker containers should be short lived. The reason for this is that (security) updates are applied when building the container, and never afterwards. For obvious reasons you should keep your systems up-to-date at all times. Therefore, the azure-pipelines file contains a schedule that can be tweaked to your liking. By default it is set to deploy a fresh container every week. If you need another schedule, use the following syntax:

"0 0 * * *"  > Daily at midnight
"0 12 * * 0" > Every Sunday at 12:00.
"0 12 1 * 0" > Every month on the first day at 12:00

Did not find the correct syntax that you need? Have a look at this great website: https://crontab.guru/ which translates the syntax to a human readable form for you.

Caviats

There are a few considerations here, with the current way of deploying things.

1. The Pipeline required to deploy the private agent will only run on a Microsoft Hosted agent. Or you will require 2 private pools. A private pool cannot update itself, as it literally overwrites itself halfway during the deployment.
2. With that in mind. If you have any running jobs during the auto update time, the jobs will be killed and the pipelines associated with it will fail. Make sure to not schedule runs when the redeployment process is ongoing.

Manual workaround for the schedule

You can create a deployment gate for your environment. You can set this gate to manual approval, which means that the new agent will never be deployed automatically. Read more about this here: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/approvals?view=azure-devops&tabs=check-pass

Conclusion

Running and scaling Azure Devops agents is easy when they are automated. You can use schedules to run updates. Throw away your VM scale set and replace it with these Docker agents today. 🚀🚀