Invoke TFS 2015 REST API from build- and release-tasks

I ran into a technical challenge writing a custom task for TFS 2015, which I solved by calling the TFS 2015 REST API from my build- and release task. I’ll try to explain why and how I did that in this post. There might be other options as well, please feel free to share them with me.

Background of the problem
I was writing a task for Release Management where I needed to update a file that had already been published to my artifact file share during the build. This XML file is intended to contain all kinds of information regarding the build but I wanted to expand its use with Release-information as well. That way we can easily track build-and-release information from the central deployment-repository. I found that there is no fixed variable available during the release which contains the artifact location! I assume that will be exposed in a next version, but I did not want to wait for it so this is what I came up with.

The solution for my problem
As mentioned: I want to update a file on the artifact share, but as we will have 400+ build definitions soon, each of the developers could potentially enter other artifact paths for their builds, so I am not able to just “assume” the path where their artifacts will be written to (although we do have solid agreements of the file-and-folder structure). I read the documentation for TFS 2015 REST API and after some attempts on my own test-environment I finally found the REST-method that would provide me with the exact artifact information. After I managed to find the specific details directly in the browser, I only needed to adjust my existing task to automate it.

1. Update the task to connect to the API
The API-path that I was specifically interested in: http://{tfslocation}/{teamProjectName}/_apis/build/builds/{buildId}/artifacts. This request will provide me with a list of all artifacts that the build created. In my case I need to find an artifact based on a fixed filename, so that should not be too hard. I did find it difficult to connect without having to enter a fixed username and password combination though. I found a few examples on the web about authentication on the Invoke-RestMethod function, which worked fine technically, but they were all using a username and password string: that’s not something I’d like to use in a business-environment. After several attempts, these two method worked, and I chose to use the latter one.

$apiPath = "{0}{1}/_apis/build/builds/{2}/artifacts" -f ($Env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI, $Env:SYSTEM_TEAMPROJECT, $Env:BUILD_BUILDID)
Write-Host "Now calling REST API to get Artifact-details: $apiPath" 
			
# First option: add the credentials to the header
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "aUserName", "aSuperSecretPassword")))
$jsonFromTfs = Invoke-RestMethod -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)} -Uri $apiPath
			
# Second option: UseDefaultCredentials, will authenticate with the account that runs the task.
$jsonFromTfs = Invoke-RestMethod -Uri $apiPath -UseDefaultCredentials

2. Use build variables to get the exact data
As you could already see in the PowerShell-example above, there are some variables that you can use to create the exact path to your TFS-instance. This way, even when you would migrate to another TFS, or if hostnames would change, etc. the script still works. The Uri, the Projectname en BuildId were the ones I needed for this specific REST-request.

3. Adjust the task so it can run in both Build and Release
I discovered that the on-premise TFS 2015 (we run on Update 3) does not seem to take the “visiblity” property of a task into account, meaning that a task can have that property set to Build or Release or both, and it will still be shown in the Add-task dialog in both the Build and Release pages. Therefor I decided to make my tasks compatible with both scenarios. To do so I simply check the HOSTTYPE-variable as illustrated below.

if ($Env:SYSTEM_HOSTTYPE -eq "build") { Write-Host "Task runs in BUILD" } 
if ($Env:SYSTEM_HOSTTYPE -eq "release")  { Write-Host "Task runs in RELEASE" }

Working PowerShell-snippet
The following snippet works for me. Of course there is more to it then just this code, but it probably makes clear how I call the API and read the data that it returns. In my case I would open the filename that the API returns, read its data and add more data along the way, but that is not what this post is about. If you’re interested in the task, just let me know.

$apiPath = "{0}{1}/_apis/build/builds/{2}/artifacts" -f ($Env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI, $Env:SYSTEM_TEAMPROJECT, $Env:BUILD_BUILDID)
Write-Host "Now calling REST API to get Artifact-details: $apiPath" 

$jsonFromTfs = Invoke-RestMethod -Uri $apiPad -UseDefaultCredentials

if ($jsonFromTfs.value[0] -ne $null)
{
	$artifactName = $jsonFromTfs.value[0].name
	$artifactPath = $jsonFromTfs.value[0].resource.data
	$artifactTotal = "{0}{1}\{2}" -f ($artifactPath, $artifactName, $artifactFile)
				
	Write-Host "We found the paths, now see if the file can be located."
	if (Test-Path $artifactTotal)
	{
		Write-Host "File was found! Now update Release-information ($artifactTotal)"
	}
}

If you need additional information about these tasks, don’t hesitate to contact me. Hope this helps other DevOps out there as well.

Writing a custom TFS 2015 build task

At my current assignment I am working purely in a DevOps role. The biggest chunk of my day-to-day work is the migration of an on-premise TFS 2013 instance to on-premise TFS 2015, including process template changes, migration to the new build eco-system (so moving away from XAML templates) and introducing new functionality as SonarQube etc. As the new build-system is fairly new, there is not that much information on the web yet about writing your own build steps, so I thought it could be helpful to others as well if I wrote down my findings.

TFS 2015 Build Definition
TFS 2015 Build Definition

For all publishing activities of my custom build-steps, I use the TFS CLI which can be downloaded here: https://github.com/Microsoft/tfs-cli.

The basics of custom buildsteps
A build-step in TFS2015 technically consists of a single folder with a manifest (task.json) which you use to configure the properties, an executable Powershell script that performs the actual task during the build and an icon file that is displayed in the list of build-steps for your definition. To make it easy for you, a command is available to generate a blueprint with you need at once:

tfx build tasks create --auth-type basic

It all starts with defining the task: you need to specify the form fields needed to configure the task, by adding textboxes, checkboxes, file- and folder-pickers and picklists (selectboxes). Each form-field can have a help-text which is a handy feature to assist the user to fill the fields with the appropriate values. Make sure to use a unique id for each of your tasks (GUID) and after each change you make to the task.json file, update the version number to make sure each of the buildagents will update their local copies of the task. If you re-use the same versionnumber, local copies of the script are not being updated, hence strange behaviour will occur. Here’s the code for the example I will use in the rest of this post:

{
	"id": "CD6A4B69-CE6A-4A09-A866-FEA612B80702",
	"name": "RobinPaardekamDebug",
	"friendlyName": "Robin Paardekam Debug Task",
	"description": "Expand variables that are available during the build and try all kinds of stuff locally.",
	"author": "Robin Paardekam, Studio 010 - Digitale Media",
	"helpMarkDown": "RobinPaardekamDebug v0.0.1. This buildstep will write all environment variables and their values so you can easily find the one that you are looking for. There are also some other controls on this page, but they are just intended for demo-purposes. Nothing is mandatory on this task.",
	"category": "Utility",
	"visibility": [
		"Build",
		"Release"
	],
	"demands": [ ],
	"version": {
		"Major": "0",
		"Minor": "0",
		"Patch": "1"
	},
	"groups": [
	{
		"name": "Dynamic",
		"displayName": "Dynamic Lists (demo)",
		"isExpanded": false
	}],
	"minimumAgentVersion": "1.95.0",
	"instanceNameFormat": "Display all build environment variables",
	"inputs": [
	{
		"name": "connectedServiceTfs",
		"type": "connectedService:Generic",
		"label": "TFS Service Endpoint",
		"required": false,
		"helpMarkDown": "Create an endpoint to TFS if you want to populate the pickLists in the Dynamic Lists section. E.g. for Test: http://tfstest/tfs"
	},
	{
		"name": "TfsProject",
		"type": "pickList",
		"label": "TFS Project",
		"defaultValue": "",
		"required": false,
		"groupName": "Dynamic",
		"helpMarkDown": "This list is populated with Projectnames from the TFS REST API. You are allowed to enter another value as well.",
		"properties": {
			"EditableOptions": "True"
		}
	},
	{
		"name": "TfsProjectTeam",
		"type": "pickList",
		"label": "TFS Project.Team",
		"defaultValue": "",
		"required": false,
		"groupName": "Dynamic",
		"helpMarkDown": "This list is populated with Teamnames for the selected Team Project from the TFS REST API. You are not allowed to enter another value.",
		"properties": {
			"EditableOptions": "False"
		}
	},
	{
		"name": "TfsProjectBuilddef",
		"type": "pickList",
		"label": "TFS Project.BuildDefs",
		"defaultValue": "",
		"required": false,
		"groupName": "Dynamic",
		"helpMarkDown": "This list is populated with Build Definitions for the selected Team Project from the TFS REST API. You are not allowed to enter another value.",
		"properties": {
			"EditableOptions": "False"
		}
	}],
	"sourceDefinitions": [
	{
		"target": "TfsProject",
		"endpoint": "http://tfsserver/tfs/DefaultCollection/_apis/projects#$(connectedServiceTfs)",
		"selector": "jsonpath:$.value[*].name",
		"authKey": "$(connectedServiceTfs)"
	},
	{
		"target": "TfsProjectTeam",
		"endpoint": "https://tfsserver/tfs/DefaultCollection/_apis/projects/$(TfsProject)/teams#$(connectedServiceTfs)",
		"selector": "jsonpath:$.value[*].name",
		"authKey": "$(connectedServiceTfs)"
	},
	{
		"target": "TfsProjectBuilddef",
		"endpoint": "https://tfsserver/tfs/DefaultCollection/$(TfsProject)/_apis/build/definitions#$(connectedServiceTfs)",
		"selector": "jsonpath:$.value[*].name",
		"authKey": "$(connectedServiceTfs)"
	}],
	"execution":
	{
		"PowerShell": {
			"target": "$(currentDirectory)\\executeTask.ps1",
			"argumentFormat": "",
			"workingDirectory": "$(currentDirectory)"
		}
	}
}

As you can see in the final block of the manifest, a PowerShell file is defined that will run when this task is started. Now you should start writing the actual logic for the task, so you will need to receive all input from the build-step and start processing it in your own piece of code. At my client we chose to only use PowerShell but you are also allowed to use NodeJS if you plan to target other platforms then Windows. One thing to be aware of: each of the input fields that are defined in the task manifest should be defined as ‘input parameters’ for the executable script. The task will fail when it is being run when one of the fields is not received by the build step itself. For this specific example, I used the following PowerShell script:

[cmdletbinding()]
param (
    [string]$cwd,
    [string]$connectedServiceTfs,
    [string]$TfsProject,
    [string]$TfsProjectTeam,
    [string]$TfsProjectBuilddef
)

Write-Verbose "Importing modules"
import-module "Microsoft.TeamFoundation.DistributedTask.Task.Internal"
import-module "Microsoft.TeamFoundation.DistributedTask.Task.Common"

Write-Host "INPUT (User) workingDir: $cwd"
Write-Host "INPUT (User) TfsProject: $TfsProject"
Write-Host "INPUT (User) TfsProjectTeam: $TfsProjectTeam"
Write-Host "INPUT (User) TfsProjectBuilddef: $TfsProjectBuilddef"

$environmentVars = get-childitem -path env:*

foreach($var in $environmentVars)
{
 $keyname = $var.Key
 $keyvalue = $var.Value
 
 Write-Output "${keyname}: $keyvalue"
}

Write-Host ("##vso[task.complete result=Succeeded;]DONE")

exit 0

The script above will display all server-variables that are available at build-time, which is a neat little helper when you are working on writing new tasks. The pickList values from the task are not actually used, their values are just being displayed as this is just a demo.

Passing variables between build steps
Once you have created one or more buildsteps for your business-needs, you will likely run into the situation that (string-)output from one step is needed as input in a following step. When I worked on TFS 2013 build workflows this wasn’t an issue at all: you create a variable in the correct scope and it is available for you to consume it. However: a task runs in its own context so the regular PowerShell variables are not persistent through out the rest of the build-process. There are some tricks to fix that: using VSO Logging Commands, I found the command below, which enabled me to access the parameter in a next buildstep.

Write-Host "##vso[task.setvariable variable=VariableName;]VariableValue"

In a following step you can access this variable using the statement in a way as illustrated below:

Write-Host "The following variable was found: $(VariableName)"

The above and many more task logging commands are documented here. Another cool command I hope to discuss on short term is the “task.addattachment” which can be used to add specific content on the build output front-page (aka. timeline).

Consuming data from REST APIs
One of the coolest things I have done with my custom build-steps so far is getting data from external sources to be displayed in the picklists of my build-steps. E.g. I have some configuration values that are stored in a custom made application that are needed to define how the build-step behaves, or another example (used in the script above) is to fill the pickLists with values that are coming from TFS itself. Let’s look a bit closer on how to achieve this:

The “sourceDefinitions” block in the manifest file defines where to get the data from. However: they are dependent on the Service Endpoint defined as the first inputfield, called “connectedServiceTfs”. This control will show all Service Endpoints configured for the current Team Project (unfortunately there is not an easy way to share Service Endpoint between Team Projects at this moment). If you haven’t configured any Service Endpoints yet (the pickList filters for Generic Endpoints only) you can do so using the manage-link at the righthand side.

TFS 2015 Generic Endpoint TFS Rest API
TFS 2015 Generic Endpoint TFS Rest API

As illustrated in the image above, you need to name the endpoint (so you can easily distinguish them when your list grows in the future) and you need to add a service address, username and password. As this blogpost is only addressing on-premise TFS 2015, this is all you will need. For VSTS you’ll need to do more manual preparations to use Personal Access Tokens etc but that is out of scope for this particular example. In this case, the main thing is to save the credentials so the task-page can easily connect once you define your build.

Now that you have defined your Generic Service Endpoint, the task should be able to populate the fields defined in your manifest: TfsProject, TfsProjectTeam and TfsProjectBuilddef. When I started writing the task I assumed that I could simply request the Service address from the selected endpoint, avoiding the fixed paths in my manifest. Unfortunately I did not manage to get the endpoint address, probably for security purposes. For now I can live with the hardcoded paths, although it would be great to dynamically change the source of the pickList, based on the endpoint. If you can come up with a fix for this, I am looking forward to any response!

TFS 2015 Custom Build Task
TFS 2015 Custom Build Task

The task contains 1 additional groupbox for the dynamic pickLists. Once you have selected a Team Project from the first list, the other 2 boxes can also be populated accordingly. Using the Webdeveloper tools in Internet Explorer or Chrome (press F12) you can troubleshoot any issues when your lists remain empty. As mentioned earlier: there is no real use in this task currently for these 3 pickList values, but I think it shows how easily you can consume a REST service from your task page.

As you see in the selector-property in the manifest, you have to define an jsonpath for selecting the appropriate values. If you’ve got experience with XML and XPath this shouldn’t be too much of a problem. It might help to use a site like JsonPath.com to debug your specific rest-calls.

Adding information to buildoutput frontpage
As mentioned earlier in this post, another cool VSO Command is the “task.addattachment”. It allows you to
upload and attach summary markdown to current timeline record. It took me some trial-on-error to get this working, but eventually I got it working as expected. The following snippet will create a local text-file with the information I want to display on the build-frontpage and upload and attach it to the timeline.

### Add additional information to the timeline.
  $logFilePath = "{0}\BuildDemo_Front_{1}.txt" -f ($Env:BUILD_BINARIESDIRECTORY, $Env:BUILD_BUILDID)
  $text = "This is just some not-important content to show how to add Log-output to the build-dashboard. Make it useful!"
  $text | Set-Content "$logFilePath"
  Write-Host "##vso[task.addattachment type=Distributedtask.Core.Summary;name=Demo Output On Timeline;]$logFilePath"
### End additional information on timeline script

The snippet above will add a new caption on your timeline called “Demo Output On Timeline” followed by the text that you’ve declared in the $text-variable. It can be useful for adding build-specific URLs, network-paths etc. so your users won’t have to dig through the (verbose-)logs but can easily click the links on the timeline. Note that you can use markdown-syntax here!

Uploading your new task to TFS
Once you have all your files ready, you can publish the task to your TFS-application server. Note that you will need to enable Basic Authentication on the tfs Virtual Directory in IIS as that is currently the only way you can talk to the tfs-API when running TFS2015 on-premise. The following command will start publication of your task (first answer the three questions “Address of your TFS instance”, “Username”, “Password”) to the application server, making your build-step available when editing a definition. Just to be clear: I am NOT addressing any XAML-build stuff in this blogpost. Everything in this post is about the new build ecosystem.

tfx build tasks upload --auth-type basic --task-path ./FolderWithYourBuildStep

The TFS Rest Api documentation can be found here. Update September 2016: I wrote another blog post regarding TFS 2015 build and release tasks, mainly focussing on the TFS REST API.