Greetings, everyone!

In this blog post we will talk about building a React application with TeamCity using one of the newest TeamCity features –  Docker integration, introduced in  TeamCity 2017.2 EAP1.

We’ve created an example on GitHub based on Create React App and Storybook, and set up a demo project on the public TeamCity server with the VCS Root pointing to the project on GitHub.

We used the Command Line build step for our build configuration.

Command Line Build Step

We are going to execute the following script:

#!/bin/bash
set -e -x

yarn
yarn test-ci
yarn build
yarn build-storybook

Here yarn is used for installing dependencies and running scripts, but it is possible to use npm as well.

Our command line will run in a docker container – in TeamCity 2017.2 EAP the Command Line runner settings have the Docker settings section where you can specify the image for the container:

TeamCity will start the container and run this build step in it. node:latest in the Docker settings section stands for Docker’s official node image):

Report tabs

test-ci in our script runs tests with the jest-teamcity-reporter plugin (see this commit).
In the line react-scripts test --env=jsdom --coverage --testResultsProcessor=jest-teamcity-reporter, the --coverage option turns on generating a code coverage report, which will be displayed in a dedicated build tab. TeamCity has the report tabs feature, and on our server the Root project – the parent of the demo project – has a predefined Code Coverage tab which expects the report to be at coverage.zip!index.html:

All we have to is specify the artifact path coverage/lcov-report => coverage.zip in the General Settings section of our build configuration:

We also added the artifact path for the build directory (build => build), so that it is available in the artifacts and can be deployed to some server later, which is out of scope for this post.

After running the build, the Code Coverage report is available on the build results page.

In our build the build-storybook script generates a living style guide for your component base using Storybook. It will be handy to use another build tab to display it, so the storybook-static => stories artifact path has been added. Then we set up a custom tab in the “React App” project settings using the Report tabs section:

Note that this Stories tab available on the build results page is generated per build, so you can see how your components change over time and, for example, detect at which point something started looking and/or behaving in a strange way.

Your feedback is always welcome in our forum and tracker!

Happy building!

When building, testing, and deploying an application with TeamCity, you may need to run a build under a specific user account, different from the one used to run the build agent.

TeamCity RunAs Plugin Overview

The teamcity-runas plugin can be used to run TeamCity build steps under a specified user account on Windows or Linux.

The plugin is compatible with TeamCity 10.0 and later.

Installing Plugin

System Administrators can download the runAs.zip from here and install it on the TeamCity server as usual.

Configuring TeamCity Agents

On Windows

The teamcity-runas plugin requires the TeamCity agent’s account to have certain permissions. Depending on the way you run the agent, granting additional privileges may be required.

• If you are running the TeamCity agent as a Windows service under the Local System account, it has all required permissions and the teamcity-runas plugin works out-of-the-box.
• If you are starting the agent as a Windows service using a specified Windows account (for example AgentUser), it must have administrative privileges, must have the ability to replace a process level token (SeAssignPrimaryTokenPrivilege), and act as a part of the operating system (SeTcbPrivilege) or needs to work under the Local Service account. These privileges can be added via a security policy using Control Panel | Administrative Tools | Local Security Policy | Local Policies | User Rights Assignment | Replace a process level token. See Microsoft documentation for details.
  You can also do it using the provided scripts, e.g.: runGrantAccess.cmd BuildUser.
The changes will not take effect until the next login, so you’ll have to restart the agent.

• If you are starting the TeamCity agent as a Windows console application, your Windows user account must have administrative privileges.

On Linux

• With Debian/Ubuntu/Linux, the recommended setup is to run the TeamCity agent as ROOT, as in this case the account has all the required privileges and the teamcity-runas plugin works out-of-the-box (see the related section below).
• If you run the TeamCity agent as a user different from ROOT, you need to install socat using the following command:
  sudo apt-get update && sudo apt-get install socat

Using Plugin

After the server restart, the plugin provides the Run As build feature and you can add it to your build configuration specifying the username and password.

For Windows the following username formats are acceptable:〈username〉or〈username@domain〉or〈domain\username〉.

For Windows-based build agents it is possible to specify the Windows Integrity Level, the logging level, and additional arguments.

Once the build is run, the plugin will run all the commands under the specified user account.

Checking Account Used to Run Build

You can enable the teamcity.runAs.log.enabled= true parameter of a build configuration (set to “false” by default) and TeamCity will print the required information into the build log.

If the credentials are invalid, the build fails with “Authentication failure” error:

Alternatively, to make sure the teamcity-runas plugin is configured properly, you can get the user whose account was used to run the build by checking the environment variables: USERNAME (for Windows) / USER, LOGNAME (for Linux).

For example, you can add a command line build step with the set USERNAME command on Windows / echo "$USER" echo “$LOGNAME” on Linux. This will print the user whose account was used to run the build into the log.

Accessing Files created by RunAs User

It is important to note that when running a build under a specified user account, all processes run in the environment specific for this user.

It means that all files created during the build (e.g. compiled classes, object files, executables, dlls, Gradle/Maven cache files, TeamCity artifacts, some test statistics and so on) will have the teamcity-runas user as the file owner.

The TeamCity agent must have the full control over the files created in the build checkout directory and ideally, over files in other locations, e.g. m2 or .gradle directories as well. Depending on the way you configured the agent (see Configuring TeamCity Agents above), the access level will differ and granting additional permissions may be required.

For Windows, the account with administrative permissions used to run the TeamCity build agent will have sufficient rights to manage the files.

For Linux, if the TeamCity agent is not running under ROOT, it may encounter problems accessing artifacts in the build checkout directory. In this case the necessary permissions can be granted by running the following command:
chmod -R a+rwX %teamcity.build.checkoutDir%.
It can be added as the last step of the build executed under the ‘Run As’ user.

Install the plugin on your TeamCity server, give it a try, and let us know what you think about it.

Happy building!

The TeamCity Team

When performing integration tests and deployments, build scripts need credentials to access external servers and services. Traditionally passwords are stored on the TeamCity server as secure parameters. But although secrets are masked in the UI, encrypted at rest, and protected from being exposed in the build log as plain text, this often does not provide a high enough level of security.

HashiCorp Vault offers a unified approach to managing secrets and credentials, allows auditing access, and helps with password rotation.

Integration with Vault

Today we are presenting a new plugin to help build scripts interact with Vault and obtain credentials dynamically.

The plugin allows connecting TeamCity to Vault, requesting new credentials when a build starts, passing them to the build script, and revoking them immediately when the build finishes.

Usage

We expect you have installed and configured:

• a Vault server. Refer to the Getting Started guide to learn how to launch a testing Vault instance.
• A TeamCity server. Refer to the Getting Started guide for details.

Perform the following:

• Download the plugin from the plugin repository, and install it to TeamCity server.
• Configure secret backends in Vault to obtain secrets, for example AWS credentials, or generic secrets.
• Create an AppRole in Vault for the TeamCity server to access these backends.
• Create a connection to Vault in your TeamCity project:
• Next in your build configuration declare new build parameters which refer to the Vault secrets to be used in this build:

The parameter value has a special syntax:%vault:PATH!KEY%where PATH is the secret name, and KEY is the specific value inside.
At the moment these values can be used in the build parameter declaration only, and cannot be specified in build steps.

• Run a build step. Deployment scripts can read credentials from environment variables, and do not need to perform any Vault-related actions:

Besides AWS, any other Vault backends can be used. In this case:

• Declare a configuration parameter with a value from generic backend:
• In a build step refer to this  parameter explicitly:

How It Works

• The TeamCity server stores the AppRole ID and secret, and keeps the secret encrypted at rest. When required, it can be rotated manually in the UI, or via the REST API.
• When a new build is started, the TeamCity server requests a one-time response wrappingtoken in Vault, and sends it to a build agent.
  The AppRole secret never leaves the TeamCity server machine.
• The build agent requests the actual secrets from Vault.
  The actual credentials are obtained from Vault by a build agent machine directly, and never transferred to the TeamCity server.
• The build parameters are resolved and the secrets are passed to environment variables and system properties.
• The build steps are started, and they refer to these environment variables and system properties.
• If secret values appear in a build log, they are automatically replaced by asterisks.
• Right after the build finish, the credentials are explicitly revoked in Vault.

Security Considerations

It is advised that you limit access to TeamCity project settings. If a person can modify build configuration settings in a TeamCity project, they can get access to the secrets. Dynamic credentials are revoked after a build, but for generic secrets this may cause more risks. To minimize the risks, split build configurations for CI and deployments into separate projects with different permissions.

Make sure build agent machines are connected to the TeamCity server via HTTPS. Only one-time tokens are transferred over network, and if they are stolen, credentials will be immediately revoked. However, it is a good practice to encrypt all traffic between the server and agents.

Feedback Wanted

This is an initial release, and we continue to improve features and user experience, and encourage you to give it a try and tell us what do you think!

The plugin is open-sourced, and published on GitHub.

If you are attending HashiConf in Austin, TX, USA next week, visit the JetBrains sponsor booth for the demonstration of this plugin and our other plugins for integrating Packer and Terraform with TeamCity and IntelliJ-based IDEs.

Starting from its very first version, TeamCity has provided support for IntelliJ platform-based IDEs; and since they are products from JetBrains, this integration is the most powerful one when compared to other IDE integrations.

The main idea behind the integration is the ability to take advantage of all the features provided by TeamCity as a continuous integration server without leaving the context of the IDE: using the integration, you can configure notifications, run builds and tests, get the build status, view your changes, assign investigations and perform other actions available in TeamCity right from the IntelliJ IDEA. Besides, the integration allows cross navigation between the IDE and build results in TeamCity; for non-DVCS it allows you to test your local changes and see how they integrate with the source code before the commit, which ensures your changes do not break anything.

In this blog post we’ll go over the setup of the TeamCity plugin with IntelliJ IDEA.
TeamCity provides the same level of support for both IntelliJ IDEA editions: the free community edition and the paid ultimate edition. All the IntelliJ platform-based IDEs are also supported: JetBrains RubyMine, JetBrains PyCharm, JetBrains PhpStorm/WebStorm, and AppCode. The versions compatibility information is provided in the TeamCity documentation.

So, first we need to install the TeamCity plugin.

Installing the Plugin

There are two ways to install a plugin.

Installing from the plugins repository

You can install the plugin right from the IDE (connection to the Internet is required) using File | Settings | Plugins. Click Browse JetBrains Plugins and search for TeamCity Integration. Double-click on the listed plugin to download and install it.

Alternatively, you can download the plugin directly from the TeamCity server, which is especially useful if the Internet access is restricted in your organization.

Installing from the TeamCity server

Log in to the TeamCity server, in the right top corner, click the arrow next to your username and select My Settings & Tools on the right of your name. On the right, you will see TeamCity Tools.

1. Click on the download link near IntelliJ Platform plugin and download TeamCity-IDEAplugin.zip to your local machine.
2. After this, in IDEA, go to File | Settings | Plugins | Install plugin from disk… and in the explorer window, select the downloaded ZIP archive.
3. Click OK.

Next we’ll restart the IDE is required to activate the plugin.

After the restart, the TeamCity menu is available at the top of the screen.

At the bottom on the left the TeamCity tool window  is visible, on the right we now have the TeamCity ‘Logged out’ status icon.
Now we can connect to TeamCity.

Logging in to TeamCity

To log in to the TeamCity server, click TeamCity | Log in… and provide the required credentials:

After you log in, if you installed the plugin from the IDE, you might have to update the plugin to the version corresponding to the TeamCity server you are connected to using the TeamCity | Update plugin menu item.

Now let’s see the integration in action.

Configuring Notifications

Next we’ll configure notifications which the TeamCity server will send to our IDE. This can be done using TeamCity | Options | Edit Notification Rules or by clicking a link in the TeamCity tool window asking you the specify the projects you want to watch: either of the options redirects you to the Notification Rules page in the Web UI of the TeamCity server you are connected to. By default, TeamCity sends out IDE notifications to the user if a build containing their changes fails. You can add your own rules here:

TeamCity will track the changes by the username specified in the VCS Usernames section.

Open Files from TeamCity in IDE

When looking at changes in the TeamCity Web UI, you can open changed files from in the IDE:

Clicking the IDE link opens up the specific file in the active IDE, i.e. the IDE we are currently connected to.

You can also work with all changes: in this case the patches are downloaded to the IDE, and they can be applied to the working directory as required.

Working with TeamCity from IDE

At the bottom on the left the TeamCity tool window has the Projects, My Changes, and My Investigations tabs, which correspond to the options available in the TeamCity Web UI.

The Projects tab displays projects and build configurations we decided to monitor. It also allows us to configure more visible projects by redirecting us to the TeamCity Web UI.

Let’s run a build from the IDE: right-clicking the build configuration name (Build in the image below) opens a menu allowing you to add a build to the queue, view pending changes, assign and investigation to any user registered on the TeamCity server you are connected to, mark an investigation as fixed, open the build configuration home page in the browser.

The My Changes tab displays all changes made by you and detected by TeamCity according to the specified username:

You can filter the changes and see only those related to the project you are currently working on:

It is easy to see the details of a specific commit in the IDE or quickly open the TeamCity change details page in the browser by right-clicking on the change and selecting the required option:

If you are assigned to investigate problems (typically build failures caused by your changes) in TeamCity, you will your investigations in the My Investigations tab.
As you can see, the TeamCity plugin enables you to run builds, view change details, assign investigations and perform other actions available in TeamCity without leaving IntelliJ IDEA.

Committing Pre-Tested Changes to Non-DVCS

In our project we are using the most popular version control at the moment – Git, and we use Git branches to test how our changes integrate with the existing code. In this part we’ll talk about the feature that can help if your project uses centralized version controls (like Subversion (SVN) or Perforce).
Using the pre-tested commit feature, you can prevent committing defective code into a build, so the entire team’s process is not affected. First the developer runs a build in TeamCity with the local changes on the developer’s machine and if this remote run is successful, the changes are committed to the version control automatically.

“Remote” means that the build will not run on the developer machine, but on a TeamCity build agent and this build will include the changes on our developer’s machine and the code in the version control: this way we can see how our changes integrate with other changes in the repository and be sure that our changes will not break the code.

If the remote run (also called ‘Personal Build’) fails, we will get a notification and can roll the changes back or fix the issues.
To trigger a remote run, we’ll use the TeamCity | Remote Run menu item. In the dialog that is displayed, select all changes you made locally or cherry-pick just the changes you want to test on the build server like we did. It is possible to select when to commit the changes: our changes will be committed if no builds fail. All whats left is to click the OK button and the remote run starts:

Debugging Java Code Remotely

We already mentioned that TeamCity integration supports all IntelliJ-based IDEs: it is programming language agnostic. However, some of its features are limited to a particular technology, and the next feature we’ll talk about is Java-specific.

The TeamCity plugin for IntelliJ IDEA allows you to debug your tests remotely on the TeamCity agent machine from the IDE on the local developer machine. The remote debug feature is useful if the agent environment is unique in some aspect, which causes a test to fail, and it is difficult to reproduce the problem locally.

You can use this feature if the following requirements are met:

1. an IntelliJ IDEA run configuration (Java application run configuration /JUnit run configuration /JTestNG run configuration) on the local developer machine with the TeamCity plugin for IntelliJ IDEA is installed
2. a build configuration on the TeamCity Server with the IntelliJ IDEA Project runner as one of the build steps
3. a remote TeamCity agent to run this build available to the local machine by socket connection

To start debugging, select the TeamCity | Remote Debug menu item. You will be asked to select the run configuration to debug.
(Alternatively, you can right-click tests and select the Debug Remotely on TeamCity Agent option from the context menu.)

Now we click Debug Remotely on TeamCity agent. The build is added to the queue and the standard IntelliJ IDEA debug tool window appears.The debug tool window works in the listening mode, i.e. it waits for the agent to connect to it. Once the agent connects, the Java process on the agent is paused and the Agent Attached notification appears in the IDE. Now we can set some breakpoints, and actually start the debug session by clicking Start either in the notification popup or in the debug tool window.

When the build is finished, you can view failed tests in the IDE:

Once the JVM process exits, another notification popup appears in the IDE:

The debug session is not finished yet, it is possible to either repeat or finish it. Selecting Repeat will rerun the same build step again, which is much faster than starting a new debug session.

Well, we’ve covered some of the most popular features of the TeamCity-IDEA integration and really hope we managed to persuade you to try the integration.

Your feedback is most welcome as usual herehere in the comments or ourissue tracker.

Happy integrating and building!

We’re often asked how to run different build steps in different branches. In fact, this has already been possible in TeamCity for quite some time (since version 9.1), but it seems we need to do a better job explaining how to use this feature.

Let’s assume that you have a build configuration where building of feature branches is enabled.

First, enable Versioned settings for the project where this build configuration resides. TeamCity will ask for a VCS root where the settings are to be stored. For simplicity’s sake, let’s store the settings in the same VCS repository where other project files are located.

For the Versioned settings, make sure the option When build starts is set to the use settings from VCS value. TeamCity will then take the settings from a branch used by the build instead of the current settings.

Once versioned settings are enabled, TeamCity will commit the .teamcity directory with the current project settings to the default branch of the selected VCS root. To start customizing the settings in our feature branch, we need to merge the .teamcity directory from the default branch.

But before making any changes in the .teamcity directory in your feature branch, bear in mind that not all the changes there will be applied. For instance, if you add a new project or a build configuration, such changes will be ignored. This happens because TeamCity does not support different sets of projects and build configurations in feature branches.

Besides, there is a chicken and egg problem. Currently TeamCity loads settings from .teamcity while the build sits in the queue. But to load the settings from a repository, TeamCity needs to know this repository settings (VCS roots). Thus VCS roots are always taken from the current project settings.

The same story is with build triggers and snapshot dependencies. TeamCity needs triggers to decide when to place builds into the queue. And TeamCity needs snapshot dependencies to create a build chain (graph of builds) before the builds are going to the queue. Hence the changes in .teamcity affecting these settings are ignored too.

But there are plenty of other settings whose changes will affect the build behavior:

• build number pattern
• build steps
• system properties and environment variables
• requirements
• build features (except Automatic merge and VCS labeling)
• failure conditions
• artifact publishing rules
• artifact dependencies

By default, the settings are stored in the XML format. Since there is no publicly available documentation for this format, it is hard to guess how to change build steps correctly, or how to add a build feature. This is what a command line build step looks like in XML:

<build-runners>
 <runner id="RUNNER_76" name="" type="simpleRunner">
   <parameters>
     <param name="script.content" value="echo &quot;Hello world!&quot;" />
     <param name="teamcity.step.mode" value="default" />
     <param name="use.custom.script" value="true" />
   </parameters>
 </runner>
</build-runners>

What can help here is seeing how TeamCity generates these settings. For instance, get a sandbox project on your TeamCity server for such experiments, and browse the audit page for settings difference after each change:

Another option is to use Kotlin DSL (see the Settings format option on the Versioned settings page). In this case, instead of .xml files, .kt files will be stored under the .teamcity folder.

A Kotlin DSL project can be opened in IntelliJ IDEA Community (free and open-source IDE from JetBrains). Kotlin DSL files look much more concise, and IDE auto-completion makes it much easier to work with them, especially in the recent TeamCity versions. This is what the same command line build step looks like in Kotlin DSL:

steps {
    script {
        scriptContent = """echo "Hello world!""""
    }
}

Some additional hints:

• Since TeamCity 2017.2 you can browse Kotlin DSL documentation using the following URL: <TeamCity server URL>/app/dsl-documentation/index.html
• If a build loads settings from the VCS repository, you should see something like this in the build log:
  
• Actual settings used by the build are stored in the buildSettings.xml file under the hidden build artifacts. This file can be helpful if something does not work as expected.
  
• If you’re using remote runs instead of feature branches, then all the same abilities are also available, provided that the .teamcity directory is  present in your repository and versioned settings are configured as described above.

Finally, one more thing to keep in mind. If changes under .teamcity in your feature branch were temporary, you should not forget to revert the settings during the merge with the default branch. For instance, if you removed some long running steps or switched off some tests for convenience.

Happy building!

Greetings, everyone!

We are proud to announce that TeamCity 10.0 EAP 2 build 41463 has just been released!

One of the most exciting features of this EAP is the ability to create TeamCity projects and build configurations programmatically: using DSL based on the Kotlin language. You can now define TeamCity project settings using Kotlin, check them into your version control and TeamCity will execute your Kotlin script and produce projects and build configurations with the desired settings. See our Release notes for more information.

This EAP build also comes with improved algorithm for VCS changes polling. As you know, TeamCity polls repositories for changes according to the configured interval. Although this approach is quite reliable and easy to maintain, in case of large installations with thousands of VCS roots it imposes a great load on both the TeamCity server and VCS repository servers. To avoid this, you can set up a commit hook, but still you’d need to manually adjust checking-for-changes intervals in VCS roots, and to constantly monitor that the hook works. Fortunately, starting with this EAP, TeamCity will increase the checking-for-changes interval automatically, once it determines that changes were detected as a result of a commit hook. If TeamCity detects a non-working commit hook, the checking-for-changes interval will be decreased. Our Release Notes provide details.

Among other things, the latest TeamCity version is more pleasant to look at, as it comes with new refreshed icons in the Web UI! All the more reason to check out the full list of this EAP’s  great features, and give this version a test drive! Download  build 41463 and don’t hesitate to install it on a trial server.

All our EAP releases come with a  60-day Enterprise evaluation license, and we hope you’ll have something to say about the new version once you have tried this build.

Happy building!

Greetings, everyone!

Our team is excited to announce the arrival of TeamCity 10.0, the jubilee version of our continuous integration and deployment server!

The number ten is regarded as the most perfect of numbers, as it contains the life and the nothing, the Unit that did it all, and the zero, symbol of the matter and the Chaos, of which all came out; it represents a new beginning at a higher level.

We literally follow the symbolism: TeamCity 1.0 saw the light in 2006, and 10 years (and BTW over 16 000 bugfixes!) later, TeamCity 10.0 is here! This TeamCity version comes with advanced features revolutionizing the way you create your projects and builds, making you TeamCity installation more scalable, expanding your horizons with new and improved integrations, all packed in the revised UI.

While so many of our users love the simplicity and ease of configuring projects and builds via the UI, for repetitive tasks like copying projects, build configurations, etc. the programmatic approach is by far more efficient. With TeamCity 10.0 Kotlin-based DSL you’ll enter a new world of creating and maintaining your build infrastructure, dynamically applying all your changes to the projects and builds, skipping the need to use the UI.

In this release another take on scalability has been made – now you can set up a two-node TeamCity, distributing the workload between two servers.

Issue trackers integration has been improved: TFS, GitHub, and Bitbucket are supported out of the box

We’ve made significant progress making VCS-related improvements:
cross-platform support for TFS is finally here; TeamCity works with Perforce streams now and more!

This is just a teaser – for the full list of features see our What’s New  and Release Notes, and download TeamCity 10.0! Remember to check the Upgrade Notes before you install the new version!

Register for our webinar to see the new TeamCity 10.0 features in action!

Happy upgrading and building!

P.S. We’ve also adopted a tradition of giving a decennial retrospective of the TeamCity UI, so take a look at our site to see how pretty our GUI grown to be!

Greetings, everyone!

Since TeamCity 10.0 it has become possible to use commit hooks with a TeamCity server. Now, when a VCS change is detected via a commit hook, TeamCity automatically increases the VCS repository polling interval, reducing the load on both the TeamCity server and VCS repository. And obviously the presence of the hook greatly decreases the time needed to detect a change.

Unfortunately, installation and configuration of commit hooks is not an easy task and for on-premises VCS repositories it requires administration skills. At the same time, popular VCS hostings such as GitHub, support installation of commit hooks via their REST API, and installation of commit hooks for them can be a lot simpler, provided that we use such an API from TeamCity…

So today, we’d like to announce one more TeamCity plugin whose task is to install and maintain GitHub commit hooks (do not worry, we plan to add support for other VCS hosting services too).

The plugin supports both GitHub.com and GitHub Enterprise. The plugin does not install a webhook automatically for GitHub.com because a webhook requires a connection from GitHub.com to the TeamCity server, and in the majority of cases, when TeamCity is installed in the intranet, such a connection is blocked by a firewall. In case of GitHub Enterprise, the plugin will install a webhook automatically for any TeamCity project created from a URL or via GitHub integration.

The plugin works with the GitHub REST API and has to make API calls to GitHub on behalf of the current user, so it requires a GitHub connection configured in the project or its parent.

The plugin is quite simple, basically it does three things:

• It shows a suggestion to install a GitHub webhook if it finds a GitHub repository in a project without such a webhook:
• It provides a new action in the project actions menu for webhook installation enabling you to install or reinstall a webhook at any time:
• It checks the status of all of the installed webhooks and raises a warning via the health report if some problem is detected:

The plugin is open source, distributed under the Apache 2.0 license. Most of the code is written in Kotlin. The source code is published on GitHub.

The plugin relies on the new TeamCity API and will only work with TeamCity 10.0 and later. Download the plugin, install it on your TeamCity server and try the plugin. We’ll appreciate your feedback!

Happy building!

Configuring TeamCity to use GitHub as the source code repository has always been easy, especially since the feature ‘create from URL‘ was first introduced. TeamCity 10 has brought a number of improvements related to integration with GitHub, which are worth a special mention.

Project-level GitHub Connection in TeamCity

We’ve been gradually improving the integration, and in TeamCity 10 you can now set up a connection to GitHub on the project level:

Once the connection is set up, you can create a project or a build configuration pointing to a GitHub repository:

Besides, with the connection configured, a small GitHub icon becomes active in several places where a repository URL can be specified: create project from URL, create Git VCS root, GitHub issue tracker, create create VCS root from URL:

On clicking the icon, TeamCity will list GitHub repositories available to the user.

Configuring a GitHub connection is useful for the organization administrator who can create a parent project and configure a connection to GitHub there once; thus, all the TeamCity users of the organization will see a list of GitHub repositories URLs in the TeamCity Web UI. It makes setting up a subproject, a Git VCS root, or a GitHub issue tracker extremely easy.

GitHub Issue Tracker Integration in TeamCity

Built-in integration with GitHub issue tracker was also introduced in TeamCity 10. It is configured on the dedicated page of the project settings. If a project level connection to GitHub is configured, you can simply select the repository URL from the list available on clicking the GitHub icon:

If no connection is configured, the URL can be specified manually.

The rest is easy – select the type of authentication and provide the required information, tell TeamCity which strings should be recognized as references to issues: for GitHub, the regex syntax is used, e.g. #(\d+). TeamCity will resolve the issue number mentioned in a VCS comment and will display a link to this issue in the Web UI (e.g. on the Changes page, Issues tab of the build results page).

TeamCity Build Status in GitHub

Developers find it handy to view the status of their commits right in the repository. Earlier there were several external plugins allowing you to publish the TeamCity build status on GitHub, and in TeamCity 10 we delivered a bundled build feature, Commit Status Publisher, which automatically attaches the build status to your GitHub pull requests:

Besides, using the TeamCity REST API, you can publish the status icon of your TeamCity build in the README for your repository:

TeamCity Commit Hooks

All the above-mentioned improvements are bundled with TeamCity 10, but we should also mention the TeamCity Commit Hooks Plugin, not bundled with TeamCity yet. This plugin is compatible with TeamCity 10 or later. Its task is to install webhooks for GitHub repositories specified in TeamCity VCS roots.
Installed GitHub webhooks greatly decrease the time required for TeamCity to detect a change. As an additional benefit, webhooks reduce the load on both the TeamCity server and the GitHub server.

Take the advantage of the TeamCity-GitHub improved integration and let these features make your experience of Continuous Integration & Delivery with GitHub and TeamCity nice and smooth.

The .NET Core development platform is the next wave of .NET ecosystem evolution which brings cross-platform targets for your existing or new products. Microsoft ships .NET Core SDK packages with a runtime for supported platforms and command line tools (.NET CLI) to build, test, and pack such projects.

The .NET Core support plugin for TeamCity aims to bring streamline support for .NET Core CLI by providing .NET Core build steps, CLI detection on the build agents, and auto-discovery of build steps in your repository.

Installation

The plugin is available for download from the TeamCity plugin gallery and can be installed as an additional TeamCity plugin. Next the .NET Core SDK has to be installed on your build agent machines. The plugin uses the PATH environment variable to determine the .NET CLI tools location, but you can override it by specifying the DOTNET_HOME environment variable as follows: DOTNET_HOME=C:\Program Files\dotnet\

You can check that plugin has been properly initialized by navigating to the build agent parameters:

Building and testing .NET Core projects

The .NET Core plugin supports auto-discovery of build steps, and usually all you need is to use the Create project from URL wizard:

If you want to tune some parameters in your build steps, the .NET Core plugin provides details about configuration options and hints at their possible values:

Specify project build version

While building and creating project packages, the .NET CLI tools use the version number defined in the project.json file. If you want to provide a custom build number, you can update your project.json by leveraging the File Content Replacer build feature as follows:

Here you need to specify the target project.json files to process, fill in (\{[^{]*\"version\"\s*:\s*\")([^\"]+)(\") in the Find what and $1%build.number%$3 in the Replace with fields. Then, after executing the vcs checkout, the File Content Replacer will set the version, for instance %build.number% in the previous example.

Note: Currently .NET CLI tools support projects with project.json-based structure, but the upcoming versions will rely on the new csproj structure.

Testing .NET Core projects

The xUnit framework provide dotnet test runner which report the information about test results on the fly:

The NUnit test runner does not support it out of the box, but you can vote for the following issue. If you need any other test framework support, please leave a comment in the related issue.

Deploying .NET Core projects

.NET Core libraries

Your projects can contain a number of shared libraries which can be reused in dependent project builds. Instead of artifact dependencies, it is recommended that you use the build-in TeamCity NuGet feed. In this the specified build artifacts paths need to include the produced NuGet packages.

Tip: You can use the NuGet Dependency Trigger to start a build when a package has been updated in some nuget feed.

Publishing ASP.NET Core projects

Such projects can be deployed to a wide range of platforms. As a result of executing the dotnet publish build step, you will receive the published package data in your file system. Then you need to decide how to deploy it to the target system.

For Windows servers, the Web Deploy utility can be used. Alternatively, you can deploy the package with the auto-generated %profile%-publish.ps1 or default-publish.ps1 PowerShell scripts. More details on how to provide proper arguments to these scripts can be found in the Web Publishing documentation.

For other server platforms you can publish package by leveraging TeamCity Deployer build steps like FTP/SSH/SMB.

Feedback

You are welcome to try the plugin for TeamCity, and your feedback will be really appreciated, so please share your thoughts in the comments to this blog post, in the forum or issue tracker.

Happy building modern .NET Core projects!

This is a five part series on working with Kotlin to create Configuration Scripts for TeamCity.

1. An Introduction to Configuration Scripts
2. Working with Configuration Scripts
3. Creating Configuration Scripts dynamically
4. Extending the TeamCity DSL
5. Testing Configuration Scripts

With TeamCity 10, we brought the ability to define configurations using Kotlin as opposed to using XML or the User Interface.

This provides several benefits amongst which are

• A human-readable and editable configuration script, not relying on the user interface
• Ability to do diffs on changes to a configuration script
• Ability to check in your configuration scripts alongside your source code¹
• Ability to dynamically create configuration scripts, as well as test them

Why Kotlin?

You might ask why Kotlin and not some other language? For those not familiar with Kotlin, it is a statically-typed language developed by us at JetBrains and open sourced under the Apache 2 license. It targets the JVM, JavaScript (and we also have native in the works). It’s designed to be a pragmatic language that cuts down boilerplate code, remains expressive, and facilitates tooling. Kotlin provides a series of features that allow the creation of DSL’s (Domain Specific Languages) and TeamCity is a perfect fit for this purpose. Kotlin enables the benefits we’ve outlined and we’ll see.

Why not something more common such as YAML? Based on our experience, we believe that while YAML is great for the simpler setups, at some point it does fall short, and can lose clarity when it comes to defining more complex configuration scripts. We wanted to provide a solution that works for the simplest to the most demanding of scenarios, and that’s why we’ve gone with Kotlin. It’s important to understand though that we’re not providing just a DSL for creating configuration scripts. Kotlin is a programming language and as such we have the ability to write any kind of code in our configuration scripts (which, of course, like anything can be also abused). This enables many scenarios such as those we’ll see when creating dynamic configurations.

What is needed

Given Kotlin is a proper language, you might be wondering what tooling is required to edit configuration scripts. In principle any editor will do. Kotlin is built so that it can be used with any editor or IDE. In fact we ship support for not only IntelliJ IDEA (both Ultimate and the free OSS community editions), but also Eclipse and NetBeans. If Vim or another editor is your thing, you can also use that along with a command line compiler.

For the purpose of this blog post series, we’ll be using IntelliJ IDEA.

Creating a first script

While we can start with a completely blank script, the easiest way to create a Kotlin build script is to take an existing TeamCity project and either

• Download it as a Kotlin Script
• Enable Versioned Settings

The first is a great way to start playing with Kotlin configuration scripts and get a feel for them. The second option not only provides us with the configuration scripts checked in to source control, but it’s actually a required step for us to use Kotlin build scripts in production.

The recommended approach is to use the second option once we’re ready to enable Kotlin scripting in our projects. The first option, as mentioned, is great for discovering how things work and getting familiar with Kotlin configuration scripts.

Download Settings as a Kotlin Script

In order to see what a build configuration looks like in Kotlin, simply Edit Project Settings on the selected project and click on the Actions menu, picking the entry Download Configuration Script in Kotlin format

This downloads a zip file that contains a Maven project which can be obtained in IntelliJ IDEA. Note that the folder it contains is prefixed with a dot, which indicates it’s hidden under MacOS. The idea is that this folder can essentially be placed in version control at some point (TeamCity marks its configuration files as hidden).

Enable Versioned Settings

The second option, which is required for us to actually use Kotlin configuration scripts, is to enable Versioned Settings. This is done under Versioned Settings and selecting Enable, and Kotlin as the file format.

As soon as we activate this, the TeamCity UI will no longer allow any changes and all the configuration will be stored in version control and modifications have to take place via the build script.

These script files will have the same folder layout as that of the downloaded zip file (first option). For instance, the following are the files checked in to version control for the Joda-Time project. We can see that once again it’s a Maven project containing a series of Kotlin files (.kt). 

Opening the script in IntelliJ IDEA

Once we have the configuration script (be it as a zip file or checked in to version control on enabled versioned settings), we can open it in IntelliJ IDEA, and Maven will pull down the correct dependencies. Our project layout should look like the one below

Don’t feel overwhelmed by the number of files in there. In fact, TeamCity really only needs one file, which is settings.kts. Let’s examine each file in more detail.

settings.kts

kts is a Kotlin Script file, different from a Kotlin file (.kt) in that it can be run as a script. As mentioned above, all the code relevant to the TeamCity configuration could be stored in this single file, but it is divided into several files to provide a better separation of concerns.

This file contains usually two lines

version indicates the TeamCity version, and project() is the main entry point to the configuration script. It represents a function call, which takes as parameter a Project, representing the entire TeamCity project.

The parameter GitExtensions.Project is an object in Kotlin. Think of objects in Kotlin as you would in JavaScript. Or to compare it to Java, they’d be a Singleton, a single instance of a Kotlin class. While Kotlin script could work with singletons, having a first-class support for objects, makes things much easier.

In this case, Project is the name of the actual object, and GitExtensions is the package name.

Project.kt

This file contains the GitExtensions.Project object, and is where all the magic happens. If you look at the script layout, it’s basically a code representation of the build steps we’re accustomed to seeing the TeamCity user interface

which would correspond to the following entries in the UI, in addition to VCS roots and Build Types.

GitExtensions_HttpsGithubComJetbrainsGitextensions.kt

This object defines the VCS root configuration. It’s important to note that we could have just placed all this information directly in the code above, as opposed to making the call vcsRoot(GitExtensions_…).  However, the advantage to doing this, as we’ll see, is not only to make the code cleaner and separate concerns, but also to provide  reusability.

GitExtensions_Main.kt

Finally, we have the actual meat of where the build happens. This object defines the build steps, failure conditions, and everything else you’d come to expect for a build configuration in TeamCity

Summary

In this first part we’ve seen the components of a TeamCity configuration script. In the next part we’ll dive a little deeper into the DSL, modify the script, and see some of the benefits that Kotlin and IntelliJ IDEA already start providing us in terms of guidance via code assistants.

[1] While enabling version control storage for settings has been available since version 9, it was only available in XML format. TeamCity 10 brings Kotlin to this.

This is part two of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

1. An Introduction to Configuration Scripts
2. Working with Configuration Scripts
3. Creating Configuration Scripts dynamically
4. Extending the TeamCity DSL
5. Testing Configuration Scripts

In the first part we saw the basics of Configuration Scripts and how to get started. Now we’ll dive a little deeper into TeamCity’s DSL and see what it provides us in terms of building configuration scripts.

Examining the DSL

The DSL comes in a series of packages with the main one being the actual DSL which is contained in the configs-dsl-kotlin-{version}.jar file. Examining it, we see that it has a series of classes that describe pretty much the entire TeamCity user interface.

At any time you can navigate to a definition using for instance IntelliJ IDEA, and it will prompt you to download sources or decompile.  The sources are available from the actual TeamCity server your pom.xml file points to, and they’re a great way to see all the potential of the DSL.

The entry point to a project, as we saw in the previous post, is the Project object. From here we can define all the settings such as the basic project properties, the VCS roots, build steps, etc. There are a few basic parameters that should be set:

• Uuid: it’s the internal ID TeamCity maintains. It is unique across the server. It’s not recommended that this value be changed as TeamCity uses it internally to associate data with the project.  
• extId: this is the user-friendly ID used in URLs, in the UI, etc. and can be changed if required.
• parentId: represents the extId of a project where this project belongs, defaulting to the value _Root for top-level projects.
• name: the name of the project

and optionally

• description: description for the project

Beyond the above, everything else is pretty much optional. Of course, if that’s the only thing we define, then not much is going to happen during the build process.

Modifying the build process

The code below is an excerpt from the configuration script for the Spek project (parts of the configuration are omitted for brevity). This particular build compiles the code and runs some tests using Gradle.

steps {
   gradle {
       name = "Snapshot Build"
       tasks = "clean jar test"
       jdkHome = "%env.JDK_18_x64%"
   }
}


triggers {
   vcs {
       branchFilter = "+:<default>"
       perCheckinTriggering = true
       groupCheckinsByCommitter = true
   }
}

Let’s extend this build to add a feature to that TeamCity has, Build Files Cleaner also known as Swabra. This build features makes sure that files left by the previous build are removed before running new builds.

We can add it using the features function. As we start to type, we can see that the IDE provides us with completion:

The features function takes in turn a series of feature functions, each of which adds a particular feature. In our case, the code we’re looking for is

features {
   feature {
       type = "swabra"
   }
}

The resulting code should look like

steps {
   gradle {
       name = "Snapshot Build"
       tasks = "clean jar test"
       useGradleWrapper = true
       enableStacktrace = true
       jdkHome = "%env.JDK_18_x64%"
   }
}

triggers {
   vcs {
       branchFilter = "+:<default>"
       perCheckinTriggering = true
       groupCheckinsByCommitter = true
   }
}

features {
   feature {
       type = "swabra"
   }
}

And this works well. The problem is that, if we want to have this feature for every build configuration, we’re going to end up repeating the code. Let’s refactor it to a better solution.

Refactoring the DSL

What we’d ideally like is to have every build configuration automatically have the Build Files Cleaner feature without having to manually add it. In order to do this, we could introduce a function that wraps every build type with this feature. In essence, instead of having the Project call

buildType(Spek_Documentation)
buildType(Spek_Publish)
buildType(Spek_BuildAndTests)

we would have it call

buildType(cleanFiles(Spek_Documentation))
buildType(cleanFiles(Spek_Publish))
buildType(cleanFiles(Spek_BuildAndTests))

For this to work, we’d need to create the following function

fun cleanFiles(buildType: BuildType): BuildType {
   buildType.features {
       feature {
           type = "swabra"
       }
   }
   return buildType
}

Which essentially takes a build type, adds a feature to it, and returns the build type. Given that Kotlin allows top-level functions (i.e. no objects or classes are required to host a function), we can place it anywhere or create a specific file to hold it.

Now let’s extend this function to allow us to pass parameters to our feature, such as the rules to use when cleaning files.

fun cleanFiles(buildType: BuildType, rules: List<String>): BuildType {
   buildType.features {
       feature {
           type = "swabra"
           param("swabra.rules", rules.joinToString("\n"))
       }
   }
   return buildType
}

We pass in a list of files which are then passed in as parameter to the feature. The joinToString function allows us to concatenate a list of strings using a specific separator, in our case the carriage return.   

We can improve the code a little so that it only adds the feature if it doesn’t already exist:

fun cleanFiles(buildType: BuildType, rules: List<String>): BuildType {
   if (buildType.features.items.find { it.type == "swabra" } == null) {
       buildType.features {
           feature {
               type = "swabra"
               param("swabra.rules", rules.joinToString("\n"))
           }
       }
   }
   return buildType
}

Generalizing feature wrappers

The above function is great in that it allows us to add a specific feature to all build configurations. What if we wanted to generalize this so that we could define the feature ourselves? We can do that by passing a block of code to our cleanFiles function, which we’ll also rename to something more generic.

fun wrapWithFeature(buildType: BuildType, featureBlock: BuildFeatures.() -> Unit): BuildType {
   buildType.features {
       featureBlock()
   }
   return buildType
}

What we’re doing here is creating what’s known as a higher-order function, a function that takes another function as  a function. In fact this is exactly what features, feature, and many of the other TeamCity DSL’s are.

One particular thing about this function, however, is that it’s taking a special function as a parameter, which is an extension function in Kotlin. When passing in this type of parameter, we refer to it as Lambdas with Receivers (i.e. there is a receiver object that the function is applied on).

This then allows us to make the calls to this function in a nice way referencing  feature directly

buildType(wrapWithFeature(Spek_BuildAndTests, {
   feature {
       type = "swabra"
   }
}))

Summary

In this post we’ve seen how we can modify TeamCity configuration scripts using the extensive  Kotlin-based DSL.  What we really have in our hands is a full programming language along with all the features and power that it provides. We can encapsulate functionality in functions to re-use, we can use higher-order functions  as well as other things that open up many possibilities.

In the next part we’ll see how to use some of this to create scripts dynamically.

This is part three of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

1. An Introduction to Configuration Scripts
2. Working with Configuration Scripts
3. Creating Configuration Scripts dynamically
4. Extending the TeamCity DSL
5. Testing Configuration Scripts

We saw in the previous post how we could leverage some of Kotlin’s language features to reuse code. In this part, we’re going to take advantage of the fact that we are in fact dealing with a full programming language and not just a limited DSL, to create a dynamic build configuration.

Build Configurations based on parameters

The scenario is the following: we have an HTTP server that we need to test on different platforms with a range of concurrent connections on each one. This generates potentially a lot of different build configurations that we’d need to create and maintain.

Instead of doing this manually, what we can do is write some code to have our Kotlin DSL Configuration Script generate all the different build configurations for us.

Let’s say that we have a list of Operating Systems and a range of concurrent connections for each one. The first thing to do is to create a data class that represents this information

data class BuildParameters(val name: String, val operatingSystem: String, val connections: Int)

which in essence is like a regular class but provides a series of benefits such as a nicer string representation, equality and copy operations amongst other things.

Now let’s imagine we have the different platforms we want to test on represented as a list of BuildParameters

val operatingSystems = listOf(
            BuildParameters("Windows Build", "windows", 30),
            BuildParameters("MacOS Build", "osx", 20),
            BuildParameters("Linux Build", "ubuntu", 20)
    )

what we’d like to do, is iterate over this list and create a new build type for each combination. In essence, if our standard Project is

object Project : Project({
    uuid = "9179636a-39a3-4c2c-aa7e-6e2ea7cfbc5b"
    extId = "Wasabi"
    parentId = "_Root"
    name = "Wasabi"
    vcsRoot(Wasabi_HomeGitWasabiGitRefsHeadsMaster)
    buildType(WasabiBuild)
})

what we want to do is create a new buildType for each entry in the list

object Project : Project({
    uuid = "9179636a-39a3-4c2c-aa7e-6e2ea7cfbc5b"
    extId = "Wasabi"
    parentId = "_Root"
    name = "Wasabi"

    vcsRoot(Wasabi_HomeGitWasabiGitRefsHeadsMaster)

    val operatingSystems = listOf(
            BuildParameters("Windows Build", "windows", 30),
            BuildParameters("MacOS Build", "osx", 20),
            BuildParameters("Linux Build", "ubuntu", 20)
    )

    operatingSystems.forEach {
        buildType(OperatingSystemBuildType(it))
    }
})

Creating a base Build Type

Any parameter passed into buildType needs to be of the type BuildType. This means we’d need to inherit from this class and at the same time provide some parameters to it (BuildParameters)

class OperatingSystemBuildType(buildParameters: BuildParameters) : BuildType() {
    init {
        val paramToId = buildParameters.name.toExtId()
        uuid = "53f3d94a-20c6-43a2-b056-baa19e55fd40-$paramToId"
        extId = "BuildType$paramToId"
        name = "Build for ${buildParameters.name}"

        vcs {
            root(Wasabi.vcsRoots.Wasabi_HomeGitWasabiGitRefsHeadsMaster)

        }
        steps {

            gradle {
                tasks = "clean build"
                useGradleWrapper = true
                gradleWrapperPath = ""
                gradleParams = "-Dconnections=${buildParameters.connections}"
            }
        }
        triggers {
            vcs {
            }
        }

        requirements {
            contains("teamcity.agent.jvm.os.name", buildParameters.operatingSystem)
        }
    }

}

The code is actually a pretty standard Configuration Script that defines a Gradle build step, has a VCS definition and defines a VCS trigger. What we’ve done is just enhance it somewhat.

To begin with,  we’re using the BuildParameters name property to suffix it to the uuid, extId and name. This guarantees a unique ID per build configuration as well as provides us a name to identify it easily. To make sure the values are properly formatted, we use the TeamCity DSL defined extension function to String, named toExtId() which takes care of removing any forbidden characters.

In the steps definition, we pass in certain parameters to Gradle, which in our case is the number of concurrent connections we want to test with. Obviously, this is just a sample, and the actual data being passed in can be anything and used anywhere in the script.

Finally, we also use the BuildParameters operatingSystem property to define Agent requirements.

Summary

The above is just a sample of what can be done when creating dynamic build scripts. In this case, we created multiple build configurations, but we could just as well have created multiple steps, certain VCS triggers, or whatever else could come in useful. The important thing to understand is that at the end of the day, Kotlin Configuration Script isn’t just merely a DSL but a fully fledged programming language.

In the next part of this series, we’ll see how we can extend the Kotlin Configuration Scripts (hint – it’s going to involve extension functions).

This is part four of the five-part series on working with Kotlin to create Configuration Scripts for TeamCity.

1. An Introduction to Configuration Scripts
2. Working with Configuration Scripts
3. Creating Configuration Scripts dynamically
4. Extending the TeamCity DSL
5. Testing Configuration Scripts

TeamCity allows us to create build configurations that are dependent on each other, with the dependency being either snapshots or artifacts. The configuration for defining dependencies is done at the build configuration level. For instance, assuming that we have a build type Publish that has a snapshot dependency on Prepare Artifacts, we would define this in the build type Publish in the following way

class Publish : BuildType({
    uuid = "53f3d94a-20c6-43a2-b056-baa19e55fd40"
    extId = "Test"
    name = "Build"
    vcs {
        . . .
    }
    steps {
        . . . 
    }
    dependencies {
        dependency(Prepare_Artifacts) {
            snapshot {
                onDependencyFailure = FailureAction.FAIL_TO_START
            }
        }
    }
})

and in turn if Prepare Artifacts had dependencies on previous build configurations, we’d define these in the dependencies segment of its build configuration.

TeamCity then allows us to visually see this using the Build Chains tab in the user interface

Defining the pipeline in code

Pipeline is a sequence of phases, phase is a set of buildTypes, each buildType in a phase depends on all buildTypes from the previous phase. This can handle simple but common kind of build chains where some build produces an artifact, several builds test it in parallel and the final build deploys the result if all its dependencies are successful.

It would often be beneficial to be able to define this build chain (or build pipeline) in code, so that we could describe what’s going on from a single location. In essence, it would be nice to be able to define the above using the Kotlin DSL as so

object Project : Project ({
   . . . 
   pipeline {
        phase("Compile all clients") {
            + HttpClient
            + FluentHc
            + HttpClientCache
            + HttpMime
        }
        phase("Prepare artifacts") {
            + PrepareArtifacts
        }
        phase("Deploy and publish to CDN") {
            + Publish
        }
   }
})

Defining this at the Project.kt level in our DSL, would give us a good oversight of what’s taking place.

The issue is though that currently the TeamCity DSL does not provide this functionality. But that’s where Kotlin’s extensibility proves quite valuable as we’ll see.

Creating our own Pipeline definition

Kotlin allows us to create extension functions and properties, which are a means to extend a specific type with new functionality, without having to inherit from these. When passing extension functions as arguments to other functions (i.e. higher-order functions), we get what we call in Kotlin Lambdas with Receivers, something we saw in this series already when Generalising feature wrappers in the second part of this series. We apply the same concept here to create our pipeline DSL

class Pipeline {
    val phases = arrayListOf<Phase>()

    fun phase(description: String = "", init: Phase.() -> Unit = {}) {
        val newPhase = Phase()
        newPhase.init()
        phases.lastOrNull()?.let { prevPhase ->
            newPhase.buildTypes.forEach {
                it.dependencies {
                    for (dependency in prevPhase.buildTypes) {
                        snapshot(dependency)
                    }
                }
            }
        }
        phases.add(newPhase)
    }
}

class Phase {
    val buildTypes = hashSetOf<BuildType>()

    operator fun BuildType.unaryPlus() {
        buildTypes.add(this)
    }
}

fun Project.pipeline(init: Pipeline.() -> Unit = {}) {
    val pipeline = Pipeline()
    pipeline.init()
    //register all builds in pipeline
    pipeline.phases.forEach { phase ->
        phase.buildTypes.forEach { bt ->
            this.buildType(bt)
        }
    }
}

What the code above is doing is define a series of new constructs, namely pipeline and phase.

The code then takes the contents of what’s passed into each of these and defines, under the covers, the dependencies. In essence, it’s doing the same thing we would do in each of the different build configurations, but from a global perspective defined at the Project level.

In order to pass in the configuration to the pipeline as we saw earlier, we merely reference the specific build type (HttpClient, Publish, etc.), assigning it to a variable

val HttpClient = BuildType(....)

Flexibility of defining our own pipeline constructs

It’s important to understand that this is just one of many ways in which we can define pipelines. We’ve used the terms pipeline and phase. We could just as well have used the term stage to refer to each phase, or buildchain to refer to the pipeline itself (and thus align it with the UI). In addition to how we’ve named the constructs, and more importantly, is how the definition is actually constructed. In our case, we were interested in having this present in the Project itself. But we could just as well define a different syntax that is used at the build type level. The ability to easily extend the TeamCity DSL with our own constructs, provides us with this flexibility.

Recently we’ve been experimenting around using TeamCity as a Debian repository and we’d like to share some tips and tricks we’ve come up with in the process.

We used the TeamCity tcDebRepository plugin to build *.deb packages and serve package updates to Debian/Ubuntu servers. The plugin, the special prize winner of the 2016 TeamCity plugin contest, works like a charm, but we’ve encountered a few difficulties with the software used to build and package .deb files, hence this blog post.

It’s important to note that tcDebRepository is not capable of serving packages to recent Ubuntu versions due to the lack of package signing. This is planned to be resolved in version 1.1 of the plugin (the plugin compatibility details).

We do not intend to cover the basics of TeamCity and Debian GNU/Linux infrastructure. To get acquainted with TeamCity, this video might be helpful. Building Debian packages is concisely described in the Debian New Maintainers’ Guide.

Everything below applies to any other Debian GNU/Linux distribution, e.g. Astra Linux.

Prerequisites and Project Setup

We arbitrarily chose 4 packages to experiment with:

• autotools-dev (Git)
• bash (Git, Bazaar)
• debhelper (Git)
• dpkg (Git)

Our setup uses:

1. The free Professional version of TeamCity 10 with 3 agents managed by Debian 8.0 (Jessie).
2. The tcDebRepository plugin installed as usual.
3. Besides the Teamcity agent software, each of the agents required installing:
   • the build-essential package as a common prerequisite
   • the dependencies (from the  build-depends and build-depends-indep categories) required for individual packages.

Next we created a project in TeamCity with four build configurations (one per package):

Configuring VCS Roots

When configuring VCS Roots in TeamCity, types of Debian GNU/Linux packages had to be taken into account.

Debian packages can be native and non-native (details).
The code of native packages (e.g. debhelper, dpkg), developed within the Debian project, contains all the meta-information required for building (the debian/ directory in the source code tree).
The source code of non-native packages (e.g. bash) is not related to Debian, and an additional tree of their source code and meta-information with patches (contained in the debian/ directory) has to be maintained.

Therefore, for native packages we configured one VCS Root (pointing to Debian); whereas non-native packages needed two roots.

Next we configured checkout rules: the difference from the usual workflow is that in our case the artifacts (Debian packages) will be built outside the source code, one directory higher. This means that the checkout rules in TeamCity should be configured so that the source code of the dpkg package is checked out not into the current working directory, but into a subdirectory with the same name as the package, i.e. pkgname/. This is done by adding the following line to the checkout rules:
+:.=>pkgname
Finally, a VCS trigger was added to to every configuration.

Artifact paths

In terms of TeamCity, the binary packages we are about to build are artifacts, specified using artifact paths in the General Settings of every build configuration:

For native packages, the pkgname.orig.tar.{gz,bz2,xz,lzma} and pkgname.debian.tar.{gz,bz2,xz,lzma} will not be created.

Build Step 1: Integration with Bazaar

Building a package with Bazaar-hosted source code (bash in our case) requires integration with the Bazaar Version Control System not supported by TeamCity out-of-the box. There is an a 3rd party TeamCity plugin to support Bazaar, but it has at least two problems:

1. agent-side checkout is not supported
2. the plugin is based on bzr xmlls and bzr xmllog (the bzr commands invoked by the plugin), but these commands are provided by external modules not necessarily available in the default Bazaar installation

The TeamCity server we used is running Windows, so we decided not to opt for installing Bazaar on the server side. Instead, to build the bash package, we added a build step using the Command Line Runner and a shell script for Bazaar integration:

#!/bin/bash

export LANG=C
export LC_ALL=C

set -e

rm -rf bash/debian
bzr branch http://bazaar.launchpad.net/~doko/+junk/pkg-bash-debian bash/debian
major_minor=$(head -n1 bash/debian/changelog | awk '{print $2}' | tr -d '[()]' | cut -d- -f1)
echo "Package version from debian/changelog: ${major_minor}"
tar_archive=bash_${major_minor}.orig.tar
rm -f ${tar_archive} ${tar_archive}.bz2
# +:.=>bash checkout rule should be set for the main VCS root in TeamCity
tar cf ${tar_archive} bash
tar --delete -f ${tar_archive} bash/debian
# Required by dpkg-buildpackage
bzip2 -9 ${tar_archive}

This approach will not allow us to see the changes in one of the two trees of the source codes and run build automatically on changes, but it is ok as the first try.

Build Step 2. Initial configuration

We used the Command Line Runner to invoke dpkg-buildpackage. The -uc and -us keys in the image below indicate that digital signatures are currently out of scope for our packages. If you wish to sign the packages, you’ll have to load the corresponding pair of GnuPG keys on each of the agents.

Note that the “Working directory” is not set to the current working directory, but to the subdirectory with the same name as the package (bash in this case) where the source code tree will be checked out and dpkg-buildpackage will be executed. If the VCS Root is configured, the “Working directory” field can be filled out using the TeamCity directory picker, without entering the directory name manually:

After the first build was run, we encountered some problems.

Build problems resolution

Code quality

For some packages (bash in our case), two code trees were not synchronized, i.e. they corresponded to slightly different minor versions, which forced us to use tags rather than branches from the main code tree. Luckily, TeamCity allows building on a tag and the “Enable to use tags in the branch specification” option can be set in such case:

Dependencies

Here is another problem we faced. When running the first build, dpkg-buildpackage exits with code 3 even though all the required dependencies were installed (see the prereqs section above).

There are a few related nuances:

• The most common case for continuous integration is building software from Debian Unstable or Debian Experimental. So, to meet all the dependencies required for the build, TeamCity agents themselves need to run under Debian Unstable or Debian Experimental (otherwise dpkg-buildpackage will be reporting that your stable dependencies versions are too old). To resolve the error, we added the -d parameter:
  dpkg-buildpackage -uc -us -d
• A special case of old dependencies is the configure script, created by the GNU Autotools that are newer than those currently installed on your system. dpkg-buildpackage is unable to detect this, so if the build log shows messages about the missing m4 macros, you need to to re-generate the configure script using the of GNU Autotools currently installed on the agent. This was done by adding the following command as the build step executed before dpkg-buildpackage:
  autoreconf -i

Broken unit tests

When working with Debian Unstable or Debian Experimental, unit test may fail (as it happened in our case). We choose to ignore the failing test and still build the package by running dpkg-buildpackage in a modified environment:
DEB_BUILD_OPTIONS=nocheck dpkg-buildpackage -uc -us

Other build profiles are described here.

Final Stage

After tuning all of the build configurations, our builds finally yielded expected artifacts (the dpkg package is used as an example):

If you do not want each incremental package version to increase the number of artifacts of a single build (giving you dpkg_1.18.16_i386.deb, dpkg_1.18.17_i386.deb, etc.), the working directory should be selectively cleaned up before starting a new build. This was done by adding the TeamCity Swabra build feature to our build configurations.

Tuning up Debian repository

The tcDebRepository plugin adds the “Debian Repositories” tab in the project settings screen, which allows you to create a Debian repository. Once the repository is created, artifact filters need to be configured: at the moment each filter is added manually. Filters can be edited, deleted or copied.

The existing artifacts will not be indexed, therefore after the configuration of the Debian repository, at least one build must be run in every build configuration. Then we can view the available indexed packages:

When adding the repository to the /etc/apt/sources.list, all these packages are visible on the client side. Note that the packages are not digitally signed, which prevents packages from being available on recent versions of Ubuntu. Package signing is planned for tcDebRepository version 1.1.

(!) If you are compiling for several architectures (i386, x32, amd64, arm*), it is worth having several build configurations corresponding to the same package with different agent requirements, or, in addition to the VCS Trigger, using the Schedule Trigger with the “Trigger build on all enabled and compatible agents” option enabled.

Happy building!

* We’d like to thank netwolfuk , the author of the tcDebRepository  plugin, for contributing to this publication.
This article, based on the original post in Russian, was translated by Julia Alexandrova.

There is sometimes a need to define a common build step in a template, so that this step will be executed either before all build configuration steps or after them.

Here’s a simple example. I’m building a lot of typical Maven projects, and for most of the part their build is the same: just run mvn clean package and upload the binaries to the maven repository. However, for some of the builds, I need the additional steps to run in between mvn package and mvn deploy.

In TeamCity, the requirement above can be implemented with the aid of Build Configuration Templates. Starting with version 2018.1, for a given template it is possible to define build steps and then define their placement in respect to build steps defined in a build configuration. The build configuration steps are represented as a placeholder in the Reorder Build Steps dialog. The template steps can be placed before or after this placeholder.

This looks like a neat solution to the problem I described above!

Even more: it is possible to override the inherited build steps right in the build configuration. For the rare occasions, it makes sense! For instance, what if for one of the build configurations I want to upload the binaries to some different repository than the one defined in the template? I could just add altDeploymentRepository parameter to the build step – voila!

The build step will be marked as overridden in the list of all steps of the build configuration.

Ever since TeamCity 2018.2, a test run in TeamCity can be associated with some supplementary information (metadata), in addition to a test’s status, execution time, and output. This information can be used to provide things like extra logs, screenshots, numeric values, and tags.

You can now use service messages to report this kind of additional test data in TeamCity and then view it using the TeamCity Web UI. Consult the corresponding documentation for details.

On GitHub, you can find a sample project that demonstrates the use of metadata with tests. In TeamCity, create a project from the URL using this GitHub repository, and then run the build. Once the build completes, on the Tests tab, next to the test name, you will notice a paperclip icon indicating that the test results include additional metadata.

By clicking the icon, you will open a popup displaying the attached metadata. The same information is presented on the build overview screen. Notice that a numeric value renders a 2D-graph from the historical results.

In the demo project, see CodeTest.kt for the examples of the metadata reporting code.

The Kotlin-based DSL is becoming more popular for defining build configurations in TeamCity. We are continuing to improve the user experience for Kotlin, both in the TeamCity UI and the IDE plugin.

Once the versioned settings option is enabled for the project, we recommend making new changes in the Kotlin configuration rather than in the UI. However, it might be tempting to make the changes in the UI instead of figuring out how to define something in Kotlin. For such occasions, we have added a little UI helper to assist with the DSL definitions.

TeamCity 2018.2 introduces a new option in the UI to help anyone who uses the Kotlin DSL. You will find the new “View DSL” toggle in build configuration settings in the UI. Click the toggle to view the Kotlin DSL definition. The DSL representation opens and highlights the setting in focus.

Check out the screencast demonstrating the new feature in action:

Installing new plugins has just become a lot simpler! You don’t have to restart the server to enable a newly uploaded plugin anymore.

The TeamCity server now integrates with the JetBrains Plugins Repository. The new integration simplifies the installation of plugins for TeamCity.

By clicking the “Browse plugins repository” button in your TeamCity server, you will share some information (URL, server id, and version) with the plugin repository. With this information, the plugin repository is able to suggest installing the selected plugin version directly to your TeamCity server. The screencast below demonstrates the plugin installation procedure.

Aside from this, if you have disabled some of the bundled plugins on your TeamCity server, enabling them also does not require a server restart.

It is also possible to enable a periodical check for plugin updates on your TeamCity server. Once any updates are found in the plugin repository, they can be easily installed through the web interface. However, you’ll still need to restart the server to apply the plugin updates.

Aside from this, if you have disabled some of the bundled plugins on your TeamCity server, enabling them also does not require a server restart.

It is also possible to enable a periodical check for plugin updates on your TeamCity server. Once any updates are found in the plugin repository, they can be easily installed through the web interface. However, you’ll still need to restart the server to apply the plugin updates.

Finally, there is good news for TeamCity plugin developers. If you develop your plugins with the help of our Maven SDK or the Gradle plugin by Rod MacKenzie, you can benefit from reloading plugins without restarting the server. In addition to this, when TeamCity is started from these SDKs, it will work in a mode where the plugins can be enabled or disabled instantly, also without the need for a server restart.

It’s official! TeamCity Google Cloud Deployment Manager template is now available on our site as an easy way to deploy the TeamCity server to Google Cloud.

The template has been around for some time, and its most recent version provides secure deployment of a TeamCity server out of the box.

This post is a short guide to TeamCity GCP deployment. Leveraging the template, you will create a production-ready TeamCity deployment in 10 minutes!
You can also read our post on the same topic in the Google Cloud Blog.

Pre-requisites

1. Make sure you have the permissions of the Project Owner role.
2. Enable the required Google Cloud API’s and you’re good to go:
   > gcloud services enable deploymentmanager.googleapis.com sqladmin.googleapis.com iam.googleapis.com cloudresourcemanager.googleapis.com runtimeconfig.googleapis.com

TeamCity Deployment

Here you have two options: go for a Quick Start or create a secure TeamCity deployment.

A secure TeamCity deployment automatically provides an HTTPS endpoint for the TeamCity server with pre-configured transport security. This helps protect your data and passwords. Creating a secure TeamCity deployment will require a bit more time but, on the bright side, you won’t have to set up the reverse proxy manually!

Quick Start

To quickly create a test TeamCity deployment, use the following command:

> gcloud deployment-manager deployments create teamcity --template https://raw.githubusercontent.com/JetBrains/teamcity-google-template/master/teamcity.jinja --properties zone:<zone>

Notice the <zone> property: it is required for the deployment and refers to the physical location of your computing resources.

To check which zones are available, list all and choose one of them:

> gcloud compute zones list

Refer to the Google Cloud Documentation for more information.

Secure TeamCity Deployment

A secure deployment requires you to have a domain name from a domain registrar of your choice. You should be able to configure an A/CNAME record for this domain.

1. Create an external IP and configure an A record for your domain pointing to that IP. Execute the nslookup <domainName> command on your computer to display the target IP address.
   
2. Create a new TeamCity deployment:
   > gcloud deployment-manager deployments create <deploymentName> --template https://raw.githubusercontent.com/JetBrains/teamcity-google-template/master/teamcity.jinja --properties zone:<zone>,ipAddress:<ipAddress>,domainName:<domainName>,domainOwnerEmail:<domainOwnerEmail>

The command above should specify the following properties:

• deploymentName: a unique name for the deployment, e.g. “teamcity”
• zone: the deployment zone
• ipAddress: the external IP address
• domainName: your domain name

The domainOwnerEmail property is optional; set it to your email to be notified in the case of problems with an SSL certificate retrieval.

Launching TeamCity

Deploying TeamCity will take a few minutes. After the deployment is complete, a link to the TeamCity server should appear in the gcloud command output.

The TeamCity server will have two pre-installed Google Cloud plugins. Refer to this blog post for more information about their configuration and functionality.

Happy building in Google Cloud!