AspNetDeploy: CI + Deployment tool (preview)

Introduction

Every time I have to set up continuous integration and automatic deployment, I want to cry. I miss magic.

I want to have a tool that will be aware of my source control, projects, their versions and deployment options to different environments without spending hours writing XML configs or command line scripts.

Every single existing tool seems working wrong for me: one can build but cannot deploy, other can build and deploy but I have to write tons of scripts.

So I decided to start AspNetDeploy.com project, an open source tool that will work right.

Sources: GitHub

Why?

Given a source control with ASP.NET solution and a bunch of servers I want simple thing to happen:

1. Take sources and understand what's inside solution and how to build it automatically
2. Form bundles from projects which cannot be deployed separately
3. Form environments and simple deployment plan with easy to maintain variables
4. One click deploy with other team members approve.
5. Straightforward new version / hotfix mechanism

Alternatives

• TeamCity can build but cannot deploy, octopus deploy integration is not straightforward
• Octopus deploy can deploy (surprise! :) ) and aware of ASP.NET but cannot build and package versioning is not straightforward
• Bamboo can build and deploy all the things, but one cannot simply build and deploy ASP.NET app without unclear batch scripting and NAnt xml
• CruiseControl is doomed
• TFS scares me (and can't work with SVN afaik)
• We do not like to store source code online (we are not alone right?) and do not live in clouds, so online build-services won't do the trick.

One day I started to google how to build solution, take sources etc. and it turned out every single operation can be programmed with few lines of code. Moreover, one day i came across this beautiful icon:  And my thought was "This is it! I have to make continuous integration tool with this magical smooth icon!" :)

This is where AspNetDeploy story starts!

 

General view

SourceControlManager

Is responsible for taking sources and parsing projects.

Notice project types, build and package timing

The idea is simple: 

First we add source control (VCS root in terms of TeamCity) one per project, after that we add versions. Once AspNetDeploy take sources it search for *.sln files, parse them and then parse project files to see what projects are inside each solution.

Take sources

Every time source control manager is up to take sources it asks ISourceControlRepositoryFactory for right implementation of ISourceControlRepository and call LoadSources

namespace AspNetDeploy.SourceControls
{
    public class SourceControlRepositoryFactory : ISourceControlRepositoryFactory
    {
        public ISourceControlRepository Create(SourceControlType type)
        {
            switch (type)
            {
                case SourceControlType.Svn:
                    return new SvnSourceControlRepository();

                case SourceControlType.Git:
                    return new GitSourceControlRepository();

                case SourceControlType.FileSystem:
                    return new FileSystemSourceControlRepository();

                default:
                    throw new AspNetDeployException("Unknown SourceControlType: " + type);
            }
        }
    }
}

SvnSourceControlRepository using SharpSvn

Before taking sources we have to check, are we loading sources first time or there is existing SVN folder and calling Update will be enough.

Instead of making table per class database structure, I decided to make generic Property table for storing specific settings for source controls and source control versions.

public LoadSourcesResult LoadSources(SourceControlVersion sourceControlVersion, string path)
{
    NetworkCredential credentials = new NetworkCredential(
                sourceControlVersion.SourceControl.GetStringProperty("Login"),
                sourceControlVersion.SourceControl.GetStringProperty("Password"));

    using (SvnClient client = new SvnClient())
    {
        client.Authentication.DefaultCredentials = credentials;

        if (!Directory.Exists(path))
        {
            return this.LoadSourcesFromScratch(sourceControlVersion, path, client);
        }

        return this.LoadSourcesWithUpdate(path, client);
    }
}

Taking sources first time:

private LoadSourcesResult LoadSourcesFromScratch(SourceControlVersion sourceControlVersion, string path, SvnClient client)
{
    SvnUpdateResult result;
    Directory.CreateDirectory(path);

    string uriString = this.GetVersionURI(sourceControlVersion);

    client.CheckOut(new Uri(uriString), path, out result);

    SvnInfoEventArgs info;
    client.GetInfo(path, out info);

    return new LoadSourcesResult
    {
        RevisionId = info.LastChangeRevision.ToString(CultureInfo.InvariantCulture)
    };
}

Calling update on existing folder

private LoadSourcesResult LoadSourcesWithUpdate(string path, SvnClient client)
{
    SvnUpdateResult result;

    try
    {
        client.Update(path, out result);
    }
    catch (SvnWorkingCopyException e)
    {
        client.CleanUp(path);
        client.Update(path, out result);
    }

    SvnInfoEventArgs info;
    client.GetInfo(path, out info);

    return new LoadSourcesResult
    {
        RevisionId = info.LastChangeRevision.ToString(CultureInfo.InvariantCulture)
    };
}

 

Parse solution file

As you may noticed, regular sln file look like this:

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 2013
VisualStudioVersion = 12.0.31101.0
MinimumVisualStudioVersion = 10.0.40219.1
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "WebUI", "WebUI\WebUI.csproj", "{EE1686C9-1D29-4D7F-AB8A-E05A70003A5C}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "WebClient", "WebClient\WebClient.csproj", "{24F17881-DD9F-4007-B66B-70E645BDFDC6}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Helpers", "PathHelper\Helpers.csproj", "{6D11DE3A-7E2D-4223-902A-411093EB02A3}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Model", "Model\Model.csproj", "{395E2908-7FBD-4153-A332-4A92DEF6FE3E}"
EndProject
Project("{00D1A9C2-B5F0-4AF3-8072-F6C62B433612}") = "Database", "Database\Database.sqlproj", "{78F8DB8E-207F-4FBD-A5A3-EE8ECFCCB351}"
EndProject

....

Project

• TypeGuid
  • E3E379DF-F4C6-4180-9B81-6769533ABE47 – MVC 4
  • E53F8FEA-EAE0-44A6-8774-FFD645390401 – MVC 3
  • F85E285D-A4E0-4152-9332-AB1D724D3325 – MVC 2
  • 603C0E0B-DB56-11DC-BE95-000D561079B0 – MVC 1
• Name
• Path
• Id

This information is good but not enough to understand what is it project exactly, we need to go deeper:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="12.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
  <PropertyGroup>
    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
    <ProductVersion>
    </ProductVersion>
    <SchemaVersion>2.0</SchemaVersion>
    <ProjectGuid>{EE1686C9-1D29-4D7F-AB8A-E05A70003A5C}</ProjectGuid>
    <ProjectTypeGuids>{349c5851-65df-11da-9384-00065b846f21};{fae04ec0-301f-11d3-bf4b-00c04f79efbc}</ProjectTypeGuids>
    <OutputType>Library</OutputType>
    <AppDesignerFolder>Properties</AppDesignerFolder>
    <RootNamespace>InnovativeManagementSystems.BackgroundCMS.WebUI</RootNamespace>
    <AssemblyName>InnovativeManagementSystems.BackgroundCMS.WebUI</AssemblyName>
    <TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
    <UseIISExpress>false</UseIISExpress>
    <IISExpressSSLPort />
    <IISExpressAnonymousAuthentication />
    <IISExpressWindowsAuthentication />
    <IISExpressUseClassicPipelineMode />
    <TargetFrameworkProfile />
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
    <DebugSymbols>true</DebugSymbols>
    <DebugType>full</DebugType>
    <Optimize>false</Optimize>
    <OutputPath>bin\</OutputPath>
    <DefineConstants>DEBUG;TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
    <DebugType>pdbonly</DebugType>
    <Optimize>true</Optimize>
    <OutputPath>bin\</OutputPath>
    <DefineConstants>TRACE</DefineConstants>
    <ErrorReport>prompt</ErrorReport>
    <WarningLevel>4</WarningLevel>
  </PropertyGroup>
  <ItemGroup>
    <Content Include="Resources\Layout\bootstrap-theme.css" />
    <Content Include="Resources\Layout\bootstrap-theme.min.css" />
    <Content Include="Resources\Layout\bootstrap.css" />
    <Content Include="Resources\Layout\bootstrap.min.css" />

  .....

</Project>

• ProjectTypeGuids
  • FAE04EC0-301F-11D3-BF4B-00C04F79EFBC – Class library
  • A9ACE9BB-CECE-4E62-9AA4-C7E7C5BD2124 – Database
  • 00D1A9C2-B5F0-4AF3-8072-F6C62B433612 – Database again
  • 3AC096D0-A1C2-E12C-1390-A8335801FDAB – Test project
  • 349C5851-65DF-11DA-9384-00065B846F21 – Web project
  •  
• OutputType
  • Exe – Console app
  • WinExe – Windows app
  • Database – Database project
• OutputPath – this is where compiled binaries will end up, this folder among others will be inluded in package
• ItemGroup / Content / Include – are files to be included in publication

Logic

1. Get types from ProjectTypeGuids
2. Look at OutputType to see if it Console, Windows app or Database project
3. Look for UseIISExpress, if exists – web project
   – look at TypeGuid in sln file to see what kind of MVC version it is

Bundles

Bundle is a set of projects to be deployed together.

Each bundle has to have bundle versions where deployment steps are defined. When new bundle version is created, deployment steps seamlessly got converted to work with new version's projects and settings.

Building a bundle produce package, which can be manually or automatically deployed to first environment in a chain test -> qa -> live and then promoted to next environment with one click.

 

Building solution with MSBuild

It all starts from BuildServiceFactory

namespace AspNetDeploy.BuildServices
{
    public class BuildServiceFactory : IBuildServiceFactory
    {
        private readonly INugetPackageManager nugetPackageManager;

        public BuildServiceFactory(INugetPackageManager nugetPackageManager)
        {
            this.nugetPackageManager = nugetPackageManager;
        }

        public IBuildService Create(SolutionType project)
        {
            return new MSBuildBuildService(this.nugetPackageManager); // only one build service at this time
        }
    }
}

MSBuildService

Well, this code is more like proof of concept than state-of-art but it does what it intended to do:

using System;
using System.Collections.Generic;
using System.IO;
using AspNetDeploy.Contracts;
using AspNetDeploy.Model;
using Microsoft.Build.Evaluation;
using Microsoft.Build.Execution;
using Microsoft.Build.Framework;

namespace AspNetDeploy.BuildServices.MSBuild
{
    public class MSBuildBuildService : IBuildService
    {
        private readonly INugetPackageManager nugetPackageManager;

        public MSBuildBuildService(INugetPackageManager nugetPackageManager)
        {
            this.nugetPackageManager = nugetPackageManager;
        }

        public BuildSolutionResult Build(string solutionFilePath, Action<string> projectBuildStarted, Action<string, bool, string> projectBuildComplete, Action<string, string, string, int, int, string> errorLogger)
        {
            ProjectCollection projectCollection = new ProjectCollection();

            Dictionary<string, string> globalProperty = new Dictionary<string, string>
            {
                {"Configuration", "Release"}, 
                {"Platform", "Any CPU"}
            };

            BuildRequestData buildRequestData = new BuildRequestData(solutionFilePath, globalProperty, null, new[] { "Rebuild" }, null);

            BuildParameters buildParameters = new BuildParameters(projectCollection);
            buildParameters.MaxNodeCount = 1;
            buildParameters.Loggers = new List<ILogger>
            {
                new NugetPackageRestorer(nugetPackageManager, Path.GetDirectoryName(solutionFilePath)),
                new MSBuildLogger(projectBuildStarted, projectBuildComplete, errorLogger)
            };

            BuildResult buildResult = BuildManager.DefaultBuildManager.Build(buildParameters, buildRequestData);

            return new BuildSolutionResult
            {
                IsSuccess = buildResult.OverallResult == BuildResultCode.Success
            };
        }
    }
}

Using loggers, we can hook to ProjectStarted, ProjectFinished and ErrorRaised events. This allows us to measure build time of each project as well as to point specific project where build failed.

One of a problems I faced was missing NuGet packages which are not part of solutions and not in source control. This packages logic probably should have been placed somewhere in SourceControlManager.

Restoring packages with NuGet

It turned out it is easier to call NuGet.exe to pull all missing packets since it has no C# API like MSBuild.

namespace BuildServices.NuGet
{
    public class NugetPackageManager : INugetPackageManager
    {
        private readonly IPathServices pathServices;

        public NugetPackageManager(IPathServices pathServices)
        {
            this.pathServices = pathServices;
        }

        public void RestorePackages(string packagesConfigPath, string solutionDirectory)
        {
            Process process = new Process();
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.RedirectStandardOutput = true;

            process.StartInfo.FileName = this.pathServices.GetNugetPath();
            process.StartInfo.Arguments = string.Format(
                "install \"{0}\" -source \"{1}\" -solutionDir \"{2}\"",
                packagesConfigPath,
                "https://www.nuget.org/api/v2/",
                solutionDirectory);

            process.Start();

            process.WaitForExit();
        }
    }
}

call sample:

this.nugetPackageManager.RestorePackages(@"C:\MySolution\MyProject\packags.config", @"C:\MySolution");

(packagesConfigPath may differ from solutionDirectory)

 

To be continued

• Deployment steps
• Packaging projects
• Environments and machines
• Variables
• Deploying 
  • Running agents as windows services with hosted WCF interface
  • Uploading packages and deployment steps
  • Making secure connection
• Logging
• Users and roles
• Putting all together

Whats next?

I'm looking for feedback and for brave people to try to run all this. Probably someone will be interested to join this project.

History

To be updated