Referencing native projects in Visual Studio (MSBuild).

• Download sample - 1.2 MB
• Download Source - 28.1 KB

Introduction

This is a fourth article in series about integrating third party tools and libraries into Visual Studio. In my first article I've explained how to create custom property pages for Visual Studio Property dialog box. Second article covered internal structure and elements of the property sheet. Third article explains how to create custom builds by example of building Boost library. In this, fourth article I'll explain how to integrate custom build into Visual Studio project reference system.

Rationale

Every C++ project consists of several smaller subprojects and libraries. These are used to link to at either compile or run time and should be properly referenced. If all of the projects are created inside Visual Studio (MSBuild) referencing is taken care of by the MSBuild. But when a project or library comes from outside, we have to resort to manual configuration in order to integrate it properly. 

Ideally we should be able to integrate a project into MSBuild by adding a reference to it in Visual Studio:

Wouldn't it be nice if we could do this with any library? All of the lib files would be automatically added to LINK command and all of the DLLs are copied to Output directory so they could be linked to at run time? What about being able to debug not just our code but the library code as well?

In this article I will show you how this could be accomplished. I will use Boost library to demonstrate how it could be integrated into any project with no need for manual library manipulation or paths setting. I am assuming you already know how Boost could be built. If not read this article. 

Background

When Visual Studio adds reference from one project to another it adds record into main project like this:

<ProjectReference Include="...\boost.vcxproj">
    <Project>{9cd23c68-ba74-4c50-924f-2a609c25b7a0}</Project>
    ...
</ProjectReference>

For details on how references are added please follow this link.

During build of main project MSBuild tries to resolve and build all of project's dependencies listed in the ProjectReference sections. It locates listed subprojects and calls following Targets on each of them to gather required information:

GetTargetPath
GetNativeManifest
GetResolvedLinkLibs
GetCopyToOutputDirectoryItems

I will briefly explain what each of them does.

GetTargetPath

This Target returns full path to the assembly/library bulit by the project. At design time Visual Studio uses this file to determine if reference is correct and the output file could be found. If assembly is of managed type it also queries it for extra information.
Theoretically as long as this path points to an existing file, reference system is happy and reports reference as valid. We cant use this and return path to any arbitrary file to indicate that reference is OK. I've decided to return path to Jamroot file to indicate which source this build uses to create the library:

<Target Name="GetTargetPath" DependsOnTargets="GetNativeTargetPath"  Returns="@(NativeTargetPath)" >
  <ItemGroup>
    <NativeTargetPath Include="$(BoostRoot)\Jamroot" Condition="'$(DesignTimeBuild)' == 'true'" />
  </ItemGroup>
</Target>

GetNativeTargetPath

Native projects must implement GetNativeTargetPath to return list of output files. At run time this target is called and returns list of DLL libraries this project builds as well as all of the dependent DLLs it binds to.

<Target Name="GetNativeTargetPath" DependsOnTargets="GetBuiltLibs" Returns="@(NativeTargetPath)" >
  <MSBuild Projects="@(ProjectReference)" Targets="GetNativeTargetPath" >
    <Output ItemName="NativeTargetPath" TaskParameter="TargetOutputs"/>
  </MSBuild>
  <ItemGroup>
    <NativeTargetPath Include="$(OutputDir)\lib\*boost*%(BuiltLibs.Identity)*.dll" />
  </ItemGroup>
</Target>

The code calls GetNativeTargetPath for every project it references and adds its own DLLs created during the build.

It is worth noting that dependency DLLs could be copied locally to master project's output directory alone with boost files based on the setting set in Add References dialog box:

Setting Copy Local to true will copy dependencies as well.

GetNativeManifest

If for some reason subproject has to redistribute Manifest files alone with its libraries, this Target returns a list of manifest files. The parent project will simply copy these manifests into output directory.

Boost does not require any manifests so it will do nothing:

<Target Name="GetNativeManifest" />

GetResolvedLinkLibs

This Target returns a list of all the link libraries project exposes as well as all the libraries it links to. This list is added to LINK command so these lib files could be resolved and linked to by a master project. Boost library would have a lib file for each module it builds.

For us to return correct list of libraries we have to get the list of libraries current configuration requires and then resolve these to actual lib files. We do that in two steps:

• call b2 with current options alone with --show-libraries command (GetBuiltLibs)
• resolve references to libraries and add them to return list (GetResolvedLinkLibs)

<Target Name="GetBuiltLibs" DependsOnTargets="BuildJamTool" Returns="@(BuiltLibs)" >
  <Exec Command="b2.exe @(boost-options, ' ') --show-libraries" ... />
    
    <ReadLinesFromFile Condition="Exists('$(TempFile)')" File="$(TempFile)">
      <Output TaskParameter="Lines" ItemName="RawOutput" />
    </ReadLinesFromFile>
    <Delete Condition="Exists('$(TempFile)')" Files="$(TempFile)"/>
    
    <ItemGroup>
      <BuiltLibs Include="$([Regex]::Match(%(RawOutput.Identity), (?&lt;=\-\s)(.*) ))" />
    </ItemGroup>
  </Target>

Please note: sample code here and throughout the article is simplified for clarity

<Target Name="GetResolvedLinkLibs" DependsOnTargets="GetBuiltLibs" Returns="@(LibFullPath)">
  <ItemGroup>
    <LibFullPath Include="$(OutputDir)\lib\*boost*%(BuiltLibs.Identity)*.lib">
      <ProjectType>StaticLibrary</ProjectType>
      <ProjectType Condition="'$(boost-link)'!=''">$(boost-link)</ProjectType>
      <FileType>lib</FileType>
      <ResolveableAssembly>false</ResolveableAssembly>
    </LibFullPath>
  </ItemGroup>
  <MSBuild Projects="@(ProjectReference)" Targets="GetResolvedLinkLibs" >
    <Output ItemName="LibFullPath" TaskParameter="TargetOutputs"/>
  </MSBuild>
</Target>

It is important to note that returned items should have metadata added to it. This metadata is used by build engine to determine how to link to these files. Each item should at least have ProjectType and FileType set on them.
ProjectType could be StaticLibrary or DynamicLibrary. FileType may contain either lib or dll.

GetCopyToOutputDirectoryItems

This target returns a list of all content files that are require to be copied to output folder of the main project. It could be any type of file. In case of Boost library we dont cave any content:

<Target Name="GetCopyToOutputDirectoryItems" />

Now if we add boost project as reference it will register as valid and provide all the information parent project could require.

Building with boost

We start off with creating a simple console application with very original name Sample. Everyone knows how to create a console app in Visual Studio so I'll skip the instructions.

Add project boost to the solution.

Go to properties for project Sample and Add Reference to the boost project. You should see something like this:

As you can see in this picture project boost is correctly referenced and points to Boost library installed at D:\Boost directory. Assembly Name, Culture, Version, and Description are not available because Boost is not a managed assembly. 

It is worth to note that Copy Local property determines if library should be copied to output directory of the super project. If subproject builds managed assembly or just a lib file it all works fine. But if result of subproject is a native DLL or multiple libraries the whole process breaks. We fixed it by redefining GetCopyToOutputDirectoryItems. Now to control whether DLLs are copied to output directory of master project or not we need to add extra property on General tab of Boost property page:

Setting this property to No will disable the copy. This setting is only applicable when Boost library is built as shared and has no effect on static build. 

Incremental Build

Every time we build, b2 checks configuration and determines if it needs to build some of the components. When Boost is used to develop other projects it is very rare that library itself has changed in any way. So checking for changes is rather redundant. I've added an option to the General tab of the property page which disables that check:

When this option is Yes or empty, checking for rebuild conditions is delegated to Visual Studio. It checks list of output libraries against a list of configured libraries and against project file itself. If any of the libraries have been deleted or project settings have been changed it will run the build. Otherwise it will skip it, saving almost half a minute during each build. To re enable the check set this option to No. 

Delegating these checks to Visual Studio requires following components:

Build Outputs

List of built libraries could be inferred by examining output of command: b2 --show-libraries. Once we have the list we could verify what libraries already present by calling Target GetBoostOutputs.

<Target Name="GetBoostOutputs" DependsOnTargets="GetBuiltLibs" Returns="@(BoostOutputs)" >

  <ItemGroup>
    <BoostOutputs Include="$(OutputDir)\lib\*boost*%(BuiltLibs.Identity)*.lib" >
       <Library>%(BuiltLibs.Identity)</Library>
    </BoostOutputs>
    <ExistingLibs Include="%(BoostOutputs.Library)" />
    <BoostOutputs Include="@(BuiltLibs)" Exclude="@(ExistingLibs)" 
                  Condition="'@(ExistingLibs->Count())'!='@(BuiltLibs->Count())'" />
    <BoostOutputs Include="%(RootDir)%(Directory)%(Filename).dll"                  
                  Condition="'@(BoostOutputs0>Filename->StartsWith(&#34;boost_&#34;))'=='true' And 
                             '%(BoostOutputs.Library)'!='' And '$(boost-link)'=='DynamicLibrary'" />
  </ItemGroup>
</Target>

As you can see above we get a list of libraries from GetBuiltLibs target and search for all the lib files with names that follow following pattern: *boost*<library-name>*.lib. It returns a list containing both link libraries for DLL as well as static libraries.
In next step we create inner join on a list of libraries with built files and use it to filter out missing libraries.
Next we add missing libraries to the BoostOutputs to trigger buld if required.
In following step we add dll libraries.

This list will be checked by the Build Target to determine if anything needs to be executed.

Settings

We still need to specify one setting in the application consuming Boost library. We need to tell it where all these headers are. It is done by adding $(BOOST_BUILD_PATH) to the list of Additional Include Directories (assuming the environment variable has been set). 

Debugging Boost

One of the purposes of this article was to demonstrate how integration with Visual Studio allows seamless debugging of not only the application itself but Boost library as well.

I used example from boost\libs\lockfree\examples\queue.cpp to demonstrate this ability.

boost::atomic_int producer_count(0);
boost::atomic_int consumer_count(0);

boost::lockfree::queue<int> queue(128);

const int iterations = 10000000;
const int producer_thread_count = 4;
const int consumer_thread_count = 4;

void producer(void)
{
    for (int i = 0; i != iterations; ++i) {
        int value = ++producer_count;
        while (!queue.push(value))
            ;
    }
}

boost::atomic<bool> done(false);

void consumer(void)
{
    int value;
    while (!done) {
        while (queue.pop(value))
            ++consumer_count;
    }

    while (queue.pop(value))
        ++consumer_count;
}

int _tmain(int argc, _TCHAR* argv[])
{
    using namespace std;
    cout << "boost::lockfree::queue is ";
    if (!queue.is_lock_free())
        cout << "not ";
    cout << "lockfree" << endl;

    boost::thread_group producer_threads, consumer_threads;

    for (int i = 0; i != producer_thread_count; ++i)
        producer_threads.create_thread(producer);

    for (int i = 0; i != consumer_thread_count; ++i)
        consumer_threads.create_thread(consumer);

    producer_threads.join_all();
    done = true;

    consumer_threads.join_all();

    cout << "produced " << producer_count << " objects." << endl;
    cout << "consumed " << consumer_count << " objects." << endl;
    return 0;
}

Setting breakpoint at line 59 (Sample.cpp): on producer_threads.join_all(); call allows us to step into join_all (thread_group.hpp)

void join_all()
{
    BOOST_THREAD_ASSERT_PRECONDITION( ! is_this_thread_in() ... );
    boost::shared_lock<shared_mutex> guard(m);
    for(std::list<thread*>::iterator it=threads.begin(),end=threads.end(); it!=end; ++it)
    {
        if ((*it)->joinable())
            (*it)->join();
    }
}

Stepping into (*it)->joinable() on line 117 will open thread.cpp on line 445:

bool thread::joinable() const BOOST_NOEXCEPT
{
    detail::thread_data_ptr local_thread_info = (get_thread_info)();
    if(!local_thread_info)
    {
       return false;
    }
    return true;
}

and let you step through that as well. Location of the cpp file is inferred from debug information embedded into the library. Since stored path is current and relative to the project, Visual Studio does not require any additional information to resolve the file.

Using the code

I've provided two downloads: Sample and Source.

Source download contains three files (project, targets and xml) required to build Boost library. Copy it into a directory. Specify location of Boost library's root and build.

Sample download contains a Solution with two projects: Boost-MSBuild and Sample. When built and executed these will allow you to see the functionality described it this article in action.

History

03/182015 - Published
03/20/2015 - Corrected few omissions
03/31/2015 - Added info for native projects