Building Vagrant Boxes with VeeWee on TravisCI

(Pro Tip: you can safely skip the first 3 paragraphs.)

We’ve all been there: You push some .travis.yml commits and your clone gets parachuted into VM Land – only to find that things don’t go quite as expected. As the credits roll, you can’t help but feel a little anger towards your clone. How could it just blindly follow the script when things went wrong and not even try to improvise or troubleshoot? You wonder – how can I get a Pastrana-level of confidence before I push that clone out the door?

How can I run the Travis box locally, before I push my commits? (Let’s pretend I didn’t just spend days investigating this and then just now find a blog entry from 3 months ago which had a more polished solution for most of this topic and then I had to switch gears at the last minute and only cover an obscure corner of the topic, mmmkay? Embarrassed but undeterred, I move forward…)

We will continue by reviewing the original problem I faced that drove me to explore running a Travis box locally. You will probably never run across the problem. But hey – at least it’s unlikely that someone has already blogged about this topic!

The Goal: I Wanted to Build Vagrant Boxes with VeeWee on TravisCI

Q: Why run VeeWee on Travis?
A: The transparency offered by Travis would allow people to trust the published binaries. The binary boxes would be published by Travis at the end of the build to the GitHub repo’s "download" section.

So, I started by proving out that I could run VirtualBox inside Travis - https://travis-ci.org/veewee-community/travis-vagrant-up/builds/3427898/#L293 (compare the before/after directory listings).

$ ls
  README.md  vagrant_1.0.5_i686.deb  Vagrantfile
$ /opt/vagrant/bin/vagrant ssh --command "touch /vagrant/smash_it_with_a_hammer"
$ ls
  README.md  smash_it_with_a_hammer  vagrant_1.0.5_i686.deb  Vagrantfile

Encouraged, I continued.

I added VeeWee to the mix, and ran into this:

Starting a webserver :7122

Executing your script (veewee vbox build 'mybox' --auto --debug --nogui) 
took longer than 25 minutes and was terminated.

For more information about the test timeouts please checkout the section 
Build Timeouts at http://about.travis-ci.org/docs/user/build-configuration/.
Done. Build script exited with: 1

It kept sticking on the "Starting a webserver :7122″ line until the Travis time limit kicked in and stopped the build. Either the step was legitimately taking too long or there was some kind of error that didn’t show up in the log. What was the build trying to tell me? I needed to get inside the box’s head. But wait, the box had no head – it was headless.

Lending Travis a Helping Head

I needed to run Travis locally and see what might be trying to pop up on the GUI. So, I started by upping this Vagrantfile:

Vagrant::Config.run do |config|

  config.vm.box = "headed-travis-ruby-box"
  config.vm.box_url = "http://files.travis-ci.org/boxes/provisioned/travis-ruby.box"
  config.vm.network :hostonly, "192.168.33.10"
  config.ssh.username = "travis"

end

(Coffee break: that Vagrant box download is over 3 gig.)

After studying Travis’ GUI testing documentation, I SSHed-in to my newly upped box and ran some stuff to enable a virtual X session and publish it over VNC. I also enabled a applet-based browser-view of the VNC session:

$ sudo apt-get install x11vnc vnc-java xterm openbox
$ export DISPLAY=:99.0
/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid 
--make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16
$ openbox-session &
$ x11vnc -display :99 -httpdir /usr/share/vnc-java -httpport 5800 -loop &

Then, I started running the commands from my .travis.yml file:

language: ruby
rvm:
  - "1.9.3"
install:
  - sudo echo "deb http://download.virtualbox.org/virtualbox/debian precise contrib" | 
    sudo tee -a /etc/apt/sources.list
  - wget http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
  - sudo apt-get update
  - sudo apt-get -y install linux-headers-$(uname -r)
  - sudo apt-get -y install virtualbox-4.2
  - VBoxManage -v
  - gem install veewee
script:
  - veewee vbox define 'mybox' 'ubuntu-11.10-server-amd64'
  - date
  - veewee vbox build 'mybox' --auto --debug --nogui
  - date

Once I ran the "veewee vbox build" command, it just sat there. I browsed to the virtual X session in my browser and watched to see what popped up. Ah, finally I was getting somewhere:

I spun my wheels for a while thinking I needed to fork VeeWee and suppress the warning to continue the build process, but ran into a snag revealed by the VirtualBox code:

bool UIMessageCenter::warnAboutVirtNotEnabled64BitsGuest(bool fHWVirtExSupported)
{
  ...
    else
        return messageOkCancel(mainWindowShown(), Error,
            tr("<p>VT-x/AMD-V hardware acceleration is not available on your system. "
                "Your 64-bit guest will fail to detect a 64-bit CPU and will "
                "not be able to boot."),
            0 /* pcszAutoConfirmId */,
            tr("Close VM"), tr("Continue"));
}

Translation: No "pcszAutoConfirmId" means no way to suppress! Actually, it didn’t matter that I couldn’t suppress the message because the VM really couldn’t have run anyways – here’s what happens if I actually select "continue" to proceed past that dialogue:

I was so dumb because when I first did the proof-of-concept for running VirtualBox in Travis, Vagrant upped a 32-bit box, and now I was trying to create a 64-bit box. Well, the Travis box is 32-bit (for now) and VirtualBox has decided not to offer nested VT. (Nested VT, afaik, would allow the 64-bit inside 32-bit scenario) Bummer.

So, I changed the VeeWee template choice to request a 32-bit box… Hoorah! After 65 minutes, the Vagrant box was all built. But when I ran it in the actual Travis service instead of locally, this is what I saw:

Starting a webserver :7122
Serving file /home/travis/builds/veewee-community/travis-veewee-up/definitions/mybox/preseed.cfg
Waiting for ssh login on 127.0.0.1 with user vagrant to sshd on port => 7222 to work, timeout=10000 sec
.......................................................................................................
.......................................................................................................

Executing your script (veewee vbox build 'mybox' --auto --debug --nogui) 
took longer than 25 minutes and was terminated.

Oh well, it timed out! But at least it got past the earlier snag and would have built given enough time. I’m not complaining though, I can completely understand the reasoning behind the time limits Travis enforces. A 65 minute build on a free Travis account type is a burden on the other people trying to build stuff.

So, that’s where I got stuck. I’m just glad I didn’t actually promise you a real solution, in which case you might have been angry at me. Thanks.

– Luke Patterson