Deploy Static Web Page from GitLab to Amazon S3 using Jenkins

Introduction

This article provides instructions to setup a basic deployment using Jenkins to Amazon S3 by pulling the source code from GitLab and pushing it to a S3 bucket. Deployment automation using Jenkins saves us from manually uploading files to the S3 bucket using AWS online console. In addition to saving time, this method pushes the exact code that was committed in the latest Git push, therefore avoiding upload of wrong files. This is useful for static web site stored on S3 having a large number of files.

Pre-requisite

OS: Mac OS but the main ideas are OS agnostic.

It is assumed that you already have a static webpage on AWS S3. If not, you can check out instructions from AWS documentation.

You should also have your project added in GitLab. If not, you can have a look at their documentation.

Install Jenkins

1. Download Jenkins and install (Current LTS version as of this writing is 2.89.2)
2. After the installation, change the Update Site url from https to http:

   Jenkins->Plugin Manager->Advanced (tab)->Update Site: http://updates.jenkins.io/update-center.json

3. Install Git Plugin (3.7.0)

   Jenkins->Plugin Manager->Available (tab)

Optional. Install Green Balls plugin to make the blue success ball green.

Setup to Pull Project from GitLab

Establish SSH authentication between your local machine and GitLab.

1. Get the Private Key inside the file /.ssh/id_rsa and create a Git Credential.

   This site shows you how to create a Pair of Ssh Key if you do not have one: https://docs.gitlab.com/ce/ssh/README.html

2. Create a Git credential in Jenkins.

   Jenkins->Credentials->System->Global credentials (unrestricted) -> Add Credentials

   1. Kind: SSH Username with private key
   2. Select Private Key and paste the private key
   3. Press OK

   

3. Add Git username and email to Jenkins Git properties.
   1. Manage Jenkins->Configure System
   2. Add your Git username and email under Git plugin section

   

4. Create a project/job in Jenkins.

   Jenkins->New Item

   1. Enter any name for this Job
   2. Select Freestyle Project
   3. Press OK

   

5. Configure Source Code Management to pull the project from Git.
   1. Fill in Git project URL and select credential to use (created in step 2) under Source Code Management tab.
   2. Press Save.

   

6. At this point, you can test to see if the Jenkins job can retrieve your project from Git by clicking Build Now.

   

   If successful, you should see a blue or green ball. You can check the logs by clicking on the blue/green ball.

   

   To check if the project is correctly pulled from git to Jenkins workspace, click on Workspace. You should see the same files that are in your Git repository.

   

   Pushing Project to Amazon S3

7. Create a user to allow access to your S3 bucket.
   1. In the online AWS console->IAM (service)->Users
   2. Click Add user
   3. Enter any username you want
   4. Check Programmatic access
   5. Click Next Permission
   6. Select tab Attach Policy Directly
   7. Search for 's3'
   8. Select AmazonS3FullAccess
   9. Click Next Review
   10. Click Create user
   11. Write down the user Access Key ID and Secret Access Key as they will not be available anymore after this point.
   12. Click Close

   

   

   

   

8. Install and configure S3cmd (command line client to access S3).

   In a terminal:

   1. Install PIP if not already installed, to use it to install S3CMD: sudo easy_install pip
   2. Install S3cmd: sudo pip install s3cmd
   3. Configure S3cmd: s3cmd —configure
   4. Enter the Access Key ID and Secret Access Key and accept default for other questions. (Except if your region is other than US, enter a different region.)
   5. Type which s3cmd to find the location of S3cmd (mine is "/usr/local/bin/s3cmd")
9. Update the Jenkins job to use S3cmd to push code to S3:
   1. Return to the job created in Jenkins and click Configure
   2. Go to the Build section
   3. Enter:

      PATH="/usr/local/bin:$PATH"
      s3cmd sync . s3://optimus.com --delete-removed --exclude '.git/'

      Explanation:
      
      The first line tells Jenkins where to find s3cmd command by adding its location to PATH.
      The second line uses s3cmd to copy all (.) to a S3 bucket name optimus.com
      --delete-removed will remove deleted files from S3
      --exclude '.git/' excludes the git directory to sync to S3.

   4. Click Save
   5. Click Build Now

By checking the job’s log, you should see that Jenkins pulled the code from Git and pushed it to AWS S3.

That’s it! I hope this helped you.

History

• 1^st January, 2018: Initial version