CodeBuild cache for Jekyll build

For the past couple of years, this blog is run from AWS with iterative improvements to make publishing easier for me. Using a static site framework like Jekyll minimized the need for a “Linux Apache MySQL PHP” (LAMP) stack which used to be the de-facto standard. The problem is that because of the amount of content, the build times to publish were steadily increasing. In this post, I’ll describe the relatively easy fix that resulted in a 70% improvement.

Jekyll is an awesome, content management system that requires very little infrastructure. The entire project with extras takes up less than 150MB on disk. My workflow is to develop and test new content on my workstation and checking changes with git when appropriate. Once I’m ready to publish, I push my local repository changes upstream to a CodeCommit repository which kicks off a CodeBuild project. (For more information on how this is configured, check out the post Magic of CloudWatch Events and CodeBuild) When I first started, the complete build and publication to S3 was taking on average 90 seconds which was great. Since then, due to the addition of a couple of extra Jekyll plugins, the time now averages 225 seconds. That’s a 250% increase! I knew that I had to wrangle this issue before it became worse.

The first step in any performance evaluation is to benchmark each step. Thankfully, CodeBuild tracks the start, and end times for each step of the build so I didn’t have to look far. Once I looked at the data, it was pretty obvious where the problem was.

It was time to dig into the build phase of the project to understand why it was taking so long. Enter the CloudWatch Log integration with CodeBuild. For each build in a project, CodeBuild creates (if configured properly) a specific log stream for each build. The contents of the log stream includes any stdout/stderr output during every build phase. As soon as I reviewed the log stream for the last build, I knew why the build phase was taking so long.

In my buildspec.yaml for the project, because I’m using the default AWS container for the build, I am required to install my runtime (in this case ruby), and its respective dependencies. This process also includes bundler, all of the Jekyll gems, and their dependents. Have you figured it out yet? For each build, the container was having to reach out to the Internet, pull all of the gems, and install them before it could even start processing my treasured blog posts. If I could find a way to cache them, I could improve the build time considerably!

Enter CodeBuild’s Cache functionality.

There are two options: local caching, and S3 Local caching wasn’t viable for me as the builds are too infrequent for it to make sense, so I went down the path of using a specific S3 bucket for this cache.

Since I use CloudFormation to manage all of my resources, I updated the template with a couple of changes. The first was to add the S3 bucket:

CodeBuildCacheS3:
  Type: AWS::S3::Bucket
  Properties:
    BucketName:
      !Sub '${AWS::AccountId}-codebuild-cache'

Next was to add the appropriate permissions to get and put objects into the bucket by the role assumed by CodeBuild during execution:

Effect: Allow
Action:
  - s3:PutObject
  - s3:GetObject
Resource: !Sub
  - '${CodeBuildCacheS3Arn}/*'
  - { CodeBuildCacheS3Arn: !GetAtt CodeBuildCacheS3.Arn }

The last piece was adding the relevant cache block to the CodeBuild project:

Cache:
  Type: S3
  Location: !Ref CodeBuildCacheS3

Once I updated the stack, I was eager to re-run the same build to observe the changes by adding the cache…

I did, and didn’t notice any difference at all. I went back to look at the logs and found an entry that indicated a cache miss.

[Container] Unable to download cache: NoSuchKey: The specified key does not exist.

I continued reading through and discovered the following entries in the post build phase:

[Container] Entering phase POST_BUILD
[Container] Uploading S3 cache...
[Container] Phase complete: POST_BUILD State: SUCCEEDED

Awesome! A process within the post build phase was able to upload the contents I had specified to be cached to S3. The file had a unique name using the standard format for Universally Unique Identifiers (UUID). After downloading the file to my local machine, I observed that it was a simple unencrypted tarball.

After unpacking it, I discovered a flat structure (no directories) containing files with numbers for names and a file labeled codebuild.json.

codebuild.json is a properly formatted JSON object with the following header:

{"version":"1.0","content":{"files":[

The header is then followed by JSON objects indicating the path of the file with its proper name. Based on my interpretation and comparison, each JSON object maps to the file in the package starting from the first file number in the package.

Once the cache was populated, each subsequent build and publish was averaging 60 seconds. I was really pleased with how easily I was able to optimize this build process. While this build is small, the impact caching has on large scale builds is significant. For those of you that are using CodeBuild, and are struggling with long build times, I highly recommend looking into using the cache option.