Securing an nginx Container with letsencrypt

One of the things I wanted to automate with my new website was deployment, including fetching SSL certificates and installing them in nginx. I came into this project knowing nothing about shell scripts and Docker. Now I know a _little_about each. With the web going increasingly https I knew I had to jump this hurdle to be production ready.

Thankfully I found a blog post that helped me get started. All I had to do was make some adaptations to my setup. While that post is very much talking about his environment I wanted this post to generalize for others (be it a static site or one on a separate container like mine).

My sample project uses a Vapor web server, but you can sub out anything you wish for that. Here’s the sample on GitHub

We start with a Dockerfile that will build up our nginx container:

# build from the official Nginx image
FROM nginx

# install essential Linux packages
RUN apt-get update -qq && apt-get -y install apache2-utils curl

# sets the environment domain variable for use in other files
ARG build_domain
ENV DOMAIN $build_domain

# where we store everything SSL-related
ENV SSL_ROOT /var/www/ssl
 
# where Nginx looks for SSL files
ENV SSL_CERT_HOME $SSL_ROOT/certs/live
 
# copy over the script that is run by the container
COPY ./nginx/web_cmd.sh /tmp/
RUN ["chmod", "+x", "/tmp/web_cmd.sh"]

# establish where Nginx should look for files
ENV WEB_ROOT /home/hello-world/Public/

# Set our working directory inside the image
WORKDIR $WEB_ROOT

# copy over static assets
COPY Public /home/hello-world/Public

# copy our Nginx config template
COPY ./nginx/server.conf /tmp/docker_example.nginx

# substitute variable references in the Nginx config template for real values from the environment
# put the final config in its place
RUN envsubst '$DOMAIN:$WEB_ROOT:$SSL_ROOT:$SSL_CERT_HOME' < /tmp/docker_example.nginx > /etc/nginx/conf.d/default.conf

# Define the script we want run once the container boots
# Use the "exec" form of CMD so Nginx shuts down gracefully on SIGTERM (i.e. `docker stop`)
CMD [ "/tmp/web_cmd.sh" ]

This file had to be modified a bit from the referenced blog post above. The script he used was updated to a new name and support current letsencrypt standards. I also wanted to inject the domain in from the outside. So the build_domain argument comes in to this script and creates an environment variable out of it. Pretty handy!

The last line here calls a script which preps everything for the SSL procedure:

#!/usr/bin/env bash

# initialize the dehydrated environment
setup_letsencrypt() {

  # create the directory that will serve ACME challenges
  mkdir -p .well-known/acme-challenge
  chmod -R 755 .well-known

  # See https://github.com/lukas2511/dehydrated/blob/master/docs/domains_txt.md
  echo "$DOMAIN www.$DOMAIN" > domains.txt

  # See https://github.com/lukas2511/dehydrated/blob/master/docs/wellknown.md
  echo "WELLKNOWN=\"$WEB_ROOT.well-known/acme-challenge\"" >> config

  # fetch stable version of dehydrated
  curl "https://raw.githubusercontent.com/lukas2511/dehydrated/v0.6.2/dehydrated" > dehydrated
  chmod 755 dehydrated
}

# creates self-signed SSL files
# these files are used in development and get production up and running so dehydrated can do its work
create_pems() {
  openssl req \
      -x509 \
      -nodes \
      -newkey rsa:1024 \
      -keyout privkey.pem \
      -out $SSL_CERT_HOME/fullchain.pem \
      -days 3650 \
      -sha256 \
      -config <(cat <<EOF
[ req ]
prompt = no
distinguished_name = subject
x509_extensions    = x509_ext
 
[ subject ]
commonName = $DOMAIN
 
[ x509_ext ]
subjectAltName = @alternate_names
 
[ alternate_names ]
DNS.1 = localhost
IP.1 = 127.0.0.1
EOF
)
 
  openssl dhparam -out dhparam.pem 2048
  chmod 600 *.pem
}

# if we have not already done so initialize Docker volume to hold SSL files
if [ ! -d "$SSL_CERT_HOME" ]; then
  mkdir -p $SSL_CERT_HOME # _HOME
  chmod 755 $SSL_ROOT
  chmod -R 700 $SSL_ROOT/certs
  cd $SSL_CERT_HOME
  create_pems
  cd $SSL_ROOT
  setup_letsencrypt
fi

# if we are configured to run SSL with a real certificate authority run dehydrated to retrieve/renew SSL certs
if [ "$CA_SSL" = "true" ]; then

  # Nginx must be running for challenges to proceed
  # run in daemon mode so our script can continue
  nginx

  # accept the terms of letsencrypt/dehydrated
  ./dehydrated --register --accept-terms

  # retrieve/renew SSL certs
  ./dehydrated --cron

  # copy the fresh certs to where Nginx expects to find them
  cp $SSL_ROOT/certs/$DOMAIN/fullchain.pem $SSL_ROOT/certs/$DOMAIN/privkey.pem $SSL_CERT_HOME

  # pull Nginx out of daemon mode
  nginx -s stop
fi

# start Nginx in foreground so Docker container doesn't exit
nginx -g "daemon off;"

The gist of what’s going on here is that we are setting up SSL. This script can be invoked in either a dev environment or a production one (that variable is configured in the docker-compose file below). This is handy for local development where a self-signed cert will do. For production it will go through the whole process of phoning the letsencrypt service and getting real certificates.

We then assemble our server config file that gets copied into the container. Here’s that template:

server {
	server_name $DOMAIN;
	listen 443 ssl;

	root $WEB_ROOT;

	# configure SSL
	ssl_certificate $SSL_CERT_HOME/fullchain.pem;
	ssl_certificate_key $SSL_CERT_HOME/privkey.pem;
    ssl_dhparam $SSL_CERT_HOME/dhparam.pem;

	try_files $uri @proxy;

	location @proxy {
		proxy_pass http://web:8080;
		proxy_pass_header Server;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_pass_header Server;
		proxy_connect_timeout 3s;
		proxy_read_timeout 10s;
	}
}

server {
  # many clients will send unencrypted requests
  listen 80;

  # accept unencrypted ACME challenge requests
  location ^~ /.well-known/acme-challenge {
    alias $SSL_ROOT/.well-known/acme-challenge/;
  }

  # force insecure requests through SSL
  location / {
    return 301 https://$host$request_uri;
  }
}

I’m no nginx expert and largely this is copied from the other blog post. You’ll hopefully notice the environment variables throughout the file (like $DOMAIN). They get substituted out in our Dockerfile.

Another easily overlooked part of this file is the line proxy_pass http://web:8080. This host is used by Docker as a token. The web host will be setup as a service in our Docker compose file.

Speaking of the compose file, here it is:

version: "3.3"
services:
  web:
    image: jsorge/hello-world
    volumes:
      - ./Public:/app/Public
  nginx:
    build:
      context: .
      dockerfile: ./nginx/Dockerfile-nginx
      args:
        build_domain: hello.world
    environment:
      CA_SSL: "true"
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./Public:/home/hello-world/Public
      - ./nginx/ssl:/var/www/ssl
      - ./nginx/logs:/var/log/nginx
    depends_on:
      - web

The webservice sets up our web server, and then the nginx service configures our container by running the Dockerfile. We expose the ports needed by nginx and also install the volumes.

If you wanted to simplify and use a static site, you could pass the whole directory of your site as a volume and put it in the nginx server’s root directory. I’ve not done that myself, but it seems logical for static sites. In that case you wouldn’t need the web service at all, and in the server config file you’d probably use localhost  for the proxy pass value.

I hope this has been helpful, and taken some of the pain out of getting SSL up and running on your website. I’m happy to help with any questions you might have, via email or Twitter.

Nerd Snipe - 2 Apps, 1 Extension

I’m working on a project for Lyft that will bring in a new app extension to both our passenger and driver apps. The functionality between both apps is going to be identical so sharing code should be maximized. I am trying to figure out how to get it working to make just a single extension that can be used on both apps. This is more difficult than you might imagine, because extensions can be fiddly. They need to share the same version and build strings, and building in Xcode revealed this hidden gem:

Embedded binary's bundle identifier is not prefixed with the parent app's bundle identifier.

Ugh. This begged the question of how I could possibly get it to work. Off to chase a snipe. Let’s start with a sample project, shall we?

I’m no shell script wizard, but it seemed that a run script build phase should do the trick. It needs to run in the host app prior to anything else happening. This means it’s the second build phase (dependencies always need to be built first). Here’s what I came up with:

#! /bin/sh

APPEX_DIR=$BUILT_PRODUCTS_DIR/SharedExtension.appex
INFO_PLIST=$APPEX_DIR/Info.plist
suffix=sharedextension

if [ "$IS_EMPLOYEE" = YES ] ; then
bundleID=io.taphouse.EmployeeApp.$suffix
shortVersion=$VERSION_EMPLOYEE
version=$VERSION_EMPLOYEE.$BUILD_NUMBER_EMPLOYEE
else
bundleID=io.taphouse.CustomerApp.$suffix
shortVersion=$VERSION_CUSTOMER
version=$VERSION_CUSTOMER.$BUILD_NUMBER_CUSTOMER
fi

plutil -replace CFBundleIdentifier -string "$bundleID" $INFO_PLIST
plutil -replace CFBundleShortVersionString -string "$shortVersion" $INFO_PLIST
plutil -replace CFBundleVersion -string "$version" $INFO_PLIST

I’ve extracted out the short version and build values as build settings in each main app target and those get funneled into the script. In this sample I’ve hard-coded each app’s bundle ID to prefix the extension, but at Lyft these are build settings as well. (Pro-tip: make good use of xcconfig files because editing this stuff in Xcode is a huge pain, not to mention the possible merge conflicts that could arise in your project file.)

I’ve got a build setting in one of my apps indicating what kind of app it is (in this case that is the IS_EMPLOYEE setting). Checking that will tell me the environment the script is running in. I setup 3 variables for each app and replace their values in the extension’s already built Info.plist file. This is important, because it’s too late to change the file in my source directory.

So now I’ve got an Info.plist file with all the proper values. Each app will build and run in the simulator. 🎉!

But we don’t write apps for the simulator. We write them for phones. Let’s give that a shot.

0x16b257000 +MICodeSigningVerifier _validateSignatureAndCopyInfoForURL:withOptions:error:: 147: Failed to verify code signature of /private/var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.KCRuNY/extracted/EmployeeApp.app/PlugIns/SharedExtension.appex : 0xe8008001 (An unknown error has occurred.)

Dang. Looking at the build steps for my extension reveals that there’s a code signature step at the very end and changing the Info.plist afterwards seems to break that signature. So how do we get this done?

Well, I don’t know. My best guess is that I might need to figure out how to re-sign the extension after manipulating its Info.plist. Is this possible? Reasonable to do?

What I’ll probably wind up doing is have individual extensions (one for each app) and a shared framework that backs each. I think it will be easier to get going and more resilient to changes in tooling in the future.

Unless you’ve got a way for me to get this done… 🙂

Finny Turns 2

2 years ago yesterday, I was unable to attend Xcoders. I got a frantic call from Emily that her water broke, and I got a ride to the hospital from a good buddy. 24 hours later Finnian was born.

Today he turns 2. It’s hard to believe how big he’s getting. Watching him grow from newborn to our floppy haired jolly boy has been a real blessing. Happy birthday, buddy!

This is My New Site; It Looks Like The Old Site

I have launched my new website. If you’ve been around here before you might not notice anything different (if you’re a new reader – welcome! 👋). But I assure you, it’s different.

I was running my site on Ghost. And while it was great for me for the last few years I found it lacking lately (mostly I want to combine microblogging and full-length blogging). So I decided to build my own engine.

I call it Maverick, and my ambitions for this project are pretty lofty. It’s not a finished product yet; far from it. But I realized that it was close enough to replace Ghost that I should just pull the trigger. Real artists ship, after all. In fact, this is the first thing I’ve shipped on my own since Scorebook back in November, 2014.

Maverick is powered by Vapor, which is a web framework written in Swift. This appealed to me on a few levels but the main one is that I already know Swift since I’m an iOS developer by trade. Not learning a whole new language is a big win for me (as I’ll explain in later posts there were several other things I had to learn to get here).

I started writing the intro post which went into some technical detail but realized that maybe I’d gone too far after 1100 words. So I’m going to go a different route and break things up into different posts. I wanted to get this inaugural post out there to commemorate the occasion.

If you’re curious about Maverick, it’s fully open source. Things are a little confusing, but there are 2 repos: the first is for the actual application, and the second is for my website. I’ll leave you with links to the two repos. Cheers!

Proxying Vapor 3 Using nginx and Docker

Vapor is a framework for running server side code, all built in Swift. There’s a great post by bygri about how to build a Vapor app using Docker. If any of what I’ve just said is confusing, I highly recommend reading that post before getting started here.

During my building of a couple of Vapor apps, I’ve found getting the sites running while hitting localhost:8080 to be really easy. The hard part is putting that site behind a domain locally and even more, getting it deployed to a server. So I’m going to put a huge disclaimer here: I’m no expert in nginx or Docker. I’ve been able to piece things together using web searches and a lot of conversation in the Vapor Discord room. There’s a great channel for Docker specifically, where I’ve been hugely helped by @bygri. He’s good people.

This post’s goal is purely to show how I got my Vapor app - which worked in development - deployed to production and successfully proxied by nginx. There will be a few extra bits I learned along the way as well (just to sweeten the deal for you).

My initial thought process was this:

  1. Everything for my web app lives inside of the git repository.
  2. I have a script to run for local development
  3. When it’s time to deploy, I clone the repo to my server, and run a script to boot it all up there.

Turns out I was a bit mistaken. If you read through the post I linked at the top, you’ll notice that he has separate Dockerfiles for development and production. I skipped over that part, much to my initial peril (so don’t do that).

What I did was create a repository on the Docker Hub which would hold a named, built image. That image is built with the following Dockerfile:

# Build image
FROM swift:4.1 as builder
RUN apt-get -qq update && apt-get -q -y install \
  && rm -r /var/lib/apt/lists/*
WORKDIR /app
COPY . .
RUN mkdir -p /build/lib && cp -R /usr/lib/swift/linux/*.so /build/lib
RUN swift build -c release && mv `swift build -c release --show-bin-path` /build/bin

# Production image
FROM ubuntu:16.04
RUN apt-get -qq update && apt-get install -y \
  libicu55 libxml2 libbsd0 libcurl3 libatomic1 \
  tzdata \
  && rm -r /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /build/bin/Run .
COPY --from=builder /build/lib/* /usr/lib/
COPY Resources/ ./Resources/
EXPOSE 8080
ENTRYPOINT ./Run serve -e prod -b 0.0.0.0

If you have discerning eyes you’ll notice that this Dockerfile is almost exactly the same as the one in the post  I referenced above (go read it and come back if you didn’t earlier). That’s because I copied it from him 😊. The one addition I had to make was COPY Resources/ ./Resources. What this does is copies the Resources directory from my local drive and into the Docker image.

I was initially skeptical that I needed the EXPOSE directive since nginx would proxy over there by default, but I’ve confirmed that it’s not exposed on my server and this will make for good documentation, so 🤷‍♂️.

I built that image using docker build -t jsorge/taphouse.io. Then I can push the image using docker push jsorge/taphouse.io. These 2 commands will build the image and tag it locally, then push it up to Docker Hub. I highly suggest scripting this stuff (I’m becoming partial to Makefiles to get it all done).

Now let’s flip our environment to the production server. I don’t need much to bring in the Vapor app since it’s already built and hosted. I just need to reference it and mount the Public folder.

But then comes the fun part. I’ll catch you on the flipside of the compose file:

version: "3.3"
services:
  web:
	image: jsorge/taphouse.io
	volumes:
	  - ./Public:/app/Public
  nginx:
	image: nginx:alpine
	restart: always
	ports:
	  - 80:80
	  - 443:443
	volumes:
	  - ./Public:/home/taphouse/Public
	  - ./nginx/server.conf:/etc/nginx/conf.d/default.conf
	  - ./nginx/logs:/var/log/nginx
	depends_on:
	  - web

Ok, so what’s going on here? We’re grabbing the nginx image from alpine (a trusted provider of nginx). I’m honestly not sure what the restart command does but it showed up on my DuckDuckGo results of examples. But the rest I understand.

ports: I’m exposing ports 80 and 443 to get http and https traffic listened to (the syntax for ports is <external>:<container-internal>). I could pick other internal ports to listen to and update my nginx configs appropriately but didn’t feel like rocking the boat too much. I plan on adding TLS support via letsencrypt but that’s a problem for another day.

volumes: I’m sure people familiar with Docker won’t need this explained (or any of this compose file really) but this part blows my mind a bit. The volumes directive basically puts a link from the local machine to inside the container. The link could be a file or a folder; that part doesn’t matter. So to get the root of my site working in nginx, I mount the Public folder that exists locally on my machine into the container at the specified path. The syntax for this is <local-path>:<container-path>.

I also created the path of nginx/logs on my server, and mounted it as the log directory in the container. This allows me to have the logs from the container persisted to my server volume and easily read them as they come in. This is super cool!

depends_on: This one is pretty cool. It establishes the dependency chain between your containers and starts them up in the proper order to satisfy them. In this case, nginx depends on my Vapor app (called web).

The nginx configuration file is the last link in the chain to get us going. This is what mine looks like:

server {
	server_name taphouse.io;
	listen 80 default_server;

	root /home/taphouse/Public/;

	try_files $uri @proxy;

	location @proxy {
		proxy_pass http://web:8080;
		proxy_pass_header Server;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_pass_header Server;
		proxy_connect_timeout 3s;
		proxy_read_timeout 10s;
	}
}

I got this by finding the suggested configuration for Vapor 2. Curiously this page hasn’t made its way to the Vapor 3 docs but I’m guessing that has something to do with them wanting you to deploy on Vapor Cloud. I’m removing my tinfoil hat now.

As far as nginx configuration files go, this one looks pretty standard. I don’t know the exact ins and outs of what’s going on but the point of note is here: proxy_pass http://web:8080;. Remember how I said earlier that Docker Compose provides its own networking? Well it turns out I can use my Vapor container name as the host and it will resolve everything inside the container network. Super cool!

From here I got these files on the server and ran my make server command (which aliases to docker-compose -f docker-compose-prod.yml up --build -d) and tried hitting http://taphouse.io. It worked! 🤯

The next thing on my list is to get acme.sh working so that TLS is up and running, and I can disable port 80. That’s my current holdup, but when I get that done I’ll be sure to write that up as well.

If I made any grievous errors or you have general feedback, I’d love to hear it. I’m still very green when it comes to Docker, nginx, and server admin stuff in general.