"It works fine locally. What's gone wrong?" – the inevitable (and eternal?) dilemma of deployment, as if the actual coding wasn't hard enough (even today with the help of AI)! Software development can be a real maze - solving one issue often unravels another:

Building software is like having a giant house of cards in our brains Idan Gazit on Developer Experience

With countless moving parts to juggle, tools that ease the transition from code to shipped product are real game-changers. For me, Vercel's developer experience did just that. It would seamlessly deploy and host my NextJS app (this site!) from a single 'Git push'. The zero-downtime deployments, and knowing it rolls back if something goes wrong gave me peace of mind. I could just focus on the creating, not managing servers!

From Serverless to Self-host NextJS

Although Serverless hosting provides so much convenience, the intricate billing model brought a layer of complexity that ultimately compelled me to revert to self-hosting. After experiencing the awesomeness of Vercel though, I couldn't just go back to an old manual workflow.

This post therefore walks you through how I set up a self-hosted NextJS app with a development workflow powered by GitHub Actions. It gives:

• Zero downtime – no downtime between deployments

• 'Roll back' – if the build fails, use the working version

• Auto Deploy - 'git push' to main

• Logs – See the View all the build logs and see build progress

You'll also find a couple script files you can copy and paste to create a deploy experience along the lines of Vercel's.

With the recent discussions around serverless pricing vs self hosting, I hope the post saves you some time if you're trying self-hosted NextJS.

Improved infrastructure pricing – Vercel

We’re reducing pricing on Vercel fundamentals like bandwidth and functions.

...but first, enter Blue Green deployment: 👀

Blue Green Deployment

Before anything, let's understand the general workflow.
The app will use a 'Blue Green' deployment model, where we'll have 2 copies of the Next app side-by-side. Here's how it works:

• 2 versions of the same app - one called blue, one called green

• 1 live version at a given time (say the green one), and changes get deployed to the 'sleeping' version (or the blue one)

• If the build is successful, the sleeping app is turned on so is live. The other app is turned off, and is now the 'old version'.

This diagram from Candost's blog illustrates it really well:

What is Blue-Green Deployment

The Blue-Green Deployment is an application deployment technique used in continuous delivery. The new service is deployed and put into production gradually.

Overall it seems safe - if the build fails, nothing happens. There's no downtime as the new build must work before the old one is turned off.

Here's another article that explains blue green deployment in more depth - when I came across the concept, I thought this was all beyond me, but in practice I found it more straightforward.

What is blue green deployment?

Blue green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production.

Let's see how it's done in practice:

Server Setup and Deployment Guide

To get started with this 'blue green' setup, this guide assumes you have a server running, preferably NGINX (but it should be possible with others like Apache). I'm not going to walk you through setting that stuff up, but how to configure it all. The rest of the 'guide' is split into 2 parts:

1. Server Setup

• Next Apps: Setting up the 2 NextJS apps (blue and green)

• NGINX/PM2: Configure the NGINX server and PM2 processes

2. Deployment Guide

• Deploy: Add a deploy script to build the new deployment

• GitHub Actions: Configure GitHub Actions to trigger/log the build

Let's begin the server setup with the NextJS app:

Part 1: Server Setup

I started out with a single Next app running on the server using NGINX and PM2 first. So let's do that first, then add the second Next app to create the blue green workflow.

A. Green App

Follow these 3 parts to set up the green app:

• Next App (green): The initial app was cloned from the Prototypr GitHub repo (give it a star ⭐) - and the .next folder is built using npm run build. For the deploy script to work, call this folder next-app-green.

• PM2 (port 3000): The initial app is running on PM2, using the command below. If you're following my process, make sure your app name is "green", and you specify port 3000. That's important for later!

pm2 start npm --name "green" -- run start -- --port 3000

• NGINX (proxy_pass): The initial NGINX location block looked like below. It uses `proxy_pass` which points to the PM2 (the process manager) running on port 3000. Make sure you have this working first, then we'll make a couple tweaks for blue/green.

{
  location /
        {
        proxy_pass http://localhost:3000; 
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        }
}

Be sure your app is turned on by running PM2 ls - it will say 'online' in the status column.

B. Blue app

Here's the bits for the blue app, and to get the blue green concept set up:

• Next App (blue): Now clone the same Next app Github repo into a new directory called it next-app-blue (the name is important for the deploy script in part 2)

• PM2 (port 3001): start your blue version with the following command. Ensure the --name flag is blue, and the --port flag is 3001:

pm2 start npm --name "blue" -- run start -- --port 3001

• NGINX (upstream): this is the cool part! With 1 additional block in the NGINX, we can have both apps running at the same time with these 2 changes:

  1. Add an 'upstream' block is at the top, that directs to both of our running PM2 processes (localhost:3000 and localhost:3001).

  2. Change the original proxy_pass to http:// + the name of the upstream block. In this case it is http://bluegreen, but you can call it anything.

upstream bluegreen {
    server localhost:3000 weight=1;
    server localhost:3001 weight=1;
}

server
{
        server_name yourdomain.com;
location /
        {
        proxy_pass http://bluegreen;
        #proxy_pass http://localhost:3000; 
        ...
        }

}

It's called Load Balancing

This is actually called load balancing - the requests to the domain will be sent to either of your running apps. If one of the apps is stopped, all requests will go to the other running app. Load balancing sounded really complicated to me, but it was done it with those 2 changes. Here's more info on it with NGINX:

Simple Load Balancing | NGINX

An example NGINX configuration that performs simple load balancing.

What Is Load Balancing? How Load Balancers Work

Load balancing is the distribution of incoming network traffic accross a group of backend servers.

Last step for this section: now both apps running, stop one of them using pm2 stop green (or blue). At this stage, running pm2 ls will produce a table like below - 1 app online, the other stopped (doesn't matter which way around):

All the traffic is therefore going through 1 app.

Part 2: Deployment Guide

With that Blue/Green server setup done, there's 2 parts left. You can paste my scripts for both of these, but there's a little bit of configuring to do to set up SSH keys for GitHub:

1. Deploy Script: Add a deploy.sh script on the server that will create the latest build, and fail gracefully if anything goes wrong.

2. GitHub Actions: Set up GitHub actions to trigger a deployment when you push to the main branch

1. Deploy Script

This is the script that will be triggered by a GitHub action when you push to your main branch. Find it below- you can copy paste it into a file called deploy.sh (or whatever), and put it in your user's home directory.

The script seems long, but I've added comments to explain what's happening. Here's an overview:

• Checks live app: It first determines which app is running in PM2 - if green is running, set the deploy folder to blue. If blue is running, deploy to green folder.

• Builds new deployment: Then it navigates to the deployment folder, runs git pull, npm install, and next build. If anything fails, it will display in the console. (if nvm isn't installed you'll get an error)

• Starts new deployment: If everything builds without errors, it will start the new app and stop the old one. Otherwise, it will just fail, leaving your live app still running.

If you've set up the folders with the same names as outlined throughout, you'll be able to paste this into your deploy.sh file and it should work with minimal changes. You should test it after you paste it in:

#!/bin/bash

#both app folders start with 'next-app'
LIVE_DIR_PATH=/var/www/html/next-app

#add the suffixes to the app folders
BLUE_DIR=$LIVE_DIR_PATH-blue
GREEN_DIR=$LIVE_DIR_PATH-green

# set up variables to store the current live and deploying to directories
CURRENT_LIVE_NAME=false
CURRENT_LIVE_DIR=false

DEPLOYING_TO_NAME=false
DEPLOYING_TO_DIR=false

# this checks if the green and blue apps are running
GREEN_ONLINE=$(pm2 jlist | jq -r '.[] | select(.name == "green") | .name, .pm2_env.status' | tr -d '\n\r')
BLUE_ONLINE=$(pm2 jlist | jq -r '.[] | select(.name == "blue") | .name, .pm2_env.status' | tr -d '\n\r')

# if green is running, set the current live to green and 'deploying to' is blue
if [ "$GREEN_ONLINE" == "greenonline" ]; then
    echo "Green is running"
    CURRENT_LIVE_NAME="green"
    CURRENT_LIVE_DIR=$GREEN_DIR
    
    DEPLOYING_TO_NAME="blue"
    DEPLOYING_TO_DIR=$BLUE_DIR
fi

# if blue is running, set the current live to blue and 'deploying to' is green
if [ "$BLUE_ONLINE" == "blueonline" ]; then
    echo "Blue is running"
    CURRENT_LIVE_NAME="blue"
    CURRENT_LIVE_DIR=$BLUE_DIR
    
    DEPLOYING_TO_NAME="green"
    DEPLOYING_TO_DIR=$GREEN_DIR
fi

# if both green and blue are running, set 'deploying to' to blue
if [ "$GREEN_ONLINE" == "greenonline" ] && [ "$BLUE_ONLINE" == "blueonline" ]; then
    echo "Both blue and green are running"
    DEPLOYING_TO_DIR=$BLUE_DIR
fi

# display the current live and deploying to directories
echo "Current live: $CURRENT_LIVE_NAME"
echo "Deploying to: $DEPLOYING_TO_NAME"

echo "Current live dir: $CURRENT_LIVE_DIR"
echo "Deploying to dir: $DEPLOYING_TO_DIR"

# Navigate to the deploying to directory
cd $DEPLOYING_TO_DIR || { echo 'Could not access deployment directory.' ; exit 1; }

# use the correct node version
# Load nvm
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm

nvm use 18.17.0 || { echo 'Could not switch to node version 18.17.0' ; exit 1; }

# load the .env file
source .env.local || { echo 'The ENV file does not exist' ; exit 1; }


# Pull the latest changes
# clear any changes to avoid conflicts (usually permission based)
git reset --hard || { echo 'Git reset command failed' ; exit 1; }
git pull -f origin main || { echo 'Git pull command failed' ; exit 1; }

# install the dependencies
npm install --legacy-peer-deps || { echo 'npm install failed' ; exit 1; }
# Build the project
npm run build || { echo 'Build failed' ; exit 1; }

# Restart the pm2 process
pm2 restart $DEPLOYING_TO_NAME || { echo 'pm2 restart failed' ; exit 1; }

# add a delay to allow the server to start
sleep 5

# check if the server is running
DEPLOYMENT_ONLINE=$(pm2 jlist | jq -r '.[] | select(.name == "$DEPLOYING_TO_NAME") | .name, .pm2_env.status' | tr -d '\n\r')

if [ "$DEPLOYMENT_ONLINE" == "$DEPLOYING_TO_NAMEonline" ]; then
    echo "Deployment successful"
else
    echo "Deployment failed"
    exit 1
fi

# stop the live one which is out of date
pm2 stop $CURRENT_LIVE_NAME;

Here it is in a GitHub repo for easier access:

GitHub - Prototypr/nextjs-deploy-script: Script to deploy NextJS with NGINX and PM2

Script to deploy NextJS with NGINX and PM2. Contribute to Prototypr/nextjs-deploy-script development by creating an account on GitHub.

That script was hard for someone who never wrote a bash script before! If you have any improvements, they're welcome.

Before going to the next step, it's a good idea to make sure it works. Log into your server and run it with ./deploy.sh. Fix any errors, and run pm2 ls again to see if the correct app is running.

2. Set up GitHub Actions

After ensuring the deploy script works, and it is sitting in your user's home directory (cd ~/), we now need a GitHub action to run it when pushing to the main branch. You can learn how to do it by pasting my action, or here's a decent guide if you like reading:

Handling Continuous Integration And Delivery With GitHub Actions — Smashing Magazine

In this article, Aleem Isiaka explains how you can deploy a basic NodeJS application to Heroku, automate and schedule a deployment to run at certain intervals of the day, and use other actions from the Github Actions community.

Create the deploy.yml workflow file

From the above article, this is a practical part to do now:

From the root of our application, create a folder named .github/workflows that will contain all the GitHub action workflows inside this folder create a file named action.yml, this file will hold the instructions for our deployment process

In my workflow folder, I have a file called deploy.yaml - you can copy its contents to your own yaml file:

prototypr-frontend/.github/workflows/deploy.yml at main · Prototypr/prototypr-frontend

Open source publishing platform, built with Next.js and Strapi CMS backend. Uses Tiptap / Prose-mirror editor. 💙 - Prototypr/prototypr-frontend

Upon inspecting this deploy.yml, you can see in the first few lines that it gets triggered when pushing to the main branch. It also needs these 2 environmental secrets to deploy on your server:

SERVER_HOST: ${{ secrets.SERVER_HOST }}

SERVER_USER: ${{ secrets.SERVER_USER }}

SERVER_SSH_PRIVATE_KEY: ${{ secrets.SERVER_SSH_PRIVATE_KEY }}

Let's set that bit up:

Set up your server access secrets

The 3 secrets needed are these:

• SERVER_HOST is your server's IP address

• SERVER_USER is the username of the user where the deploy.sh script is

• SERVER_SSH_PRIVATE_KEY: your user's key to ssh into the server.

Add them to the environments secret section of your repo as shown here:

Steps 1,2 and 3 of the following article by Gourab shows you how to do this part. That's what I followed to set up the secrets, and generate the ssh key needed for SERVER_SSH_PRIVATE_KEY.

Automating Next.js Application Deployment with GitHub Actions

Are you ready to supercharge your Next.js application deployment? With GitHub Actions, the process…

That's pretty much it - pushing to the main branch should trigger the deploy script on your server, and you can watch the build progress from GitHub itself. All the logs and progress from the deploy.sh script will be logged in GitHub. I don't think that happens if you use a webhook rather than SSH, which is why I think this is great:

That's all I got for now, didn't expect this post to be so long. Below are some issues I ran into.

Star us on GitHub ⭐

If this helped you, give the repository a star:

GitHub - Prototypr/prototypr-frontend: Open source publishing platform, built with Next.js and Strapi CMS backend. Uses Tiptap / Prose-mirror editor. 💙

Open source publishing platform, built with Next.js and Strapi CMS backend. Uses Tiptap / Prose-mirror editor. 💙 - Prototypr/prototypr-frontend

I'm trying out this GitHub sponsors thing to make more resources like this, so check that out too if you want to support open source efforts:

Sponsor @Prototypr on GitHub Sponsors

Support Prototypr’s open source work

Troubleshooting

Here are a couple issues I had connecting to my server from GitHub actions

Host key verification failed:

Once that's set up all should be ready to go, or you might get this error:

Run mkdir -p ~/.ssh/ Host key verification failed.

The "Host key verification failed" error typically means that the SSH client was unable to verify the identity of the server. This can happen if the server's host key has changed or if it's the first time you're connecting to the server and its host key is not in your known_hosts file. (copilot response)

I fixed it by adding GitHub to my server's known hosts by following this:

GitHub’s SSH key fingerprints - GitHub Docs

Public key fingerprints can be used to validate a connection to a remote server.

Then added this to the deploy.yaml:

# Fetch the server's host key and add it to known_hosts
   ssh-keyscan -H ${{ secrets.SERVER_HOST }} >> ~/.ssh/known_hosts

Permission denied (publickey):

Other issues I had when connecting via ssh was was:

Load key "/home/runner/.ssh/id_rsa": error in libcrypto @: Permission denied (publickey).

This happened because I pasted the wrong key to SERVER_SSH_PRIVATE_KEY, or pasted it incorrectly by missing out (or including) the ssh-rsa part of the key (can't remember how its should be, but pasting the keys correctly fixed the permission denied issue.

fin.

GitHub - Prototypr/prototypr-frontend: Open source publishing platform, built with Next.js and Strapi CMS backend. Uses Tiptap / Prose-mirror editor. 💙

Open source publishing platform, built with Next.js and Strapi CMS backend. Uses Tiptap / Prose-mirror editor. 💙 - Prototypr/prototypr-frontend