[Next.js](https://nextjs.org/ "Next.js") Image Component [next-image](https://nextjs.org/docs/pages/api-reference/components/image  "next-image") is a feature introduced in Next.js version 10.0.0 to optimize images and improve the performance of your web-application.

When you use the Next.js Image Component, it automatically optimizes and serves images in modern image formats that improves the performance of your web application. It supports various image sources, such as local images, images from the web, and third-party sources.

However you cannot transform image, e.g. crop images, which is the reason I was looking for a solution which enables my personal website [mxd.codes](https://mxd.codes "mxd.codes") to resize images to my needs.

## imgproxy

[imgproxy](https://imgproxy.net/ "imgproxy") is an open-source image processing server designed to simplify the resizing, cropping, and manipulation of images on the fly. It is often used as part of a web application's infrastructure to ensure efficient delivery of images with optimized sizes and quality. 

Key features of imgproxy include:

1. **On-the-Fly Image Processing:** Imgproxy allows you to resize, crop, rotate, and perform other image manipulations on the fly, based on the URL parameters. This enables efficient delivery of images in various sizes and formats without having to store multiple versions of the same image.

2. **Security:** Imgproxy provides security features such as URL signature generation. This helps prevent unauthorized access and abuse of the image manipulation service.

3. **Performance:** Imgproxy is designed to be performant and can efficiently handle high loads of image processing requests.

4. **Integration with Existing Storage:** Imgproxy can be integrated with various storage solutions, including Amazon S3, Google Cloud Storage, and more.

## Deploy imgproxy with Docker Compose

While searching for a way to deploy imgproxy with docker I found a [imgproxy Docker Compose Project on GitHub](https://github.com/shinsenter/docker-imgproxy/blob/main/docker-compose.yml "imgproxy docker compose") where I changed minor things like the volumes and the web-server configuration.

You can copy this ```docker-compose.yml```file and paste it into Portainer or save it manually in a folder on your server.

```docker
version: '3'

################################################################################
# Ultra Image Server
# A production grade image processing server setup powered by imgproxy and nginx
#
# Author: Mai Nhut Tan <shin@shin.company>
# Copyright: 2021-2023 SHIN Company https://code.shin.company/
# URL: https://shinsenter.github.io/docker-imgproxy/
################################################################################

networks:
################################################################################
  default:
    driver: bridge


services:
################################################################################
  web:
    image: nginx:alpine
    container_name: imgproxy-nginx
    restart: always
    volumes:
      - /data/containers/imgproxy:/var/www/html:ro
      - /etc/imgproxy/imgproxy-nginx.conf:/etc/nginx/conf.d/default.conf:ro
    ports:
      - 8080:80
    links:
      - imgproxy:imgproxy
    environment:
      NGINX_ENTRYPOINT_QUIET_LOGS: 1

################################################################################
  imgproxy:
    restart: unless-stopped
    image: darthsim/imgproxy:${IMGPROXY_TAG:-latest}
    container_name: imgproxy_app
    security_opt:
      - no-new-privileges:true
    volumes:
      - /data/containers/imgproxy:/var/www/html:ro
    expose:
      - 8080
    healthcheck:
      test: ["CMD", "imgproxy", "health"]
    environment:
      ### See:
      ### https://docs.imgproxy.net/configuration/options

      ### log and debug
      IMGPROXY_LOG_LEVEL: "warn"
      IMGPROXY_ENABLE_DEBUG_HEADERS: "false"
      IMGPROXY_DEVELOPMENT_ERRORS_MODE: "false"
      IMGPROXY_REPORT_DOWNLOADING_ERRORS: "false"

      ### timeouts
      IMGPROXY_READ_TIMEOUT: 10
      IMGPROXY_WRITE_TIMEOUT: 10
      IMGPROXY_DOWNLOAD_TIMEOUT: 10
      IMGPROXY_KEEP_ALIVE_TIMEOUT: 300
      IMGPROXY_MAX_SRC_FILE_SIZE: 33554432 # 32MB
      IMGPROXY_MAX_SRC_RESOLUTION: 48

      ### image source
      IMGPROXY_TTL: 2592000 # client-side cache time is 30 days
      IMGPROXY_USE_ETAG: "false"
      IMGPROXY_SO_REUSEPORT: "true"
      IMGPROXY_IGNORE_SSL_VERIFICATION: "true"
      IMGPROXY_LOCAL_FILESYSTEM_ROOT: /home
      IMGPROXY_SKIP_PROCESSING_FORMATS: "svg,webp,avif"

      ### presets
      IMGPROXY_AUTO_ROTATE: "true"
      #IMGPROXY_WATERMARK_PATH: /home/noimage_thumb.jpg
      IMGPROXY_PRESETS: default=resizing_type:fit/gravity:sm,logo=watermark:0.5:soea:10:10:0.15,center_logo=watermark:0.3:ce:0:0:0.3

      ### compression
      IMGPROXY_STRIP_METADATA: "true"
      IMGPROXY_STRIP_COLOR_PROFILE: "true"
      IMGPROXY_FORMAT_QUALITY: jpeg=80,webp=70,avif=50
      IMGPROXY_JPEG_PROGRESSIVE: "false"
      IMGPROXY_PNG_INTERLACED: "false"
      IMGPROXY_PNG_QUANTIZATION_COLORS: 128
      IMGPROXY_PNG_QUANTIZE: "false"
      IMGPROXY_MAX_ANIMATION_FRAMES: 64
      IMGPROXY_GZIP_COMPRESSION: 0
      IMGPROXY_AVIF_SPEED: 8

      ### For URL signature
      IMGPROXY_KEY: IMGPROXY_KEY_KEY
      IMGPROXY_SALT: IMGPROXY_KEY_SALT
      IMGPROXY_SIGNATURE_SIZE: 32
    network_mode: "host"  
```

You will also need a nginx-configuration file for imgproxy which should be saved to ```/etc/imgproxy/imgproxy-nginx.conf```. Of course you can also store the file anywhere else but be sure to change the volume in the docker-compose.yml.


```nginx
upstream upstream_imgproxy  {
    server    imgproxy:8080;
    keepalive 16;
}

server {
        server_name _;

        location / {
                proxy_pass http://upstream_imgproxy;
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection 'upgrade';
                proxy_set_header Host $host;
        }

}
```

Now you can  deploy the stack with

```bash
docker-compose up -d --build --remove-orphans --force-recreate
```
or on Portainer. 

Your imgproxy instance should be now running on http://localhost:8080 which you already can use.

![Screenshot 2024-01-12 113947.png](/uploads/Screenshot_2024_01_12_113947_ed6e757944.png)

But I wanted to integrate it within my personal site built with Next.js so I also had to modify the nginx-configuration for my personal site.
So i used the existing configuration [Nginx reverse proxy with caching for Next.js with imgproxy](https://gist.github.com/hlubek/02955a3f28db168417884b5397ce07c0 "") and copied it to /etc/nginx/sites-available/default.

```nginx
# Based on https://steveholgado.com/nginx-for-nextjs/

# - /var/cache/nginx sets a directory to store the cached assets
# - levels=1:2 sets up a two‑level directory hierarchy as file access speed can be reduced when too many files are in a single directory
# - keys_zone=STATIC:10m defines a shared memory zone for cache keys named “STATIC” and with a size limit of 10MB (which should be more than enough unless you have thousands of files)
# - inactive=7d is the time that items will remain cached without being accessed (7 days), after which they will be removed
# - use_temp_path=off tells NGINX to write files directly to the cache directory and avoid unnecessary copying of data to a temporary storage area first
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=STATIC:10m inactive=7d use_temp_path=off;

upstream nextjs_upstream {
  server localhost:3000;
}

upstream imgproxy_upstream {
  server localhost:8080;
}

server {
  listen 80 default_server;

  server_name _;

  server_tokens off;

  gzip on;
  gzip_proxied any;
  gzip_comp_level 4;
  gzip_types text/css application/javascript image/svg+xml;

  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection 'upgrade';
  proxy_set_header Host $host;
  proxy_cache_bypass $http_upgrade;

  # Imgproxy paths can contain multiple slashes (e.g. local:///image/file.jpg)
  merge_slashes off;

  location /img/ {

    proxy_cache STATIC;

    proxy_pass http://imgproxy_upstream/;

    # For testing cache - remove before deploying to production
    add_header X-Cache-Status $upstream_cache_status;
  }

  location /_next/static {
    proxy_cache STATIC;
    proxy_pass http://nextjs_upstream;

    # For testing cache - remove before deploying to production
    add_header X-Cache-Status $upstream_cache_status;
  }

  location /static {
    proxy_cache STATIC;

    # Ignore cache control for Next.js assets from /static, re-validate after 60m
    proxy_ignore_headers Cache-Control;
    proxy_cache_valid 60m;

    proxy_pass http://nextjs_upstream;

    # For testing cache - remove before deploying to production
    add_header X-Cache-Status $upstream_cache_status;
  }

  location / {
    proxy_pass http://nextjs_upstream;
  }
}
```
With this configuration all requests with the path `/img/` will be redirected to the imgproxy instance and all other paths to my personal-website.

You can test the configuration with `sudo nginx -t` and restart nginx when the test is successfull with `sudo systemctl restart nginx`.

Now when you access [https://mxd.codes/img/](https://mxd.codes/img/ "imgproxy instance link") you will be redirected to the imgroxy instance and when you access [https://mxd.codes](https://mxd.codes "personal website link") you will be redirected to my personal website.

The last missing piece is a custom image loader for the Next.js site.

## Custom image loader for imgproxy

You can configure a custom `loaderFile` in your `next.config.js` like the following:

```javascript
images: {
        loader: "custom",
        loaderFile: "./src/utils/loader.js",
}
```

This must point to a file relative to the root of your Next.js application. The file must export a default function that returns a string:

```javascript
export default function imgproxyLoader({ src, width, height, quality }) {

  const path =
    `/size:${width ? width : 0}:${height ? height : 0}` +
    `/resizing_type:fill` +
    (quality ? `/quality:${quality}` : "") +
    `/sharpen:0.5` +
    `/plain/${src}` +
    `@webp`

  const host = process.env.NEXT_PUBLIC_IMGPROXY_URL

  const imgUrl = `${host}/insecure${path}`

  return imgUrl
}
```

Now all images you serve with `next/image` will use your custom loader which will be using imgproxy to transform and optimize your images for your Next.js site.

Recently I also started to deploy my personal site with docker so the whole docke-compose.yml now looks like the following, while the nginx configuration file remains the same:

```yaml
version: "3"
    
services:
  nextjs:
    image: mxdcodes/personal-website:latest
    container_name: personal-website
    restart: always
    ports:
      - "3000:3000"
    environment:
      NODE_ENV: production
    network_mode: "host"  
    
  imgproxy:
    restart: unless-stopped
    image: darthsim/imgproxy:${IMGPROXY_TAG:-latest}
    container_name: imgproxy_app
    security_opt:
      - no-new-privileges:true
    volumes:
      - /data/containers/imgproxy/www:/home:cached
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD", "imgproxy", "health"]
    environment:
      ### See:
      ### https://docs.imgproxy.net/configuration/options

      ### options
      IMGPROXY_ALLOWED_SOURCES: https://mxd.codes/
      
      ### log and debug
      IMGPROXY_LOG_LEVEL: "warn"
      IMGPROXY_ENABLE_DEBUG_HEADERS: "false"
      IMGPROXY_DEVELOPMENT_ERRORS_MODE: "false"
      IMGPROXY_REPORT_DOWNLOADING_ERRORS: "false"

      ### timeouts
      IMGPROXY_READ_TIMEOUT: 10
      IMGPROXY_WRITE_TIMEOUT: 10
      IMGPROXY_DOWNLOAD_TIMEOUT: 10
      IMGPROXY_KEEP_ALIVE_TIMEOUT: 300
      IMGPROXY_MAX_SRC_FILE_SIZE: 33554432 # 32MB
      IMGPROXY_MAX_SRC_RESOLUTION: 48

      ### image source
      IMGPROXY_TTL: 2592000 # client-side cache time is 30 days
      IMGPROXY_USE_ETAG: "false"
      IMGPROXY_SO_REUSEPORT: "true"
      IMGPROXY_IGNORE_SSL_VERIFICATION: "false"
      IMGPROXY_LOCAL_FILESYSTEM_ROOT: /home
      IMGPROXY_SKIP_PROCESSING_FORMATS: "svg,webp,avif"

      ### presets
      IMGPROXY_AUTO_ROTATE: "true"
      #IMGPROXY_WATERMARK_PATH: /home/noimage_thumb.jpg
      IMGPROXY_PRESETS: default=resizing_type:fit/gravity:sm,logo=watermark:0.5:soea:10:10:0.15,center_logo=watermark:0.3:ce:0:0:0.3

      ### compression
      IMGPROXY_STRIP_METADATA: "true"
      IMGPROXY_STRIP_COLOR_PROFILE: "true"
      IMGPROXY_FORMAT_QUALITY: jpeg=80,webp=70,avif=50
      IMGPROXY_JPEG_PROGRESSIVE: "false"
      IMGPROXY_PNG_INTERLACED: "false"
      IMGPROXY_PNG_QUANTIZATION_COLORS: 128
      IMGPROXY_PNG_QUANTIZE: "false"
      IMGPROXY_MAX_ANIMATION_FRAMES: 64
      IMGPROXY_GZIP_COMPRESSION: 0
      IMGPROXY_AVIF_SPEED: 8

      ### For URL signature
      IMGPROXY_KEY: KEY
      IMGPROXY_SALT: SALT
      IMGPROXY_SIGNATURE_SIZE: 32
    network_mode: "host"  
```

How to transform and optimize images with imgproxy hosted with docker for your Next.js application.

Optimizing images for Next.js sites with imgproxy and docker

Recently I went out of storage for my homelab so I bought an used NAS (Synology DS214 play) to have some more capacities for [Proxmox](https://www.proxmox.com/de/ "Proxmox") Backups and OpenStreetMap. I still had a 1TB hdd lying around at home, which I now use for proxmox backups. 

To have some redundancy (and to learn something new) I decieded to copy the Proxmox backups to the cloud, in particular to an Azure Storage Account with [AzCopy](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) and in the following I will describe with more details how I was able to do it.

Overall this article will cover the following informations:
- Creating an Azure Storage Account
- Getting started with AzCopy
- Creating a bash-script to copy the Proxmox backups to an Azure Storage Container

## Creating an Azure Storage Account

First off all you need an active Azure subscription and an storage account to be able to store your backups. In the [Azure Portal](https://portal.azure.com "Azure Portal") you can search for the service "Storage Accounts" which you will need.

![azure_storage_account_e81efc2682.png](https://cms.mxd.codes/uploads/azure_storage_account_e81efc2682_260acf4298.png)

In the service "Storage Accounts" you can create a new storage account. For the storage account you will need 
- an active azure subscription,
- a ressource group (create one if you don't have one it, e.g. RG-HOMELAB),
- a storage account name,
- selecting a region and 
- selecting redundancy (e.g. LRS;)
- Access tier "cold" (See Advanced)

![storage_account_basics_e82f1d8a4d.png](https://cms.mxd.codes/uploads/storage_account_basics_e82f1d8a4d_6f72c73f59.png)
![storage_account_advanced_02f6609c40.png](https://cms.mxd.codes/uploads/storage_account_advanced_02f6609c40_50a3cf6f7c.png)

You can keep all the other settings as default. After your Storage Account has been deployed you can add a lifecycle rule from "Lifecycle Management" which will move files from the "cold" access tier to the archive storage. 

![storage_lifecycle_management_e113a6efb8.png](https://cms.mxd.codes/uploads/storage_lifecycle_management_e113a6efb8_d83047917d.png)

For example I created a rule which moves all new files after one day to the archive storage tier.

![storage_lifecycle_lifecycle_rule_85a074cae0.png](https://cms.mxd.codes/uploads/storage_lifecycle_lifecycle_rule_85a074cae0_1542a982cc.png)

![storage_lifecycle_lifecycle_rule_2_5486e872ee.png](https://cms.mxd.codes/uploads/storage_lifecycle_lifecycle_rule_2_5486e872ee_fbf78cef24.png)

By storing files in archive storage instead of in the regular "cold" access tier you can actually save about 82%. But keep in mind that accessing data in the Archive storage is more expensive than in the cold (or any other) storage tier.

Also you could create another rule which will for example will delete all all blobs which were created 365 days ago.

![storage_pricing_8998b65a68.png](https://cms.mxd.codes/uploads/storage_pricing_8998b65a68_9635c1cb24.png)

Please have a look at [https://azure.microsoft.com/en-us/pricing/details/storage/blobs/](https://azure.microsoft.com/en-us/pricing/details/storage/blobs/ "Azure Storage Pricing") for uptodate Azure Storage pricing.

After the storage account has been configured you will need to create a Container where the actual files will be stored.
Go to  "Data storage" -> "Containers" and create a Container.
![create storage container.png](/uploads/create_storage_container_8ee34fe39e.png)
Again name it however you want.

Due to the fact that the current version of AzCopy V10 does not support Azure AD authorization in cron jobs I used an SAS token to be able to upload files to the container. You can create a SAS token in the container at "Shared access tokens".

![storage_shared_access_token_1646ebf346.png](https://cms.mxd.codes/uploads/storage_shared_access_token_1646ebf346_8b4b61c9f6.png)

For the Shared access token you will need to select the permissions for Add/Create/Write and select an expiry date for security reasons. Then you can generate the SAS token and URL. Copy that **Blob SAS URL** because you will need it for the upload script.

## Getting started with AzCopy

> AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account. This article helps you download AzCopy, connect to your storage account, and then transfer data. (https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10)

To get AzCopy for Linux you have to download a tar file and decompress the tar file anywhere you like. You can then just use AzCopy because it's an executable file, so nothing has to be installed.

```bash
#Download AzCopy
cd ~
wget https://aka.ms/downloadazcopy-v10-linux
 
#Expand Archive
tar -xvf downloadazcopy-v10-linux
 
#(Optional) Remove existing AzCopy version
rm /usr/bin/azcopy
 
#Move AzCopy to the destination you want to store it
cp ./azcopy_linux_amd64_*/azcopy /usr/bin/

# Remove Azcopy from home
rm -r downloadazcopy-v10-linux
rm -r azcopy_linux_amd64_10.16.2/

```

By adding the azcopy file location as system path you can just type azcopy from any directory on your system.
You can add it to your system path with:

```bash
nano ~/.profile
```
and then adding these lines:

```bash
export PATH=/usr/bin/azcopy:$PATH
```

Lastly, update your system variables:

```bash
source ~/.profile
```

## Creating a bash-script to synchronize the Proxmox backup directory to an Azure Storage Container

The only piece missing now is the script which will upload the the Proxmox backup files to the previously created azure storage container after the backup task has finished. 
For copying the backups to Azure we will use azcopy **copy** because acopy uses less memory and incurs lower billing costs because a copy operation does not need to index the source or destination before moving files in comparison to azcopy **sync**.
Azcopy also compares file names and last modified timestamp to only upload new or changed files to the storage container, which overall will reduce bandwidth usage and it will work perfectly with the previously created lifecycle rule.

For automatically starting the upload after the backup has finished  we can use a hook script for vzdump. Therefore you need to add the following line to the end of the "/etc/vzdump.conf" file.
```bash
script: /home/youruser/scripts/upload-backups-to-azure.sh
```
Afterwards you can create the script which will upload the files with:
```bash
cd ~
mkdir scripts
cd scripts
nano upload-backups-to-azure.sh
```
Then copy paste the following content into the file and replace the content for src with the location of your dumps. Note that there is "/*" at the end of src so that only the files inside the directory will be copied. Also replace token with the **Blob SAS URL**.

```bash
#!/bin/bash
# Script to upload Proxmox backups to Azure Storage

src="/mnt/pve/xyz/dump/*"
token="Blob SAS URL"

dobackup(){
  echo "Uploading Proxmox backups from $src to Azure..."
  azcopy copy "$src" "$token" --overwrite=false
  echo "Finished Uploading!"
}

if [ "$1" == "job-end" ]; then
  dobackup
fi

exit 0
```

Close the file and make it executable for the user with:

```bash
chmod +x ~/scripts/upload-backups-to-azure.sh
```

Now the next time your backup task has finished the files will be automatically uploaded to your Azure storage container.
Due to the  hook script you can check the status of the copy process in the proxmox ui.

This article will explain to you how you can create an Azure Storage Account, prepare AzCopy locally and how to create a bash script which will be used to copy your local Proxmox Backups via AzCopy to an Azure Storage Account by a cost effective way.

How to copy your Proxmox backups with AzCopy to Azure Storage Containers

With [Gatsby 4](https://www.gatsbyjs.com/gatsby-4/) bringing in Server-Side Rendering (SSR) and Deferred Static Generation (DSG) you need an alternative methode to just hosting static files. Each page using SSR or DSG will be rendererd after a user requests it so there has be a server in the background which will handle these requests and build the pages if needed.

In this post i will show you how you can deploy your Gatsby site with SSR and/or DSG on your own server with a CI/CD pipeline via PM2 and Github Webhooks.

Therefore i will be using

- Gatsby 4,Nginx,
- Nginx
- PM2 and
- Github webhooks.

## Setup your server

First of all you need an server with root access. I strongly recommend to have a look at the guide ["Initial Server Setup with Ubuntu 18.04"](https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-18-04) from the DigitalOcean community which will lead you through the process of:

- Logging in and set up root user access to your server with SSH
- Creating a new user
- Granting Administrative Privileges to the new user
- Setting up a basic firewall
- Giving your regular user access to the server with SSH key authentication.

After you have done that you can continue by installing all necessary dependencies on your server.
Install Node.js

Again there is an guide by DigitalOcean which will help you [installing Node.js using PPA](https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04#installing-using-a-ppa).

After completing

- install Node.js, NPM and
- the "build-essential package"

you will have to change npm's default directory.

Create a `.npm-global` directory and set the path to this directory for `node_modules`:

```bash
cd ~
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
```

Create (or modify) a `~/.profile` and add the following line:

```bash
sudo nano ~/.profile

# set PATH so global node modules install without permission issues
export PATH=~/.npm-global/bin:$PATH
```

Now you have to update your system variables:

```bash
source ~/.profile
```

Now you should be able to check your installed Node.js version with:

```bash
node -v
```

## Install git

Check if git is already installed with:

```bash
git --version
```

If it isn't installed yet you can install it with

```bash
sudo apt install git
```

and configure Git with

```bash
git config --global user.name "Your Name"
git config --global user.email "youremail@domain.com"
```

After git is installed and configured you can deploy your Gatsby site by cloning it from Github.

## Deploy from Github

It is important that you are loggin in as **non-root** user for the following steps.

```bash
cd ~
git clone https://github.com/your-githubuser/your-gatsby-repo.git path your-gatsby-site
```

After you have deployed your project (optionally with environment variables) you can install all dependencies and build your Gatsby site with:

```bash
cd ./your-gatsby-site/
npm install
npm run build
```

Now you should have a copy of your local project/Gatsby site on your remote server.

Next you are going to setup PM2 which will be used to keep your site alive and restart it with every reboot.

## Setup PM2

You can install PM2 with:

```bash
npm install pm2@latest -g
```

You will need to create/configure an `ecosystem.config.js` file which will restart the default Gatsby server.

```bash
cd ~
pm2 init
sudo nano ecosystem.config.js
```

Copy/paste the template and replace the content.

```javascript
module.exports = {
  apps: [
    {
      name: 'gatsby-site',
      cwd: ' /home/your-name/my-gatsby-site',
      script: 'npm',
      args: 'serve',
      env: {
        //NODE_ENV: 'production',
      },
    },
    // optionally a second project
],};
```
With

```bash
cd ~
pm2 start ecosystem.config.js
```

you can start your server which will run on the Port 9000.

You can always check the status with:

```bash
pm2 status
```

After the server reboots this PM2 should be always automatically be restarted. For that you are going to need a small Startup script which you can also copy/paste.
Generate and configure a startup script to launch PM2:

```bash
cd ~
pm2 startup systemd

[PM2] Init System found: systemd
[PM2] To setup the Startup Script, copy/paste the following command:
**sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u your-name --hp /home/your-name**
```

- Copy/paste the generated command:

```bash
**sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u your-name --hp /home/your-name**

[PM2] Init System found: systemd
Platform systemd

. . .


[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
   $ pm2 save

[PM2] Remove init script via:
   $ pm2 unstartup systemd
```

- And save the new PM2 process list and environments. Then Start the service with systemctl.

```bash
pm2 save

[PM2] Saving current process list...
[PM2] Successfully saved in /home/your-name/.pm2/dump.pm2
```

If you reboot your server now with `sudo reboot` the script should be automatically restart your Gatsby site. Give it a try!

## Setup Github Webhook

One thing missing now is an continuos integration and continuos delivery (CI/CD) pipeline which you will setup using Github webhooks.

Therefore you need to create a new Webhook in your repository.

The following articles provide additional information to the steps below:

- [GitHub Creating Webhooks Guide](https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks#setting-up-a-webhook) and
- [DigitalOcean Guide to GitHub WebHooks](https://www.digitalocean.com/community/tutorials/how-to-use-node-js-and-github-webhooks-to-keep-remote-projects-in-sync).

You need to create a server script which will do something if it is triggered by the Github webhook.

```bash
cd ~
mkdir NodeWebHooks
cd NodeWebHooks
sudo nano webhook.js
```

The script is going to create a server running on Port 8100. (Your Github webhook should be of course sending the webhook to something like http://server-ip:8100.)

If it gets triggered by a webhook it will

- go into your repo `~/my-gatsby-site/`,
- pull the latest commits,
- install all dependencies,
- build a new version of the site and
- restart the server via the PM2 script.

```javascript
const secret = "your-secret-key";
const repo = "~/my-gatsby-site/";

const http = require('http');
const crypto = require('crypto');
const exec = require('child_process').exec;

const BUILD_CMD = 'npm run build';
const PM2_CMD = 'pm2 restart gatsby-site';

http.createServer(function (req, res) {
    req.on('data', function(chunk) {
        let sig = "sha1=" + crypto.createHmac('sha1', secret).update(chunk.toString()).digest('hex');

        if (req.headers['x-hub-signature'] == sig) {
            exec('cd ' + repo + ` && git pull && npm install && ${BUILD_CMD} && ${PM2_CMD}`);
        }
    });

    res.end();
}).listen(8100);
```

You will need to allow communication on Port 8100 with:

```bash
sudo ufw allow 8100/tcp
sudo ufw enable
```

Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup

Earlier you setup PM2 to restart your Gatsby site whenever the server reboots or is started. You will now do the same for the webhook script.

Run echo $PATH and copy the output for use in the next step.

```bash
echo $PATH

/home/your-name/.npm-global/bin:/home/your-name/bin:/home/your-name/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
```

Create a webhook.service file:

```bash
cd ~
sudo nano /etc/systemd/system/webhook.service
```
    
In the editor, copy/paste the following script, but make sure to replace your-name in two places with your username. Earlier, you ran echo ```$PATH```, copy this to the Environment=PATH= variable, then save and exit:

```bash
[Unit]
Description=Github webhook
After=network.target

[Service]
Environment=PATH=your_path
Type=simple
User=your-name
ExecStart=/usr/bin/nodejs /home/your-name/NodeWebHooks/webhook.js
Restart=on-failure

[Install]
WantedBy=multi-user.target
```
    
Enable and start the new service so it starts when the system boots:

```bash
sudo systemctl enable webhook.service
sudo systemctl start webhook
```
    
Check the status of the webhook:

```bash
sudo systemctl status webhook
```

You can test your webhook with these [instructions](https://www.digitalocean.com/community/tutorials/how-to-use-node-js-and-github-webhooks-to-keep-remote-projects-in-sync#step-4-testing-the-webhook "Test Webhook").

The Gatsby server is now running on your-ip:9000 and you implemented a CI/CD pipeline via PM2 and Github Webhooks but you still can't access your website via a domain because you need to configure a webserver like Nginx.

## Configure Nginx

I am using Cloudflare to manage DNS for my domains but you can do this with every other provider also.

Create two A Records which will point `your-domain.com` and `www.your-domain.com` to the IP-adress of your server.

After that you will need to configure Nginx.

The following instructions are based on [How To Install Nginx on Ubuntu 18.04 [Quickstart]](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04-quickstart).

- Update your local package index:

```bash
sudo apt update
```

- install Nginx:

```bash
sudo apt install nginx
```
    
and adjust the Firewall:

```bash
sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'
```

You should now be able to see the Nginx landing page on `http://your_server_ip`.

## Setting up Server Blocks
    
Create the directory for your-domain.com, using the -p flag to create any necessary parent directories:

```bash
sudo mkdir -p /var/www/your-domain.com/html
```
    
Assign ownership of the directory:

```bash
sudo chown -R $USER:$USER /var/www/your-domain.com/html
```
    
The permissions of your web roots should be correct if you haven’t modified your umask value, but you can make sure by typing:

```bash
sudo chmod -R 755 /var/www/example.com
```
    
Make a new server block at /etc/nginx/sites-available/your-domain.com:

```bash
sudo nano /etc/nginx/sites-available/example.com
```
    
Copy/Paste the following Gatsby-nginx configuration and update the server_name sections:

```nginx
server {
    # Listen HTTP
    listen 80;
    listen [::]:80;

    server_name your-domain.com www.your-domain.com;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    # Listen HTTP
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name your-domain.com www.your-domain.com;

    # SSL config
    include snippets/self-signed.conf;
    include snippets/ssl-params.conf;

    # Proxy Config
    location / {
        proxy_pass http://localhost:9000
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass_request_headers on;
    }
    location ~ /.well-known {
       allow all;
    }
}
```

Save the file and close it when you are finished.

Enable the file by creating a link from it to the sites-enabled directory:

```bash
sudo ln -s /etc/nginx/sites-available/your-domain.com /etc/nginx/sites-enabled/
```
    
Test for syntax errors:

```bash
sudo nginx -t
```
    
and finally enable the changes:

```bash
sudo systemctl restart nginx
```

Nginx should now be serving your gatsby site on your domain name. That means if you have a look at `http://your-domain.com` you should see your Gatsby site.

In the end should deny traffic to Port 9000 because Nginx is handling the requests with:

```bash
cd ~
sudo ufw deny 9000
```

To install SSL, you will need to [install and run Certbot by Let's Encrypt](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04).



With Gatsby 4 bringing in Server-Side Rendering (SSR) and Deferred Static Generation (DSG) you need an alternative methode to just hosting static files. Each page using SSR or DSG will be rendererd after a user requests it so there has be a server in the background which will handle these requests and build the pages if needed.

How to deploy your GatsbyJS site on your own server

This article shows you how can host your Next.js site on a (virtual private) server with Nginx and a CI/CD pipeline via PM2 and Github Webhooks.

## Setup your server

First of all you need an server with root access.
I strongly recommend to have a look at the guide ["Initial Server Setup with Ubuntu 18.04"](https://www.digitalocean.com/community/tutorials/initial-server-setup-with-ubuntu-18-04 "Initial Server Setup with Ubuntu 18.04") from the DigitalOcean community which will lead you through the process of:

1. Logging in and set up root user access to your server with SSH
2. Creating a new user
3. Granting Administrative Privileges to the new user
4. Setting up a basic firewall
5. Giving your regular user access to the server with SSH key authentication.

After you have done that you can continue by installing all necessary dependencies on your server.

### Install Node.js

Again there is an guide by DigitalOcean which will help you [installing Node.js using PPA](https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04#installing-using-a-ppa "Installing Node.jsUsing a PPA"). 

After completing 
+ **install Node.js, NPM and**
+ **the "build-essential package"**

you will have to change npm's default directory.

+ Create a `.npm-global` directory and set the path to this directory for `node_modules`:

```bash
cd ~
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'
```

+ Create (or modify) a `~/.profile` and add the following line:
```bash
sudo nano ~/.profile
```

```bash
# set PATH so global node modules install without permission issues
export PATH=~/.npm-global/bin:$PATH
```
Now you have to update your system variables:

```bash
source ~/.profile
```

Now you should be able to check your installed Node.js version with:
 ```bash
node -v
```

### Install git

Check if git is already installed with:
```bash
git --version
```

If it isn't installed yet you can install it with

```bash
sudo apt install git
```

and configure Git with 

```bash
git config --global user.name "Your Name"
git config --global user.email "youremail@domain.com"
```
After git is installed and configured you can deploy your project by cloning it from Github.

## Deploy from Github

It is important that you are loggin in as **non-root** user for the following steps.

```bash
cd ~
git clone https://github.com/your-name/your-project-repo.git path
```

Create a `.env`  on the server if you are using one locally and copy/paste your content.

After you have deployed your project (optionally with environment variables) you can install all dependencies and build your Next.js site with:
```bash
cd ./my-project/
npm install
NODE_ENV=production npm run build
```

Now you should have a copy of your local project/Next.js site on your remote server.

Next you are going to setup PM2 which will be used to keep your site alive and restart it after every reboot.

## Setup PM2

You can install PM2 with:
```bash
npm install pm2@latest -g
```

You will need to create/configure an `ecosystem.config.js` file which will restart the default Next.js server.
```bash
cd ~
pm2 init
sudo nano ecosystem.config.js
```

Copy/paste the template and replace the content.
```javascript
module.exports = {
  apps: [
    {
      name: 'next-site',
      cwd: ' /home/your-name/my-nextjs-project',
      script: 'npm',
      args: 'start',
      env: {
        NEXT_PUBLIC_...: 'NEXT_PUBLIC_...',
      },
    },
    // optionally a second project
],};
```

With 

```bash
cd ~
pm2 start ecosystem.config.js
```

you can start your server which will run on the `Port 1337`.

 You can always check the status with:

```bash
pm2 status next-site
```

After the server reboots this PM2 should be always automatically be restarted. For that you are going to need a small Startup script which you can also copy/paste.

+ Generate and configure a startup script to launch PM2:

```bash
cd ~
pm2 startup systemd

[PM2] Init System found: systemd
[PM2] To setup the Startup Script, copy/paste the following command:
**sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u your-name --hp /home/your-name**
```

+ Copy/paste the generated command:

```bash
**sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u your-name --hp /home/your-name**

[PM2] Init System found: systemd
Platform systemd

. . .


[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
   $ pm2 save

[PM2] Remove init script via:
   $ pm2 unstartup systemd
```

+ And save the new PM2 process list and environments. Then Start the service with `systemctl`.

```bash
pm2 save

[PM2] Saving current process list...
[PM2] Successfully saved in /home/your-name/.pm2/dump.pm2
```

If you reboot your server now with `sudo reboot` the script should be automatically restart your Next.js site. Give it a try!

## Setup Github Webhook

One thing missing now is an continuos integration and continuos delivery (CI/CD) pipeline which you will setup using Github webhooks. 

Therefore you need to create a new Webhook in your repository.

The following articles provide additional information to the steps below: 
+ [GitHub Creating Webhooks Guide](https://docs.github.com/en/developers/webhooks-and-events/webhooks/creating-webhooks#setting-up-a-webhook "GitHub Creating Webhooks Guide") and 
+ [DigitalOcean Guide to GitHub WebHooks](https://www.digitalocean.com/community/tutorials/how-to-use-node-js-and-github-webhooks-to-keep-remote-projects-in-sync "DigitalOcean Guide to GitHub WebHooks"). 

You need to create a server script which will do something if it is triggered by the Github webhook.

```bash
cd ~
mkdir NodeWebHooks
cd NodeWebHooks
sudo nano webhook.js
```

The script is going to create a server running on `Port 8100`. 
(Your Github webhook should be of course sending the webhook to something like `http://server-ip:8100`.)

If it gets triggered by a webhook it will 
+ go into your repo `~/my-nextjs-project/`, 
+ pull the latest commits, 
+ install all dependencies, 
+ build a new version of the site and 
+ restart the server via the PM2 script.

```javascript
const secret = "your-secret-key";
const repo = "~/my-nextjs-project/";

const http = require('http');
const crypto = require('crypto');
const exec = require('child_process').exec;

const BUILD_CMD = 'npm install && NODE_ENV=production npm run build';
const PM2_CMD = 'pm2 restart next-site';

http.createServer(function (req, res) {
    req.on('data', function(chunk) {
        let sig = "sha1=" + crypto.createHmac('sha1', secret).update(chunk.toString()).digest('hex');

        if (req.headers['x-hub-signature'] == sig) {
            exec('cd ' + repo + ` && git pull && npm install && ${BUILD_CMD} && ${PM2_CMD}`);
        }
    });

    res.end();
}).listen(8100);
```

You will need to allow communication on `Port 8100` with:

```bash
sudo ufw allow 8100/tcp
sudo ufw enable

Command may disrupt existing ssh connections. Proceed with operation (y|n)? y
Firewall is active and enabled on system startup
```

Earlier you setup PM2 to restart the services (your Next.js site) whenever the server reboots or is started. You will now do the same for the webhook script.

- Run echo $PATH and copy the output for use in the next step.

```bash
echo $PATH

/home/your-name/.npm-global/bin:/home/your-name/bin:/home/your-name/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
```

+ Create a webhook.service file:

```bash
cd ~
sudo nano /etc/systemd/system/webhook.service
```

+ In the editor, copy/paste the following script, but make sure to replace your-name in two places with your username. Earlier, you ran echo $PATH, copy this to the Environment=PATH= variable, then save and exit:

```bash
[Unit]
Description=Github webhook
After=network.target

[Service]
Environment=PATH=your_path
Type=simple
User=your-name
ExecStart=/usr/bin/nodejs /home/your-name/NodeWebHooks/webhook.js
Restart=on-failure

[Install]
WantedBy=multi-user.target
```

+ Enable and start the new service so it starts when the system boots:

```bash
sudo systemctl enable webhook.service
sudo systemctl start webhook
```

+ Check the status of the webhook:

```bash
sudo systemctl status webhook
```

You can test your webhook with these [instructions](https://www.digitalocean.com/community/tutorials/how-to-use-node-js-and-github-webhooks-to-keep-remote-projects-in-sync#step-4-testing-the-webhook "Test Webhook").

The Next.js server is now running on *your-ip:3000* and you implemented a CI/CD pipeline via PM2 and Github Webhooks but you still can't access your website via a domain because you need to configure a webserver like Nginx.

## Configure Nginx

I am using Cloudflare to manage DNS for my domains but you can do this with every other provider also.

+ Create two `A` Records which will point `your-domain.com` and `www.your-domain.com` to the IP-adress of your server.

After that you will need to configure Nginx. 

_The following instructions are based on [How To Install Nginx on Ubuntu 18.04 [Quickstart]](https://www.digitalocean.com/community/tutorials/how-to-install-nginx-on-ubuntu-18-04-quickstart "How To Install Nginx on Ubuntu 18.04 [Quickstart]")._

+ Update your local package index:

```bash
sudo apt update
```

+ install Nginx:

```bash
sudo apt install nginx
```

+ and adjust the Firewall:

```bash
sudo ufw allow 'Nginx Full'
sudo ufw delete allow 'Nginx HTTP'
```

You should now be able to see the Nginx landing page on `http://your_server_ip`.

### Setting up Server Blocks

+ Create the directory for ```your-domain.com```, using the -p flag to create any necessary parent directories:

```bash
sudo mkdir -p /var/www/your-domain.com/html
```
+ Assign ownership of the directory:

```bash
sudo chown -R $USER:$USER /var/www/your-domain.com/html
```

+ The permissions of your web roots should be correct if you haven’t modified your `umask` value, but you can make sure by typing:

```bash
sudo chmod -R 755 /var/www/example.com
```

+ Make a new server block at `/etc/nginx/sites-available/your-domain.com`: 

```bash
sudo nano /etc/nginx/sites-available/example.com
```

+ Copy/Paste the following nextjs-nginx configuration and update the server_name sections:

```nginx
server {
    # Listen HTTP
    listen 80;
    listen [::]:80;

    server_name your-domain.com www.your-domain.com;

    # Redirect HTTP to HTTPS
    return 301 https://$host$request_uri;
}

server {
    # Listen HTTP
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name your-domain.com www.your-domain.com;

    # SSL config
    include snippets/self-signed.conf;
    include snippets/ssl-params.conf;

    # Proxy Config
    location / {
        proxy_pass http://localhost:3000
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_pass_request_headers on;
    }
    location ~ /.well-known {
       allow all;
    }
}
```

Save the file and close it when you are finished.

+ Enable the file by creating a link from it to the sites-enabled directory:

```bash
sudo ln -s /etc/nginx/sites-available/your-domain.com /etc/nginx/sites-enabled/
```

+ Test for syntax errors:

```bash
sudo nginx -t
```

+ and finally enable the changes:
```bash
sudo systemctl restart nginx
```

Nginx should now be serving content on your domain name. That means if you have a look at `http://your-domain.com` you should see your Next.js site.


In the end should deny traffic to `Port 3000` with: 

```bash
cd ~
sudo ufw deny 3000
```

+ To install SSL, you will need to [install and run Certbot by Let's Encrypt](https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-18-04 "install and run Certbot by Let's Encrypt") .


_This guide is also using parts of [Strapi Deployment on DigitalOcean](https://strapi.gitee.io/documentation/3.0.0-beta.x/deployment/digitalocean.html "Strapi Deployment on DigitalOcean") which helped me a lot setting up Strapi and Next.js on a server in a proper way._ 

This article shows you how can host your Next.js site on a (virtual private) server with Nginx, a CI/CD pipeline via PM2 and Github Webhooks.

Hosting NextJS on a private server using PM2 and Github webhooks as CI/CD

With Google Cloud (Build) you can automate your whole workflow from building your Gatsby site up to deploying your site to Firebase hosting.

**What will you need?**

+ [Firebase Hosting](https://firebase.google.com/products/hosting "Firebase Hosting")
+ [Cloud Build](/cloud-google/cloud-build "Cloud Build")
+ [Cloud Scheduler](/cloud-google/scheduler "Cloud Scheduler") (optional)

## Firebase Hosting

To set up Firebase you will need an Google Cloud Account which has billing enabled and at least one project.
You can add the Firebase SDK with:

```bash
npm install --save firebase
```

and configure it with

```bash
firebase init
```

In the process you can setup a new project which will create a new `.firebaserc` file if it doesn't exist yet and enable **Hosting**.

![Firebase_init_1_c3dd56b33d.png](https://cms.mxd.codes/uploads/Firebase_init_1_c3dd56b33d_c68c4c50c3.png)

As public directory you have to set your **public** folder. After you have set your public folder you can reject all oncoming messages.

> You can also have a look at the [Google Firebase Docs](https://firebase.google.com/docs/cli#install-cli-windows "Google Firebase Docs") for setting up Firebase.

In the end it will also create a **firebase.json** where you can copy/paste the following to optimize your hosting for a Gatsby site.

```json
{
  "hosting": {
    "public": "public",
    "ignore": [
      "firebase.json", 
      "**/.*", 
      "**/node_modules/**"],
    "headers": [
      {
        "source": "**/*",
        "headers": [
          {
            "key": "cache-control",
            "value": "cache-control: public, max-age=0, must-revalidate"
          }
        ]
      },
      {
        "source": "static/**",
        "headers": [
          {
            "key": "cache-control",
            "value": "public, max-age=31536000, immutable"
          }
        ]
      },
      {
        "source": "**/*.@(css|js)",
        "headers": [
          {
            "key": "cache-control",
            "value": "public, max-age=31536000, immutable"
          }
        ]
      },
      {
        "source": "sw.js",
        "headers": [
          {
            "key": "cache-control",
            "value": "cache-control: public, max-age=0, must-revalidate"
          }
        ]
      },
      {
        "source": "page-data/**",
        "headers": [
          {
            "key": "cache-control",
            "value": "cache-control: public, max-age=0, must-revalidate"
          }
        ]
      }
    ],
  }
}
```
When you have build your Gatsy site and **public** folder is present you can upload it to your Firebase Hosting with:

```bash
firebase deploy
```

If you want to use your custom domain (for example mxd.codes) you have to go to Hosting in the Firebase console where you will find DNS-records to point your domain to your Firebase Hosting.
You can also create a second domain (for example www.gis-netzwerk.com) to redirect automatically to your root domain.

In your project settings you can find your Firebase configurations which look like:

```javascript
const firebaseConfig = {
  apiKey: "apiKey",
  authDomain: "{.firebaseapp.com",
  databaseURL: "https:// projectId.firebaseio.com",
  projectId: " projectId",
  storageBucket: " projectId.appspot.com",
  messagingSenderId: "1",
  appId: "2",
  measurementId: "G-123"
};
```

You probably want to save these as environment variables.

## Cloud Build

To create a CI-/CD-Pipeline you have to activate Cloud Build for your account. The console itself is quite clear in comparison to AWS CodeBuild.
You will see Dashboard which is displaying some basic informations, the history of your Cloud Builds, Triggers and Options.

You just will have to create a new trigger which will trigger a new build everytime new content is pushed to your linked GitHub repository..

First of all you have to connect your repository. After that you can create the trigger.

![Cloud_Build_Trigger_settings_c6be399330.png](https://cms.mxd.codes/uploads/Cloud_Build_Trigger_settings_c6be399330_9340cf2334.png)

Important options are:
+ Event: **Push to branch**
+ Branch: **^master$** (or whatever branch you want to build)
+ Cloud Build Configuration file: **cloudbuild.yaml**

After that you can create all environment variables as **Substitution variables**.
You will notice that all variables have to start with an underscore.
Because of that we will need a small workaround in the cloudbuild.yaml configuration file.

For now you can just create your substitions variables and add the underscore to your default variable names.

To be able to deploy via firebase you will need to authorize Firebase with a '$_TOKEN'.
You can retrive this token on your local machine with:

```bash
firebase login:ci
```

A new page will be opened in your prefered browser where you will have to login with your Google account to get the token.
Once you have the token you can also add it as substition variable.

```bash
_TOKEN : {TOKEN VALUE}
```

If you have created all substitution variables you can check again if they are inserted correct and create the new trigger.

### Cloudbuild.yaml

Cloud build needs the cloudbuild.yaml to know what it should do.
If you have inserted the path like above mentioned you will need a cloudbuild.yaml in your root directory.

You can copy the following into it:

```yaml
steps: 
# Install dependencies
  - name: node:10.16.0
    id: Installing dependencies...
    entrypoint: npm
    args: ["install"] 
    waitFor: ["-"] # Begin immediately

# Install Firebase   
  - name: node:10.16.0 
    id: Installing Firebase...
    entrypoint: npm 
    args: ["install", "firebase-tools"]
    waitFor:
      - Install dependencies...

# Create file with env-variables
  - name: node:10.16.0
    id: Creating Envirnonment variables...
    entrypoint: npm
    args: ["run", "create-env"]
    env:
    - "CLIENT_EMAIL=${_CLIENT_EMAIL}"
    - "PRIVATE_KEY=${_PRIVATE_KEY}"
    - "MAIL_CHIMP=${_MAIL_CHIMP}"
    - "GA_ID=${_GA_ID}"
    - "GA_VIEW_ID=${_GA_VIEW_ID}"
    - "IG_TOKEN=${_IG_TOKEN}"
    - "FIREBASE_API_KEY=${_FIREBASE_API_KEY}"
    - "FIREBASE_APP_ID=${_FIREBASE_APP_ID}"
    - "FIREBASE_AUTH_DOMAIN=${_FIREBASE_AUTH_DOMAIN}"
    - "FIREBASE_DB_URL=${_FIREBASE_DB_URL}"
    - "FIREBASE_MEASUREMENT_ID=${_FIREBASE_MEASUREMENT_ID}"
    - "FIREBASE_MESSAGE_SENDER_ID=${_FIREBASE_MESSAGE_SENDER_ID}"
    - "FIREBASE_PROJECT_ID=${_FIREBASE_PROJECT_ID}"
    - "FIREBASE_STORAGE_BUCKET=${_FIREBASE_STORAGE_BUCKET}"
    - "GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true"
    waitFor: ["-"] # Begin immediately

# Gatsby build
  - name: node:10.16.0
    id: Building Gatsby site...
    entrypoint: npm
    args: ["run", "build"]
    waitFor:
      - Installing dependencies...
      - Creating Envirnonment variables...

# Deploy
  - name: node:10.16.0 
    id: Deploying to Firebase...
    entrypoint: "./node_modules/.bin/firebase" 
    args: ["deploy", "--project", "$PROJECT_ID", "--token", "$_TOKEN"]
    waitFor:
      - Installing Firebase...
      - Building Gatsby site...
    
timeout: 30m0s
```

The cloudbuild.yaml is basically divided into six parts which will
+ Install dependencies,
+ Install Firebase,
+ Create file with env-variables,
+ Build the Gatsby site and
+ Deploy it to Firebase

As you can see you will create a file with the environment variables which will map the substitutions variables with your "default" variables.
Everything else should be pretty self-explanatory.

> Cloud Build will timeout the build by default after 10 minutes. So if your build is gonna take longer you will have to set up a custom timeout like in the cloudbuild.yaml above. You can also set a timeout for each step.

Another import point is that you will have to add the Plugin for Google Analytics Reporting Api as dynamic plugin like for example the following because otherwise you will get errors during your build.

```javascript
if (
  process.env.CLIENT_EMAIL &&
  process.env.PRIVATE_KEY &&
  process.env.GA_VIEW_ID
) {
  const startDate = new Date()
  startDate.setMonth(startDate.getMonth() - 3)
  dynamicPlugins.push(
    /*{
      resolve: `gatsby-plugin-guess-js`,
      options: {
        GAViewID: process.env.GA_VIEW_ID,
        jwt: {
          client_email: process.env.CLIENT_EMAIL,
          private_key: process.env.PRIVATE_KEY.replace(/\\n/g, "\n"),
        },
        period: {
          startDate,
          endDate: new Date(),
        },
      }
    },*/
    {
      resolve: `gatsby-source-google-analytics-reporting-api`,
      options: {
        email: process.env.CLIENT_EMAIL,
        key: process.env.PRIVATE_KEY.replace(/\\n/g, "\n"),
        viewId: process.env.GA_VIEW_ID,
        startDate: `2009-01-01`,
      }
    }
  )
}

module.exports = {
  plugins: [
    plugins...
  ].concat(dynamicPlugins),
};

Due to the fact that Code Build uses Linux machines small and capital letters are important (Windows doesnt care).
That means if you import a component like

```javascript
import MyComponent from "../Mycomponent"
```

and the actual folder name is ```MyComponent``` your build will fail.

## Speeding up your builds

+ **Image optimization**

If you aren't using preoptimized images yet you should consider to crop, resize images etc. **before** you will build the site because it can basically save a lot of time (depending on the amount of images).

Before i optimized my images a build on Google Cloud took about ~ 1800 sec.
After i optimized all my images for posts with Python ([Image optimization with Python](/articles/scaling-and-cropping-images-using-python "Image optimization with Python")) the build time went down to ~ 620 sec. So actually just a third.

## Set Cloud Scheduler (optional)

With Cloud Scheduler you can trigger a build automatically after a specific time.
The first 3 jobs a month a free and after that you have to pay $0.10 for every job after that.

(You can also use Cloud Scheduler with your default trigger (Push to branch) or without).

The trigger you created before has an ID and with a POST request you can start the trigger anytime you want.
To get the ID of the trigger you have to open the Cloud Shell, type ```gcloud beta builds triggers list``` and search for **id**. Copy that.

The URL for the POST request looks like (without []):
`https://cloudbuild.googleapis.com/v1/projects/[PROJECT_ID]/triggers/[TRIGGER_ID]:run`
Now you have to create a new job in Cloud Schedule.

![Cloud_Scheduler_Settings_1d117b6db6.png](https://cms.mxd.codes/uploads/Cloud_Scheduler_Settings_1d117b6db6_28caea6e1d.png)

```cron
0 3 * * *
```
will trigger a build every day at 3 am MESZ. 

The URL for the POST request looks like (without []):
`https://cloudbuild.googleapis.com/v1/projects/[PROJECT_ID]/triggers/[TRIGGER_ID]:run
`

In Text you will need

```json
{
  "branchName": "master"
}
```

and you will authorize the job with your service account.



Ultimate guide to automate your Gatsby builds with Google Cloud Build, deploying to Firebase and optional Cloud Scheduler.

Build and deploy your Gatsby site with Google Cloud Build to Firebase

With the free [contingent for AWS](https://aws.amazon.com/en/free/free-tier/ "contingent for AWS") you always get an active [AWS code pipeline](https://aws.amazon.com/codepipeline/?did=ft_card&trk=ft_card "AWS Codepipeline") per month and [100 minutes AWS codebuild](https://aws.amazon.com/codebuild/?did=ft_card&trk=ft_card "100 minutes AWS codebuild" ) per month.

So you can set up a continuous integration and continuous delivery pipeline for free or with more than 100 build minutes a month for relatively little money, which triggers a build with every push to a GitHub repository whoch will be automatically deployed to S3 and optionally also invalidate CloudFront Cache.

## CodeBuild

First of all you need a new build project in CodeBuild.
In the project configuration you can assign a name for it and select GitHub as the source provider under **Source**.

Depending on whether the repository is public or not, you then select "Public Repository" and enter the repository URL or you link your GitHub account and give CodeBuild the necessary rights to access the repository.

An environment image must now be selected under **Environment**. For GatsbyJS this would be a "managed image" and the operating system "Amazon Linux 2". "Standard" is selected as the runtime (s) and "aws / codebuild / amazonlinux2-x86_64-standard: 2.0" as the image.

![codebuild_umgebung_e32ebcd3d2.png](https://cms.mxd.codes/uploads/codebuild_umgebung_e32ebcd3d2_34c630be29.png)

Now a new service role can be created automatically (which is required) so that CodeBuild has the necessary rights for the AWS account.

> This service role can also be assigned rights for CodePipeline, so that this service role can be used for CodeBuild and CodePipeline.
If environment variables are used, these can be specified under "Additional configuration" in the environment. You can also make sure that "3GB RAM, 2vCPUs" is really selected, since only this option is included in the free contingent.

**Buildspec** now uses a buildspec file in YAML format. For a Gatsby site this should somehow look like the following:

```yaml
version: 0.2
phases:
    install:
        runtime-versions:
            nodejs: 12
        commands:
            - 'touch .npmignore'
            - 'npm install -g gatsby'
    pre_build:
        commands:
            - 'npm install'
    build:
        commands:
            - 'npm run build'
    post_build:
        commands:       
            - 'find public -type f -regex ".*\.$htm\|html\|txt\|text\|js\|css\|json$$" -exec gzip -f -k {} \' ## sofern cloudfront nicht automatisch die dateien komprimiert
artifacts:
    base-directory: public
    files:
        - '**/*'
    discard-paths: no
cache:
    paths:
        - '.cache/*'
        - 'public/*'
```

The buildspec.yml file only needs to be placed in the root directory so that CodeBuild can find it.
In addition, the build script must of course still be available in "package.json".

```json
"build": "gatsby build",
```

The default settings can be retained under **Artifacts**.
With CloudWatch you have the possibility to save logs for CodeBuild in an S3 bucket.

> There may be additional costs!

If all settings have now been entered correctly, the build project can be created.
The only thing missing now is the code pipeline that triggers a build and deployed it in the S3 bucket.

## CodePipeline

For this you switch to CodePipeline and create a new pipeline in which a name and a service role are selected first.

At Source you can now log in with a GitHub account and link the respective repository with a branch.
You now have two options to trigger a build.

+ GitHub-Webhooks and
+ AWS CodePipeline

Then you choose the build provider "AWS CodeBuild" and the previously created project or (if you have not already done this) create a new project.

After a build, the public/ folder can also be automatically deployed to an S3 bucket with AWS CodeDeploy.
_Alternatively_ you can also skip this step and [gatsby-plugin-s3](https://github.com/jarizhttps://www.gatsbyjs.com-plugin-s3 "gatsby-plugin-s3"), which also optimizes caching.

```bash
npm i gatsby-plugin-s3
```
bzw.
```bash
yarn add gatsby-plugin-s3
```

Now the configuration in gatsby-config.js and the deployment script are missing

```javascript
plugins: [
  {
      resolve: `gatsby-plugin-s3`,
      options: {
          bucketName: 'my-website-bucket'
      },
  },
]
```

```javascript
"scripts": {
    ...
    "deploy": "gatsby-plugin-s3 deploy --yes"
}
```
The deployment script "npm run deploy" must then of course be added to the buildspec file under post-build commands.
The CodePipeline should now look something like this:

![Code_Pipeline_b8acf52436.png](https://cms.mxd.codes/uploads/Code_Pipeline_b8acf52436_43c7fc9910.png)

Experience shows that a build for around 100 pages takes around 10 minutes if you have some pictures per page.

Every time CodePipeline detects a push to the GitHub repository, a build is automatically triggered and made available on an S3 bucket.

With `aws cloudfront create-invalidation --distribution-id DISTRIBUTION_ID --paths / * ` the CloudFront cache can also be invalidated.

With the free tier for AWS you always get one active AWS code pipeline per month and 100 minutes of AWS code build per month with which you can create a CI / CD pipeline for a GatsbyJS site.

GatsbyJS with CI/CD Pipeline via Codebuild 

With [AWS](https://aws.amazon.com/ "AWS") (and in particular the [free AWS contingent](https://aws.amazon.com/en/free/ "free AWS account")) you have the option of a static website with a custom domain for a few Hosting cents a month including CDN via CloudFront and CI / CD integration.

Before I switched completely to AWS, I had a common shared hosting option that cost me around € 72 a year. With this option I had
+ 250 GB SSD storage space,
+ free unlimited traffic,
+ 6 domains included,
+ 250 GB SSD mail storage space,
+ 25 MySQL databases,
+ and, and, and.

On the whole, much more than I need to run my static [GatsbyJS](https://www.gatsbyjs.com/) website.

So why shouldn't I only use and pay for resources that I ultimately need and also get some cloud computing experience?

## Creating S3-Buckets 

The basis for hosting on AWS is formed by [S3 Buckets](https://s3.console.aws.amazon.com/s3/home "S3 Buckets"). 
Buckets are "containers" on the web where you can save files.
In order for redirects from subdomains such as **www.mxd.codes** to **mxd.codes** to work, you need a bucket for each domain.

First of all, you create an S3 bucket for the root domain. In my case, the bucket name is the domain name mxd.codes and you select a region (for example EU (Frankfurt)).
The default settings can be kept under Options, unless you want to save different versions or access protocols.
So that everyone can access the website content later, remove the tick that is present by default at "Block *any* public access", check the bucket settings again and finally create it.

In each bucket you can or should enter a bucket policy that further defines access.
To do this, click on the name of the bucket and go to "Permissions" -> "Bucket Policy".

The following guideline must then be saved for public access.
```json
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "AllowPublicRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::mxd.codes/*"
        }
    ]
}
```
**"mxd.codes" has to be replace with your bucket name!**

If everything was done correctly, the permissions should now look something like this:

![bucket_richtline_d32ff17da6.png](https://cms.mxd.codes/uploads/bucket_richtline_d32ff17da6_173be5d676.png)

In the bucket settings you have to activate the "hosting a static website" and specify an index document and an error document.
For GatsbyJS that would be **index.html** and **404.html**.

![bucket_hosting_6f40697460.png](https://cms.mxd.codes/uploads/bucket_hosting_6f40697460_500bdd971a.png)

Now the S3 bucket for the subdomain **www.mxd.codes** is still missing.
So create a new bucket with the name of the subdomain **www.mxd.codes** with public access and add the bucket policy.

In the settings for "hosting a static website" you use "redirect requests" and enter the target bucket **mxd.codes** and you can enter **https** as a protocol, because later on the content of the static website is delivered via CloudFront, which can be encrypted with SSL certificates.

![bucket_redirect_740544f6d5.png](https://cms.mxd.codes/uploads/bucket_redirect_740544f6d5_5b708ee0ba.png)

The buckets are now created and correctly configured for the operation of a static website including redirect.

## CloudFront Distributions

With the free AWS contingent, 50 GB of data transfer can be burned per month in Cloudflare.

With a page size of generous 4 MB, this is enough for 12,500 page views per month and should therefore be more than sufficient for a website with average traffic. So why not take a free CDN with you?

If the costs incurred after the free year should deter someone, you still have the option to switch to another CDN provider such as [CloudFlare](https://www.cloudflare.com "CloudFlare").

At [CloudFront](https://console.aws.amazon.com/cloudfront/ "CloudFront") you have to create a web distribution for **each** bucket.
As the origin domain name, a bucket can **not** be selected from the dropdown list, but the end point of the bucket from S3 must be copied.

![cloudfront_origin_f1e2731789.png](https://cms.mxd.codes/uploads/cloudfront_origin_f1e2731789_3f2eb47d64.png)

In the distribution for the bucket **mxd.codes**, for example "mxd.codes.s3-website.eu-central-1.amazonaws.com" is specified as origin.
"Origin ID" will then be filled in automatically. In "Viewer Protocol Policy" **Redirect HTTP to HTTPS** is selected because users should only be able to access the website via HTTPS.  "Compress Objects Automatically" **Yes** can be selected, so that CloudFront will zip all files automatically.
Under "Alternate Domain Names (CNAMES)" you have to specify the bucket for the root domain for the distribution for the root domain. For example mxd.codes

At "SSL-Certifacte" you can now create for the two domains **mxd.codes** and **www.mxd.codes** two free Amazon SSL certificate via the [Certificate Manager (ACM )](https:/https://aws.amazon.com.amazon.com/en/certificate-manager/ "Certificate Manager (ACM)"). To do this, add your two domains in ACM.

![acm_domain_ee3a548091.png](https://cms.mxd.codes/uploads/acm_domain_ee3a548091_38bd644d47.png)

You can now have this validated using a DNS or email method. If you have included your domain in Route 53, you can do it more or less automatically by simply following the instructions.

Back in the CloudFront Distribution Creation you only have to specify a "Default Root Object". -> index.html
If you don't do this, CloudFront always shows an "Access Denied" message in XML format when you access your domain (atleast that was my case).

Finally, the distribution must of course still be activated under "Distribution State".

First distribution finished. The same procedure now for the subdomain **www.mxd.codes** with the corresponding "Origin Domain Name" (**Bucket end point!**)

_This can take up to 20 minutes_
(If you want to clear your CloudFront cache and have already installed AWS CLI, you can do this with the following command:

```bash
aws cloudfront create-invalidation --distribution-id DEINE_DISTRIBUTION ID --paths "/*"
```

In the meantime, you can create the redirects for Cloudfront in Route 53.

## Route 53 DNS settings

In [Route 53]("Route 53") you need a hosted zone (= 0.50€ per month). Then A (and provided that in the CLoudFrontDistribution IPv6 is
activated (which it is by default), as well as an AAAA) data record can be created.

That means you basically need 4 "alias" data sets:
+ mxd.codes A "CloudFront URL for root domain"
+ mxd.codes AAAA "CloudFront URL for root domain"
+ www.mxd.codes A "CloudFront URL for subdomain"
+ www.mxd.codes AAAA "CloudFront URL for subdomain"

Exceptionally, you can select the CDN url from the dropdown list.
With "Routing Guideline" and "Assess the state of the target" you can leave the default settings, unless you want to experiment.

Now if you wait a little you should be redirect from
+ http://wwww.mxd.codes,
+ https://wwww.mxd.codes,
+ wwww.mxd.codes,
+ http://mxd.codes and
+ mxd.codes
to **https://mxd.codes** .


With AWS (and in particular the free AWS contingent) you have the option of a static website with a custom domain for a few Hosting cents a month including CDN via CloudFront and CI/CD integration.