mailcow-dockerized-docs/docs/third_party-borgmatic.md

Borgmatic Backup

Introduction

Borgmatic is a great way to run backups on your Mailcow setup as it securely encrypts your data and is extremely easy to set up.

Due to it's deduplication capabilities you can store a great number of backups without wasting large amounts of disk space. This allows you to run backups in very short intervals to ensure minimal data loss when the need arises to recover data from a backup.

This document guides you through the process to enable continuous backups for mailcow with borgmatic. The borgmatic functionality is provided by the borgmatic Docker image by b3vis. Check out the README in that repository to find out about the other options (such as push notifications) that are available. This guide only covers the basics.

Setting up borgmatic

Create or amend docker-compose.override.yml

In the mailcow-dockerized root folder create or edit docker-compose.override.yml and insert the following configuration:

version: '2.1'

services:
  borgmatic-mailcow:
    image: b3vis/borgmatic
    hostname: mailcow
    restart: always
    dns: ${IPV4_NETWORK:-172.22.1}.254
    volumes:
      - vmail-vol-1:/mnt/source/vmail:ro
      - crypt-vol-1:/mnt/source/crypt:ro
      - redis-vol-1:/mnt/source/redis:ro,z
      - rspamd-vol-1:/mnt/source/rspamd:ro,z
      - postfix-vol-1:/mnt/source/postfix:ro,z
      - mysql-socket-vol-1:/var/run/mysqld/:z
      - borg-config-vol-1:/root/.config/borg:Z
      - borg-cache-vol-1:/root/.cache/borg:Z
      - ./data/conf/borgmatic/etc:/etc/borgmatic.d:Z
      - ./data/conf/borgmatic/ssh:/root/.ssh:Z
    environment:
      - TZ=${TZ}
      - BORG_PASSPHRASE=YouBetterPutSomethingRealGoodHere
    networks:
      mailcow-network:
        aliases:
          - borgmatic

volumes:
  borg-cache-vol-1:
  borg-config-vol-1:

Ensure that you change the BORG_PASSPHRASE to a secure passphrase of your choosing.

For security reasons we mount the maildir as read-only. If you later want to restore data you will need to remove the ro flag prior to restoring the data. This is described in the section on restoring backups.

Create data/conf/borgmatic/etc/config.yaml

Next, we need to create the borgmatic configuration.

source mailcow.conf
cat <<EOF > data/conf/borgmatic/etc/config.yaml
location:
    source_directories:
        - /mnt/source
    repositories:
        - user@rsync.net:mailcow
    exclude_patterns:
        - '/mnt/source/postfix/public/'
        - '/mnt/source/postfix/private/'
        - '/mnt/source/rspamd/rspamd.sock'

retention:
    keep_hourly: 24
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 6

hooks:
    mysql_databases:
        - name: ${DBNAME}
          username: ${DBUSER}
          password: ${DBPASS}
          options: --default-character-set=utf8mb4
EOF

Creating the file in this way ensures the correct MySQL credentials are pulled in from mailcow.conf.

This file is a minimal example for using borgmatic with an account user on the cloud storage provider rsync.net for a repository called mailcow (see repositories setting). It will backup both the maildir and MySQL database, which is all you should need to restore your mailcow setup after an incident. The retention settings will keep one archive for each hour of the past 24 hours, one per day of the week, one per week of the month and one per month of the past half year.

Check the borgmatic documentation on how to use other types of repositories or configuration options. If you choose to use a local filesystem as a backup destination make sure to mount it into the container. The container defines a volume called /mnt/borg-repository for this purpose.

!!! note If you do not use rsync.net you can most likely drop the remote_path element from your config.

Create a crontab

Create a new text file in data/conf/borgmatic/etc/crontab.txt with the following content:

14 * * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1

This file expects crontab syntax. The example shown here will trigger the backup to run every hour at 14 minutes past the hour and log some nice stats at the end.

Place SSH keys in folder

Place the SSH keys you intend to use for remote repository connections in data/conf/borgmatic/ssh. OpenSSH expects the usual id_rsa, id_ed25519 or similar to be in this directory. Ensure the file is chmod 600 and not world readable or OpenSSH will refuse to use the SSH key.

Bring up the container

For the next step we need the container to be up and running in a configured state. To do that run:

docker-compose up -d

Initialize the repository

By now your borgmatic container is up and running, but the backups will currently fail due to the repository not being initialized.

To initialize the repository run:

docker-compose exec borgmatic-mailcow borgmatic init --encryption repokey-blake2

You will be asked you to authenticate the SSH host key of your remote repository server. See if it matches and confirm the prompt by entering yes. The repository will be initialized with the passphrase you set in the BORG_PASSPHRASE environment variable earlier.

When using any of the repokey encryption methods the encryption key will be stored in the repository itself and not on the client, so there is no further action required in this regard. If you decide to use a keyfile instead of a repokey make sure you export the key and back it up separately. Check the Exporting Keys section for how to retrieve the key.

Restart container

Now that we finished configuring and initializing the repository restart the container to ensure it is in a defined state:

docker-compose restart borgmatic-mailcow

Restoring from a backup

Restoring a backup assumes you are starting off with a fresh installation of mailcow, and you currently do not have any custom data in your maildir or your mailcow database.

Restore maildir

!!! warning Doing this will overwrite files in your maildir! Do not run this unless you actually intend to recover mail files from a backup.

!!! note "If you use SELinux in Enforcing mode" If you are using mailcow on a host with SELinux in Enforcing mode you will have to temporarily disable it during extraction of the archive as the mailcow setup labels the vmail volume as private, belonging to the dovecot container exclusively. SELinux will (rightfully) prevent any other container, such as the borgmatic container, from writing to this volume.

Before running a restore you must make the vmail volume writeable in docker-compose.override.yml by removing the ro flag from the volume. Then you can use the following command to restore the maildir from a backup:

docker-compose exec borgmatic-mailcow borgmatic extract --path mnt/source --archive latest

Alternatively you can specify any archive name from the list of archives (see Listing all available archives)

Restore MySQL

!!! warning Running this command will delete and recreate the mailcow database! Do not run this unless you actually intend to recover the mailcow database from a backup.

To restore the MySQL database from the latest archive use this command:

docker-compose exec borgmatic-mailcow borgmatic restore --archive latest

Alternatively you can specify any archive name from the list of archives (see Listing all available archives)

After restoring

After restoring you need to restart mailcow. If you disabled SELinux enforcing mode now would be a good time to re-enable it.

To restart mailcow use the follwing command:

docker-compose down && docker-compose up -d

If you use SELinux this will also trigger the re-labeling of all files in your vmail volume. Be patient, as this may take a while if you have lots of files.

Useful commands

Manual archiving run (with debugging output)

docker-compose exec borgmatic-mailcow borgmatic -v 2

Listing all available archives

docker-compose exec borgmatic-mailcow borgmatic list

Break lock

When borg is interrupted during an archiving run it will leave behind a stale lock that needs to be cleared before any new operations can be performed:

docker-compose exec borgmatic-mailcow borg break-lock user@rsync.net:mailcow

Where user@rsync.net:mailcow is the URI to your repository.

Now would be a good time to do a manual archiving run to ensure it can be successfully performed.

Exporting keys

When using any of the keyfile methods for encryption you MUST take care of backing up the key files yourself. The key files are generated when you initialize the repository. The repokey methods store the key file within the repository, so a manual backup isn't as essential.

Note that in either case you also must have the passphrase to decrypt any archives.

To fetch the keyfile run:

docker-compose exec borgmatic-mailcow borg key export --paper user@rsync.net:mailcow

Where user@rsync.net:mailcow is the URI to your repository.