Skip to content

# Borgmatic Backup

## Introduction

Borgmatic is a great way to run backups on your Mailcow setup as it securely encrypts your data and is extremely easy to set up.

Due to it's deduplication capabilities you can store a great number of backups without wasting large amounts of disk space. This allows you to run backups in very short intervals to ensure minimal data loss when the need arises to recover data from a backup.

This document guides you through the process to enable continuous backups for mailcow with borgmatic. The borgmatic functionality is provided by the borgmatic Docker image by b3vis. Check out the `README` in that repository to find out about the other options (such as push notifications) that are available. This guide only covers the basics.

## Setting up borgmatic

### Create or amend `docker-compose.override.yml`

In the mailcow-dockerized root folder create or edit `docker-compose.override.yml` and insert the following configuration:

    version: '2.1'

    services:
      borgmatic-mailcow:
        image: b3vis/borgmatic
        hostname: mailcow
        restart: always
        dns: ${IPV4_NETWORK:-172.22.1}.254
        volumes:
          - vmail-vol-1:/mnt/source/vmail:ro
          - crypt-vol-1:/mnt/source/crypt:ro
          - redis-vol-1:/mnt/source/redis:ro,z
          - rspamd-vol-1:/mnt/source/rspamd:ro,z
          - postfix-vol-1:/mnt/source/postfix:ro,z
          - mysql-socket-vol-1:/var/run/mysqld/:z
          - borg-config-vol-1:/root/.config/borg:Z
          - borg-cache-vol-1:/root/.cache/borg:Z
          - ./data/conf/borgmatic/etc:/etc/borgmatic.d:Z
          - ./data/conf/borgmatic/ssh:/root/.ssh:Z
        environment:
          - TZ=${TZ}
          - BORG_PASSPHRASE=YouBetterPutSomethingRealGoodHere
        networks:
          mailcow-network:
            aliases:
              - borgmatic

    volumes:
      borg-cache-vol-1:
      borg-config-vol-1:

Ensure that you change the `BORG_PASSPHRASE` to a secure passphrase of your choosing.

For security reasons we mount the maildir as read-only. If you later want to restore data you will need to remove the `ro` flag prior to restoring the data. This is described in the section on restoring backups.

### Create `data/conf/borgmatic/etc/config.yaml`

Next, we need to create the borgmatic configuration.

    source mailcow.conf
    cat <<EOF > data/conf/borgmatic/etc/config.yaml
    location:
        source_directories:
            - /mnt/source
        repositories:
            - user@rsync.net:mailcow
        exclude_patterns:
            - '/mnt/source/postfix/public/'
            - '/mnt/source/postfix/private/'
            - '/mnt/source/rspamd/rspamd.sock'

    retention:
        keep_hourly: 24
        keep_daily: 7
        keep_weekly: 4
        keep_monthly: 6

    hooks:
        mysql_databases:
            - name: ${DBNAME}
              username: ${DBUSER}
              password: ${DBPASS}
              options: --default-character-set=utf8mb4
    EOF

Creating the file in this way ensures the correct MySQL credentials are pulled in from `mailcow.conf`.

This file is a minimal example for using borgmatic with an account `user` on the cloud storage provider `rsync.net` for a repository called `mailcow` (see `repositories` setting). It will backup both the maildir and MySQL database, which is all you should need to restore your mailcow setup after an incident. The retention settings will keep one archive for each hour of the past 24 hours, one per day of the week, one per week of the month and one per month of the past half year.

Check the borgmatic documentation on how to use other types of repositories or configuration options. If you choose to use a local filesystem as a backup destination make sure to mount it into the container. The container defines a volume called `/mnt/borg-repository` for this purpose.

Note

If you do not use rsync.net you can most likely drop the `remote_path` element from your config.

### Create a crontab

Create a new text file in `data/conf/borgmatic/etc/crontab.txt` with the following content:

    14 * * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1

This file expects crontab syntax. The example shown here will trigger the backup to run every hour at 14 minutes past the hour and log some nice stats at the end.

### Place SSH keys in folder

Place the SSH keys you intend to use for remote repository connections in `data/conf/borgmatic/ssh`. OpenSSH expects the usual `id_rsa`, `id_ed25519` or similar to be in this directory. Ensure the file is `chmod 600` and not world readable or OpenSSH will refuse to use the SSH key.

### Bring up the container

For the next step we need the container to be up and running in a configured state. To do that run:

    docker-compose up -d

### Initialize the repository

By now your borgmatic container is up and running, but the backups will currently fail due to the repository not being initialized.

To initialize the repository run:

    docker-compose exec borgmatic-mailcow borgmatic init --encryption repokey-blake2

You will be asked you to authenticate the SSH host key of your remote repository server. See if it matches and confirm the prompt by entering `yes`. The repository will be initialized with the passphrase you set in the `BORG_PASSPHRASE` environment variable earlier.

When using any of the `repokey` encryption methods the encryption key will be stored in the repository itself and not on the client, so there is no further action required in this regard. If you decide to use a `keyfile` instead of a `repokey` make sure you export the key and back it up separately. Check the Exporting Keys section for how to retrieve the key.

### Restart container

Now that we finished configuring and initializing the repository restart the container to ensure it is in a defined state:

    docker-compose restart borgmatic-mailcow

## Restoring from a backup

Restoring a backup assumes you are starting off with a fresh installation of mailcow, and you currently do not have any custom data in your maildir or your mailcow database.

### Restore maildir

Warning

Doing this will overwrite files in your maildir! Do not run this unless you actually intend to recover mail files from a backup.

If you use SELinux in Enforcing mode

If you are using mailcow on a host with SELinux in Enforcing mode you will have to temporarily disable it during extraction of the archive as the mailcow setup labels the vmail volume as private, belonging to the dovecot container exclusively. SELinux will (rightfully) prevent any other container, such as the borgmatic container, from writing to this volume.

Before running a restore you must make the vmail volume writeable in `docker-compose.override.yml` by removing the `ro` flag from the volume. Then you can use the following command to restore the maildir from a backup:

    docker-compose exec borgmatic-mailcow borgmatic extract --path mnt/source --archive latest

Alternatively you can specify any archive name from the list of archives (see Listing all available archives)

### Restore MySQL

Warning

Running this command will delete and recreate the mailcow database! Do not run this unless you actually intend to recover the mailcow database from a backup.

To restore the MySQL database from the latest archive use this command:

    docker-compose exec borgmatic-mailcow borgmatic restore --archive latest

Alternatively you can specify any archive name from the list of archives (see Listing all available archives)

### After restoring

After restoring you need to restart mailcow. If you disabled SELinux enforcing mode now would be a good time to re-enable it.

To restart mailcow use the follwing command:

    docker-compose down && docker-compose up -d

If you use SELinux this will also trigger the re-labeling of all files in your vmail volume. Be patient, as this may take a while if you have lots of files.

## Useful commands

### Manual archiving run (with debugging output)

    docker-compose exec borgmatic-mailcow borgmatic -v 2

### Listing all available archives

    docker-compose exec borgmatic-mailcow borgmatic list

### Break lock

When borg is interrupted during an archiving run it will leave behind a stale lock that needs to be cleared before any new operations can be performed:

    docker-compose exec borgmatic-mailcow borg break-lock user@rsync.net:mailcow

Where `user@rsync.net:mailcow` is the URI to your repository.

Now would be a good time to do a manual archiving run to ensure it can be successfully performed.

### Exporting keys

When using any of the `keyfile` methods for encryption you MUST take care of backing up the key files yourself. The key files are generated when you initialize the repository. The `repokey` methods store the key file within the repository, so a manual backup isn't as essential.

Note that in either case you also must have the passphrase to decrypt any archives.

To fetch the keyfile run:

    docker-compose exec borgmatic-mailcow borg key export --paper user@rsync.net:mailcow

Where `user@rsync.net:mailcow` is the URI to your repository.

Back to top