From 32f5c3158377a2658dbd46e0a16f737f258ee894 Mon Sep 17 00:00:00 2001 From: Felix Kaechele Date: Fri, 16 Apr 2021 19:36:22 -0400 Subject: [PATCH] Third-party: Borgmatic Backup This guide describes setting up automatic recurring backups using borgmatic. This is a good alternative to juggling tar archives. Signed-off-by: Felix Kaechele --- docs/third_party-borgmatic.md | 247 ++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 248 insertions(+) create mode 100644 docs/third_party-borgmatic.md diff --git a/docs/third_party-borgmatic.md b/docs/third_party-borgmatic.md new file mode 100644 index 000000000..4d9de3d01 --- /dev/null +++ b/docs/third_party-borgmatic.md @@ -0,0 +1,247 @@ +# Borgmatic Backup + +## Introduction + +Borgmatic is a great way to run backups on your Mailcow setup as it securely encrypts your data and is extremely easy to +set up. + +Due to it's deduplication capabilities you can store a great number of backups without wasting large amounts of disk +space. This allows you to run backups in very short intervals to ensure minimal data loss when the need arises to +recover data from a backup. + +This document guides you through the process to enable continuous backups for mailcow with borgmatic. The borgmatic +functionality is provided by the [borgmatic Docker image by b3vis](https://github.com/b3vis/docker-borgmatic). Check out +the `README` in that repository to find out about the other options (such as push notifications) that are available. +This guide only covers the basics. + +## Setting up borgmatic + +### Create or amend `docker-compose.override.yml` + +In the mailcow-dockerized root folder create or edit `docker-compose.override.yml` and insert the following +configuration: + +```yaml +version: '2.1' +services: + borgmatic-mailcow: + image: b3vis/borgmatic + restart: always + dns: ${IPV4_NETWORK:-172.22.1}.254 + volumes: + - vmail-vol-1:/mnt/source/vmail:ro + - mysql-socket-vol-1:/var/run/mysqld/:z + - ./data/conf/borgmatic/etc:/etc/borgmatic.d:Z + - ./data/conf/borgmatic/state:/root/.config/borg:Z + - ./data/conf/borgmatic/ssh:/root/.ssh:Z + environment: + - TZ=${TZ} + - BORG_PASSPHRASE=YouBetterPutSomethingRealGoodHere + networks: + mailcow-network: + aliases: + - borgmatic +``` + +Ensure that you change the `BORG_PASSPHRASE` to a secure passphrase of your choosing. + +For security reasons we mount the maildir as read-only. If you later want to restore data you will need to remove +the `ro` flag prior to restoring the data. This is described in the section on restoring backups. + +### Create `data/conf/borgmatic/etc/config.yaml` + +Next, we need to create the borgmatic configuration. + +```shell +source mailcow.conf +cat < data/conf/borgmatic/etc/config.yaml +location: + source_directories: + - /mnt/source + repositories: + - user@rsync.net:mailcow + remote_path: borg1 + +retention: + keep_hourly: 24 + keep_daily: 7 + keep_weekly: 4 + keep_monthly: 6 + +hooks: + mysql_databases: + - name: ${DBNAME} + username: ${DBUSER} + password: ${DBPASS} + options: --default-character-set=utf8mb4 +EOF +``` + +Creating the file in this way ensures the correct MySQL credentials are pulled in from `mailcow.conf`. + +This file is a minimal example for using borgmatic with an account `user` on the cloud storage provider `rsync.net` for +a repository called `mailcow` (see `repositories` setting). It will backup both the maildir and MySQL database, which is +all you should need to restore your mailcow setup after an incident. The retention settings will keep one archive for +each hour of the past 24 hours, one per day of the week, one per week of the month and one per month of the past half +year. + +Check the [borgmatic documentation](https://torsion.org/borgmatic/) on how to use other types of repositories or +configuration options. If you choose to use a local filesystem as a backup destination make sure to mount it into the +container. The container defines a volume called `/mnt/borg-repository` for this purpose. + +!!! note + If you do not use rsync.net you can most likely drop the `remote_path` element from your config. + +### Create a crontab + +Create a new text file in `data/conf/borgmatic/etc/crontab.txt` with the following content: + +``` +14 * * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic --stats -v 0 2>&1 +``` + +This file expects crontab syntax. The example shown here will trigger the backup to run every hour at 14 minutes past +the hour and log some nice stats at the end. + +### Place SSH keys in folder + +Place the SSH keys you intend to use for remote repository connections in `data/conf/borgmatic/ssh`. OpenSSH expects the +usual `id_rsa`, `id_ed25519` or similar to be in this directory. Ensure the file is `chmod 600` and not world readable +or OpenSSH will refuse to use the SSH key. + +### Bring up the container + +For the next step we need the container to be up and running in a configured state. To do that run: + +```shell +docker-compose up -d +``` + +### Initialize the repository + +By now your borgmatic container is up and running, but the backups will currently fail due to the repository not being +initialized. + +To initialize the repository run: + +```shell +docker-compose exec borgmatic-mailcow borgmatic init --encryption repokey-blake2 +``` + +You will be asked you to authenticate the SSH host key of your remote repository server. See if it matches and confirm +the prompt by entering `yes`. The repository will be initialized with the passphrase you set in the `BORG_PASSPHRASE` +environment variable earlier. + +When using any of the `repokey` encryption methods the encryption key will be stored in the repository itself and not on +the client, so there is no further action required in this regard. If you decide to use a `keyfile` instead of +a `repokey` make sure you export the key and back it up separately. Check the [Exporting Keys](#exporting-keys) section +for how to retrieve the key. + +### Restart container + +Now that we finished configuring and initializing the repository restart the container to ensure it is in a defined +state: + +```shell +docker-compose restart borgmatic-mailcow +``` + +## Restoring from a backup + +Restoring a backup assumes you are starting off with a fresh installation of mailcow, and you currently do not have +any custom data in your maildir or your mailcow database. + +### Restore maildir + +!!! warning + Doing this will overwrite files in your maildir! Do not run this unless you actually intend to recover mail + files from a backup. + +!!! note "If you use SELinux in Enforcing mode" + If you are using mailcow on a host with SELinux in Enforcing mode you will have to temporarily disable it during + extraction of the archive as the mailcow setup labels the vmail volume as private, belonging to the dovecot container + exclusively. SELinux will (rightfully) prevent any other container, such as the borgmatic container, from writing to + this volume. + +Before running a restore you must make the vmail volume writeable in `docker-compose.override.yml` by removing +the `ro` flag from the volume. +Then you can use the following command to restore the maildir from a backup: + +```shell +docker-compose exec borgmatic-mailcow borgmatic extract --path mnt/source --archive latest +``` + +Alternatively you can specify any archive name from the list of archives (see +[Listing all available archives](#listing-all-available-archives)) + +### Restore MySQL + +!!! warning + Running this command will delete and recreate the mailcow database! Do not run this unless you actually + intend to recover the mailcow database from a backup. + +To restore the MySQL database from the latest archive use this command: + +```shell +docker-compose exec borgmatic-mailcow borgmatic restore --archive latest +``` + +Alternatively you can specify any archive name from the list of archives (see +[Listing all available archives](#listing-all-available-archives)) + +### After restoring + +After restoring you need to restart mailcow. If you disabled SELinux enforcing mode now would be a good time to +re-enable it. + +To restart mailcow use the follwing command: + +```shell +docker-compose down && docker-compose up -d +``` + +If you use SELinux this will also trigger the re-labeling of all files in your vmail volume. Be patient, as this may +take a while if you have lots of files. + +## Useful commands + +### Manual archiving run (with debugging output) + +```shell +docker-compose exec borgmatic-mailcow borgmatic -v 2 +``` + +### Listing all available archives + +```shell +docker-compose exec borgmatic-mailcow borgmatic list +``` + +### Break lock + +When borg is interrupted during an archiving run it will leave behind a stale lock that needs to be cleared before any +new operations can be performed: + +```shell +docker-compose exec borgmatic-mailcow borg break-lock user@rsync.net:mailcow +``` + +Where `user@rsync.net:mailcow` is the URI to your repository. + +Now would be a good time to do a manual archiving run to ensure it can be successfully performed. + +### Exporting keys + +When using any of the `keyfile` methods for encryption you **MUST** take care of backing up the key files yourself. The +key files are generated when you initialize the repository. The `repokey` methods store the key file within the +repository, so a manual backup isn't as essential. + +Note that in either case you also must have the passphrase to decrypt any archives. + +To fetch the keyfile run: + +```shell +docker-compose exec borgmatic-mailcow borg key export --paper user@rsync.net:mailcow +``` + +Where `user@rsync.net:mailcow` is the URI to your repository. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 328ea89d4..5ec91ef3e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -121,6 +121,7 @@ nav: - 'Windows Phone': 'client/client-windowsphone.md' - 'Manual configuration': 'client/client-manual.md' - 'Third party apps': + - 'Borgmatic Backup': 'third_party-borgmatic.md' - 'Exchange Hybrid Setup': 'third_party-exchange_onprem.md' - 'Gitea': 'third_party-gitea.md' - 'Gogs': 'third_party-gogs.md'