April 10, 2019
Building a Backup Appliance Resilient to Generic Ransomware Threats
The following article describes our attempt to build a dedicated backup appliance which has a particular set of benefits over generic NAS boxes in terms of data protection from unauthorized access, modification and removal. Using a modern NAS box as a backup appliance gives us access to great functionality like storage scale-ability, data redundancy and resiliency (RAID volumes), data accessibility (CIFS/NFS/AFP shares) and so on. With all these benefits in mind, we must admit that the bare fact of data accessibility adds additional risks of “unwanted” access to data (weak passwords, password reuse, credentials theft by malware, or even just malware accessing a legitimately open network share). Regardless of the initial attack vector, malicious actors, etc., we aim to remove well-known remote access methods from backup appliances while having better control over OS security updates, vulnerabilities tracking and backup storage encryption at rest.
Overall we have multiple copies of our backups on-site and in the cloud, so we’ve sought to have something like one week of “most actual” mission critical backups always at hand with an additional layer of security. Let’s say we have 60 GB of such data (with 20 GB being additionally encrypted, so not subject to compression), with an expected compression rate of 50% and 10% of daily changes in this data set. In the model of incremental backup with periodic fulls, 1 week of these such backups (1 full + 6 increments per week) would consume (40 GB * 0.5 + 20 GB) * (1 + 6 * 0.1) = 64 GB. Having a full synthetic backup each week would require an additional (40 GB * 0.5 + 20 GB) = 40 GB of space, giving a total requirement of 104 GB. In fact, due to retention policy implementation, it will be 2 full weeks of backups before the oldest one can be removed, giving us 128 GB requirement.
Selection of a single-board computer to implement the project was simple for us – one of our developers had Raspberry PI 3 Model B+ serving as a home media center that we could lend on the initial phase. The board has enough CPU power to do most modern tasks including encryption. However, the board design with both storage and network sitting on the same USB 2.0 limits overall data processing bandwidth to theoretical 480 Mbit/s. Having the same bus bandwidth shared for incoming data and writing it to storage would leave us with only half bandwidth so any USB thumb stick with 30 MB/s writer speed and decent capacity could serve us as storage.
In turn, for our project, we went with:
- Single-board computer ARMv7/ARMv8 (Raspberry Pi 3 Model B+)
- 32/64-bit Linux based OS with minimal set of services and software
- NAKIVO Backup & Replication
- 16+ GB microSD card for OS & software install
- 128+ GB of storage for backups
- 1.5-2A 5V power supply
Download OS image and prepare microSD card
There are multiple options of OS selection for the purpose of building a backup appliance on top of Raspberry Pi or similar boards. For sake of simplicity in this article, Raspbian Stretch Lite maintained by Raspberry Pi Foundation will be used (note that image is 32-bit ARMv7 at the moment of this writing while hardware can run native 64-bit ARMv8).
First, we download the OS image archive (https://downloads.raspberrypi.org/raspbian_lite_latest):
$ wget --no-check-certificate -O image.zip
Saving to: `image.zip'
2018-11-01 17:41:38 (908 KB/s) - `image.zip' saved [368317240/368317240]
Next it is recommended that we verify the hash of downloaded image file vs published for Raspbian Stretch Lite (https://www.raspberrypi.org/downloads/raspbian/):
$ sha256sum image.zip
Extract the image from archive and write it to microSD card (assuming block device for microSD card to be /dev/mmcblk0, it may be different in your case so consult tools like lsblk to find block device of appropriate size):
$ unzip image.zip
$ sudo dd if=*raspbian-stretch-lite.img of=/dev/mmcblk0 bs=4MiB
445+0 records in
445+0 records out
1866465280 bytes (1,9 GB, 1,7 GiB) copied, 103,985 s, 17,9 MB/s
$ sudo partprobe
Up to this point, it is easy to do the same stuff from Windows host using browser and various tools to writer image file to microSD card (like Rufus, https://rufus.ie/en_IE.html). The reminder of this section is possible only from Linux host, but you should be able to do most of tweaks once you boot and login into OS.
Let’s harden our deployment by renaming the default pi account with the custom user name marabu (don’t select already-used names like backup, that default password will remain intact), enforce sudo to require a password for renamed user and tighten SSH login options to disable password-based login in favor of key-based authentication (assuming you have private and public key pair generated with something like: ssh-keygen -b 4096 -f appliance.key):
$ sudo su
$ mount /dev/mmcblk0p2 /mnt
$ sed -i ‘/^pi:/s/pi/marabu/’ /mnt/etc/shadow
$ sed -i ‘/^pi:/s/pi/marabu/g’ /mnt/etc/passwd
$ sed -i -e '/^pi:/s/pi/marabu/' -e '/[:,]pi$/s/pi/marabu/' /mnt/etc/group
$ rm /mnt/etc/sudoers.d/010_pi-nopasswd
$ mv /mnt/home/pi /mnt/home/marabu
$ echo ‘PermitRootLogin no’ >/mnt/etc/ssh/sshd_config
$ echo ‘PasswordAuthentication no’ >>/mnt/etc/ssh/sshd_config
$ echo ‘ChallengeResponseAuthentication no’ >>/mnt/etc/ssh/sshd_config
$ echo ‘PubkeyAuthentication yes’ >>/mnt/etc/ssh/sshd_config
$ echo ‘AuthorizedKeysFile /etc/ssh/authorized_keys’ >>/mnt/etc/ssh/sshd_config
$ cat appliance.key.pub >/mnt/etc/ssh/authorized_keys
$ chown 1000:0 /mnt/etc/ssh/authorized_keys
$ chmod 440 /mnt/etc/ssh/authorized_keys
$ umount /mnt
You can also change the default SSH port with additional line, like:
$ echo ‘Port 6809’ >>/mnt/etc/ssh/sshd_config
As an optional step, if you don’t have display and keyboard, you can guess or look up IP address that board will receive upon boot with DHCP configuration you can trigger SSH server to be started on first boot:
$ sudo mount /dev/mmcblk0p1 /mnt
$ sudo touch /mnt/ssh
$ sudo umount /mnt
Another optional step would be to disable WiFi and/or Bluetooth capabilities on Raspberry Pi 3 specifically, if you don’t plan to use them (for reduced power consumption, etc):
$ sudo su
$ mount /dev/mmcblk0p1 /mnt
$ echo ‘dtoverlay=pi3-disable-wifi’ >>/mnt/config.txt
$ echo ‘dtoverlay=pi3-disable-bt’ >>/mnt/config.txt
$ umount /mnt
First boot and initial configuration
Now let’s put the microSD card into Raspberry Pi board, plug Ethernet cable and apply power (optionally connect HDMI monitor and USB keyboard). If everything was done correctly on previous step system should boot, receive network settings over DHCP and enable SSH login for user marabu with private key. Otherwise (or if you have skipped user renaming and SSH configuration), you should see OS login prompt with default user pi and password raspberry.
First things first, let’s change default password for renamed user marabu (default password is raspberry, be sure to remember new password):
Changing password for marabu.
(current) UNIX password:
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Now let’s check our IP address obtained via DHCP (you may need to do manual configuration of static IP address at this point if you don’t have active DHCP server in your network):
$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.30.30.101 netmask 255.255.255.0 broadcast 10.30.30.255
Raspberry Pi does not have real-time clock hardware and requires the correct time to be set manually on each boot or synchronized from the network. So let’s set the correct time zone for the system via raspi-config → “Localization Options” → “Change timezone”. After that you can verify your date/time settings with timedatectl tool and verify that network synchronization is enabled.
Updating and installing additional software
It is time to start updating installed software, configuring unattended security upgrades and getting additional packages that will be needed on backup appliance:
$ sudo apt-get update
$ sudo apt-get -y dist-upgrade
$ sudo apt-get -y install unattended-upgrades
$ sudo apt-get -y install cryptsetup open-iscsi parted ntfs-3g lvm2
This will install the latest patches and packages needed to create encrypted storage for backups (cryptsetup) and tools most commonly used to recover files from Linux/Windows VM backups (parted, open-iscsi, ntfs-3g and lvm2). Default configuration for unattended upgrades would install only security updates; you can review and tweak desired sources in /etc/apt/apt.conf.d/50unattended-upgrades.
The final preparation step is to disable any “extra” networks and plug-and-play services that are not needed for our deployment:
$ sudo systemctl stop avahi-daemon.socket
$ sudo systemctl disable avahi-daemon.socket
$ sudo systemctl stop avahi-daemon.service
$ sudo systemctl disable avahi-daemon.service
$ sudo systemctl stop lvm2-lvmetad.socket
$ sudo systemctl disable lvm2-lvmetad.socket
$ sudo systemctl stop lvm2-lvmetad.service
$ sudo systemctl disable lvm2-lvmetad.service
$ sudo sed -i '/use_lvmetad = /s/1/0/' /etc/lvm/lvm.conf
At this point we should have a system with a minimal set of exposed network ports: SSH and DHCP client (you can use netstat -antu to verify this).
Prepare encrypted backup storage
Once again, we are not going to focus on redundancy and resiliency of storage for our backup, just as we aren’t zeroing in on attempts to minimize backup/recovery time. Our backup appliance is designed to be a secure node with short time retention of backups. We start with a USB 128 GB thumb drive and expect it to be able to hold 1-2 weeks of mission critical backups. An external USB HDD/SSD or even Pi extension board that allows an SSD attachment (like http://www.raspberrypiwiki.com/index.php/X820) would not be fully utilized due to platform limitations (most of the content ends up on the same USB 2.0 bus with 480 Mbit/s theoretical speed limit).
With all this said, let’s get our USB thumb drive connected to Raspberry Pi (if not already done) and configured (we assume the device to appear on /dev/sda, but you can check dmesg or lsblk output to identify it). First we’ll do a secure wipe of the device with random data:
$ sudo dd if=/dev/urandom of=/dev/sda bs=4MiB
$ sudo partprobe
Next let’s do a full disk encryption with the cryptsetup tool using Linux Unified Key Setup (https://guardianproject.info/code/luks/), which should allow us to have an enterprise grade security setup with multiple access keys and means to revoke/change them as needed. First, we set up the primary key (for slot 0) that is not expected to be used/known to anyone except the Master Admin or may be even the CEO, and add a secondary key to slot 1 to be used by backup software.
$ sudo cryptsetup luksFormat /dev/sda
This will overwrite data on /dev/sda irrevocably.
Are you sure? (Type uppercase yes): YES
$ sudo cryptsetup luksAddKey /dev/sda
Enter any existing passphrase:
Enter new passphrase for key slot:
It is recommended that you back up the LUKS header from the device and have it securely stored:
$ sudo cryptsetup luksHeaderBackup /dev/sda --header-backup-file appliance-luks.hdr
Finally, let’s open the encrypted storage and create an ext4 file system on it with all space allocated to regular users (as our backup software will run under dedicated credentials, not the root account). You can select a different file system or settings for your deployment.
$ sudo cryptsetup luksOpen /dev/sda crypt
Enter passphrase for /dev/sda:
$ sudo mkfs.ext4 -m 0 /dev/mapper/crypt
$ sudo cryptsetup luksClose crypt
Download and install NAKIVO Backup & Replication
At this point we have secure box with minimal set of services, secure storage and ready to be turned into dedicated backup appliance running NAKIVO Backup & Replication software.
$ wget -O install.sh https://d96i82q710b04.cloudfront.net/res/product/NAKIVO_Backup_Replication_v8.5.1_NAS_arm_v7.sh
$ sudo sh ./install.sh -C --eula-accept -i /opt/nakivo
NAKIVO Backup & Replication installed successfully.
$ rm install.sh
This should be it for now on Raspberry Pi side, and we can shift to any browser accessing https://YOUR_RASPBERRY_IP:4443 for backup software configuration. For your first login, you will need to accept a self-signed certificate in your browser and configure user and password:
We’ll skip first two steps of welcome wizard where you discover your environment to protect and confirm you have at least one Transporter. Instead, let’s jump to the backup repository configuration step to create the encrypted repository on /dev/sda that we have prepared. We want to create a repository of Incremental with full backups type to avoid periodic repository maintenance and have backups independent from each other (with the cost of additional storage used).
A few final considerations
We are mostly done with our backup appliance, but there are a few other things that we found worth mentioning for consideration. Our system still has SSH configured and running that we use to do a clean shutdown of the system after backup runs. But there are ways to get rid even of SSH:
- Hardware button on GPIO to shut down the system gracefully
- Post-job script that schedules a system shutdown
- Modified /opt/nakivo/transporter/scripts/encrytped_unmount.sh (that is invoked when we are done with the repository) that schedules a system shutdown
We chose the first option that looked like this at the development stage:
With the following script to serve it (stored as /opt/nakivo/button.sh with marabu:root ownership and 770 permissions):
echo "20" > /sys/class/gpio/export
echo "out" > /sys/class/gpio/gpio20/direction
echo "1" > /sys/class/gpio/gpio20/value
echo "21" > /sys/class/gpio/export
echo "in" > /sys/class/gpio/gpio21/direction
while ( true ); do
if [ "x$VAL" = "x1" ]; then
systemctl stop nkv-dirsvc
systemctl stop nkv-bhsvc
shutdown -h now
Auto-load of script is done with the line ‘/opt/nakivo/button.sh &’ in /etc/rc.local just before ‘exit 0’. SSH service is stopped and disabled with commands:
$ sudo systemctl disable ssh
$ sudo systemctl stop ssh
SSH service can be re-enabled at any moment if needed by creating an empty file ssh on boot (first) partition of microSD card before powering up the board.
Instead of a conclusion
For the project we decided to go with full NAKIVO Backup & Replicaion installation on the Raspberry Pi, that is fully functional WebUI, management stuff and Transporter (component responsible for backup repository handling and data manipulations). If it wasn’t 480 Mbit/s general board bandwidth limit, we would probably install Transporter only deployment to squeeze maximum performance from the system for data manipulation tasks while having WebUI and other stuff running in a virtual appliance or the likes of one (NAKIVO Backup & Replicaion allows such deployment options). The same bandwidth limitation removes most storage options from the table, so if you choose to go with some hardware, pay attention to its design considerations and limitations.
Overall, we have managed to build backup appliance within the reasonable budget of $150 with 128 GB USB thumb drive storage for compactness. Though, at the price of such a USB thumb drive, you can get USB HDD with 1+ TB capacity, so we encourage you to weigh out your design and options before implementation.