July 11, 2016
File Recovery from Encrypted Volumes in AWS EC2
Migration from the production environment to AWS EC2 raises a lot of questions, ranging from pricing to services availability. One of the critical aspects of migration for us was deployment security and, to a big extent, confidentiality of the critical data that we move to AWS.
Once moved outside of our premises, the data must be encrypted in transit and at rest. An analysis of AWS EC2 encrypted volumes led us to the conclusion that any encrypted volume and its snapshot can be decrypted in the EC2 environment as long as the decryption key can be accessed by the account used to log in. So, if the account is compromised, an attacker would be able to access data on encrypted volume(s), and even detached volumes and volume snapshots. This was truly a “show stopper” for us.
Full Disk (Partition) Encryption Within AWS EC2 Instance
After looking at our options, we have selected the approach with the full disk (partition) encryption within AWS EC2 instance relying on Linux encryption techniques, namely cryptsetup facility with LUKS key management. This gives us the following benefits:
- More control over who can decrypt the volumes (even though you would need to unlock them manually on instance maintenance and reboot)
- The ability to revoke compromised and retired passwords
- Rebooted instance, detached volume, and any volume backup would not be decrypted
- Higher chance that even Amazon itself would not be able to eavesdrop on data at rest
We cannot encrypt everything including the boot volume in this manner since the lack of console access would not allow entering a password at boot time. But this approach is well suited for separate disks/partitions with critical data. As a side note, there is still a chance to decrypt the data inside a running EC2 instance, but that is another story of “physical” instance protection.
So what about file recovery in such a configuration? We are using our own product, NAKIVO Backup & Replication, to back up our environment at AWS and we have the file recovery feature, that enables restoring files from “cold backups”. However, file recovery from encrypted disks must fail (and actually does) since the data is encrypted and no “default” passwords/keys are available.
As in many other cases – if something cannot be done directly in the product interface, it is still possible for an administrator to solve the task at hand. Let's see how:
- Open a file recovery session in NAKIVO Backup & Replication for the backup and recovery point of interest
- Log in via SSH into the Transporter that is assigned to the repository
- Identify the attached encrypted disk/partitions to recover from
- Open the encrypted disk/partition by providing the right passwords
- Mount the decrypted volume
- Find and recover files you need
- Unmount the decrypted volume
- Close the encrypted disk/partition
- Close the file recovery session in NAKIVO Backup & Replication
File Recovery Steps
So let’s quickly walk through the steps with a bit more detail and open a backup recovery point for file recovery in NAKIVO Backup & Replication:
Log in via SSH (either with an ssh client or the putty utility, depending on your preferences).
Check that disks from backup are mounted:
>iscsiadm -m session
Identify disk and partition within the system view (via dmesg or lsblk):
>dmesg | tail -n 50
Where you should see a disk label (like sdc) and its partitions (like sdc1, sdc2, …) with sizes. Additionally, the parted tool can be used for additional verification that the right disk/partition is selected:
>parted /dev/sdc print
Once you know which disk/partition you need to decrypt and mount, run the following command and enter a valid password when prompted (this should create a new decrypted device entry point under /dev/mapper with the name that you specified as a third argument to the command):
>cryptsetup luskOpen /dev/sdc2 recover
Create (or use an existing) folder and mount the decrypted drive into it:
>mkdir 1; mount /dev/mapper/recover 1
Find and grab the files you need!
When you’re done, do the proper teardown:
> umount /dev/mapper/recover
> cryptsetup luskClose recover
Now you can close the file recovery wizard in NAKIVO Backup & Replication, so the software would do its own cleanup steps.
The same technique can be applied to other encryption types and features of file systems not directly available from NAKIVO Backup & Replication (like BitLocker, TrueCrypt, etc.). Just remember that the Transporter OS should be capable of supporting the feature and be brave to try.