Encryption for Amazon’s Elastic File System (EFS)

Posted on

AWS now has the Elastic File System (EFS) available, but many enterprise features are missing of which encryption might be one of the most glaring. However, by chaining together some useful open source components, you can have an encrypted file system today using Amazon’s Elastic File System as the storage backend.

There are several solutions available, but one that meets many of the requirements we have seen for customers is EncFS. EncFS provides encryption of file names, directory names, and file contents. EncFS relies on each client performing a FUSE mount to present the decrypted file system to the operating system.

This provides the advantage that you can amortize the cost of encryption and decryption across the clients which mount the file system, and you do not need external virtual appliances to implement this solution. The drawback is that each of the clients must be able to access the root encryption key for the volume in order to mount the file system.

Once EncFS is installed on an NFS client, the process of creating and mounting a file system is pretty straightforward:

(more…)

Why EFS might not be production-ready (yet)

Posted on

Amazon’s Elastic File System has been the long awaited storage solution for shared file access in the public cloud with cross-zone availability and integration into the AWS console and API. The announcement last week of the general availability in three regions was exciting, but there is a single detail that could be a non-starter for many enterprises considering EFS for a production deployment.

There is not a missing feature or critical bug. Rather, there is something in the documentation (more precisely, something not in the documentation) that concerns me more than anything else.

(more…)

HOWTO: Generating pseudorandom test data without cheating

Posted on

The Problem

The dd command can be your friend when generating test data files for storage systems; however, when used incorrectly you can end up with heavily skewed results in either an overly optimistic or pessimistic manner.

/dev/zero

Many times /dev/zero is used as the source of test data. This is, of course, the perfectly compressible pattern so that if the transport layer or storage system take advantage of compression your results can be overly optimistic from what can be expected in production. Also, some systems are able to TRIM or otherwise discard zero-based patterns as a form of reverse thin-provisioning so that zeros are not an acceptable test pattern.

/dev/zero is fast, but it is not an acceptable pattern to test storage systems with:

[ec2-user@ip-172-31-1-118 ~]$ dd if=/dev/zero of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 0.0683285 s, 15.7 GB/s

/dev/urandom

The easy alternative of /dev/urandom is great at producing data that is not compressible; however, it is quite slow at generating that data as it favors cryptographic strength over speed. Using /dev/urandom as a live stream of test data most likely will result in failing to stress the storage system as it will be mostly idle waiting on the random data generator to produce data.

/dev/urandom produces an acceptable test pattern, but it is too slow to test storage systems with (/dev/zero is over 1,000 times faster):

[ec2-user@ip-172-31-1-118 ~]$ dd if=/dev/urandom of=/dev/null bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 85.2168 s, 12.6 MB/s

(more…)

Automatically mounting an Amazon Elastic File System (EFS) on boot

Posted on
Amazon announced the general availability of EFS in three regions yesterday. Here is a short demonstration for how you can automatically mount an Elastic File System within newly launched Linux instances over NFS using cloud-init user data.

User data template (see it on GitHub):

#!/bin/bash

##############################################################################
# USER CONFIGURATION
##############################################################################
EFS_FILE_SYSTEM_ID="fs-XXXXXXXX"
EFS_MOUNT_POINT="/mnt/efs"

# Make sure nfs-utils is installed already (CentOS / Amazon Linux / RHEL)
yum -y install nfs-utils

# Discover environment
INSTANCE_AZ="$(curl http://169.254.169.254/latest/meta-data/placement/availability-zone)"
INSTANCE_REGION="${INSTANCE_AZ:0:-1}"
EFS_DNS="${INSTANCE_AZ}.${EFS_FILE_SYSTEM_ID}.efs.${INSTANCE_REGION}.amazonaws.com"

# Determine if already in fstab before proceeding
if grep -Fq "${EFS_DNS}" /etc/fstab
then
	echo "Mount already exists"
else
	# Mount point must exist before mounting
	mkdir -p ${EFS_MOUNT_POINT}
	# Add to fstab
	echo "${EFS_DNS}:/ ${EFS_MOUNT_POINT} nfs nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab
	# Mount new file system
	mount ${EFS_MOUNT_POINT}
fi