SSL Ciphers on ELB

For some of the apps 'in our portfolio' we run PCI compliance scans. I won't name names, but we use McAfee for this particular one. These scans are great, but they have uncanny ability to always find something.

This time they did find something we are not particularly happy with. The problem they reported was that we had weak SSL ciphers enabled on our ELBs.

Of we course we are not alone with this problem, and AWS recently added support for this particular problem. They were happy to announce the support of SSL ciphers a few weeks ago.

But, as is often the case with AWS, how to actually make this work is somewhere inbetween Developer Guid, API Specification, Jeff's wise words and people like us struggling and helping each other on the forum.

We are currently not in production yet, but we are testing if this is the way to go. If you have some sort of shell with the cmdline tools set up, you can follow along with this on your own ELBs.

Be default we all of the following enabled

name=Protocol-SSLv2,value=false

name=Protocol-TLSv1,value=true

name=Protocol-SSLv3,value=true

name=DHE-RSA-AES256-SHA,value=true

name=DHE-DSS-AES256-SHA,value=true

name=DHE-RSA-CAMELLIA256-SHA,value=true

name=DHE-DSS-CAMELLIA256-SHA,value=true

name=ADH-AES256-SHA,value=false

name=ADH-CAMELLIA256-SHA,value=false

name=AES256-SHA,value=true

name=CAMELLIA256-SHA,value=true

name=PSK-AES256-CBC-SHA,value=true

name=EDH-RSA-DES-CBC3-SHA,value=true

name=EDH-DSS-DES-CBC3-SHA,value=true

name=ADH-DES-CBC3-SHA,value=false

name=DES-CBC3-SHA,value=true

name=DES-CBC3-MD5,value=false

name=PSK-3DES-EDE-CBC-SHA,value=true

name=KRB5-DES-CBC3-SHA,value=true

name=KRB5-DES-CBC3-MD5,value=true

name=DHE-RSA-AES128-SHA,value=true

name=DHE-DSS-AES128-SHA,value=true

name=DHE-RSA-SEED-SHA,value=true

name=DHE-DSS-SEED-SHA,value=true

name=DHE-RSA-CAMELLIA128-SHA,value=true

name=DHE-DSS-CAMELLIA128-SHA,value=true

name=ADH-AES128-SHA,value=false

name=ADH-SEED-SHA,value=false

name=ADH-CAMELLIA128-SHA,value=false

name=AES128-SHA,value=true

name=SEED-SHA,value=true

name=CAMELLIA128-SHA,value=true

name=RC2-CBC-MD5,value=false

name=PSK-AES128-CBC-SHA,value=true

name=ADH-RC4-MD5,value=false

name=IDEA-CBC-SHA,value=true

name=RC4-SHA,value=true

name=RC4-MD5,value=true

name=PSK-RC4-SHA,value=true

name=KRB5-RC4-SHA,value=true

name=KRB5-RC4-MD5,value=true

name=EDH-RSA-DES-CBC-SHA,value=true

name=EDH-DSS-DES-CBC-SHA,value=true

name=ADH-DES-CBC-SHA,value=false

name=DES-CBC-SHA,value=true

name=DES-CBC-MD5,value=false

name=KRB5-DES-CBC-SHA,value=true

name=KRB5-DES-CBC-MD5,value=true

name=EXP-EDH-RSA-DES-CBC-SHA,value=true

name=EXP-EDH-DSS-DES-CBC-SHA,value=true

name=EXP-ADH-DES-CBC-SHA,value=false

name=EXP-DES-CBC-SHA,value=true

name=EXP-RC2-CBC-MD5,value=true

name=EXP-KRB5-RC2-CBC-SHA,value=true

name=EXP-KRB5-DES-CBC-SHA,value=true

name=EXP-KRB5-RC2-CBC-MD5,value=true

name=EXP-KRB5-DES-CBC-MD5,value=true

name=EXP-ADH-RC4-MD5,value=false

name=EXP-RC4-MD5,value=true

name=EXP-KRB5-RC4-SHA,value=true

name=EXP-KRB5-RC4-MD5,value=true

We don't want that, we want to be more restrictive lik ELBSample-ELBDefaultNegotiationPolicy, one of the defaults AWS offers when creating an ELB. The totally shitty thing is that these policies are ELB specific, and not account wide.

name=Protocol-SSLv2,value=false

name=EDH-DSS-DES-CBC3-SHA,value=false

name=DHE-RSA-CAMELLIA128-SHA,value=false

name=DES-CBC-MD5,value=false

name=KRB5-RC4-SHA,value=false

name=ADH-CAMELLIA128-SHA,value=false

name=EXP-KRB5-RC4-MD5,value=false

name=ADH-RC4-MD5,value=false

name=PSK-RC4-SHA,value=false

name=PSK-AES128-CBC-SHA,value=false

name=EXP-EDH-RSA-DES-CBC-SHA,value=false

name=CAMELLIA128-SHA,value=false

name=DHE-DSS-AES128-SHA,value=false

name=EDH-RSA-DES-CBC-SHA,value=false

name=DHE-RSA-SEED-SHA,value=false

name=KRB5-DES-CBC-MD5,value=false

name=DHE-RSA-CAMELLIA256-SHA,value=false

name=ADH-DES-CBC3-SHA,value=false

name=DES-CBC3-MD5,value=false

name=EXP-KRB5-RC2-CBC-MD5,value=false

name=EDH-DSS-DES-CBC-SHA,value=false

name=KRB5-DES-CBC-SHA,value=false

name=PSK-AES256-CBC-SHA,value=false

name=ADH-AES256-SHA,value=false

name=KRB5-DES-CBC3-SHA,value=false

name=AES128-SHA,value=true

name=DHE-DSS-SEED-SHA,value=false

name=ADH-CAMELLIA256-SHA,value=false

name=EXP-KRB5-RC4-SHA,value=false

name=EDH-RSA-DES-CBC3-SHA,value=false

name=EXP-KRB5-DES-CBC-MD5,value=false

name=Protocol-TLSv1,value=true

name=PSK-3DES-EDE-CBC-SHA,value=false

name=SEED-SHA,value=false

name=DHE-DSS-CAMELLIA256-SHA,value=false

name=IDEA-CBC-SHA,value=false

name=RC2-CBC-MD5,value=false

name=KRB5-RC4-MD5,value=false

name=ADH-AES128-SHA,value=false

name=RC4-SHA,value=true

name=AES256-SHA,value=true

name=Protocol-SSLv3,value=true

name=EXP-DES-CBC-SHA,value=false

name=DES-CBC3-SHA,value=true

name=DHE-RSA-AES128-SHA,value=false

name=EXP-EDH-DSS-DES-CBC-SHA,value=false

name=EXP-KRB5-RC2-CBC-SHA,value=false

name=DHE-RSA-AES256-SHA,value=false

name=KRB5-DES-CBC3-MD5,value=false

name=RC4-MD5,value=true

name=EXP-RC2-CBC-MD5,value=false

name=DES-CBC-SHA,value=false

name=EXP-ADH-RC4-MD5,value=false

name=EXP-RC4-MD5,value=false

name=ADH-DES-CBC-SHA,value=false

name=CAMELLIA256-SHA,value=false

name=DHE-DSS-CAMELLIA128-SHA,value=false

name=EXP-KRB5-DES-CBC-SHA,value=false

name=EXP-ADH-DES-CBC-SHA,value=false

name=DHE-DSS-AES256-SHA,value=false

name=ADH-SEED-SHA,value=false

We saved this file in strict-ciphers.txt and to create our policy for the specific ELB

$ awk 'BEGIN{

        cmd = "elb-create-lb-policy staging-9apps-net --policy-name Strict-ELBNegotiationPolicy --policy-type SSLNegotiationPolicyType"

    }{

        cmd = cmd " --attribute \x22"$1"\x22"

    }END{system( cmd)}' strict-ciphers.txt

$ awk 'BEGIN{

        cmd = "elb-create-lb-policy staging-9apps-org --policy-name Strict-ELBNegotiationPolicy --policy-type SSLNegotiationPolicyType"

    }{

        cmd = cmd " --attribute \x22"$1"\x22"

    }END{system( cmd)}' strict-ciphers.txt

Enabling it for the staging-9apps-net and staging-9apps-org ELBs

$ elb-set-lb-policies-of-listener staging-9apps-net \

    --lb-port 443 \

    --policy-names Strict-ELBNegotiationPolicy

$ elb-set-lb-policies-of-listener staging-9apps-org \

    --lb-port 443 \

    --policy-names Strict-ELBNegotiationPolicy

We hope this helps everyone along a bit, with getting the ciphers they want.

Tomcat (JVM) in CloudWatch

Not all of us have Beanstalk. Too bad AWS keeps the goodies like this and ElastiCache to themselves in US-East. But we like Beanstalk quite a lot, so much we wrote about it, beginning of this year.

But, we are also running Java things outside of Beanstalk. And we do like the various CloudWatch metrics Beanstalk provides on its instances. Wouldn't it be nice to have at least that?

Luckily most Java installs come with jstat, a utility to query memory usage on a JVM. With some simple scripting we now add various JVM memory statistics to CloudWatch. (We used PHP, but any other environment/language with an AWS SDK will do.)

CloudWatch

We wanted a quick, but solid scripted solution. So we divided several of the tasks over different scripts. The PHP script adds different metrics to CloudWatch. We could optimize the solution by having the PHP script adding all metrics in one 'session'. But this works fine.

<?php
    require_once 'AWSSDKforPHP/sdk.class.php';

    define('AWS_KEY', '');
    define('AWS_SECRET_KEY', '');
    define('AWS_ACCOUNT_ID', '');

    $jstat = array(
        'S0' => array('SurvivorSpace0Utilization','Percent'),
        'S1' => array('SurvivorSpace1Utilization','Percent'),
        'E'  => array('EdenSpaceUtilization','Percent'),
        'O'  => array('OldSpaceUtilization','Percent'),
        'P'  => array('PermanentSpaceUtilization','Percent'),
        'YGC'  => array('YoungGenerationGCEvents','Count'),
        'YGCT'  => array('YoungGenerationGCTime','Seconds'),
        'FGC'  => array('FullGenerationGCEvents','Count'),
        'FGCT'  => array('FullGenerationGCTime','Seconds'),
        'GCT'  => array('TotalGCTime','Seconds')
    );

    $cw = new AmazonCloudWatch();
    $cw->set_region(AmazonCloudWatch::REGION_EU_W1);

    $dimensions = array(
        array( 'Name' => 'InstanceIdentifier',
        'Value' => $argv[1])
    );
    $timestamp = date( DATE_RFC822);

    $response = $cw->put_metric_data('9Apps/Smartfox', array(
         array(
             'MetricName' => $jstat[$argv[2]][0],
             'Dimensions' => $dimensions,
             'Value' => $argv[3],
             'Timestamp' => $timestamp,
             'Unit' => $jstat[$argv[2]][1],
         )));
    if( 200 != $response->status) {
        print_r( $response);
    }
?>

jstat

You can get all sorts of memory information from jstat, in all shapes and forms. We are mostly interested in Utilization of different 'spaces'. Utilization fits nicely on CloudWatch's Percentage metric, we don't have to do any calculation. This bash script get the jstat information, parses it and fires off the PHP script. It is basically a wrapper for use in crontab.

#!/bin/bash

/usr/bin/jstat -gcutil -t `pgrep java` 1 1 | \
    /bin/sed '/^Timestamp/d' | \
    instance_id=`/usr/bin/curl --retry 3 -s -S -f http://169.254.169.254/latest/meta-data/instance-id` \
awk '{
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " S0 " $2);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " S1 " $3);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " E " $4);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " O " $5);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " P " $6);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " YGC " $7);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " YGCT " $8);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " FGC " $9);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " FGCT " $10);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " GCT " $11);
}'

Crontab

We want to run this every minute. We could get and aggregate several measurements per minute, but this is fine for our purpose.

* * * * *     /root/collect-jvm-stats.sh > /dev/null 2>&1

Counting non InnoDB with CloudWatch

With WordPress we run the risk our tables are being 'altered' to MyISAM. We suspect plugins, with plugin developers dis-respecting our choices. It appears to be innocent, harmless. But we loose the ability to 'Point in Time Restore' of RDS. So, potentially devastating.

We created a small bash script to count the nr of non InnoDB tables.

#!/bin/bash

E_BADARGS=65

if [ ! -n "$1" ]
then
  echo "Usage: `basename $0` hostname username password"
  exit $E_BADARGS
fi

hostname=$1
identifier=${hostname/%.*}
username=$2
password=$3

echo "getting the databases from ${hostname}"
databases=`mysql -N -u ${username} -p${password} -h ${hostname} -e 'show databases where \`Database\` not in ("mysql", "innodb", "information_schema", "performance_schema")'`

total=0

for database in ${databases}
do
    echo "getting the nr of tainted tables from ${database}"
    tainted_tables=`mysql -N -u ${username} -p${password} -h ${hostname} -e "show table status from ${database} where Engine <> \"InnoDB\"" | wc -l`
    total=$((total + tainted_tables))
done

echo "we have ${total} tainted tables"
php put-status.php ${identifier} ${total}

We report the identifier and nr to CloudWatch with a PHP script.

<?php
require_once 'AWSSDKforPHP/sdk.class.php';

define('AWS_KEY', '');
define('AWS_SECRET_KEY', '');
define('AWS_ACCOUNT_ID', '');

$cw = new AmazonCloudWatch();

$dimensions = array(
    array( 'Name' => 'DBInstanceIdentifier',
        'Value' => $argv[1])
);
$timestamp = date( DATE_RFC822);

$response = $cw->put_metric_data('9Apps/RDS', array(
    array(
        'MetricName' => 'NonInnoDBTables',
        'Dimensions' => $dimensions,
        'Value' => $argv[2],
        'Timestamp' => $timestamp,
        'Unit' => 'Count'
    ),
));
?>

Which we put to use with simple cron job by adding something like this

# check and report non InnoDB tables
* * * * * /root/innodb.sh [hostname] [username] [password] > /dev/null 2>&1

MongoDB on AWS (RDS-style)

MongoDB is drawing crowds, lately. Some even dare to call it the new MySQL. We didn't work with it yet, although we investigated its use on GeoSpatial systems already a while ago.

Usabilla, our latest partner, and one of Amsterdam's hottest startups wants one. Apart from being fun, one of the reasons they 'want one' is that it promises to help them fight the monkey that wrecked serious havoc on my wedding day. So, we have to build a MongoDB 'thing' on Amazon AWS giving us

• high availability, and
• scalability

MongoDB will help Usabilla deal with huge key/value style datasets when they will start collecting feedback from live events. It is also a good fit because the team can continue to practice their Javascript skills.

If you do not have the patience to continue to read, but want to get down to it immediately, go get our stuff at github. We installed everything on Ubuntu, you can see our full install script in the repository. This is how to easily install the latest MongoDB

echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" \
        >> /etc/apt/sources.list    
apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
apt-get update && apt-get install mongodb-10gen

MongoDB

We want to run a MongoDB replica set. In short it gives you a distributed database, with one 'primary' and hopefully one or more 'secondaries' that can take over when the system decides that it is necessary. One obvious example of electing a new primary is failure of the old one. Another obvious one is for maintenance, at least when it is easy to force these reconfigurations.

The smallest MongoDB replica set that is recommended has one primary, one secondary, and one arbiter. The arbiter doesn't do much, but helps in electing new primaries when necessary. We don't want to rely on one arbiter, so we'll use two. (This is not that expensive, because we can run them on t1.micro instances.)

The idea is to build two different types of instances

1. regular replica set member (primary or secondary), holding data
2. arbiters, without data

Once the replica set is initiated, the members have to be able to (de)register itself. A member will join the replica set automatically. The arbiter will do nothing but vote. A new member will either do a full sync, or an incremental sync depending on the availability of backups. (The MongoDB data and log is stored on a separate EBS volume.)

Addressability (Route53)

Of course we want to be able to talk to this replica set. Most of the difficult work of talking to a set of instances has already been taken care of by MongoDB. We only need to find one available member, doesn't matter which.

Each regular member periodically checks if it is primary, and if so instructs Route53 to make mongodb.usabilla.com point to its own public DNS, if it didn't already. This way, the entry point of the replica set will most be valid most of the time. Only when the primary dies a sudden death, will it take around a minute or two for the entry point to be valid again.

Initiation (manual)

Our MongoDB replica set has two AutoScaling groups, one for regular set member and the other for arbiters. We need to start carefully, because we have bootstrap the replica set. We launch the first two regular instances. One of these instances will be the primary by running

mongo --eval "rs.initiate()"

If the initiation is done, this instance will tell Route53 to point mongodb.usabilla.com to its own public DNS name. The moment the DNS is updated, and the domain resolves properly we'll tell the second instance to join the cluster (note that we have to do this manually because we launched 2 instances simultaneously)

mongo mongodb.usabilla.com --eval "rs.add(\"`hostname`:27017\")"

Backups

Now that we have a replica set behind mongodb.usabilla.com, we have to talk about backups. The primary already has a responsibility, it takes care of the addressability of the set. The backups will be delegated to the secondaries, which is the logical thing to do.

We normally use snapshots as backups, administering the expiration with SimpleDB. This is perfect for disaster recovery (DR) but we also need to know the most recent snapshot. We added a timestamp to the item, and can now query for the most recent snapshot.

If a regular member launches, it first checks the availability of snapshots. If there are snapshots (the replica set exists) it creates a new volume from that snapshot. And tells the replica set to add it to the set.

Conclusion

There are a couple of small things we would like to do, namely

• automatic removal of stale members
• CloudWatch monitoring of member
• and CloudWatch monitoring of the replica set

But, in the meantime, we have probably the coolest MongoDB replica set in Amazon AWS!! It is very resilient, and we'll survive Chaos Monkeys easily, even those that visit on nice spring days. Of course we use Availability Zones, for extra durability.

An added bonus is that it is extremely easy to upgrade the entire system. If we want to do an upgrade, of compatible MongoDB versions, we only have to change the AutoScaling, terminate the instances one by one. Our MongoDB RDS will take care of the rest.

as-create-launch-config mongodb-mongodb-usabilla-com-lc-7 \
        --image-id ami-fd915694 \
        --instance-type m1.large \
        --group mongodb
as-update-auto-scaling-group mongodb-mongodb-usabilla-com-as-group-1 \
        --launch-configuration mongodb-mongodb-usabilla-com-lc-7 \
        --min-size 2 \
        --max-size 2

as-terminate-instance-in-auto-scaling-group i-55f03d34 -D
as-terminate-instance-in-auto-scaling-group i-e16aaa80 -D

Decaf for iPhone

Decaf for iPhoneI am very proud to announce Decaf for iPhone. It took us much longer than anticipated, but quality of the app was most important. You'll use it to manage your EC2 account, which is not something taken lightly.

With this app you can manage all your EC2 assets. You can do everything you are used to in the AWS Console, only we added our own way of navigating. In Decaf you can easily go from AMI to Image to Security Group to attached Volumes and related Snapshots. We have gone great lengths to offer you a navigation that makes use of the screen that you have.

Because mobile data networks are still not 100% reliable, and the AWS APIs take their time answering, we persisted the data on the phone. You can navigate your EC2 account with great ease. Many of the things you have in your account do not change that much. And if you do a quick refresh will get the information you interested in immediately.

Go check it out!! You can find it here, or by using the qrcode above.

We tried to stay close to our original idea as possible, but some things we do on Android are not possible. For example monitoring using background processes and the widget for CloudWatch information. We do believe iOS will support this technology in the (near) future, and we'll work hard to add these features.