Tomcat (JVM) in CloudWatch

Not all of us have Beanstalk. Too bad AWS keeps the goodies like this and ElastiCache to themselves in US-East. But we like Beanstalk quite a lot, so much we wrote about it, beginning of this year.

But, we are also running Java things outside of Beanstalk. And we do like the various CloudWatch metrics Beanstalk provides on its instances. Wouldn't it be nice to have at least that?

Luckily most Java installs come with jstat, a utility to query memory usage on a JVM. With some simple scripting we now add various JVM memory statistics to CloudWatch. (We used PHP, but any other environment/language with an AWS SDK will do.)

CloudWatch

We wanted a quick, but solid scripted solution. So we divided several of the tasks over different scripts. The PHP script adds different metrics to CloudWatch. We could optimize the solution by having the PHP script adding all metrics in one 'session'. But this works fine.

<?php
    require_once 'AWSSDKforPHP/sdk.class.php';

    define('AWS_KEY', '');
    define('AWS_SECRET_KEY', '');
    define('AWS_ACCOUNT_ID', '');

    $jstat = array(
        'S0' => array('SurvivorSpace0Utilization','Percent'),
        'S1' => array('SurvivorSpace1Utilization','Percent'),
        'E'  => array('EdenSpaceUtilization','Percent'),
        'O'  => array('OldSpaceUtilization','Percent'),
        'P'  => array('PermanentSpaceUtilization','Percent'),
        'YGC'  => array('YoungGenerationGCEvents','Count'),
        'YGCT'  => array('YoungGenerationGCTime','Seconds'),
        'FGC'  => array('FullGenerationGCEvents','Count'),
        'FGCT'  => array('FullGenerationGCTime','Seconds'),
        'GCT'  => array('TotalGCTime','Seconds')
    );

    $cw = new AmazonCloudWatch();
    $cw->set_region(AmazonCloudWatch::REGION_EU_W1);

    $dimensions = array(
        array( 'Name' => 'InstanceIdentifier',
        'Value' => $argv[1])
    );
    $timestamp = date( DATE_RFC822);

    $response = $cw->put_metric_data('9Apps/Smartfox', array(
         array(
             'MetricName' => $jstat[$argv[2]][0],
             'Dimensions' => $dimensions,
             'Value' => $argv[3],
             'Timestamp' => $timestamp,
             'Unit' => $jstat[$argv[2]][1],
         )));
    if( 200 != $response->status) {
        print_r( $response);
    }
?>

jstat

You can get all sorts of memory information from jstat, in all shapes and forms. We are mostly interested in Utilization of different 'spaces'. Utilization fits nicely on CloudWatch's Percentage metric, we don't have to do any calculation. This bash script get the jstat information, parses it and fires off the PHP script. It is basically a wrapper for use in crontab.

#!/bin/bash

/usr/bin/jstat -gcutil -t `pgrep java` 1 1 | \
    /bin/sed '/^Timestamp/d' | \
    instance_id=`/usr/bin/curl --retry 3 -s -S -f http://169.254.169.254/latest/meta-data/instance-id` \
awk '{
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " S0 " $2);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " S1 " $3);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " E " $4);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " O " $5);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " P " $6);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " YGC " $7);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " YGCT " $8);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " FGC " $9);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " FGCT " $10);
    system( "/usr/bin/php /root/put-jvm-status.php " ENVIRON["instance_id"] " GCT " $11);
}'

Crontab

We want to run this every minute. We could get and aggregate several measurements per minute, but this is fine for our purpose.

* * * * *     /root/collect-jvm-stats.sh > /dev/null 2>&1