Serverless Monitoring: What do we Monitor when the Server Goes Away?

Originally written for The SLAppForge Blog; Feb 17, 2020

Monitoring your serverless application is crucial - especially while it is handling your production load. This brings us to today's topic, how to effectively monitor a serverless application.

Serverless = Ephemeral

Serverless environments are inherently ephemeral; once the execution completes, you don't have much left behind to investigate.

There's so much exciting talk about container reuse and keep-warm, but theoretically every single one of your function could be a cold-start. Even with reuse, you don't get access to the environment to analyze the previous invocation, until the next one comes in.

So, to effectively monitor a serverless system, we have to gather as much data as possible - while it is actually handling a request.

Monitoring the Ephemeral. The Dynamic. The Inaccessible.

Serverless offers the benefit of less management, through higher levels of abstraction; for obvious reasons, that comes hand in hand with the caveat of less visibility.

In short, you pretty much have to either:

• depend on what the serverless platform provider discloses to you, or
• write your own monitoring utilities (telemetry agents, instrumentations etc.) to squeeze out more metrics from the runtime

Log Analysis

Logs are the most common means of monitoring, auditing and troubleshooting traditional applications; not surprisingly, it holds true in serverless as well.

What's in a Log?

All major serverless platforms offer comprehensive logging for their FaaS elements: CloudWatch Logs for AWS Lambda, StackDriver Logging for Google Cloud Functions, Azure Monitor Logs for Azure Functions, CloudMonitor for Alibaba Cloud Functions, and so forth.

The logs usually contain:

• execution start marker, with a unique ID for each request/invocation
• all application logs generated during execution, up until the point of completion of the invocation
• execution summary: duration, resources used, billed quota etc.

Serverless is not just FaaS; and logging is not just about function executions. Other services like storage, data-storage and networking also provide their share of logging. While these are mostly associated with access and security auditing (e.g. CloudTrail auditing), they can still be merged with FaaS logs to enable more verbose, detailed monitoring.

Distributed Logging: A Nightmare?

One problem with the inherently distributed nature of serverless systems, is that these logs are usually scattered all over the place; unlike in a traditional monolith where all logs would be neatly arranged in one or a few well-known files.

Imagine your backend consists of five Lambdas, which the web client invokes in a particular sequence to get a job done - or are coordinated through Step Functions or Destinations; the logs for a single "call" could span five log streams in five log groups.

It may sound simple enough during development; but when your production app goes viral and starts receiving hundreds of concurrent calls, tracking a single client journey through the logs could become harder than finding a needle in a haystack.

Log Aggregation

This is where log aggregation services come into play. (In fact, serverless was lucky - because log management had already received a boost, thanks to microservice architecture.) Services like Coralogix and Dashbird will ingest your logs (via push or pull) and allow you to perform filters, aggregations, summarizations etc. as if they were from one or a few sources.

With visibility to long-term data, aggregation services can - and do - actually provide more intelligent outputs; such as real-time alerts on predefined error levels/codes, and stability- or security-oriented anomaly detection through pattern recognition, machine learning etc.

Resource Usage

Even with your application logic running like clockwork, the system could start failing if it is under-provisioned and runs out of resources; or become a budget-killer if over-provisioned.

Additionally, unusual patterns in resource usage may also indicate anomalies in your applications - such as attacks, misuse and other exploits.

What can we Measure?

• memory usage
• execution time or latency: given that FaaS invocations are under strict timeout constraints, time becomes a critical resource. You do not want your function to time-out before completing its job; but you also do not want it to remain hung indefinitely over a bad database connection that is going to take two minutes to time-out on its own.
• compute power used: in many platforms, allocated compute power grows in proportion to memory, so the product of allocated memory and execution time is a good relative measure for the total compute power consumed by the request. In fact, most platforms actually bill you by GB-seconds where GB refers to memory allocation.

Resource Isolation

Serverless invocations are isolated, which means one failure cannot affect another request. Sadly, it also means that each runtime instance should be able to handle the largest possible input/request on its own, as there is virtually no resource sharing across instances; one request handler cannot "borrow" memory from another, as is the case in monolithic apps with multiple services on the same runtime/VM.

This is bad in a sense, that it denies you of the luxury to maintain shared resources (e.g. connection pools, memory-mapped file caches, etc.) to be pooled across requests. But at the same time, it means that managing and monitoring per-request resources becomes easier; each time you allocate or measure it, you are doing it for a single request.

From Logs

As mentioned before, serverless logs usually contain execution summaries stating the allocated vs. maximum memory usages, and exact vs. billed execution time.

From Runtime Itself

Your application runs as a Linux process, so it is always possible to grab resource usage data from the runtime itself - either via standard language/runtime calls (e.g. NodeJS process.memoryUsage()) or by directly introspecting OS-level semantics (e.g. /proc/self/stat). Some aspects like execution time are also provided by the serverless runtime layer - such as remaining time via context.getRemainingTimeInMillis() on Lambda.

From Serverless Platform Metrics

Most platforms keep track of resource metrics themselves. AWS CloudWatch Metrics is perhaps the best example.

While CloudWatch does not yet offer memory or compute graphs, third party tooling like SigmaDash can compute them on your behalf.

Platforms usually provide better metrics for non-FaaS systems than they do for FaaS; probably because of the same challenges they face in accurately monitoring the highly dynamic environment. But they are constantly upgrading their techniques, so we can always hope for better.

For non-compute services like storage (e.g. AWS S3), persistence (e.g. AWS DynamoDB) and API hosting (e.g. AWS API Gateway), platforms offer detailed metrics. The upside with these built-in metrics is that they offer different granularity levels, and basic filtering and summarization, out of the box. The downside, often, is that they are fairly non-real-time (generally several seconds behind on the actual request).

Also, for obvious reasons, longer analysis periods mean lesser granularity. Although you can request for higher-precision data, or report custom metrics at desired precisions, it will cost you more - and, at serverless scales, such costs can add up fairly quickly.

Invocations, Errors and Throttles

While logs and resource usages can reveal anomalies during execution, these depict more straightforward serverless monitoring metrics related to the actual end results of application invocations. In that sense, resource monitoring can be treated as oriented more towards performance, whereas error/throttle monitoring leans more towards correctness, stability and scalability.

You can usually grab these numbers from the built-in metrics of the platform itself.

Invocations-to-error Ratio

Similar to signal-to-noise ratio (SNR), this is a measure of how "efficient" or "robust" your application is; the higher it is, the more requests your application can successfully serve without running into an error.

Of course, the validity of this serverless monitoring metric depends on the actual nature and intent of your code, as well as the nature of inputs; if you receive lots of erratic or failure-prone inputs, and you are actually supposed to fail the invocation in such cases, this value would be naturally high.

Throttles

Presence of throttles could indicate one of a few things:

• You have (perhaps mistakenly) throttled your function (or the API/interface) below its regular concurrency limit.
• Your application is under a denial-of-service (DoS) attack.
• Your application's scalability goes beyond what the serverless platform can occur (maybe it has gone viral); you need to re-architect it to batch-up invocations, introduce back-offs to safely deal with rejections, etc.

Note that throttling issues are not always limited to front-facing elements; if you are asynchronously triggering a function from an internal data stream (e.g. a Lambda with a Kinesis trigger) you could run into throttling, based on the dynamics of the rest of the application.

Also, throttling is not always in front of FaaS. If a third-party API that your application invokes (e.g. Amazon's own seller APIs) gets throttled, you could run into issues even without an apparent increase in the fronting traffic, events or invocation counts. Worse, it could lead to chain reactions or cascading failures - where throttling of one service can flow down the execution path to others, and eventually to the user-facing components.

Once you are on the upper ends of scalability, only a good combination of API knowledge, fail-safe measures and serverless monitoring techniques - logs, throttles and runtime latency measurements - can save you from such scenarios.

Instrumenting

So far, all serverless monitoring techniques we discussed, have been dependent on what the runtime and platform offer by default, or out-of-shelf. However, people find these inadequate for monitoring serious production applications where:

• service health and performance are critical,
• early detection of anomalies and threats is vital, and
• alerts and recovery actions need to be as fine-grained and real-time as possible.

To get beyond what the platform offers, you usually need to instrument your runtime and grab additional serverless monitoring insights. This can be via:

• wrapper code and/or additional libraries (e.g. SignalFX or lambda-monitor),
• installation of metrics-reporter agents (e.g. New Relic APM), or
• modification of the runtime itself (e.g. via layers or custom runtimes, as done by IOpipe, DataDog and Thundra).

For FaaS, this obviously comes with implications on performance - your function runtime now has to spare some cycles to gather and report the metrics; in many cases, the tools also require you to modify your application code. Often these changes are subtle enough - such as importing a library, calling a third-party method, etc.; nevertheless, it goes against the developers' and dev-ops' pipe dream of code-free monitoring.

Realizing the demand for advanced serverless monitoring and diagnostics, cloud platforms themselves have also come forth with instrumentation options:

• AWS X-Ray ships with Lambda by default, and can be enabled via a simple configuration with no code change. Once enabled, it can report the latencies of different invocation phases: initialization, individual AWS API/SDK calls, etc. With a bit of custom code it can also capture calls to any downstream HTTP service - which can come in handy when monitoring and troubleshooting latency issues.
• Google Cloud Functions offers StackDriver Monitoring for advanced analytics via built-in timing, invocation and memory metrics; as well as custom metrics reporting and alerting.
• Azure Functions has its own Azure Application Insights service offering performance analytics .

However, the best work related to instrumentation have come from third-party providers, such as the ones mentioned earlier. These usually require you to subscribe to their services, configure your serverless platforms to report metrics to theirs, and provide analysis results and insights intelligence through their own dashboards.

Serverless Monitoring, up against Privacy and Security Concerns

The obvious problem with delegating serverless monitoring responsibilities to third parties, is that you need to guarantee the safety of the data you are sharing; after all, it is your customers' data.

Even if you implicitly trust the third party, you need to take precautions to minimize the damage in case the monitoring data gets leaked - for the sake of your users as well as the security and integrity of your application.

• Anonymizing and screening your logs for sensitive content,
• using secure channels for metrics sharing, and
• always sticking to the principle of least privilege during gathering, sharing and analysis of data

will get you started on a robust yet safe path to efficiently monitoring your serverless application.

How I made AWS CLI 300% faster! [FULL disclosure]

Yeah yeah, it's "highly experimental" and all, but still it's 3 times faster than simply running aws bla bla bla, the "plain" aws-cli.

And yes, it won't always be that fast, especially if you only run AWS CLI only about once a fortnight. But it will certainly have a clear impact once you start batching up your AWS CLI calls; maybe routine account checks/cleanups, maybe extracting tons of CloudWatch Metrics records; or maybe a totally different, unheard-of use case.

Whatever it is, I guess it would be useful for the masses some day.

Plus, as one of the authors and maintainers of the world's first serverless IDE, I have certainly had several chances to put it to good use!

The problem: why AWS CLI is "slow" for me

(Let's just call it "CLI", shall we?)

It's actually nothing to do with the CLI itself; rather it's the fact that each CLI invocation is a completely new program execution cycle.

This means:

• python (and ultimately the OS) has to load the binaries, configs, boto API definitions and so forth;
• the CLI has to initialize itself: load all supported commands definitions, prepare parsers, generate API client classes and so forth

But, as usual, the highest impact comes via network I/O:

• the CLI has to create an API client from scratch (the previous one was lost when the command execution completed)
• since the network connection to AWS is managed by the client, this means that each command creates (and then destroys) a fresh TCP connection to the AWS endpoint; which involves a DNS lookup as well (although later lookups may be served from the system cache)
• since AWS APIs almost always use SSL, every new connection results in a full SSL handshake (client hello, server hello, server cert, yada yada yada)

Now, assume you have 20 CloudWatch Log Groups to be deleted. Since the Logs API does not offer a bulk deletion option, the cheapest way to do this would be to run a simple shell script - looping aws logs delete-log-group over all groups:

for i in $(aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output text); do
    aws logs delete-log-group --log-group-name $i
done

This would run the CLI 20 times (21 to be precise, if you count the initial list API call); meaning that all of the above will run 20 times. Clearly a waste of time and resources, since we were quite clear that the same endpoint was going to be invoked in all those runs.

Try scaling this up to hundreds or thousands of batched operations; and see where it takes you!

And no, aws-shell does not cut it.

Not yet, at least.

Leaving aside the nice and cozy REPL interface (interactive user prompt), handy autocompletion, syntax coloring and inline docs, aws-shell does not give you any performance advantage over aws-cli. Every command in the shell is executed in a new AWS CLI instance - with parsers, command hierarchies, API specs and - more importantly API clients - getting recreated for every command.

Skeptical? Peek at the aws-shell sources; or better still, fire up Wireshark (or tcpdump if you dare), run a few commands in the shell REPL, and see how each command initializes a fresh SSL channel from scratch.

The proposal: what can we do?

Obviously, the CLI cannot do pretty much anything about it. It's a simple program, and whatever improvements we do, won't last until the next invocation. The OS would rudely wipe them and start the next CLI with a clean slate; unless we use some spooky (and rather discouraged) memory persistence magic to serialize and reload the CLI's state. Even then, the other OS-level stuff (network sockets etc.) will be gone, and our effort would be pretty much fruitless.

If we are going to make any impactful changes, we need to make the CLI stateful; a long-running process.

The d(a)emon

In the OS world, this usually means setting up a daemon - a background process that waits for and processes events like user commands. (A popular example is MySQL, with its mysql-server daemon and mysql-client packages.)

In our case, we don't want a fully-fledged "managed" daemon - like a system service. For example, there's no point in starting our daemon before we actually start making our CLI calls; also, if our daemon dies, there's no point in starting it right away; since we cannot recover the lost state anyway.

So we have a simple plan:

• break the CLI into a "client" and daemon
• every time we run the CLI,
  • check for the presence of the daemon, and
  • spawn the daemon if it is not already running

This way, if the daemon dies, the next CLI invocation will auto-start it. Nothing to worry, nothing to manage.

Our fast AWS CLI daemon - it's all in a subprocess!

It is easy to handle the daemon spawn without having the trouble of maintaining a second program or script; simply use subprocess.Popen to launch another instance of the program, and instruct it to run the daemon's code path, rather than the client's.

Enough talk; show me the code!

Here you go:

#!/usr/bin/python

import os
import sys
import tempfile
import psutil
import subprocess

rd = tempfile.gettempdir() + "/awsr_rd"
wr = tempfile.gettempdir() + "/awsr_wr"


def run_client():
	out = open(rd, "w")
	out.write(" ".join(sys.argv))
	out.write("\n")
	out.close()

	inp = open(wr, "r")
	result = inp.read()
	inp.close()

	sys.stdout.write(result)


def run_daemon():
	from awscli.clidriver import CLIOperationCaller, LOG, create_clidriver, HISTORY_RECORDER

	def patchedInit(self, session):
		self._session = session
		self._client = None

	def patchedInvoke(self, service_name, operation_name, parameters, parsed_globals):
		if self._client is None:
			LOG.debug("Creating new %s client" % service_name)
			self._client = self._session.create_client(
				service_name, region_name=parsed_globals.region,
				endpoint_url=parsed_globals.endpoint_url,
				verify=parsed_globals.verify_ssl)
		client = self._client

		response = self._make_client_call(
			client, operation_name, parameters, parsed_globals)
		self._display_response(operation_name, response, parsed_globals)
		return 0

	CLIOperationCaller.__init__ = patchedInit
	CLIOperationCaller.invoke = patchedInvoke

	driver = create_clidriver()
	while True:
		inp = open(rd, "r")
		args = inp.read()[:-1].split(" ")[1:]
		inp.close()

		if len(args) > 0 and args[0] == "exit":
			sys.exit(0)

		sys.stdout = open(wr, "w")
		rc = driver.main(args)

		HISTORY_RECORDER.record('CLI_RC', rc, 'CLI')
		sys.stdout.close()


if __name__ == "__main__":
	if not os.access(rd, os.R_OK | os.W_OK):
		os.mkfifo(rd)
	if not os.access(wr, os.R_OK | os.W_OK):
		os.mkfifo(wr)

	# fork if awsr daemon is not already running
	ps = psutil.process_iter(attrs=["cmdline"])
	procs = 0
	for p in ps:
		cmd = p.info["cmdline"]
		if len(cmd) > 1 and cmd[0].endswith("python") and cmd[1] == sys.argv[0]:
			procs += 1
	if procs < 2:
		sys.stderr.write("Forking new awsr background process\n")
		with open(os.devnull, 'r+b', 0) as DEVNULL:
			# new instance will see env var, and run itself as daemon
			p = subprocess.Popen(sys.argv, stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL, close_fds=True, env={"AWSR_DAEMON": "True"})
			run_client()

	elif os.environ.get("AWSR_DAEMON") == "True":
		run_daemon()
	else:
		run_client()

Yep, just 89 lines of rather primitive code - of course it's also on GitHub, in case you were wondering.

Some statistics - if you're still not buying it

"Lies, damn lies and statistics", they say. But sometimes, statistics can do wonders when you are trying to prove a point.

As you would understand, our new REPL really shines when there are more and more individual invocations (API calls); so that's what we would compare.

S3 API: s3api, not s3

Let's upload some files (via put-object):

date

for file in $(find -type f -name "*.sha1"); do
    aws s3api put-object --acl public-read --body $file --bucket target.bucket.name --key base/path/
done

date

• Bucket region: us-east-1
• File type: fixed-length checksums
• File size: 40 bytes each
• Additional: public-read ACL

Uploading 70 such files via aws s3api put-object takes:

• 4 minutes 35 seconds
• 473.5 KB data (319.5 KB downlink + 154 KB uplink)
• 70 DNS lookups + SSL handshakes (one for each file)

In comparison, uploading 72 files via awsr s3api put-object takes:

• 1 minute 28 seconds
• 115.5 KB data (43.5 KB downlink + 72 KB uplink)
• 1 DNS lookup + SSL handshake for the whole operation

A 320% improvement on latency (or 420%, if you consider bandwidth).

If you feel like it, watch the outputs (stdout) of the two runs - real-time. You would notice how awsr shows a low and consistent latency from the second output onwards; while the plain aws shows almost the same latency between every output pair - apparently because almost everything gets re-initialized for each call.

If you monitor (say, "wireshark") your network interface, you will see the real deal: aws continuously makes DNS queries and SSL handshakes, while awsr just makes one every minute or so.

Counterargument #1: If your files are all in one place or directory hierarchy, you could just use aws s3 cp or aws s3 sync in one go. These will be as performant as awsr, if not more. However in my case, I wanted to pick 'n' choose only a subset of files in the hierarchy; and there was no easy way of doing that with the aws command alone.

Counterargument #2: If you want to upload to multiple buckets, you will have to batch up the calls bucket-wise (us-east-1 first, ap-southeast-2 next, etc.); and kill awsr after each batch - more on that later.

CloudWatch logs

Our serverless IDE Sigma generates quite a lot of CloudWatch Logs - especially when our QA battalion is testing it. To keep things tidy, I prefer to occasionally clean up these logs, via aws logs delete-log-group.

date

for i in $(aws logs describe-log-groups --query 'logGroups[*].logGroupName' --output text); do
    echo $i
    aws logs delete-log-group --log-group-name $i
done

date

Cleaning up 172 such log groups on us-east-1, via plain aws, takes:

• 5 minutes 44 seconds
• 1.51 MB bandwidth (1133 KB downlink, 381 KB uplink)
• 173 (1 + 172) DNS lookups + SSL handshakes; one for each log group, plus one for the initial listing

On the contrary, deleting 252 groups via our new REPL awsr, takes just:

• 2 minutes 41 seconds
• 382 KB bandwidth (177 KB downlink, 205 KB uplink)
• 4 DNS lookups + SSL handshakes (about 1 in each 60 seconds)

This time, a 310% improvement on latency; or 580% on bandwidth.

CloudWatch metrics

I use this script to occasionally check the sizes of our S3 buckets - to track down and remove any garbage; playing the "scavenger" role:

for bucket in `awsr s3api list-buckets --query 'Buckets[*].Name' --output text`; do
    size=$(awsr cloudwatch get-metric-statistics --namespace AWS/S3 \
        --start-time $(date -d @$((($(date +%s)-86400))) +%F)T00:00:00 --end-time $(date +%F)T00:00:00 \
        --period 86400 --metric-name BucketSizeBytes \
        --dimensions Name=StorageType,Value=StandardStorage Name=BucketName,Value=$bucket \
        --statistics Average --output text --query 'Datapoints[0].Average')
    if [ $size = "None" ]; then size=0; fi
    printf "%8.3f  %s\n" $(echo $size/1048576 | bc -l) $bucket
done

Checking 45 buckets via aws (45+1 API calls to the same CloudWatch API endpoint), takes:

94 seconds

Checking 61 buckets (62 API calls) via awsr, takes:

44 seconds

A 288% improvement.

The catch

There are many; more unknowns than knowns, in fact:

• The REPL depends on serial communication via pipes; so you cannot run things in parallel - e.g. invoke several commands and wait for all of them to complete. (This, however, should not affect any internal parallelizations of aws-cli itself.)

• awsr may start acting up, if you cancel or terminate an already running command - also a side-effect of using pipes.

• awsr reuses internal client objects across invocations (sessions), so it is, let's say, "sticky"; it "remembers" - and does not allow you to override - the profile, region etc. across invocations. In order to start working with a new configuration, you should:
  • terminate the existing daemon:

    kill $(ps -ef -C /usr/bin/python | grep -v grep | grep awsr | awk '{print $2}')

  • in case the daemon might have been processing a command when it was brutally massacred; delete the pipes /tmp/awsr_rd and /tmp/awsr_wr
  • run a new awsr with the correct profile (--profile), region (--region) etc.

• awsr cannot produce interactive output - at least not yet - as it simply reads/writes from/to each pipe exactly once in a single invocation. So commands like ec2 wait and cloudformation deploy will not work as you expected.

• Currently the pipes only capture standard input and standard output; so, unless you initially launched awsr in the current console/tty, you won't be seeing any error messages (written to standard error) being generated by the underlying AWS API call/command.

• Some extensions like s3 don't seem to benefit from the caching - even when invoked against the same bucket. It needs further investigation. (Luckily, s3api works fine - as we saw earlier.)

Bonus: hands-on AWS CLI fast automation example, FTW!

I run this occasionally to clean up our AWS accounts of old logs and build data. If you are curious, replace the awsr occurrences with aws (and remove the daemon-killing magic), and witness the difference in speed!

Caution: If there are ongoing CodeBuild builds, the last step may keep on looping - possibly even indefinitely, if the build is stuck in BUILD_IN_PROGRESS status. If you run this from a fully automated context, you may need to enhance the script to handle such cases as well.

for p in araProfile meProfile podiProfile thadiProfile ; do
    for r in us-east-1 us-east-2 us-west-1 us-west-2 ca-central-1 eu-west-1 eu-west-2 eu-central-1 \
        ap-northeast-1 ap-northeast-2 ap-southeast-1 ap-southeast-2 sa-east-1 ap-south-1 ; do

        # profile and region changed, so kill any existing daemon before starting
        arg="--profile $p --region $r"
        kill $(ps -ef -C /usr/bin/python | grep -v grep | grep awsr | awk '{print $2}')
        rm /tmp/awsr_rd /tmp/awsr_wr

        # log groups
        for i in $(awsr $arg logs describe-log-groups --query 'logGroups[*].logGroupName' --output text); do
            echo $i
            awsr $arg logs delete-log-group --log-group-name $i
        done

        # CodeBuild projects
        for i in $(awsr $arg codebuild list-projects --query 'projects[*]' --output text); do
            echo $i
            awsr $arg codebuild delete-project --name $i
        done

        # CodeBuild builds; strangely these don't get deleted when we delete the parent project...
        while true; do
            builds=$(awsr $arg codebuild list-builds --query 'ids[*]' --output text --no-paginate)
            if [[ $builds = "" ]]; then break; fi
            awsr $arg codebuild batch-delete-builds --ids $builds
        done

    done
done

In closing: so, there it is!

Feel free to install and try out awsr; after all there's just one file, with less than a hundred lines of code!

Although I cannot make any guarantees, I'll try to eventually hunt down and fix the gaping holes and shortcomings; and any other issues that you or me come across along the way.

Over to you, soldier/beta user!

AWS: Some Tips for Avoiding Those "Holy Bill" Moments

Cloud is awesome: almost-100% availability, near-zero maintenance, pay-as-you-go, and above all, infinitely scalable.

But the last two can easily bite you back, turning that awesomeness into a billing nightmare.

And occasionally you see stories like:

Within a week we accumulated a bill close to $10K.

And here I unveil a few tips that we learned from our not-so-smooth journey of building the world's first serverless IDE, that could help others to avoid some "interesting" pitfalls.

Careful with that config!

One thing we learned was to never underestimate the power of a configuration.

If you read the above linked article you would have noticed that it was a simple misconfiguration: a CloudTrail logging config that was writing logs to one of the buckets it was already monitoring.

You could certainly come up with more elaborate and creative examples of creating "service loops" yielding billing black-holes, but the idea is simple: AWS is only as intelligent as the person who configures it.

(Well, in the above case it was one of my colleagues who configured it, and I was the one who validated it; so you can stop here if you feel like it ;) )

So, when you're about to submit a new config update, try to rethink the consequences. You won't regret it.

It's S3, not your attic.

AWS has estimated that 7% of cloud billing is wasted on "unused" storage - space taken up by content of no practical use: obsolete bundles, temporary uploads, old hostings, and the like.

Life in a bucket

However, it is true that cleaning up things is easier said than done. It is way too easy to forget about an abandoned file than to keep it tracked and delete it when the time comes.

Probably for the same reason, S3 has provided lifecycle configurations - time-based automated cleanup scheduling. You can simply say "delete this if it is older than 7 days", and it will be gone in 7 days.

This is an ideal way to keep temporary storage (build artifacts, one-time shares etc.) in check, hands-free.

Lifecycle configs can also become handy when you want to delete a huge volume of files from your bucket; rather than deleting individual files (which in itself would incur API costs - while deletes are free, listing is not!), you can simply set up a lifecycle config rule to expire everything in 1 day. Sit back and relax, while S3 does the job for you!

{
    "Rules": [
        {
            "Status": "Enabled",
            "Prefix": "",
            "Expiration": {
                "Days": 1
            }
        }
    ]
}

Alternatively you can move the no-longer-needed-but-not-quite-ready-to-let-go stuff into Glacier, for a fraction of the storage cost; say, for stuff under the subpath archived:

{
    "Rules": [
        {
            "Filter": {
                "Prefix": "archived"
            },
            "Status": "Enabled",
            "Transitions": [
                {
                    "Days": 1,
                    "StorageClass": "GLACIER"
                }
            ]
        }
    ]
}

But before you do that...

Ouch, it's versioned!

(Inspired by true events.)

I put up a lifecycle config to delete about 3GB of bucket access logs (millions of files, obviously), and thought everything was good - until, a month later, I got the same S3 bill as the previous month :(

Turns out that the bucket had had versioning enabled, so deletion does not really delete the object.

So with versioning enabled, you need to explicitly tell the S3 lifecycle logic to:

• discard non-current (deleted) object versions, and
• expire old delete markers

in order to completely get rid of the "deleted" content and the associated delete markers.

So much for "simple" storage service ;)

CloudWatch is your pal

Whenever you want to find out the total sizes occupied by your buckets, just iterate through your AWS/S3 CloudWatch Metrics namespace. There's no way—suprise, surprise—to check bucket size natively from S3; even the S3 dashboard relies on CloudWatch, so why not you?

Quick snippet to view everything? (uses aws-cli and bc on bash)

yesterday=$(date -d @$((($(date +%s)-86400))) +%F)
for bucket in `aws s3api list-buckets --query 'Buckets[*].Name' --output text`; do
        size=$(aws cloudwatch get-metric-statistics --namespace AWS/S3 --start-time ${yesterday}T00:00:00 --end-time $(date +%F)T00:00:00 --period 86400 --metric-name BucketSizeBytes --dimensions Name=StorageType,Value=StandardStorage Name=BucketName,Value=$bucket --statistics Average --output text --query 'Datapoints[0].Average')
        if [ $size = "None" ]; then size=0; fi
        printf "%8.3f  %s\n" $(echo $size/1048576 | bc -l) $bucket
done

EC2: sweep the garbage, plug the holes

EC2 makes it trivial to manage your virtual machines - compute, storage and networking. However, its simplicity also means that it can leave a trail of unnoticed garbage and billing leaks.

Pick your instance type

There's a plethora of settings when creating a new instance. Unless there are specific performance requirements, picking a T2-class instance type with Elastic Block Store (EBS)-backed storage and 2-4 GB of RAM would suffice for most needs.

Despite being free tier-eligible, t2.micro can be a PITA if your server could receive compute-or memory-intensive loads at some point; in these cases t2.micro tends to simply freeze (probably has to do with running out of CPU credits?), causing more trouble than it's worth.

Clean up AMIs and snapshots

We habitually tend to take periodic snapshots of our EC2 instances as backups. Some of these are made into Machine Images (AMIs) for reuse or sharing with other AWS users.

We easily forget about the other snapshots.

While snapshots don't get billed for their full volume sizes, they can add up to significant garbage over time. So it is important to periodically visit and clean up your EC2 snapshots tab.

Moreover, creating new AMIs would usually mean that older ones become obsolete; they can be "deregistered" from the AMIs tab as well.

But...

Who's the culprit - AMI or snapshot?

The actual charges are on snapshots, not on AMIs themselves.

And it gets tricky because deregistering an AMI does not automatically delete the corresponding snapshot.

You usually have to copy the AMI ID, go to snapshots, look for the ID in the description field, and nuke the matching snapshot. Or, if you are brave (and lazy), select and delete all snapshots; AWS will prevent you from deleting the ones that are being used by an AMI.

Likewise, for instances and volumes

Compute is billed while an EC2 instance is running; but its storage volume is billed all the time - right up to deletion.

Volumes usually get nuked when you terminate an instance; however, if you've played around with volume attachment settings, there's a chance that detached volumes are left behind in your account. Although not attached to an instance, these still occupy space; and so AWS charges for them.

Again, simply go to the volumes tab, select the volumes in "available" state, and hit delete to get rid of them for good.

Tag your EC2 stuff: instances, volumes, snapshots, AMIs and whatnot

It's very easy to forget what state was in the instance, at the time that snapshot was made. Or the purpose of that running/stopped instance which nobody seems to take ownership or responsibility of.

Naming and tagging can help avoid unpleasant surprises ("Why on earth did you delete that last month's prod snapshot?!"); and also help you quickly decide what to toss ("We already have an 11-05 master snapshot, so just delete everything older than that").

You stop using, and we start billing!

Sometimes, the AWS Lords work in mysterious ways.

For example, Elastic IP Addresses (EIPs) are free as long as they are attached to a running instance. But they start getting charged by the hour, as soon as the instance is stopped; or if they get into a "detached" state (not attached to a running instance) in some way.

Some prior knowledge about the service you're about to sign up for, can prevent some nasty surprises of this fashion. A quick pricing page lookup or google can be a deal-breaker.

Pay-per-use vs pay-per-allocation

Many AWS services follow one or both of the above patterns. The former is trivial (you simply pay for the time/resources you actually use, and enjoy a zero bill for the rest of the time) and hard to miss; but the latter can be a bit obscure and quite easily go unnoticed.

Consider EC2: you mainly pay for instance runtime but you also pay for the storage (volumes, snapshots, AMIs) and network allocations (like inactive Elastic IPs) even if your instance has been stopped for months.

There are many more examples, especially in the serverless domain (which we ourselves are incidentally more familiar with):

• Kinesis charges by shard-hour - even if all your shards are idle
• DynamoBB charges for storage and read/write in terms of "capacity units" - thankfully there's a non-expiring free tier!
• RDS (very similar to EC2) charges for instance runtime, whether busy or idle (Aurora Serverless seems to be trying to change this to some extent)
• KMS charges a flat fee for each customer-managed key (CMK) whether you use it or not

Meanwhile, some services secretly set up their own monitoring, backup and other "utility" entities. These, although (probably!) meant to do good, can secretly seep into your bill:

• DynamoDB sets up CloudWatch Alarms; these are left behind even after the corresponding tables have been deleted (at least when managed via CloudFormation).
• RDS automatically creates instance volume snapshots, at termination as well as during daily maintenance (esp. when deployed via the "default" CloudFormation configs; these can easily add up over your storage quotas

These are the main culprits that often appear in our AWS bills; certainly there are better examples, but you get the point.

CloudWatch (yeah, again)

Many services already—or can be configured to—report usage metrics to CloudWatch. Hence, with some domain knowledge of which metric maps into which billing component (e.g. S3 storage cost is represented by the summation of the BucketSizeBytes metric across all entries of the AWS/S3 namespace), you can build a complete billing and monitoring solution around CloudWatch Metrics (or delegate the job to a third-party service like DataDog).

CloudWatch in itself is mostly free, and its metrics have automatic summarization mechanisms so you don't have to worry about overwhelming it with age-old garbage—or getting overwhelmed with off-the-limit capacity bills.

The Billing API

Although AWS does have a dedicated Billing Dashboard, logging in and checking it every single day is not something you would add to your agenda (at least not for API/CLI minds like you and me).

Luckily, AWS offers a billing API whereby you can obtain a fairly granular view of your current outstanding bill, over any preferred time period - broken down by services or actual API operations.

Catch is, this API is not free: each invocation costs you $0.01. Of course this is negligible - considering the risk of having to pay several dozens—or even hundreds or thousands in some cases—it is worth having a $0.30/month billing monitor to track down any anomalies before it's too late.

Food for thought: with support for headless Chrome offered for Google Cloud Functions, one might be able to set up a serverless workflow that logs into the AWS dashboard and checks the bill for you. Something to try out during free time (if some ingenious folk hasn't hacked it together already).

Billing alerts

Strangely (or perhaps not ;)) AWS doesn't offer a way to put up a hard limit for billing; despite the numerous user requests and disturbing incident reports all over the web. Instead, they offer alerts for various billing "levels"; you can subscribe for notifications like "bill at x% of the limit" and "limit exceeded", via email or SNS (handy for automation via Lambda!).

My advice: this is a must-have for every AWS account. If we had one in place, we could already have saved well over thousands of dollars to date.

Organizational accounts

If you want to delegate AWS access to third parties (testing teams, contract-basis devs, demo users etc.), it might be a good idea to create a sub-account by converting your root account into an AWS organization with consolidated billing enabled.

(While it is possible to do almost the same using an IAM user, it will not provide resource isolation; everything would be stuffed in the same account, and painstakingly complex IAM policies may be required to isolate entities across users.)

Our CEO and colleague Asankha has written about this quite comprehensively so I'm gonna stop at that.

And finally: Monitor. Monitor. Monitor.

No need to emphasize on this - my endless ramblings should already have conveyed its importance.

So, good luck with that!