Idempotency for Go Lambdas, and the clever tricks behind it

A row of identical envelopes on a dim post office counter; only the one nearest a brass rubber stamp is marked PAID, and glowing circuit lines run from every envelope to a single padlock beside the stamp.

Where this came from

I was watching the re:Invent 2023 talk Building next-generation applications with event-driven architecture (API302), which is about idempotency in event-driven architectures: why it matters, and how to get it right when your system is made of queues, events and functions that retry on their own. Along the way it mentioned Powertools for AWS Lambda, the AWS toolkit that gives you idempotency, batch processing, logging and metrics out of the box.

Then I went looking for the Go version. There isn’t one. Python, TypeScript, Java and .NET get Powertools; Go doesn’t.

So I decided to give it a go (pun intended) with my trusty clanker, and the result is lambda-kit-go. It currently has two packages: idempotency and batch.

The library itself is not the most interesting part, though. While building it I found out how clever the idempotency design is, and that’s what this post is about.

Duplicates are not an edge case

The first thing to accept: in an event-driven system on AWS, you will receive the same event more than once. Almost every AWS messaging service delivers at least once:

  • SQS standard queues are at-least-once by design. If your Lambda runs longer than the visibility timeout, a second Lambda can receive the same message while the first one is still working on it.
  • Lambda async invocations (from SNS, EventBridge, S3) retry twice on errors and timeouts. A timeout doesn’t mean the work didn’t happen; the card may have been charged a millisecond before the clock ran out.
  • Producers crash after publishing but before recording that they published, and publish again.
  • API clients retry a POST after a network timeout.

Exactly-once delivery across distributed components isn’t something you can buy. What you can do is make the consumer idempotent: processing the same event twice has the same effect as processing it once. At-least-once delivery plus idempotent consumers gives you what people call “effectively once”.

The naive approach, and why it breaks

The first thing that comes to mind is something like:

if !alreadyProcessed(orderID) {
	chargeCard(order)
	markProcessed(orderID)
}

This breaks exactly when you need it: two Lambdas receive the same message at the same moment, both check, both see “not processed”, and both charge the card. Lambdas don’t share memory and may not even run on the same machine. A check followed by an act is a race.

Trick 1: a conditional write as a distributed lock

The fix is to make the check and the claim a single atomic operation, done in a place every Lambda can reach. That place is a DynamoDB table, and the operation is a conditional write: “create this record only if no live record exists”.

const createCondition = "attribute_not_exists(#id)" +
	" OR #exp <= :now" +
	" OR (#status = :inprogress AND #ipexp <= :nowms)"

DynamoDB evaluates that condition atomically on the item. If ten Lambdas try at the same instant, exactly one wins and the other nine get a conditional-check failure. There’s no lock server, no Redis cluster and no leader election: one table, one write. (You can see it in store.go.)

Here’s how it plays out when SQS hands the same message to two Lambdas, A and B, at the same time. The message is an order with the ID order-42:

t=0   A: create record for order-42   -> wins, record is INPROGRESS
t=0   B: create record for order-42   -> condition fails -> ErrInProgress
                                         B does NOT charge the card
      B reports the message as failed -> back to the queue
t=2s  A: charges the card, saves the result, record is COMPLETED
t=30s SQS redelivers -> C: condition fails, record is COMPLETED
                          -> C returns the saved result, no charge

The card is charged once, even though the message was processed three times.

One thing that confused me at first: the library doesn’t only protect DynamoDB operations. DynamoDB is just where the lock lives. What it protects is whatever your function does: charging a card, sending an email, calling an API. Your code only runs if the claim succeeds.

Trick 2: the key is prefix#sha256

To know that two events are “the same”, each one needs a key. The library builds it like this:

<prefix>#<sha-256 of the key data>

The prefix is a namespace, which defaults to the Lambda function name. It’s what allows one table to serve many functions. “Charge order-42” and “send the receipt for order-42” share the same order ID but are different operations, so they get different prefixes and never block each other.

The SHA-256 is a fingerprint of the key data: whatever you tell the library identifies the work. It’s JSON-encoded and then hashed. The same input always gives the same fingerprint, any change gives a completely different one, and the length is always 64 characters no matter how big the input is.

A concrete example: a function called charge-orders keyed on the order ID.

StepValue
Key datao.ID = order-42
As JSON"order-42"
SHA-25690140eb1a898639024d3752994bf84b1d40cfe639f70bcbdf7d047bf72274a1d
Final keycharge-orders#90140eb1…4a1d

Order order-43 hashes to c7516cc7…400c, a completely different record.

The important part is choosing what to hash. My first assumption was that it hashes the whole SQS message, and that would actually break everything. The message includes fields that change on every delivery (receipt handle, receive count), so every copy of a duplicate would get a different key. Even the message body can carry a timestamp. The rule of thumb is to ask what is the thing that must only happen once? and hash exactly that. Usually that’s a business ID, like the order ID.

If one field isn’t enough, pass a struct or a map. It gets JSON-encoded into a single string and hashed into a single key:

idempotency.Do(ctx, idem, map[string]string{"customer": o.CustomerID, "order": o.ID}, fn)

That’s better than gluing strings together yourself, because "ab" + "c" and "a" + "bc" both give "abc". JSON keeps each value quoted and separate.

Trick 3: save the response, not just a flag

The record is more than a “done” flag. It has a status (INPROGRESS or COMPLETED), and when the function succeeds, its result is saved in the record.

That matters for synchronous callers. If an API client retries a POST, it should get the same response the first call produced (same order ID, same receipt), not an error and not a second order. A duplicate just gets the saved response replayed, for as long as the record is within its expiration time (more on that next).

Trick 4: two different expiration times

Each record carries two expirations, and they solve different problems.

expiration is the deduplication window. Retries arrive within a bounded time: SQS keeps messages for 4 days by default, Lambda async retries last up to 6 hours, EventBridge retries for up to 24 hours. Size the window to the longest path a duplicate can take. It doubles as cleanup, since it’s configured as DynamoDB’s TTL, so expired records are deleted automatically for free. TTL deletion is lazy and can take a day or two, but that doesn’t matter: the conditional write already treats an expired record as absent. So the same attribute does two jobs. For the library, it’s a hard deadline checked on every write. For DynamoDB, it’s housekeeping that happens whenever. An expired record still sitting in the table is simply overwritten, and its old response is never replayed.

in_progress_expiration is about crashes. If a Lambda dies halfway through, its record stays INPROGRESS, and without a second clock the key would be blocked for the whole window. So the in-progress expiry is set to the invocation’s deadline. Once the crashed Lambda would have been killed anyway, a retry can take over.

Trick 5: an ownership token

This one came out of an AI code review, and it’s my favorite.

Picture this: invocation A is slow and goes past its deadline. Retry B takes over the key, as designed. Then A finally finishes and saves its result, overwriting B’s record. Or A fails and deletes the record, so a third invocation starts while B is still running.

The fix is that every record stores a random token for the invocation that created it, and saves and deletes only go through if the token still matches. A slow invocation that lost the key simply can’t touch the record anymore. It’s two lines of DynamoDB condition, and it closes a race I would never have spotted on my own.

Trick 6: an error that tells you the side effect already happened

There’s one failure mode no lock can fix: your function charges the card, and then saving the result to DynamoDB fails. The charge happened, but the record doesn’t say so.

If the handler returns an error, Lambda retries and the card gets charged again. AI suggested using a callback hook. Then I asked: can’t the user just see the error and decide? And yes, that’s the more Go way of doing it. Errors are values. The library returns the result together with an error matching ErrResponseNotSaved, and you decide:

receipt, err := idempotency.Do(ctx, idem, o.ID, func(ctx context.Context) (Receipt, error) {
	return charge(ctx, o)
})
if errors.Is(err, idempotency.ErrResponseNotSaved) {
	slog.ErrorContext(ctx, "charged but not recorded", "order", o.ID, "err", err)
	return receipt, nil // don't let Lambda retry and charge again
}

Lambdas get batches, not messages

Lambda service polls the queue and invokes your function with a batch of up to 10 messages by default. You can set it as low as 1, or as high as 10,000 for standard queues (FIFO queues are capped at 10).

The catch: if one message in the batch fails, by default the whole batch is retried, including the messages that already succeeded. That’s one of the most common sources of duplicates on AWS.

So the second package, batch, processes records one by one and tells Lambda exactly which ones failed:

idem := idempotency.New(dynamodbstore.New(ddb, "idempotency"), idempotency.WithKeyPrefix("charge"))

lambda.Start(batch.SQS(func(ctx context.Context, msg events.SQSMessage) error {
	var o Order
	if err := json.Unmarshal([]byte(msg.Body), &o); err != nil {
		return err
	}
	_, err := idempotency.Do(ctx, idem, o.ID, func(ctx context.Context) (struct{}, error) {
		return struct{}{}, charge(ctx, o)
	})
	return err
}))

A few details I’m happy with. FIFO queues stop at the first failure so order is kept. A panicking record fails only itself. And when the invocation gets within a second of its timeout, the remaining records are handed back to Lambda unprocessed, so the response actually gets sent. (The first version only checked whether the context was already done. A review pointed out that a Lambda context’s deadline is the moment it gets killed, so by then there’s no time left to respond.)

One warning that bites people: you have to enable ReportBatchItemFailures on the event source mapping. Without it, Lambda ignores the failure report, and the failed messages are deleted.

Testing it on real AWS

Unit tests and DynamoDB Local only get you so far. Some claims can only be checked against the real thing: that the IAM permissions in the README are enough, that Lambda honors the partial failure report, that the timeout margin works against a real invocation deadline, and that two Lambdas racing for the same order really can’t both win.

So the repo also has a deployable example: an SQS queue, a Go Lambda using batch + idempotency, the idempotency table and a least-privilege IAM role, all in Terraform. Nothing in it is reachable from the internet (no API Gateway, no function URL), which matters when you’re worried about denial-of-wallet attacks.

The clever part of the test is how it detects a double charge. Instead of charging a card, the function adds 1 to a per-order counter in a “ledger” table. That operation is deliberately not idempotent, so if the library ever let a duplicate through, the order would end up with charges = 2.

Then a small load generator throws the nasty cases at it:

  • 1,000 normal orders
  • 200 orders published 2 to 4 times, like a producer that retries
  • 50 orders sent 5 times at the same instant, so copies land in different Lambdas at once and race for the same key
  • 5 slow orders that sleep close to the 10 second timeout
  • 20 messages that aren’t even valid JSON

The first run passed everything. Six times, two real Lambdas raced for the same order and the conditional write let only one through. Here’s one of them, straight from CloudWatch:

{"level":"WARN","msg":"record failed","messageId":"16f272eb-…","err":"idempotency: invocation already in progress"}

The loser handed its message back to the queue, SQS redelivered it later, and it replayed the saved result instead of charging again.

Then I ran an AI code review on the example, and it pointed out that part of that green result was luck. A slow order (9.3 seconds on a 10 second timeout) that started even a little late would finish its sleep with almost no time left, and fire the ledger write right at the deadline. If the write landed but the client gave up waiting, the library would see an error, free the key, and the retry would charge again. The run passed only because no slow order happened to start late enough.

The fix is a rule worth stealing for any Lambda with side effects: don’t start an irreversible step unless there’s time to finish it and record it. Before charging, the function now checks that the work plus 2 seconds of headroom fits before the deadline. If it doesn’t, it bails out before charging, the key is freed cleanly, and the message is retried in a fresh invocation. The review also caught that my queue settings were too tight (AWS recommends a visibility timeout of 6x the function timeout and at least 5 receives before the dead-letter queue), so valid orders could have been pushed to the DLQ just by retrying a few times.

After the fixes, I ran it again:

Run rtovc7e: 1899 messages for 1255 orders, 20 poison

  [PASS] no order charged twice (0 doubled)
  [PASS] every order charged (0 missing)
  [PASS] no valid order in the DLQ (0 found)
  [PASS] every invalid message reached the DLQ (20 of 20)
  [PASS] IAM permissions sufficient (0 AccessDenied lines)

  Evidence the interesting paths ran:
    concurrent duplicates blocked (ErrInProgress):  3
    charges deferred by the headroom guard:         0
    records handed back near timeout:               0
    invocations that hit the hard timeout:          0

Exactly 1,255 idempotency records, one per order. The races happened again (three this time), and the slow orders all finished well inside the limit.

That also means the new guard never fired. Every slow order happened to start early in its invocation, with plenty of time left. The guard matters when earlier messages in the same batch have already used up most of the time, which in production usually means a downstream API that suddenly got slow. I didn’t force that on AWS: whether two messages end up in the same batch is up to the SQS poller, so the run would mostly be testing AWS, not the code. Unit tests cover the guard instead. They fake a deadline, check that a charge without enough time is refused before it starts, and check that the redelivery charges exactly once.

What it doesn’t solve

Idempotency isn’t magic, and a few things remain your job:

  • Choosing the key. If two duplicate events share no identifier, nothing can match them. Producers should include a business ID.
  • A crash halfway through. If the Lambda charges the card and dies before saving, the retry will charge again once the in-progress record expires. No external lock can close that gap. Pass your key downstream when the API supports it (Stripe accepts an Idempotency-Key header) so it deduplicates on its side too.
  • Ordering. Idempotency stops duplicates; it doesn’t stop events arriving out of order.

How I built it (with AI)

Same approach as always: I steer, the clanker types. AI wrote most of the code, but the interesting moments were the back-and-forth. Asking “can’t the user just see the error?” changed the API for the better. Pushing back on “you need to test it in real AWS to get a copy-paste example” exposed that two unrelated ideas had been lumped together. And two rounds of AI code review found real bugs: the takeover race, a cancelled context leaving keys blocked for an hour, and the timeout check that could never fire.

My takeaway is the same as with my previous projects: use the tool, but stay in the driver’s seat, and question everything that sounds confident.

Try it

The code is on GitHub: nixmaldonado/lambda-kit-go.

go get github.com/nixmaldonado/lambda-kit-go@v0.1.0

If you want to see it working end to end, the SQS orders example deploys with make deploy, runs the test with make test, and cleans up with make destroy.

It’s early: v0.1.0, and the API may still change before v1. Next up is probably a metrics package. Issues and feedback are very welcome.