Idempotent Receiver in Mule: charging the order only once

This page implements the Idempotent Receiver pattern in Mule 4 with the Idempotent Message Validator, in an example that runs, with MUnit tests that pin down its behavior. It is for developers who already build flows and need to make sure a repeated order is not charged twice.

The order that arrives twice

The book Enterprise Integration Patterns starts from an observation: even when the sender sends a message only once, the receiver may receive it more than once. The pattern’s question is how the receiver deals with these duplicates (Idempotent Receiver).

In the series’ order scenario, the payment consumer is the one that feels this. A client that gets no response tries again, a gateway repeats the call, someone reprocesses messages after an incident. It is always the same order, and it has to be charged only once.

The pattern in the book: remember what has already been processed

The book calls a message idempotent when it produces the same effect whether it is received once or several times, and points to two paths: remove duplicates explicitly, or define the message so that repeating it causes no problem (Idempotent Receiver). The example follows the first: the receiver keeps the ID of each processed message and ignores any message that arrives with a known ID.

The receiver facing the same message twice

The diagram shows what the receiver does when the channel delivers, twice, a message the sender sent once.

In Mule: the validator with the order key

In Mule, the pattern is a core component, the Idempotent Message Validator. It lets through only messages with an ID it has not seen yet, stores the IDs in an object store and throws MULE:DUPLICATE_MESSAGE for repeats. The ID comes from idExpression and the store from objectStore (Idempotent Message Validator).

In the example, the receiver is the POST /eip/payments/after/charges endpoint, which receives { orderId, amount }. In the series scenario, payment consumes an Anypoint MQ queue; the example uses HTTP so it runs without a broker. In both cases the validator sits in the flow, after the source.

The charge flow with the validator

The diagram shows the path of a charge request through each of the flow’s exits.

With a new ID, the flow charges, sends the payment request to the provider (the subject of the Correlation Identifier page) and answers accepted. The charge is simulated: each one becomes an entry, with a new chargeId, in a statement the tests count. With a repeated ID, the Try’s on-error-continue records the duplicate and answers 409. Other errors go to the application’s global error handling, which answers 500.

Before and after: from correlationId to orderId

The “before” endpoint, in idempotent-receiver-before.xml, uses the validator with no configuration:

<try doc:name="Charge once">
	<idempotent-message-validator doc:name="Reject duplicates (default: by correlation ID)" />
	<flow-ref doc:name="Charge" name="payment-charge-order" />
	<set-variable doc:name="Accepted" variableName="chargeResult" value="accepted" />
	<error-handler>
		<on-error-continue doc:name="Duplicate: 409" type="MULE:DUPLICATE_MESSAGE" logException="false">
			<flow-ref doc:name="Record duplicate" name="payment-record-duplicate" />
			<set-variable doc:name="Duplicate" variableName="chargeResult" value="duplicate" />
			<set-variable doc:name="409" variableName="httpStatus" value="409" />
		</on-error-continue>
	</error-handler>
</try>

In the repository: src/main/mule/eip/idempotent-receiver/idempotent-receiver-before.xml

The “after” endpoint, in idempotent-receiver-after.xml, declares a store and uses the order number as the ID. Duplicate handling is the same:

<os:object-store name="eip-processed-orders-store" doc:name="Processed order IDs"
	persistent="false" entryTtl="24" entryTtlUnit="HOURS" expirationInterval="10" expirationIntervalUnit="MINUTES" />
...
<try doc:name="Charge once">
	<idempotent-message-validator doc:name="Reject duplicates by orderId" idExpression="#[payload.orderId]" objectStore="eip-processed-orders-store" />
	<flow-ref doc:name="Charge" name="payment-charge-order" />
	<flow-ref doc:name="Request payment from provider" name="payment-request-from-provider" />
	<set-variable doc:name="Accepted" variableName="chargeResult" value="accepted" />
	<error-handler>
		...
	</error-handler>
</try>

In the repository: src/main/mule/eip/idempotent-receiver/idempotent-receiver-after.xml

Without idExpression, the ID is #[correlationId], and the series article explains why it gets it wrong. The idempotent-receiver-test-suite.xml suite shows both errors and the fix, with Mule 4.9.17 and MUnit 3.7.1:

Situation	Before: ID = correlationId	After: ID = orderId
Three different orders sent over HTTP from inside a For Each	accepted,duplicate,duplicate (idempotent-before-for-each-over-http-rejects-different-orders)	accepted,accepted,accepted (idempotent-after-for-each-over-http-accepts-different-orders)
The same order in two requests with different X-CORRELATION-ID values	Charged twice (idempotent-before-same-order-from-two-client-requests-is-charged-twice)	Charged once; the second request comes back as duplicate (idempotent-after-same-order-from-two-client-requests-is-charged-once)

The first row uses the batch client in payment-common.xml, a For Each that calls the endpoint once per order. Every call goes out with the same ID in the X-CORRELATION-ID header, the batch’s correlationId, and the listener adopts that ID. In the second row, the test simulates a client retrying by setting the ID on the HTTP Request:

<http:request doc:name="Client request CLIENT-TRY-2" config-ref="eip-self-http-request-config" method="POST" path="/eip/payments/before/charges" sendCorrelationId="ALWAYS" correlationId="ORD-IR-B-100-CLIENT-TRY-2" target="second">

In the repository: src/test/munit/eip/idempotent-receiver-test-suite.xml

The default ID blocks a duplicate in only one case: the same request repeated with the same X-CORRELATION-ID is blocked the second time (idempotent-before-same-request-repeated-is-rejected). But that depends on the client resending the header.

The trap: the ID stays stored when the charge fails

The validator stores the ID when it accepts the message, before the rest of the flow runs, and does not remove it if the flow fails afterwards. The documentation does not mention this; the behavior was observed on Mule 4.9.17 and is pinned down by the test idempotent-after-failed-charge-keeps-the-id-so-the-retry-is-a-duplicate, which makes storing the charge fail with OS:STORE_NOT_AVAILABLE, after the validator has already accepted the order, and then tries again:

<munit-tools:assert-that doc:name="First attempt failed with 500" expression="#[vars.firstStatus as Number]" is="#[MunitTools::equalTo(500)]" />
<munit-tools:assert-that doc:name="orderId still stored" expression="#[vars.idStoredAfterFailure]" is="#[MunitTools::equalTo(true)]" />
<munit-tools:assert-that doc:name="Retry answered 409" expression="#[vars.retryStatus as Number]" is="#[MunitTools::equalTo(409)]" />
<munit-tools:assert-that doc:name="Retry is a duplicate" expression="#[vars.retryResult]" is="#[MunitTools::equalTo('duplicate')]" />
<munit-tools:assert-that doc:name="Charges" expression="#[payload.charges]" is="#[MunitTools::equalTo(0)]" />
<munit-tools:assert-that doc:name="Duplicates rejected" expression="#[payload.duplicatesRejected]" is="#[MunitTools::equalTo(1)]" />

In the repository: src/test/munit/eip/idempotent-receiver-test-suite.xml

The first attempt answers 500, the orderId stays in the store, the retry gets 409 and the statement ends with zero charges: the order is never charged. It is a quieter error than a double charge, because the retry got back a 409, duplicate.

By the same rule, a failure in the request to the provider, after the charge has succeeded, also leaves the ID stored; the comment in idempotent-receiver-after.xml records this case.

In the series scenario, the trap does not depend on a client that retries. In the AUTO acknowledgment mode of the Anypoint MQ Subscriber, the message is acknowledged (ACK) only if the flow ends successfully; if there is an error, it is not acknowledged (NACK) and goes back to the queue for redelivery (ACK and NACK). Put that together with the rule from the test: the charge fails, the propagated error returns the message to the queue, and the redelivery arrives as a duplicate. The on-error-continue handles MULE:DUPLICATE_MESSAGE and the flow ends as if it had succeeded (On Error Continue), AUTO mode acknowledges the message, and it leaves the queue without a charge. It does not reach the DLQ either, since the DLQ receives a message only after the delivery attempts run out (queues). For the same reason, reprocessing the order after an incident, by moving the message from the DLQ back to the queue or by publishing it again, produces the charge only if the ID has been removed or has already expired. This path is a consequence of the observed rule and the documented behavior of AUTO mode, not a result: the repository runs without a broker and does not test it.

There are ways out, each with a cost to weigh in your scenario. None of them is in the example repository, and none was tested there.

• Remove the ID in error handling, with the Object Store Connector’s os:remove operation. It deletes the key’s value and throws OS:KEY_NOT_FOUND if the key does not exist (Object Store Connector Reference). In the test above, the key the validator stored is the plain orderId. The risk lies in where the failure happened: if the charge had already gone through and the error came from the provider, deleting the ID allows a second charge, exactly what the validator was supposed to prevent.
• Store the ID only after everything has succeeded, replacing the validator with an os:contains at the start and an os:store at the end. Between the check and the store, two copies of the same order can get through together. The Object Store documentation says it has no transactional access and recommends another solution when the same key may be changed in parallel (Object Store Connector).

Choosing the key, the time and the store

The key defines “the same message”. Here it is the orderId: a duplicate is the same order, whichever request it arrives in. The documentation also shows a hash of the payload as the ID (Idempotent Message Validator); with a hash, any field that changes between attempts, such as a send timestamp, makes the duplicate look like a new message. If an order can have more than one legitimate charge, the key has to include what tells them apart.

The time to live has to cover the whole window in which a duplicate can still arrive. Once the ID expires, the same orderId is accepted again. With no store configured, the validator keeps the IDs in memory for 5 minutes and checks for expiry every 6 seconds (Idempotent Message Validator). The example uses 24 hours and explains the choice in the XML: a client or gateway retry takes minutes, and reprocessing after an incident can take hours. The TTL cuts both ways: until the ID expires, it blocks reprocessing of an order already charged and also of an order whose charge failed, because of the trap in the previous section.

Persistence. The example’s store is in memory so that each test run starts clean. A transient store loses its data if the runtime goes down; a persistent one keeps it (Object Store Connector). Outside a Mule cluster, each replica has its own store: a runtime that goes down with an in-memory store forgets the orders already charged, and with two replicas each one accepts the same order once. The documentation warns that idempotency does not work correctly with horizontal scaling when the same order reaches different instances and the object store content is local to each one. In a Mule cluster, object stores, persistent or not, live in memory shared among the nodes, and all of them know the same IDs; that content survives redeploys and restarts as long as the cluster keeps quorum, and is lost only when every node stops (Mule clusters). In production, then, the store has to be persistent and, with more than one replica, shared among them.

On CloudHub, the persistent store can be Object Store v2: with its option enabled at deployment, it replaces the application’s default persistent object store. It keeps data for at most 30 days from creation, which limits the TTL. With several workers, it can show data inconsistencies or key conflicts, because there is no distributed locking with it on CloudHub or CloudHub 2.0; the documentation suggests a distributed key-value store as a lock (Object Store v2). That is the case of two copies of the same order arriving together at different replicas.

When not to use it: the target is already idempotent

The book’s second path makes the validator unnecessary: if the target operation is already idempotent by business key, such as a payment upsert by orderId, the duplicate changes nothing, as long as the operation has no other effects, such as an email sent or an event published. Then the validator only adds a store to size, a TTL to choose and the failure trap. The charge in this example is not like that: each call creates a new charge, and that is why the validator is there.

Running the example and the tests

git clone https://github.com/brunosouzas/mulesoft-patterns-examples.git
cd mulesoft-patterns-examples
mvn clean test

The project needs Java 17, Maven 3.9+ and the Mule 4.9 runtime. MUnit runs on the Enterprise runtime, so Maven needs access to the MuleSoft Enterprise repository (credentials in ~/.m2/settings.xml).

To call the endpoints, run the application in Anypoint Studio (port 8081) and open requests.http in VS Code (REST Client) or IntelliJ. The Idempotent Receiver entries:

### EIP idempotent receiver — same order twice: "before" charges it twice, "after" answers 409 the second time
POST {{base}}/eip/payments/after/charges
Content-Type: application/json

{ "orderId": "ORD-2001", "amount": 100 }

###
POST {{base}}/eip/payments/before/charges
Content-Type: application/json

{ "orderId": "ORD-2002", "amount": 100 }

### EIP idempotent receiver — three different orders from one For Each: before gives accepted,duplicate,duplicate
POST {{base}}/eip/payments/before/charges/batch
Content-Type: application/json

[ { "orderId": "ORD-2101", "amount": 10 }, { "orderId": "ORD-2102", "amount": 20 }, { "orderId": "ORD-2103", "amount": 30 } ]

### ...and after accepts all three
POST {{base}}/eip/payments/after/charges/batch
Content-Type: application/json

[ { "orderId": "ORD-2201", "amount": 10 }, { "orderId": "ORD-2202", "amount": 20 }, { "orderId": "ORD-2203", "amount": 30 } ]

### Charges and rejected duplicates for one order
GET {{base}}/eip/payments/ORD-2001/charges

In the repository: requests.http

Send the first two requests twice each and check each order’s statement with the GET, changing the orderId in the path: it counts the charges and the blocked duplicates.

The other patterns in the series, with the Mule component that implements each one, are in the catalog of integration patterns in Mule.