The Retry You Never Wrote

In our .NET 10 lab, a simulated payment took twelve seconds. The application timed out after thirty. Our code made one call and caught one exception, but the server recorded three charges.

The third charge came through about six seconds after the application reported the failure.

We hadn't written a retry loop. The retries came from a single line of configuration, the sort a template adds and you never look at again. If your .NET services call slow endpoints, you may already have that line too.

We ran every scenario at least twice and got the same counts. Here's what we measured, how a default setting turned one call into three charges, and why the documented fix addresses only half the problem.

The line

builder.Services.ConfigureHttpClientDefaults(http =>
{
    // Turn on resilience by default
    http.AddStandardResilienceHandler();
    // ...
});

This is from the Service Defaults project generated by the Aspire template. In an app that calls AddServiceDefaults(), every HttpClient created through IHttpClientFactory gets this handler. You can add the same call to an individual client yourself; the defaults are the same.

Three defaults matter here:

The handler treats an attempt timeout as a transient failure and retries the request. That applies to every HTTP method, including POST. Microsoft documents the risk: "if a POST request inserts a new record to a database, then making retries for such a request could lead to data duplication."

The behaviour is documented, but it is easy to miss when the code making the request says nothing about retries.

Here is what happened to our twelve-second payment:

The handler allows a fourth attempt, but the 30-second budget ran out before it could start.

The client cancelled each attempt, but our endpoint kept working. Once the call to the payment provider is in flight, the server can still finish the charge. A client timeout means the client stopped waiting.

Four scenarios

We ran the same twelve-second payment in four scenarios, changing one thing at a time: either how the server handles the caller going away or how the handler is configured.

When the server stops, nothing is charged

The second scenario looks safe: three attempts, no charges, and an error that matches what happened. Our endpoint passed HttpContext.RequestAborted to its work, so each cancelled attempt stopped before making the charge. A real endpoint can only do this until it calls the payment provider. Once that call has gone out, cancelling the request does not undo it: the provider may or may not have charged, and the outcome stays unknown until you ask the provider.

The documented fix solves half the problem

Microsoft's fix is one line:

http.AddStandardResilienceHandler(options =>
{
    options.Retry.DisableForUnsafeHttpMethods();
});

That removes the duplicates: one attempt, one charge. But the caller still gets TimeoutRejectedException, now at 10 seconds, while the payment completes at 12. The application says the payment failed even though it went through. When our application tried again, as a user pressing Pay again would, it got a second charge and a second error. We come back to that below.

Retries multiplied the charges. The attempt timeout is what turned a slow payment into one with an unknown outcome: the handler gives up on any attempt that has no response after ten seconds, even if the server is still working. In the fourth scenario, we raised the attempt timeout to 20 seconds and got one charge and an HTTP 200. We also had to raise the circuit breaker's sampling duration to 60 seconds: the handler rejects a sampling duration shorter than twice the attempt timeout. Twenty seconds gives our twelve-second payment enough time to finish. For your own calls, set it from their measured latency and from how long the caller can afford to wait. A longer timeout makes the unknown outcome rarer; only the server can rule it out.

What the logs show

Our code made one call and caught one exception. The logs told a fuller story. With the logging levels from the ASP.NET Core template, each of the two retries in the first scenario left this sequence in the log (shortened):

fail: Polly[0] Resilience event occurred. EventName: 'OnTimeout', Source: 'defaults-standard//Standard-AttemptTimeout'
warn: Polly[0] Resilience event occurred. EventName: 'OnRetry', Source: 'defaults-standard//Standard-Retry'
info: System.Net.Http.HttpClient.defaults.ClientHandler[100] Sending HTTP request POST http://127.0.0.1:5099/pay/ignores-cancel

You can spot the retry if you look for it: an OnRetry next to a POST. In the third scenario, the log has one OnTimeout and no OnRetry. It gives no indication that the payment is still running.

Turning off retries prevents the extra attempts, but the client still doesn't know what happened to the payment. To find out, it needs help from the server.

The key that makes the retry safe

When a payment times out, the client cannot tell whether it went through. The server can recognise a second attempt if the client sends an idempotency key: a value created once per payment and sent with every attempt, usually in an Idempotency-Key header.

The key belongs to the payment, not to the HTTP call. Create it when the payment is created, for example with the order, and keep it until the payment is settled. A user who presses Pay again must send the same key; a fresh one makes the second click a second payment.

On the client, with an HttpClient that has a BaseAddress:

var key = order.PaymentKey; // created once with the order, e.g. Guid.NewGuid().ToString("N")
using var request = new HttpRequestMessage(HttpMethod.Post, "/pay");
request.Headers.Add("Idempotency-Key", key);
using var response = await http.SendAsync(request);

Each retry inside the handler carries the same headers, so every attempt sends the same key. If your application retries on its own, build a new HttpRequestMessage for each call, since a sent message cannot be sent again, and give it the same key.

Here is the server code, simplified from our lab:

using System.Collections.Concurrent;

var payments = new ConcurrentDictionary<string, Lazy<Task>>();

app.MapPost("/pay", async (HttpRequest req) =>
{
    var key = req.Headers["Idempotency-Key"].ToString();
    if (string.IsNullOrEmpty(key)) return Results.BadRequest();

    // The first request with a key starts the payment. A repeat waits for that same payment.
    var payment = payments.GetOrAdd(key, _ => new Lazy<Task>(() => Task.Run(ChargeAsync)));
    await payment.Value;
    return Results.Ok(new { key, status = "charged" });
});

static async Task ChargeAsync() => await Task.Delay(TimeSpan.FromSeconds(12)); // the call to the payment provider

We ran three more scenarios with the same twelve-second payment:

With the defaults, the second attempt reached the server at 12.1 seconds. It found the payment already running and waited for it to finish. The application received HTTP 200 at 12.4 seconds, and the customer paid once. The key made the template's retry useful: the call succeeded after the timeout.

The second row shows what happens when the user presses Pay again. With automatic retries disabled and no key, our application tried once more after the timeout. That produced a second charge and a second error. In the third row, the application made the same retry but included the key. The server recognised the payment and returned its result. The customer was charged once.

What our lab version leaves out

• The keys are held in memory on one instance. In production, claim the key atomically before any work starts (a unique constraint in the database does this), so two instances cannot both start the payment. Record the payment's state durably: started, charged, failed.
• Pass the same key to the payment provider if it accepts one. Without that, a crash between the provider's charge and your own write leaves a payment only reconciliation can resolve.
• A failed payment stays cached here, and every repeat with its key gets the same failure. Production code has to tell a payment that definitely failed from one whose outcome is unknown, and treat them differently.
• Here, a repeat request waits if the payment is still running. Some APIs return 409 Conflict instead, leaving the client to try again later. In either case, a repeat must never start a second payment.
• Store a fingerprint of the request alongside the key. Reject requests that reuse the key with a different body.
• Keep keys for longer than any retry, by the handler or by a person, can arrive. A retry after the key has expired is a new payment.
• The key prevents the second charge. It does not guarantee the caller a response: every attempt can still time out. Keep such a payment pending and look up its status with the same key.

Check your own services

1. Search for AddStandardResilienceHandler, ConfigureHttpClientDefaults and AddServiceDefaults(). Every application that calls AddServiceDefaults() gives the handler to every HttpClient created by the factory, unless a client removes it with RemoveAllResilienceHandlers().
2. List the POST, PUT, PATCH and DELETE calls those clients make. For each call, ask what happens if it runs twice.
3. Check how long each call can reasonably take against the 10-second attempt timeout. Calls that take longer are cancelled and, with the defaults, sent again.
4. For calls that must not run twice, send an idempotency key and have the receiving server honour it. If you do not own the server, check whether it supports one.
5. Look for OnRetry alongside a POST in the logs. That is where you will see this happening.

What we measured

• With the defaults and a slow POST: three charges, but an error for the caller.
• With retries off: one charge, but the caller still got an error even though the payment went through.
• With an idempotency key: one charge and a successful response, with retries on or off.

The standard handler's defaults make sense for a GET that can safely run twice. For payments, your team should decide explicitly whether those defaults fit.

How we ran this

We used .NET 10.0.14 and Microsoft.Extensions.Http.Resilience 10.11.0 (Polly 8.4.2) in one ASP.NET Core process on localhost. The simulated payment endpoint takes 12 seconds. The runs quoted here had logging at the ASP.NET Core template's levels.

We ran the first four scenarios four times and the idempotency scenarios twice. The counts were the same every time. The timings in this article come from one run of each set. Between runs they moved by up to about two seconds, mostly because the retry delay includes jitter.