Malekbenz

Hi, I'm MalekBenz. I author this blog, I'm FullStack Developer, create courses & love learning, writing, teaching technology. ( Dotnet , Javascript, ASP.NET , NodeJS, SQL Server )

Messaging with RabbitMQ in .NET - Queues, exchanges, acks and dead letters

In June we sent order confirmations from a background service fed by an in-memory Channel. It made the order request fast — and it lost every queued email on each deployment, because a Channel lives in the app’s memory. For work that must happen, you need a queue that lives outside the app: a message broker. RabbitMQ is one of the most used, it’s free, and I’ve used it on both Node.js and .NET projects. Let’s build the producer and the consumer in .NET, and spend most of the time on what makes it reliable: durable queues, acknowledgements, idempotency, retries and dead letters.

The core idea: producer → exchange → queue → consumer

 Order API                 RabbitMQ                              Workers
┌──────────┐   publish   ┌──────────┐  routing   ┌──────────────┐   consume   ┌────────────┐
│ producer │ ──────────► │ exchange │ ─────────► │ queue: emails │ ──────────► │ consumer 1 │
└──────────┘ "order.     └──────────┘            └──────────────┘ ──────────► │ consumer 2 │
             created"          │                 ┌──────────────┐             └────────────┘
                               └───────────────► │ queue: stock  │ ──► stock worker
                                                 └──────────────┘
  • The producer never sends to a queue directly: it publishes to an exchange, with a routing key (order.created).
  • The exchange copies the message to every queue bound to it with a matching key.
  • Consumers read from queues. Several consumers on one queue share the work.
  • Messages wait in the queue — on disk if durable — until a consumer confirms it processed them. The API and the workers don’t even need to be running at the same time.

A broker turns “call this service now and hope it’s up” into “leave a message, it will be processed”.


Exchanges: direct, fanout, topic

Type Routing Example
direct Exact routing key match invoice.generate → the invoices queue
fanout To every bound queue, key ignored “cache invalidated” → all instances
topic Pattern match on dotted keys: * = one word, # = zero or more order.* gets order.created and order.cancelled

A topic exchange is the most flexible default: the emails queue binds to order.created, the stock queue to order.*, an audit queue to #.


Connection and channel

dotnet add package RabbitMQ.Client
var factory = new ConnectionFactory
{
    HostName = "localhost",
    UserName = "app",
    Password = builder.Configuration["RabbitMq:Password"],
    DispatchConsumersAsync = true,           // needed for async consumers
};

using var connection = factory.CreateConnection("order-api");   // a TCP connection: expensive
using var channel = connection.CreateModel();                   // a lightweight "virtual connection"
Object Cost Share it?
IConnection A real TCP connection, slow to open ✅ One per application, as a singleton
IModel (channel) Cheap, lives inside the connection ❌ Not thread-safe — one per thread / consumer

Opening a connection per message is the RabbitMQ version of the new HttpClient() mistake. Register the connection once (singleton) and create channels where needed.

Declaring the topology

channel.ExchangeDeclare("orders", ExchangeType.Topic, durable: true);

channel.QueueDeclare("emails", durable: true, exclusive: false, autoDelete: false, arguments: null);
channel.QueueBind("emails", exchange: "orders", routingKey: "order.created");

Declarations are idempotent: declaring an existing exchange or queue with the same settings does nothing, so both the producer and the consumer can declare what they need at startup. (Declaring it with different settings fails — change settings with a new queue name.)


Publishing reliably

public sealed record OrderCreated(Guid MessageId, int OrderId, string CustomerEmail, decimal Total);

public sealed class OrderEventsPublisher
{
    private readonly IConnection _connection;
    public OrderEventsPublisher(IConnection connection) => _connection = connection;

    public void PublishOrderCreated(OrderCreated message)
    {
        using var channel = _connection.CreateModel();
        channel.ConfirmSelect();                                     // enable publisher confirms

        var props = channel.CreateBasicProperties();
        props.Persistent = true;                                     // written to disk
        props.ContentType = "application/json";
        props.MessageId = message.MessageId.ToString();

        var body = JsonSerializer.SerializeToUtf8Bytes(message);
        channel.BasicPublish(exchange: "orders", routingKey: "order.created", basicProperties: props, body: body);

        channel.WaitForConfirmsOrDie(TimeSpan.FromSeconds(5));       // the broker has it, or throw
    }
}

Three settings make a message survive a broker restart:

durable exchange + durable queue + persistent message   → survives a RabbitMQ restart
publisher confirms                                       → you KNOW the broker stored it

Without confirms, BasicPublish returns immediately — if the broker is restarting at that moment, the message is lost and you don’t know.

(For production clusters, RabbitMQ’s quorum queues — arguments: { ["x-queue-type"] = "quorum" } — replicate the queue across nodes. Same code otherwise.)


Consuming in a BackgroundService

public sealed class EmailConsumer : BackgroundService
{
    private readonly IConnection _connection;
    private readonly IServiceScopeFactory _scopeFactory;
    private readonly ILogger<EmailConsumer> _logger;
    private IModel? _channel;

    public EmailConsumer(IConnection connection, IServiceScopeFactory scopeFactory, ILogger<EmailConsumer> logger)
    {
        _connection = connection;
        _scopeFactory = scopeFactory;
        _logger = logger;
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _channel = _connection.CreateModel();
        _channel.BasicQos(prefetchSize: 0, prefetchCount: 10, global: false);   // max 10 unacked messages

        var consumer = new AsyncEventingBasicConsumer(_channel);
        consumer.Received += async (_, ea) =>
        {
            try
            {
                var message = JsonSerializer.Deserialize<OrderCreated>(ea.Body.Span)!;

                await using var scope = _scopeFactory.CreateAsyncScope();
                var sender = scope.ServiceProvider.GetRequiredService<IConfirmationSender>();
                await sender.SendAsync(message, stoppingToken);

                _channel.BasicAck(ea.DeliveryTag, multiple: false);            // done: remove from the queue
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Failed to process message {MessageId}", ea.BasicProperties.MessageId);
                _channel.BasicNack(ea.DeliveryTag, multiple: false, requeue: false);   // → dead letter (below)
            }
        };

        _channel.BasicConsume(queue: "emails", autoAck: false, consumer: consumer);
        return Task.CompletedTask;
    }

    public override void Dispose()
    {
        _channel?.Close();
        base.Dispose();
    }
}

The same patterns as in the background services post: one scope per message, a try/catch around each message.


Acknowledgements: at-least-once delivery

autoAck: false is the most important argument in this post. It means: the message stays in the queue until I say I’m done.

autoAck: true   → RabbitMQ deletes the message as soon as it's delivered
                  → the consumer crashes while sending the email → message LOST
autoAck: false  → deleted only after BasicAck
                  → the consumer crashes → the message is delivered again to another consumer
Call Meaning
BasicAck(tag) Processed — delete it
BasicNack(tag, requeue: true) Failed — put it back in the queue (immediately!)
BasicNack(tag, requeue: false) Failed — drop it, or send it to the dead letter exchange if configured

Be careful with requeue: true on a message that always fails (bad data): it comes back instantly, fails again, forever — a “poison message” loop at 100% CPU.


Idempotent consumers

With manual acks, RabbitMQ guarantees at-least-once delivery — not exactly-once. If the consumer sends the email and crashes before BasicAck, the message is delivered again: the customer gets two emails.

The consumer must be idempotent: processing the same message twice has the same effect as once. The usual way: remember the message ids you’ve processed.

// Inside the handler, in the same transaction as the work when possible
if (await _db.ProcessedMessages.AnyAsync(m => m.Id == message.MessageId, ct))
    return;                                                    // already done: just ack it

await DoTheWorkAsync(message, ct);
_db.ProcessedMessages.Add(new ProcessedMessage { Id = message.MessageId, ProcessedAt = DateTime.UtcNow });
await _db.SaveChangesAsync(ct);

That’s why the publisher sets a MessageId — a unique id generated once per business event.


Prefetch and competing consumers

BasicQos(prefetchCount: 10) limits how many unacknowledged messages a consumer holds. Without it, RabbitMQ pushes the whole queue to the first consumer that connects — the others stay idle, and the first one holds thousands of messages in memory.

emails queue ──┬──► consumer A (10 in flight)
               ├──► consumer B (10 in flight)      ← "competing consumers": the work is shared
               └──► consumer C (10 in flight)

To process more, run more consumers (more instances of the worker) — no code change. Lower the prefetch for slow messages (1–5), raise it for fast ones (50–100).


Failures: retries and dead letter queues

A failed message shouldn’t disappear, and shouldn’t loop forever. Configure the queue with a dead letter exchange: every message nacked with requeue: false (or expired) goes there.

channel.ExchangeDeclare("orders.dlx", ExchangeType.Fanout, durable: true);
channel.QueueDeclare("emails.dead", durable: true, exclusive: false, autoDelete: false, arguments: null);
channel.QueueBind("emails.dead", "orders.dlx", routingKey: "");

channel.QueueDeclare("emails", durable: true, exclusive: false, autoDelete: false,
    arguments: new Dictionary<string, object> { ["x-dead-letter-exchange"] = "orders.dlx" });
emails ──(nack, requeue: false)──► orders.dlx ──► emails.dead   ← inspect, fix, replay later

For retries with delay (the mail server is down for 2 minutes), a common trick uses the same mechanism: nack into a “retry” queue that has a message TTL and dead-letters back into the main exchange:

emails ──fail──► emails.retry (x-message-ttl: 30000, x-dead-letter-exchange: orders)
                     └── after 30 s ──► back to emails

Count attempts with a header (or RabbitMQ’s x-death header) and send to the dead letter queue after N tries. Monitor the dead letter queue: messages there are work that didn’t happen.


Saving to the database and publishing

One more trap, on the producer side:

await _db.SaveChangesAsync();                // ✅ order saved
_publisher.PublishOrderCreated(message);     // 💥 broker unreachable → no email, ever

Or the other order: published, then the database save fails → an email for an order that doesn’t exist. The two operations can’t be in one transaction. The robust solution is the outbox pattern: save the message in an Outbox table in the same transaction as the order, and let a background job publish pending outbox rows (and mark them sent). Nothing is lost, and with idempotent consumers, publishing twice is harmless.


Libraries on top

RabbitMQ.Client is low-level on purpose: you saw connections, channels, acks and topology. Libraries like MassTransit or NServiceBus wrap all of it — consumers as classes, retries, dead letters, outbox, serialization — behind a simple API. Learn the raw concepts first (they explain every behavior of those libraries), then use a library for real projects if the team prefers.


Key rules to remember

Rule Why
Publish to exchanges, consume from queues Producers don’t know who listens
topic exchanges for flexible routing order.*, #
One connection per app, one channel per thread/consumer Connections are expensive, channels aren’t thread-safe
Durable exchange + durable queue + persistent message + confirms Messages survive restarts, and you know they’re stored
autoAck: false, ack after the work is done A crash doesn’t lose the message
Consumers must be idempotent (MessageId) At-least-once means duplicates happen
Set a prefetch (BasicQos) Fair work sharing, bounded memory
Never requeue: true blindly Poison messages loop forever
Dead letter exchange + monitored DLQ, delayed retries via TTL Failures are kept, visible and retryable
Outbox table for “save + publish” No lost or phantom messages

That’s it & see you soon!.

Comments