WhatsApp Had Acknowledged the Message. My Client Kept Waiting.

One of the parts of ExWapp that made me think hardest was ACK handling: the acknowledgements exchanged by WhatsApp's protocol.

It sounds like a low-level detail. A client sends a message, receives an acknowledgement, updates its local status. End of story.

Except real protocols aren't always that tidy.

There's a regression test in the repository that captures the problem rather well: the server had acknowledged a message, but my client couldn't associate that acknowledgement with the stored message. Its status stayed :pending, as though nobody had replied.

A successful send could end up looking like a timeout.

One contact, two identities

WhatsApp uses multiple addressing formats. Among other things, ExWapp has to deal with phone-number identifiers and LIDs, alternative identifiers that aren't the phone number itself.

The problem appears when I store a message under one identifier and the acknowledgement arrives with the other.

The regression fixture contains exactly this situation:

@lid "70000004444@lid"
@pn  "15550004444@s.whatsapp.net"

The message was stored under the LID, but the ACK arrived with the phone-number address.

If I look up the message using only the phone-number JID carried by the ACK, I won't find the chat stored under its LID.

What makes this particularly unpleasant is that a storage update can return :ok without actually changing a message. Everything seems to have processed the ACK correctly, yet the state never moves.

The regression test checks the precise case below. I've left out the surrounding test helpers:

store =
  store_with_pending(@lid, ["MSG_CROSS_FAMILY"])

{:ok, _state} =
  Receipt.handle_ack(
    ack(%{
      "id" => "MSG_CROSS_FAMILY",
      "from" => @pn
    }),
    self(),
    %{store: store}
  )

assert %{status: :sent} =
  Store.find_message(
    store, @lid, "MSG_CROSS_FAMILY"
  )

The lesson isn't simply that WhatsApp has two ID formats. It's that the recipient's identity, the message identifier, and the delivery state are different things.

I can't assume the server will send back exactly the same representation I used internally.

It's a nasty kind of messaging bug because the caller might retry a message that the server already accepted, simply because the first attempt appeared to fail.

Then there's the opposite problem: sending ACKs

ACKs matter on the receiving side, but the client also needs to send protocol acknowledgements of its own.

That led me to a different architectural problem.

ExWapp keeps an outbound command queue. When the connection goes down, commands can wait for the transport to come back.

A normal FIFO sounds perfectly reasonable. The first command enqueued goes out first.

But if there are thousands of ordinary sends waiting, why should a protocol acknowledgement have to sit behind every one of them?

I wanted to serialize transport writes without pretending that all writes were equally urgent.

That's where an Erlang function that many Elixir developers rarely need came in handy: :queue.in_r/2.

queue = :queue.from_list([
  {:message, 1},
  {:message, 2},
  {:message, 3}
])

queue = :queue.in_r({:ack, 42}, queue)

:queue.to_list(queue)

# [
#   {:ack, 42},
#   {:message, 1},
#   {:message, 2},
#   {:message, 3}
# ]

No second broker or distributed queue is required. Ordinary operations join the back; protocol-critical ones go to the front.

The real ExWapp queue is bounded too, with a default capacity of 10,000 items. When it is full, a priority insertion can displace an item at the back and notify the session that the displaced command was dropped.

That's not a particularly cheerful compromise: an application send may fail to make room for protocol traffic.

But the alternatives were an ever-growing queue or ACKs delayed behind a huge backlog.

I didn't like either.

A bounded queue isn't the whole backpressure story

There is an important catch here for anyone using GenServer.

The internal queue is fed through GenServer.cast. Its :ok only means the asynchronous call was issued: it doesn't guarantee the target process is still alive or has processed the command, much less that it entered the bounded queue or reached the WebSocket.

If producers send casts faster than the GenServer can process them, the process mailbox may keep growing even when state.queue has a strict size limit.

Bounding the application queue doesn't automatically bound all incoming traffic.

For sustained load, I'd watch message_queue_len, queue-full errors, and acknowledgement latency. Rate limiting may need to happen before commands are even enqueued.

I'd rather make that limitation explicit than hide it behind :ok.

What does “sent” actually mean?

That word needed a careful definition in the public API too.

A successful send_message call in ExWapp means the encrypted stanza was written by the transport. It doesn't prove the recipient read it, or even that the server has already acknowledged acceptance.

Those are later state transitions delivered through asynchronous events.

Working on this part made me realize how easy it is to model something as ordinary as “send a message” incorrectly.

Writing bytes to a socket isn't enough.

I also have to preserve cryptographic state, resolve identities, keep the information necessary for retries, prioritize protocol control messages, and avoid declaring delivery before it is confirmed.

The hard part of a messaging client isn't sending. It's knowing precisely what happened afterward.