JetStream WorkQueue consumer behaving ambiguous (droping, delay picking messages)

[zakk616]

Observed behavior

We are using nats-server v2.10.16, a cluster of three nodes configured.

We have a workqueue stream that has multiple subjects and consumers below:

We have seen messages being dropped by nats and consumer are not picking those messages, also in some cases a delay of exactly "AckWait" duration has been noticed while messages being picked by the consumer.
Ex: messages from below consumers are being picked after 10 minutes which is its AckWait.

Expected behavior

Message count of published and picked should be the same, while deference has been noticed in numbers.

Subject	Publishsed	Picked	Lost
OTHER_TRANS.ICM_REQ	27	25	2
OTHER_TRANS.NRT_SWIFT	9	9	0
OTHER_TRANS.NRT_DC_DATABASE	1	1	0
OTHER_TRANS.DNS_SWIFT	928	911	17

Server and client version

Server: 2.10.16
Client: nats.net client for C# v4.0.30319

Host environment

OS: Ubuntu 22.04.1 LTS

Steps to reproduce

No response

[zakk616]

Issue is similar to #5638, #5612 and #5705 which were likely fixed in #5639 and released with 2.10.19.
We upgraded 2.10.16 to 2.10.19. so far we couldn't reproduce this issue.

[derekcollison]

We suggest upgrading to 2.10.20, we had a small regression on R1 KV CAS operations.

[zakk616]

Upgrading to 2.10.20 would require repeating much of the effort we’ve already invested in upgrading to 2.10.19. Additionally, we are currently running R3.

[derekcollison]

Only one bug fix in 2.10.20 vs 2.10.19, so drop in replacement..