Add Advanced Queueing Support in Thin Mode

[mxdev88]

The documentation specifies that Advanced Queueing (AQ) is only supported in thick mode. The request is to add support to thin mode to be able to use asynchronous operations within the same application, which are not possible in thick mode.

[anthony-tuininga]

Work has been in progress on this front for the past number of weeks. I can't say which version will contain this support yet, but once it is available for testing I will let you know!

[cjbj]

@mxdev88 what specific AQ functionality are you using? E.g. array enq/deq, recipient lists, what kind of payload(s) (RAW, JSON), etc?

[mxdev88]

Initially on the consumer side and using deqone / deqmany with RAW payload.

[mkmoisen]

Hi @anthony-tuininga, @cjbj

I'm also excited to have AQ in thin mode, thank you for working on it.

So far I am using object payloads and RAW payloads. I would love to use JSON payloads but don't have the latest Oracle version that supports this.

I use single consumer queues, single enqueues and single dequeues. No agents, No recipient lists. I think in the future array enqueue and dequeue would be important.

I also set enqueue options and dequeue options.

[mkmoisen]

@anthony-tuininga @cjbj

Would you please explain what the cost is by using AQ in thin mode in the following manner, compared to using thick mode's native AQ support? My guess is that this work around requires twice as much network IO compared to the thick mode's native AQ support.

conn = oracledb.connect(...)
my_custom_type = conn.gettype('MYSCHEMA.MY_CUSTOM_TYPE')

while True:
    cur = con.cursor()
    
    payload = my_custom_type.newobject()
    
    cur.execute(
        '''
            DECLARE
                l_dequeue_options dbms_aq.dequeue_options_t;
                l_message_properties dbms_aq.message_properties_t,
                l_msgid RAW(16);
                l_payload my_custom_type;
            BEGIN
                dbms_aq.dequeue(
                    queue_name => 'MYQUEUE',
                    dequeue_options => l_dequeue_options,
                    payload => l_payload,
                    msgid => l_msgid
                );
                
                :payload := l_payload;
            END;
        ''',
        dict(
            payload=>payload
        )
    )
    
    do_something(payload)

My guess is that this results in twice as much network IO as compared to using native AQ in thick mode.

This thin-mode workaround will need to invoke an IO from the application server to Oracle, and then when Oracle has a new event, it will send it back over the same IO channel, for a total of two IO's per message.

With Native AQ in thick mode, I would guess that a socket is opened when the queue is created. Then the application server can go to sleep listening to the local socket. When Oracle gets a new message, it pipes it over the network to this socket, at which point the application server wakes up and reads it. So a total of one IO per message.

Is that right?

[anthony-tuininga]

I am sure there is some overhead in calling PL/SQL as opposed to using AQ natively, but I don't think it is quite as bad as you think. When you call dequeue(), if there is a message available it will be copied immediately into the output variable. If there is no message available and the mode is to wait for one to be available, then the PL/SQL block will "hang" until such time as the message is available and then copy it into the output variable. A similar thing happens when using AQ natively -- you simply avoid the overhead of using PL/SQL. In both native and PL/SQL the connection is not usable until such time as a message is available. I don't know exactly what happens internally, but that much is certain at least!

If you want to know for sure what the overhead is, you can try with thick mode -- both natively and with PL/SQL and see what the difference in performance is.

[cjbj]

@mkmoisen I don't want to get you too excited but it seems our next release (version 3.0) will have our (initial) Thin mode AQ support, since we just merged into our internal repo. @shivani-ka and her team have been working hard on this, and will continue adding functionality in future versions too. The code is not on GitHub as I type, but could be soon. The release notes say: "Added Oracle Advanced Queuing support for single enqueue and dequeue of RAW and Oracle object payload types." The fine print is that this initial Thin mode support is for classic queues (not transactional event queues). Also JSON and JMS payloads, array message queuing and dequeuing operations, and Recipient Lists are only supported in python-oracledb Thick mode in this version. As to release timing, we have a few non-AQ features close to being ready to land, and a lot of testing to do.

[mkmoisen]

@cjbj very glad to hear it. My application may be able to use it, since it uses classic queues, single dequeues, with oracle object payloads.

Thank you guys!

[mkmoisen]

@cjbj Do you know in which Oracle version "transactional event queues" was released?

The only documentation I can find for it is Oracle 21:

https://docs.oracle.com/en/database/oracle/oracle-database/21/adque/index.html

The Oracle 19 documentation does not refer to "transactional event queues":

https://docs.oracle.com/en/database/oracle/oracle-database/19/adque/index.html

However this blog post claims that transactional event queues were released in 19:

https://www.oracle.com/database/advanced-queuing/

[anthony-tuininga]

I have pushed a patch that partially implements this enhancement (note restrictions above) and have initated a build from which you can download pre-built development wheels once it completes. You can also build from source if you prefer. If you can test your scenario and provide feedback before the release, that would be appreciated!

[mkmoisen]

Hi @anthony-tuininga

How may I download a pre-built dev wheel?

For example under [Build wheels for windows-latest Python-3.13-x86](https://github.com/oracle/python-oracledb/actions/runs/13333204363/job/37242192476#logs) I clicked this link:

Artifact download URL: https://github.com/oracle/python-oracledb/actions/runs/13333204363/artifacts/2593722184

However I'm getting a 404 on github, not sure if it is public or not.

Thanks.

[anthony-tuininga]

Go here: https://github.com/oracle/python-oracledb/actions/runs/13333204363 and then click the "python-oracledb-wheels" link at the bottom under artifacts. This link: https://github.com/oracle/python-oracledb/actions/runs/13333204363/artifacts/2595043763

It is possible that you tried while the builds were ongoing. The separate wheels are built and then combined into a single artifact at the end!

[mkmoisen]

Hi @anthony-tuininga @cjbj

I have installed it on local but I am receiving this error when calling queue.deqone():

File "C:\myapp\process_queue\__init__.py", line 81, in _dequeue
    queue_entry = queue.deqone()
  File "C:\myapp\Lib\site-packages\oracledb\aq.py", line 86, in deqone
    message_impl = self._impl.deq_one()
  File "src\\oracledb\\impl/thin/queue.pyx", line 63, in oracledb.thin_impl.ThinQueueImpl.deq_one
  File "src\\oracledb\\impl/thin/protocol.pyx", line 447, in oracledb.thin_impl.Protocol._process_single_message
  File "src\\oracledb\\impl/thin/protocol.pyx", line 448, in oracledb.thin_impl.Protocol._process_single_message
  File "src\\oracledb\\impl/thin/protocol.pyx", line 440, in oracledb.thin_impl.Protocol._process_message
  File "src\\oracledb\\impl/thin/messages.pyx", line 103, in oracledb.thin_impl.Message._check_and_raise_exception
oracledb.exceptions.DatabaseError: ORA-03146: invalid buffer length for TTC field
Help: https://docs.oracle.com/error-help/db/ora-03146/

I'm using Oracle Database 19.

I'll create a script to minimally reproduce this and report back.

[mkmoisen]

@anthony-tuininga @cjbj

I'm using:

• Oracle 19.25.0.0.0,
• Python 3.13.0 64-bit on Windows
• oracledb-3.0.0b1-cp313-cp313-win_amd64.whl

Here is how I made my queue in Oracle:

CREATE OR REPLACE NONEDITIONABLE TYPE MYQUEUE_TYPE AS OBJECT (
    foo VARCHAR2(128),
    bar VARCHAR2(128)
);
/

BEGIN
    DBMS_AQADM.CREATE_QUEUE_TABLE(
        queue_table => 'MYQUEUE',
        queue_payload_type => 'MYQUEUE_TYPE'
    );

    DBMS_AQADM.CREATE_QUEUE(
        queue_name => 'MYQUEUE',
        queue_table => 'MYQUEUE'
    );

    DBMS_AQADM.START_QUEUE(
        queue_name => 'MYQUEUE'
    );
END;
/

Here is how I am calling in Python:

import oracledb
import sys
import platform

assert sys.version.startswith('3.13.0')
assert platform.architecture() == ('64bit', 'WindowsPE')

assert oracledb.version == '3.0.0b1'
assert oracledb.is_thin_mode()

conn = oracledb.connect(...)
assert conn.version == '19.25.0.0.0'

myqueue_type = conn.gettype('MYQUEUE_TYPE')

queue = conn.queue('MYQUEUE', myqueue_type)

# Error occurs with no wait or wait, just setting this because the queue is empty.
queue.deqoptions.wait = 5

print('attempting dequeue')

# Error is raised here
entry = queue.deqone()

print('dequeued')

Here is the error I'm getting while calling queue.deqone():

>>> entry = queue.deqone()
Traceback (most recent call last):
  File "<python-input-42>", line 1, in <module>
    entry = queue.deqone()
  File "C:\chat-sbp\Lib\site-packages\oracledb\aq.py", line 86, in deqone
    message_impl = self._impl.deq_one()
  File "src\\oracledb\\impl/thin/queue.pyx", line 63, in oracledb.thin_impl.ThinQueueImpl.deq_one
  File "src\\oracledb\\impl/thin/protocol.pyx", line 447, in oracledb.thin_impl.Protocol._process_single_message
  File "src\\oracledb\\impl/thin/protocol.pyx", line 448, in oracledb.thin_impl.Protocol._process_single_message
  File "src\\oracledb\\impl/thin/protocol.pyx", line 440, in oracledb.thin_impl.Protocol._process_message
  File "src\\oracledb\\impl/thin/messages.pyx", line 103, in oracledb.thin_impl.Message._check_and_raise_exception
oracledb.exceptions.DatabaseError: ORA-03146: invalid buffer length for TTC field
Help: https://docs.oracle.com/error-help/db/ora-03146/

It works fine in thick mode.

Thank you.

[shivani-ka]

@mkmoisen I am looking into this and will update once I have findings.

[cjbj]

@mkmoisen this was patched in a092cfc. Thanks for the report.

[mkmoisen]

@cjbj @anthony-tuininga @shivani-ka

I've tried the new patch and it appears to be working smoothly on my local.

Thanks!