Fix issue where query get interrupted during prepare. #973

khanaffan · 2025-01-22T22:05:50Z

Do not interrupt the query during preparation. The query can only be interrupted during the step() function.
Add an option to prepare ECSQL using a thread database connection. Experimental.
- Default to false. Not enabled.
- Thread does not wait to acquire lock on primary connection to prepare statement. Enhance faster.
- Side effect more memory as each connection load its on EC Schema when preparing ECSQL.
Query statistics now also return preparation time in nanoseconds.
Introduce a feature to automatically shut down concurrent queries when idle for a set time:
- Defaults to 30 minutes.
- Closes all threads and releases resources used by concurrent queries.
- Automatically wakes up when a new request comes in.
- New error code is added ShuttingDown which cause frontend to retry.
Allow setting the memory map I/O size for each connection, defaulting to 0 (not used):
- This is an experimental option. It improves query performance on regular files but is untested with cloud SQLite.
The statement cache size per thread can now be configured, with the default remaining at 40 (unchanged from previous).
Fix a issue where delay use for testing cause every query to delay at least once even when not testing.
Added allowTestingArgs help simulate a condition for testing.
Found another issue which cause thread dead lock during closing connection. ConcurrentQuery config is not stored on db connection anymore.
The default config look as following. Only new change is autoShutdowWhenIdealForSeconds: 1800

{
  workerThreads: 4,
  requestQueueSize: 2000,
  ignorePriority: false,
  ignoreDelay: true,
  doNotUsePrimaryConnToPrepare: false,
  autoShutdowWhenIdlelForSeconds: 1800,
  statementCacheSizePerWorker: 40,
  monitorPollInterval: 1000,
  memoryMapFileSize: 0,
  allowTestingArgs: false,
  globalQuota: { time: 60, memory: 8388608 }
}

iModelCore/ECDb/ECDb/ConcurrentQueryManagerImpl.cpp

…fank/ccq-fix

khanaffan · 2025-01-23T21:51:14Z

/azp run imodel-native

azure-pipelines · 2025-01-23T21:51:29Z

Azure Pipelines successfully started running 1 pipeline(s).

khanaffan · 2025-01-24T19:05:14Z

/azp run imodel-native

azure-pipelines · 2025-01-24T19:05:27Z

Azure Pipelines successfully started running 1 pipeline(s).

khanaffan · 2025-01-27T15:28:25Z

/azp run imodel-native

azure-pipelines · 2025-01-27T15:28:39Z

Azure Pipelines successfully started running 1 pipeline(s).

khanaffan · 2025-01-29T15:55:49Z

/azp run imodel-native

azure-pipelines · 2025-01-29T15:56:04Z

Azure Pipelines successfully started running 1 pipeline(s).

…connection.

khanaffan · 2025-02-04T14:36:07Z

@Mergifyio backport release/4.11.x release/4.10.x

mergify · 2025-02-04T14:36:16Z

backport release/4.11.x release/4.10.x

✅ Backports have been created

#988 Fix issue where query get interrupted during prepare. (backport #973) [release/4.11.x] has been created for branch release/4.11.x but encountered conflicts
#989 Fix issue where query get interrupted during prepare. (backport #973) [release/4.10.x] has been created for branch release/4.10.x but encountered conflicts

* do not interrupt query during prepare * fix * add debug arg to simulate interrupt * fix * fix a thread lockup when reading config. Config is no more stored on connection. * update api --------- Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com> (cherry picked from commit d5e7430) # Conflicts: # iModelCore/ECDb/ECDb/ConcurrentQueryManagerImpl.cpp # iModelCore/ECDb/Tests/NonPublished/ConcurrentQueryTest_V2.cpp # iModelJsNodeAddon/IModelJsNative.cpp

* do not interrupt query during prepare * fix * add debug arg to simulate interrupt * fix * fix a thread lockup when reading config. Config is no more stored on connection. * update api --------- Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com>

khanaffan · 2025-02-04T19:31:43Z

@Mergifyio backport release/4.10.x release/4.11.x

mergify · 2025-02-04T19:31:54Z

backport release/4.10.x release/4.11.x

❌ No backport have been created

#989 Fix issue where query get interrupted during prepare. (backport #973) [release/4.10.x] has been created for branch release/4.10.x
Backport to branch `` failed

GitHub error: Branch not found

#988 Fix issue where query get interrupted during prepare. (backport #973) [release/4.11.x] has been created for branch release/4.11.x

… [release/4.10.x] (#989) * Fix issue where query get interrupted during prepare. (#973) * do not interrupt query during prepare * fix * add debug arg to simulate interrupt * fix * fix a thread lockup when reading config. Config is no more stored on connection. * update api --------- Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com> (cherry picked from commit d5e7430) # Conflicts: # iModelCore/ECDb/ECDb/ConcurrentQueryManagerImpl.cpp # iModelCore/ECDb/Tests/NonPublished/ConcurrentQueryTest_V2.cpp # iModelJsNodeAddon/IModelJsNative.cpp * Fix issue where query get interrupted during prepare. (#973) * do not interrupt query during prepare * fix * add debug arg to simulate interrupt * fix * fix a thread lockup when reading config. Config is no more stored on connection. * update api --------- Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com> * unused --------- Co-authored-by: affank <khanaffan@gmail.com> Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com>

… [release/4.11.x] (#988) * Fix issue where query get interrupted during prepare. (#973) * do not interrupt query during prepare * fix * add debug arg to simulate interrupt * fix * fix a thread lockup when reading config. Config is no more stored on connection. * update api --------- Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com> (cherry picked from commit d5e7430) # Conflicts: # iModelCore/ECDb/ECDb/ConcurrentQueryManagerImpl.cpp # iModelCore/ECDb/Tests/NonPublished/ConcurrentQueryTest_V2.cpp # iModelJsNodeAddon/IModelJsNative.cpp * Fix issue where query get interrupted during prepare. (#973) * do not interrupt query during prepare * fix * add debug arg to simulate interrupt * fix * fix a thread lockup when reading config. Config is no more stored on connection. * update api --------- Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com> * fix --------- Co-authored-by: affank <khanaffan@gmail.com> Co-authored-by: Affan Khan <khanaffan@users.noreply.github.com>

khanaffan added 2 commits January 17, 2025 14:26

do not interrupt query during prepare

3a64f10

fix

29c51cc

khanaffan mentioned this pull request Jan 22, 2025

Fix issue where query get interrupted during prepare. iTwin/itwinjs-core#7592

Merged

rschili reviewed Jan 23, 2025

View reviewed changes

iModelCore/ECDb/ECDb/ConcurrentQueryManagerImpl.cpp Show resolved Hide resolved

rschili reviewed Jan 23, 2025

View reviewed changes

iModelCore/ECDb/ECDb/ConcurrentQueryManagerImpl.cpp Show resolved Hide resolved

khanaffan added 2 commits January 23, 2025 15:24

add debug arg to simulate interrupt

e368d46

Merge branch 'main' of https://github.com/iTwin/imodel-native into af…

b86e293

…fank/ccq-fix

khanaffan marked this pull request as ready for review January 23, 2025 20:55

khanaffan requested review from pmconne, calebmshafer and ColinKerr as code owners January 23, 2025 20:55

fix

f4b73d9

Merge branch 'main' into affanK/ccq-fix

dd12a16

khanaffan and others added 3 commits January 29, 2025 16:51

Merge branch 'main' into affanK/ccq-fix

b85cdc6

Merge branch 'main' into affanK/ccq-fix

5f31932

fix a thread lockup when reading config. Config is no more stored on …

7ec9358

…connection.

khanaffan requested a review from bbastings as a code owner January 31, 2025 16:44

update api

4029dca

ColinKerr approved these changes Jan 31, 2025

View reviewed changes

calebmshafer approved these changes Jan 31, 2025

View reviewed changes

khanaffan merged commit d5e7430 into main Jan 31, 2025
19 checks passed

khanaffan deleted the affanK/ccq-fix branch January 31, 2025 20:19

This was referenced Feb 4, 2025

Fix issue where query get interrupted during prepare. (backport #7592) [release/4.10.x] iTwin/itwinjs-core#7654

Merged

Fix issue where query get interrupted during prepare. (backport #7592) [release/4.11.x] iTwin/itwinjs-core#7655

Merged

This was referenced Feb 4, 2025

Fix issue where query get interrupted during prepare. (backport #973) [release/4.11.x] #988

Merged

Fix issue where query get interrupted during prepare. (backport #973) [release/4.10.x] #989

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix issue where query get interrupted during prepare. #973

Fix issue where query get interrupted during prepare. #973

khanaffan commented Jan 22, 2025 •

edited

Loading

khanaffan commented Jan 23, 2025

azure-pipelines bot commented Jan 23, 2025

khanaffan commented Jan 24, 2025

azure-pipelines bot commented Jan 24, 2025

khanaffan commented Jan 27, 2025

azure-pipelines bot commented Jan 27, 2025

khanaffan commented Jan 29, 2025

azure-pipelines bot commented Jan 29, 2025

khanaffan commented Feb 4, 2025

mergify bot commented Feb 4, 2025 •

edited

Loading

khanaffan commented Feb 4, 2025

mergify bot commented Feb 4, 2025

Fix issue where query get interrupted during prepare. #973

Fix issue where query get interrupted during prepare. #973

Conversation

khanaffan commented Jan 22, 2025 • edited Loading

khanaffan commented Jan 23, 2025

azure-pipelines bot commented Jan 23, 2025

khanaffan commented Jan 24, 2025

azure-pipelines bot commented Jan 24, 2025

khanaffan commented Jan 27, 2025

azure-pipelines bot commented Jan 27, 2025

khanaffan commented Jan 29, 2025

azure-pipelines bot commented Jan 29, 2025

khanaffan commented Feb 4, 2025

mergify bot commented Feb 4, 2025 • edited Loading

✅ Backports have been created

khanaffan commented Feb 4, 2025

mergify bot commented Feb 4, 2025

❌ No backport have been created

khanaffan commented Jan 22, 2025 •

edited

Loading

mergify bot commented Feb 4, 2025 •

edited

Loading