You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Long query with massive filter condition takes too long for VTGate(sqlparser.go) to parse the query.
Filter condition containing ts comparison connected with AND clause was repeated over 10k times. and it took over 20,000 seconds to plan the query. This occupied one gorutine at a time, leading to CPU overconsumption.
I've attached the test SQL on below reproduction steps.
When I tried the same query on v14.0.0 it worked without any issue. However, it timed out on 16.0.2 regardless of the planner version.
I tried V3, Gen4 planner each and the result was the same.
It was due to a long query with a large number of filter conditions. When I reduced the number of the AND clause in where condition, query time was shortened.
I'm assuming that the issue was due to the new ast equality api applied on sqlparser which was introduced after v16.0.0. PR link
I cannot distinguish which code caused the issue but I was able to reproduce the symptom.
I've handled this issue by setting --max_payload_size on the VTGate level. However, this might not be an optimal solution for other use cases.
It would be great if we could add some sort of timeout to limit plan time on VTGate level.
Also, it was really hard to find the query since this was logged only after it was executed(after 20,000 seconds).
set up sharded keyspace and run query.
Adding and not (test.user_id = 1 and test.user_id is not null and test.ts >= value and test.ts <= value) impacts the query planning time of vtgate. below took 56s and it has 3992 AND clause. It takes more time when more condition is added.
select*fromsharded.testwheretest.shard_key=1andtest.is_removed=1andtest.cmdin ('A','B','C')
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=1andtest.ts<=2)
---- skipped due to github comment max character limit ----and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11616andtest.ts<=11617)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11618andtest.ts<=11619)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11620andtest.ts<=11621)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11622andtest.ts<=11623)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11624andtest.ts<=11625)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11626andtest.ts<=11627)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11628andtest.ts<=11629)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11630andtest.ts<=11631)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11632andtest.ts<=11633)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11634andtest.ts<=11635)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11636andtest.ts<=11637)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11638andtest.ts<=11639)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11640andtest.ts<=11641)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11642andtest.ts<=11643)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11644andtest.ts<=11645)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11646andtest.ts<=11647)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11648andtest.ts<=11649)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11650andtest.ts<=11651)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11652andtest.ts<=11653)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11654andtest.ts<=11655)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11656andtest.ts<=11657)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11658andtest.ts<=11659)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11660andtest.ts<=11661)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11662andtest.ts<=11663)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11664andtest.ts<=11665)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11666andtest.ts<=11667)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11668andtest.ts<=11669)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11670andtest.ts<=11671)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11672andtest.ts<=11673)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11674andtest.ts<=11675)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11676andtest.ts<=11677)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11678andtest.ts<=11679)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11680andtest.ts<=11681)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11682andtest.ts<=11683)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11684andtest.ts<=11685)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11686andtest.ts<=11687)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11688andtest.ts<=11689)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11690andtest.ts<=11691)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11692andtest.ts<=11693)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11694andtest.ts<=11695)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11696andtest.ts<=11697)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11698andtest.ts<=11699)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11700andtest.ts<=11701)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11702andtest.ts<=11703)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11704andtest.ts<=11705)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11706andtest.ts<=11707)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11708andtest.ts<=11709)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11710andtest.ts<=11711)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11712andtest.ts<=11713)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11714andtest.ts<=11715)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11716andtest.ts<=11717)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11718andtest.ts<=11719)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11720andtest.ts<=11721)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11722andtest.ts<=11723)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11724andtest.ts<=11725)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11726andtest.ts<=11727)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11728andtest.ts<=11729)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11730andtest.ts<=11731)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11732andtest.ts<=11733)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11734andtest.ts<=11735)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11736andtest.ts<=11737)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11738andtest.ts<=11739)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11740andtest.ts<=11741)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11742andtest.ts<=11743)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11744andtest.ts<=11745)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11746andtest.ts<=11747)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11748andtest.ts<=11749)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11750andtest.ts<=11751)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11752andtest.ts<=11753)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11754andtest.ts<=11755)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11756andtest.ts<=11757)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11758andtest.ts<=11759)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11760andtest.ts<=11761)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11762andtest.ts<=11763)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11764andtest.ts<=11765)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11766andtest.ts<=11767)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11768andtest.ts<=11769)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11770andtest.ts<=11771)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11772andtest.ts<=11773)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11774andtest.ts<=11775)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11776andtest.ts<=11777)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11778andtest.ts<=11779)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11780andtest.ts<=11781)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11782andtest.ts<=11783)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11784andtest.ts<=11785)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11786andtest.ts<=11787)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11788andtest.ts<=11789)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11790andtest.ts<=11791)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11792andtest.ts<=11793)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11794andtest.ts<=11795)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11796andtest.ts<=11797)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11798andtest.ts<=11799)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11800andtest.ts<=11801)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11802andtest.ts<=11803)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11804andtest.ts<=11805)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11806andtest.ts<=11807)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11808andtest.ts<=11809)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11810andtest.ts<=11811)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11812andtest.ts<=11813)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11814andtest.ts<=11815)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11816andtest.ts<=11817)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11818andtest.ts<=11819)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11820andtest.ts<=11821)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11822andtest.ts<=11823)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11824andtest.ts<=11825)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11826andtest.ts<=11827)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11828andtest.ts<=11829)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11830andtest.ts<=11831)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11832andtest.ts<=11833)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11834andtest.ts<=11835)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11836andtest.ts<=11837)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11838andtest.ts<=11839)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11840andtest.ts<=11841)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11842andtest.ts<=11843)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11844andtest.ts<=11845)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11846andtest.ts<=11847)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11848andtest.ts<=11849)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11850andtest.ts<=11851)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11852andtest.ts<=11853)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11854andtest.ts<=11855)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11856andtest.ts<=11857)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11858andtest.ts<=11859)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11860andtest.ts<=11861)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11862andtest.ts<=11863)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11864andtest.ts<=11865)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11866andtest.ts<=11867)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11868andtest.ts<=11869)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11870andtest.ts<=11871)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11872andtest.ts<=11873)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11874andtest.ts<=11875)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11876andtest.ts<=11877)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11878andtest.ts<=11879)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11880andtest.ts<=11881)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11882andtest.ts<=11883)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11884andtest.ts<=11885)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11886andtest.ts<=11887)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11888andtest.ts<=11889)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11890andtest.ts<=11891)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11892andtest.ts<=11893)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11894andtest.ts<=11895)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11896andtest.ts<=11897)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11898andtest.ts<=11899)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11900andtest.ts<=11901)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11902andtest.ts<=11903)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11904andtest.ts<=11905)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11906andtest.ts<=11907)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11908andtest.ts<=11909)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11910andtest.ts<=11911)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11912andtest.ts<=11913)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11914andtest.ts<=11915)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11916andtest.ts<=11917)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11918andtest.ts<=11919)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11920andtest.ts<=11921)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11922andtest.ts<=11923)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11924andtest.ts<=11925)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11926andtest.ts<=11927)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11928andtest.ts<=11929)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11930andtest.ts<=11931)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11932andtest.ts<=11933)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11934andtest.ts<=11935)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11936andtest.ts<=11937)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11938andtest.ts<=11939)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11940andtest.ts<=11941)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11942andtest.ts<=11943)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11944andtest.ts<=11945)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11946andtest.ts<=11947)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11948andtest.ts<=11949)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11950andtest.ts<=11951)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11952andtest.ts<=11953)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11954andtest.ts<=11955)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11956andtest.ts<=11957)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11958andtest.ts<=11959)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11960andtest.ts<=11961)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11962andtest.ts<=11963)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11964andtest.ts<=11965)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11966andtest.ts<=11967)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11968andtest.ts<=11969)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11970andtest.ts<=11971)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11972andtest.ts<=11973)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11974andtest.ts<=11975)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11976andtest.ts<=11977)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11978andtest.ts<=11979)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11980andtest.ts<=11981)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11982andtest.ts<=11983)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11984andtest.ts<=11985)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11986andtest.ts<=11987)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11988andtest.ts<=11989)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11990 an
[test_sql.txt](https://github.com/vitessio/vitess/files/12601062/test_sql.txt)
d test.ts<=11991)
and not (test.user_id=1andtest.user_idis not nullandtest.ts>=11992andtest.ts<=11993)
andtest.ts>=113898andtest.parent_id=1order bytest.tsasclimit100;
Binary Version
Version: 16.0.2 (Git revision 6076fed0300de2eb7a17a5f5cd67527c29b5b3c6 branch 'HEAD') built on Wed Jun 7 00:13:43 UTC 2023 by vitess@buildkitsandbox using go1.20.3 linux/arm64
Operating System and Environment details
bash-4.4$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.8 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.8 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.8
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.8"
bash-4.4$ uname -sr
Linux 5.10.186-179.751.amzn2.aarch64
bash-4.4$ uname -m
aarch64
Log Fragments
No response
The text was updated successfully, but these errors were encountered:
Overview of the Issue
Long query with massive filter condition takes too long for VTGate(sqlparser.go) to parse the query.
Filter condition containing
ts
comparison connected withAND
clause was repeated over 10k times. and it took over 20,000 seconds to plan the query. This occupied one gorutine at a time, leading to CPU overconsumption.I've attached the test SQL on below reproduction steps.
When I tried the same query on v14.0.0 it worked without any issue. However, it timed out on 16.0.2 regardless of the planner version.
I tried V3, Gen4 planner each and the result was the same.
It was due to a long query with a large number of filter conditions. When I reduced the number of the
AND
clause in where condition, query time was shortened.I'm assuming that the issue was due to the new ast equality api applied on sqlparser which was introduced after v16.0.0.
PR link
I cannot distinguish which code caused the issue but I was able to reproduce the symptom.
I've handled this issue by setting
--max_payload_size
on the VTGate level. However, this might not be an optimal solution for other use cases.It would be great if we could add some sort of timeout to limit plan time on VTGate level.
Also, it was really hard to find the query since this was logged only after it was executed(after 20,000 seconds).
Reproduction Steps
Test Table Schema
Vschema for sharded keyspace
set up sharded keyspace and run query.
Adding
and not (test.user_id = 1 and test.user_id is not null and test.ts >= value and test.ts <= value)
impacts the query planning time of vtgate. below took 56s and it has 3992AND
clause. It takes more time when more condition is added.Binary Version
Version: 16.0.2 (Git revision 6076fed0300de2eb7a17a5f5cd67527c29b5b3c6 branch 'HEAD') built on Wed Jun 7 00:13:43 UTC 2023 by vitess@buildkitsandbox using go1.20.3 linux/arm64
Operating System and Environment details
Log Fragments
No response
The text was updated successfully, but these errors were encountered: