[Bug]: [benchmark][cluster] search, query, flush failed in DML & DQL scenario, search error message=incomplete query result #29656
Assignees
Labels
2.4-features
kind/bug
Issues or changes related a bug
stale
indicates no udpates for 30 days
test/benchmark
benchmark test
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Milestone
Comments
|
@xige-16 @yanliang567 |
sre-ci-robot
pushed a commit
that referenced
this issue
Jan 9, 2024
When the TimeTravel functionality was previously removed, it inadvertently affected the MVCC functionality within the system. This PR aims to reintroduce the internal MVCC functionality as follows: 1. Add MvccTimestamp to the requests of Search/Query and the results of Search internally. 2. When the delegator receives a Query/Search request and there is no MVCC timestamp set in the request, set the delegator's current tsafe as the MVCC timestamp of the request. If the request already has an MVCC timestamp, do not modify it. 3. When the Proxy handles Search and triggers the second phase ReQuery, divide the ReQuery into different shards and pass the MVCC timestamp to the corresponding Query requests. issue: #29656 Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
czs007
added a commit
to czs007/milvus
that referenced
this issue
Jan 9, 2024
When the TimeTravel functionality was previously removed, it inadvertently affected the MVCC functionality within the system. This PR aims to reintroduce the internal MVCC functionality as follows: 1. Add MvccTimestamp to the requests of Search/Query and the results of Search internally. 2. When the delegator receives a Query/Search request and there is no MVCC timestamp set in the request, set the delegator's current tsafe as the MVCC timestamp of the request. If the request already has an MVCC timestamp, do not modify it. 3. When the Proxy handles Search and triggers the second phase ReQuery, divide the ReQuery into different shards and pass the MVCC timestamp to the corresponding Query requests. issue: milvus-io#29656 Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Occasionally throws error: incomplete query result
argo task: fouramf-multi-vector-kfsgv client config: server: |
sre-ci-robot
pushed a commit
that referenced
this issue
Jan 11, 2024
When the TimeTravel functionality was previously removed, it inadvertently affected the MVCC functionality within the system. This PR aims to reintroduce the internal MVCC functionality as follows: 1. Add MvccTimestamp to the requests of Search/Query and the results of Search internally. 2. When the delegator receives a Query/Search request and there is no MVCC timestamp set in the request, set the delegator's current tsafe as the MVCC timestamp of the request. If the request already has an MVCC timestamp, do not modify it. 3. When the Proxy handles Search and triggers the second phase ReQuery, divide the ReQuery into different shards and pass the MVCC timestamp to the corresponding Query requests. issue: #29656 pr: #29749 Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Hybrid search errorargo task: inverted-corn-dj9g6 server: client log: 
test result: |
Occasionally throws error: incomplete query resultargo task：fouramf-multi-vector-f2hrf server： client logs: test result: test steps: |
sre-ci-robot
pushed a commit
that referenced
this issue
Feb 1, 2024
issue: #29656 /kind bug Signed-off-by: xige-16 <xi.ge@zilliz.com> --------- Signed-off-by: xige-16 <xi.ge@zilliz.com>
Recurrentargo task：inverted-corn-1706976000 server： client pod name: inverted-corn-1706976000-2027595845 
test steps: test result: |
|
|
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
sre-ci-robot
pushed a commit
to milvus-io/milvus-sdk-go
that referenced
this issue
Mar 18, 2024
…ry (#675) 1. Support retry on milvus errors. 2. Retry on ErrInconsistentRequery for search and hybrid search operations. issue: milvus-io/milvus#29656 related milvus pr: milvus-io/milvus#31343 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
|
/assign @wangting0128 |
occurred againargo task：inverted-corn-vjvws server： client pod name: inverted-corn-vjvws-3305228201 
test steps: test result: |
occurred againargo task：inverted-corn-vjvws server： client pod name: inverted-corn-vjvws-2315877231 
test steps: test result: |
|
Please use the latest version of pymilvus, as the retry logic for this error is only available in the newest version (> 2.4.0rc60). @wangting0128 |
nianliuu
added a commit
to milvus-io/milvus-sdk-java
that referenced
this issue
Mar 21, 2024
Signed-off-by: Nian Liu <nian.liu@zilliz.com>
verification passedpymilvus version: 2.4.0rc66 Since this fix is on the SDK side, please synchronize each SDK owner to add retry logic simultaneously. notifications: |
sre-ci-robot
pushed a commit
to milvus-io/milvus-sdk-java
that referenced
this issue
Mar 27, 2024
#811) Signed-off-by: Nian Liu <nian.liu@zilliz.com>
so , how can i get the pymilvus 2.4.0rc66 installation package or source code ? thanks |
only for test purpose please: https://test.pypi.org/project/pymilvus/2.4.0rc66/ |
thank you but i still get error when i use hybrid_search() in a collection with more than 2 vector field : code 2200, message=retry run out of 75 retry times, message=incomplete query result ......... i do not understand why? It seems to happen with large amounts of data (now my data less then 2 million) |
Could you please provide the version of milvus you are using? |
milvus:2.4.0-rc1 search() function works well, but hybrid_search fails all the time |
|
i suggest you retry on latest v2.3.13 or 2.4, and if it reproduced to you , please file a new issue with full logs attached. |
does milvus v2.3.13 support multi-vector search? i need this feature in my task. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue for this?
Environment
Current Behavior
argo task: fouramf-multi-vector-t6hds
test case: test_inverted_locust_hnsw_ivf_sq8_dml_dql_cluster
server:
client logs:
fouram_log.log
search error message
search_error_message.log
query error message
query_error_message.log
flush error message
flush_error_message.log
Expected Behavior
No response
Steps To Reproduce
Milvus Log
No response
Anything else?
test result:
The text was updated successfully, but these errors were encountered: