Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [restful v2]hybrid search failed with error incomplete query result, missing id %!s(int64=3551), len(searchIDs) = 10, len(queryIDs) = 1, collection=448661016962138998: inconsistent requery result #31642

Closed
1 task done
zhuwenxing opened this issue Mar 27, 2024 · 10 comments
Closed
1 task done

[Bug]: [restful v2]hybrid search failed with error incomplete query result, missing id %!s(int64=3551), len(searchIDs) = 10, len(queryIDs) = 1, collection=448661016962138998: inconsistent requery result #31642

zhuwenxing opened this issue Mar 27, 2024 · 10 comments
Assignees
@bigsheeper @czs007
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.2

Comments

@zhuwenxing
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:2.4-20240326-35664fa3-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

[2024-03-27T02:43:44.707Z] {"code":2200,"message":"incomplete query result, missing id %!s(int64=515), len(searchIDs) = 10, len(queryIDs) = 1, collection=448661016962138998: inconsistent requery result"}

i am doing perf test for hybrid search restful API. 90% requests are failed with this error

[2024-03-27T02:43:53.440Z] Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
[2024-03-27T02:43:53.441Z] --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
[2024-03-27T02:43:53.441Z] POST     /v2/vectordb/entities/hybrid_search                                            14596 13169(90.22%) |    796     550    1115    800 |    9.30        8.60
[2024-03-27T02:43:53.441Z] --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
[2024-03-27T02:43:53.441Z]          Aggregated                                                                     14596 13169(90.22%) |    796     550    1115    800 |    9.30        8.60

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/restful_v2_perf_test/detail/restful_v2_perf_test/8/pipeline

pod info

[2024-03-27T02:48:22.999Z] restful-v2-perf-test-8-etcd-0                                     1/1     Running            0                  30m     10.104.15.160   4am-node20   <none>           <none>

[2024-03-27T02:48:22.999Z] restful-v2-perf-test-8-etcd-1                                     1/1     Running            0                  30m     10.104.31.103   4am-node34   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-etcd-2                                     1/1     Running            0                  30m     10.104.19.248   4am-node28   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-kafka-0                                    1/1     Running            2 (30m ago)        30m     10.104.15.158   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-kafka-1                                    1/1     Running            2 (30m ago)        30m     10.104.31.102   4am-node34   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-kafka-2                                    1/1     Running            2 (30m ago)        30m     10.104.27.42    4am-node31   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-milvus-standalone-b97575779-wqv5v          1/1     Running            1 (29m ago)        30m     10.104.15.157   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-minio-6b89664694-br2qf                     1/1     Running            0                  30m     10.104.15.159   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-t5w1t-6bgll-97lzm                          2/2     Running            0                  33m     10.104.13.31    4am-node16   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-zookeeper-0                                1/1     Running            0                  30m     10.104.15.156   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-zookeeper-1                                1/1     Running            0                  30m     10.104.27.37    4am-node31   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-zookeeper-2                                1/1     Running            0                  30m     10.104.19.245   4am-node28   <none>           <none>

cluster: 4am
namespace: chaos-testing

Anything else?

related to #29656 (comment)
@zhuwenxing zhuwenxing added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 27, 2024
@zhuwenxing
Copy link
Contributor Author

/assign @czs007

@zhuwenxing zhuwenxing added this to the 2.4.0 milestone Mar 27, 2024
@zhuwenxing zhuwenxing added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. labels Mar 27, 2024
@bigsheeper
Copy link
Contributor

When using the RESTful interface, unlike with the SDKs (pymilvus, go-sdk), you'll need to manually retry for code:2200.

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 27, 2024
@yanliang567 yanliang567 removed their assignment Mar 27, 2024
@yanliang567
Copy link
Contributor

what causes code 2200, is it expected?

When using the RESTful interface, unlike with the SDKs (pymilvus, go-sdk), you'll need to manually retry for code:2200.

@xiaofan-luan
Copy link
Collaborator

what causes code 2200, is it expected?

When using the RESTful interface, unlike with the SDKs (pymilvus, go-sdk), you'll need to manually retry for code:2200.

Can we retry is on proxy rather than on every SDK?

This was referenced Mar 29, 2024
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue Mar 30, 2024
@bigsheeper
enhance: Retry on inconsistent requery result (#31713)
417c678 
Make retry on ErrInConsistentRequery in proxy rather than in every SDK.

issue: #31642

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Mar 30, 2024
@bigsheeper
enhance: Retry on inconsistent requery result (#31713) (#31734)
bc4a9a1 
Make retry on ErrInConsistentRequery in proxy rather than in every SDK.

issue: #31642

pr: #31713

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
@bigsheeper
Copy link
Contributor

/assign @zhuwenxing
please help verify

@bigsheeper
Copy link
Contributor

Should we remove the retry logic in SDKs in 2.4? @czs007

@zhuwenxing
Copy link
Contributor Author

still reproduced in 2.4-20240401-f3216bfe1-amd64

[2024-04-01T08:38:23.735Z] 8                  POST /v2/vectordb/entities/hybrid_search: CatchResponseError('search failed with rsp {"code":2200,"message":"incomplete query result, missing id %!s(int64=356855), len(searchIDs) = 20, len(queryIDs) = 1, collection=448778788875010685: inconsistent requery result"}')

cluster: 4am
namespace: chaos-testing
pod info

[2024-04-01T08:38:26.614Z] + kubectl get pods -o wide

[2024-04-01T08:38:26.615Z] + grep restful-v2-perf-test-28

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-8gbt3-jr8vv-hk67v                         2/2     Running            0                  101m    10.104.6.136    4am-node13   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-etcd-0                                    1/1     Running            0                  92m     10.104.15.29    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-etcd-1                                    1/1     Running            0                  92m     10.104.20.201   4am-node22   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-etcd-2                                    1/1     Running            0                  92m     10.104.34.98    4am-node37   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-kafka-0                                   1/1     Running            0                  92m     10.104.15.34    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-kafka-1                                   1/1     Running            0                  92m     10.104.16.113   4am-node21   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-kafka-2                                   1/1     Running            0                  92m     10.104.20.203   4am-node22   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-datacoord-88ccd57f5-dh957          1/1     Running            0                  92m     10.104.15.24    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-datanode-5f455c6796-97c5k          1/1     Running            0                  92m     10.104.15.22    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-indexcoord-5494dc544-d7x8c         1/1     Running            0                  92m     10.104.1.52     4am-node10   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-indexnode-8cdc7fd65-4pfc5          1/1     Running            0                  70m     10.104.13.45    4am-node16   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-indexnode-8cdc7fd65-kwjfl          1/1     Running            0                  92m     10.104.15.21    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-indexnode-8cdc7fd65-pzsl6          1/1     Running            0                  92m     10.104.4.172    4am-node11   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-proxy-7fdbc554bd-ng2lk             1/1     Running            0                  92m     10.104.15.23    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-querycoord-94c987d67-7j2zc         1/1     Running            0                  92m     10.104.1.53     4am-node10   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-querynode-76d756d9c4-spnpj         1/1     Running            0                  92m     10.104.9.122    4am-node14   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-milvus-rootcoord-75445b49bc-k89sp         1/1     Running            0                  92m     10.104.5.228    4am-node12   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-minio-0                                   1/1     Running            0                  92m     10.104.16.112   4am-node21   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-minio-1                                   1/1     Running            0                  92m     10.104.15.35    4am-node20   <none>           <none>

[2024-04-01T08:38:26.871Z] restful-v2-perf-test-28-minio-2                                   1/1     Running            0                  92m     10.104.30.222   4am-node38   <none>           <none>

[2024-04-01T08:38:26.872Z] restful-v2-perf-test-28-minio-3                                   1/1     Running            0                  92m     10.104.27.44    4am-node31   <none>           <none>

[2024-04-01T08:38:26.872Z] restful-v2-perf-test-28-zookeeper-0                               1/1     Running            0                  92m     10.104.15.37    4am-node20   <none>           <none>

[2024-04-01T08:38:26.872Z] restful-v2-perf-test-28-zookeeper-1                               1/1     Running            0                  92m     10.104.32.233   4am-node39   <none>           <none>

[2024-04-01T08:38:26.872Z] restful-v2-perf-test-28-zookeeper-2                               1/1     Running            0                  92m     10.104.34.101   4am-node37   <none>           <none>

@zhuwenxing
Copy link
Contributor Author

/unassign @zhuwenxing
/assign @bigsheeper

@sre-ci-robot sre-ci-robot assigned bigsheeper and unassigned zhuwenxing Apr 1, 2024
This was referenced Apr 17, 2024
Closed
Closed
@zhuwenxing
Copy link
Contributor Author

not a stable reproduced issue, so remove critical label

@zhuwenxing zhuwenxing removed priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. severity/critical Critical, lead to crash, data missing, wrong result, function totally doesn't work. labels Apr 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.0, 2.4.1 Apr 18, 2024
@yanliang567 yanliang567 modified the milestones: 2.4.1, 2.4.2 May 7, 2024
@stale Stale
Copy link

stale bot commented Jun 11, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Jun 11, 2024
@stale stale bot closed this as completed Jun 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bigsheeper bigsheeper

@czs007 czs007

Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.2
Development

No branches or pull requests
5 participants
@zhuwenxing @bigsheeper @czs007 @yanliang567 @xiaofan-luan