Bug Report: VTGate Segmentation Violation during rollout

[harshit-gangal]

Overview of the Issue

During vitess rollout panic is observed on the vtgate.

Reproduction Steps

during PRS sometimes there is no healthy tablet but the current primary is present for the target and states it is serving. There is a race between healthcheck updating the state vs tabletgateway finding a healthy primary to send the query to.

Binary Version

v17 and main

Operating System and Environment details

N/A

Log Fragments

2023-06-14 01:03:32.709 | E0614 01:03:32.709321       1 vtgate.go:656] Uncaught panic: |  
-- | -- | --
  |   | 2023-06-14 01:03:32.709 | runtime error: invalid memory address or nil pointer dereference |  
  |   | 2023-06-14 01:03:32.709 | runtime/panic.go:260 (0x450c95) |  
  |   | 2023-06-14 01:03:32.709 | runtime/signal_unix.go:835 (0x450c65) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/sqltypes/result.go:330 (0x132d784) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/scatter_conn.go:267 (0x132d753) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/scatter_conn.go:643 (0x1331671) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/scatter_conn.go:663 (0x133128c) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/scatter_conn.go:165 (0x132ca24) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/executor.go:1408 (0x1313d57) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/vcursor_impl.go:494 (0x133bf87) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/engine/dml.go:142 (0x10c96e2) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/engine/dml.go:91 (0x10c90c9) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/engine/update.go:72 (0x10fe811) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/vcursor_impl.go:396 (0x133ae17) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/plan_execute.go:204 (0x131dc9b) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/executor.go:402 (0x130aac4) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/plan_execute.go:99 (0x131d41d) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/plan_execute.go:178 (0x131d9f1) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/plan_execute.go:97 (0x131d2b6) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/executor.go:400 (0x130a948) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/executor.go:206 (0x1308d35) |  
  |   | 2023-06-14 01:03:32.709 | vitess.io/vitess/go/vt/vtgate/vtgate.go:448 (0x1349524) |  
.....