Istio addon not working on multinode #10248
Comments
|
/kind support |
|
I'm running into the same issue. Is this possible with istio and minikube? |
|
@mikala3 what error do u get ? have tried allocating more memory ? istio needs a lot of memory |
|
Tried again on latest minikube
Still getting the same errors |
|
@timhughes hw much memory did you allocate to your Docker Desktop and minikube ? I nticed u are using multi node --nodes=4 --cpus=4 is there a reason you are using 4 nodes ? it is unlikely that you need multi node on local cluster, how much memory your system has ? |
|
/triage needs-information |
I am on Linux so no need for Docker Desktop, My Local machine has 24 cores and 64GB ram and the disks are nvme rated at 3500MB/s so I am not worried about resources
I am attempting to run rook-ceph which requires 3 worker nodes
|
minikube start --profile=${CTX_CLUSTER1} --cpus=6 --memory=20g --driver=kvm2 --nodes=2 --addons storage-provisioner,default-storageclass,metallb |
|
This may just be a DNS issue. If I kill the coredns pod and then the istio pods, it appears to fix it. As a workaround, this is fine for me but coming up with a way if it working first go would be better |
So you are able to run with istio and minikube ( with multiple nodes)? |
|
Yes
…On Thu, 22 Apr 2021, 17:10 mikala3, ***@***.***> wrote:
This may just be a DNS issue.
If I kill the coredns pod and then the istio pods, it appears to fix it.
As a workaround, this is fine for me but coming up with a way if it
working first go would be better
So you are able to run with istio and minikube ( with multiple nodes)?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#10248 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJFZD77HPTJN7YBZNWY5J3TKBC7PANCNFSM4WQW24ZA>
.
|
|
@timhughes I wonder if the binary in this Pr helps ? http://storage.googleapis.com/minikube-builds/11731/minikube-linux-amd64 this PR by @andriyDev is doing improvements on multinode |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
|
Hi @timhughes, we haven't heard back from you, do you still have this issue? I will close this issue for now but feel free to reopen when you feel ready to provide more information. |
When trying to run istio on a multi node minikube the
istio-proxycontainers do not start.Steps to reproduce the issue:
Full output of
minikube startcommand used, if not already included:minikube-start-nodes4-cpus4-memory8g.log
minikube-addons-enable-istio-provisioner.log
minikube-addons-enable-istio.log
Full output of
minikube logscommand:minikube-logs.log
Full container logs from
istio-operatorandistio-system namespaces:istio-operator-istio-operator-6dbfd4446f-xxqnn-1611512011468363254.log
istio-system-istiod-6ccd677dc7-gmzsw-1611512018108421785.log
istio-system-istio-ingressgateway-8577c95547-v25xk-1611512014880656996.log
istio-system-prometheus-7767dfd55-kd9c8-1611512021807342019.log
istio-system-prometheus-7767dfd55-kd9c8-1611512024337276220.log
Relevant Logs from
istio-proxycontainer:2021-01-24T18:11:40.046139Z info FLAG: --binaryPath="/usr/local/bin/envoy" 2021-01-24T18:11:40.046165Z info FLAG: --concurrency="0" 2021-01-24T18:11:40.046167Z info FLAG: --configPath="/etc/istio/proxy" 2021-01-24T18:11:40.046170Z info FLAG: --connectTimeout="10s" 2021-01-24T18:11:40.046172Z info FLAG: --controlPlaneAuthPolicy="NONE" 2021-01-24T18:11:40.046175Z info FLAG: --controlPlaneBootstrap="true" 2021-01-24T18:11:40.046177Z info FLAG: --customConfigFile="" 2021-01-24T18:11:40.046178Z info FLAG: --datadogAgentAddress="" 2021-01-24T18:11:40.046180Z info FLAG: --disableInternalTelemetry="false" 2021-01-24T18:11:40.046182Z info FLAG: --discoveryAddress="istio-pilot.istio-system.svc:15012" 2021-01-24T18:11:40.046184Z info FLAG: --dnsRefreshRate="300s" 2021-01-24T18:11:40.046191Z info FLAG: --domain="istio-system.svc.cluster.local" 2021-01-24T18:11:40.046193Z info FLAG: --drainDuration="45s" 2021-01-24T18:11:40.046195Z info FLAG: --envoyAccessLogService="" 2021-01-24T18:11:40.046196Z info FLAG: --envoyMetricsService="" 2021-01-24T18:11:40.046198Z info FLAG: --help="false" 2021-01-24T18:11:40.046199Z info FLAG: --id="" 2021-01-24T18:11:40.046201Z info FLAG: --ip="" 2021-01-24T18:11:40.046203Z info FLAG: --lightstepAccessToken="" 2021-01-24T18:11:40.046204Z info FLAG: --lightstepAddress="" 2021-01-24T18:11:40.046206Z info FLAG: --lightstepCacertPath="" 2021-01-24T18:11:40.046208Z info FLAG: --lightstepSecure="false" 2021-01-24T18:11:40.046210Z info FLAG: --log_as_json="false" 2021-01-24T18:11:40.046212Z info FLAG: --log_caller="" 2021-01-24T18:11:40.046214Z info FLAG: --log_output_level="default:info" 2021-01-24T18:11:40.046215Z info FLAG: --log_rotate="" 2021-01-24T18:11:40.046218Z info FLAG: --log_rotate_max_age="30" 2021-01-24T18:11:40.046220Z info FLAG: --log_rotate_max_backups="1000" 2021-01-24T18:11:40.046223Z info FLAG: --log_rotate_max_size="104857600" 2021-01-24T18:11:40.046225Z info FLAG: --log_stacktrace_level="default:none" 2021-01-24T18:11:40.046229Z info FLAG: --log_target="[stdout]" 2021-01-24T18:11:40.046231Z info FLAG: --mixerIdentity="" 2021-01-24T18:11:40.046233Z info FLAG: --outlierLogPath="" 2021-01-24T18:11:40.046235Z info FLAG: --parentShutdownDuration="1m0s" 2021-01-24T18:11:40.046237Z info FLAG: --pilotIdentity="" 2021-01-24T18:11:40.046239Z info FLAG: --proxyAdminPort="15000" 2021-01-24T18:11:40.046241Z info FLAG: --proxyComponentLogLevel="misc:error" 2021-01-24T18:11:40.046243Z info FLAG: --proxyLogLevel="warning" 2021-01-24T18:11:40.046245Z info FLAG: --serviceCluster="istio-ingressgateway" 2021-01-24T18:11:40.046247Z info FLAG: --serviceregistry="Kubernetes" 2021-01-24T18:11:40.046249Z info FLAG: --statsdUdpAddress="" 2021-01-24T18:11:40.046251Z info FLAG: --statusPort="15020" 2021-01-24T18:11:40.046253Z info FLAG: --stsPort="0" 2021-01-24T18:11:40.046255Z info FLAG: --templateFile="" 2021-01-24T18:11:40.046257Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange" 2021-01-24T18:11:40.046259Z info FLAG: --trust-domain="cluster.local" 2021-01-24T18:11:40.046261Z info FLAG: --zipkinAddress="zipkin.istio-system:9411" 2021-01-24T18:11:40.046283Z info Version 1.5.0-c3c353285578eb68b334fc8766746b754b6b3789-Clean 2021-01-24T18:11:40.046388Z info Obtained private IP [10.244.3.3] 2021-01-24T18:11:40.046412Z info Proxy role: &model.Proxy{ClusterID:"", Type:"router", IPAddresses:[]string{"10.244.3.3", "10.244.3.3"}, ID:"istio-ingressgateway-8577c95547-v25xk.istio-system", Locality:(*envoy_api_v2_core.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersion:(*model.IstioVersion)(nil)} 2021-01-24T18:11:40.046416Z info PilotSAN []string(nil) 2021-01-24T18:11:40.046418Z info MixerSAN []string(nil) 2021-01-24T18:11:40.046742Z info Effective config: binaryPath: /usr/local/bin/envoy configPath: /etc/istio/proxy connectTimeout: 10s discoveryAddress: istio-pilot.istio-system.svc:15012 drainDuration: 45s envoyAccessLogService: {} envoyMetricsService: {} parentShutdownDuration: 60s proxyAdminPort: 15000 serviceCluster: istio-ingressgateway statNameLength: 189 tracing: zipkin: address: zipkin.istio-system:9411 2021-01-24T18:11:40.046751Z info JWT policy is third-party-jwt 2021-01-24T18:11:40.046806Z info Using user-configured CA istio-pilot.istio-system.svc:15012 2021-01-24T18:11:40.046808Z info istiod uses self-issued certificate 2021-01-24T18:11:40.046836Z info the CA cert of istiod is: -----BEGIN CERTIFICATE----- MIIC3jCCAcagAwIBAgIRAP3mz2eowkXT6AkWrglnGJQwDQYJKoZIhvcNAQELBQAw GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMTAxMjQxODExMDZaFw0zMTAx MjIxODExMDZaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3 DQEBAQUAA4IBDwAwggEKAoIBAQCV6xd981Y2QyhbxLohoqh3Vidxkr3V+uJd70Wg zKJFEepQrkLx6sL27BHRqmSfB/po6CTVYmNUFHES9z+XQPQ8DFJtg446+JlwT5u9 BVLGpuskdb2dp/YdT0ZeXSAETYMOlsT6zjK3zAWATgDGYxif8yu80jwy/DvTzIDO VRi2yIyjKTNT80G4i9M3iYjdHwGaFdFlek3ZbskhGpe75R84uXE9zR1OS+HkDTN5 FVZiENmEI62Qz6BXpmHTMX4Oy9u1nvaDFauuyxlFSdSPnm1296Pz5kesNAyB7F+r fsTEpjMtf9NNcK6NGeHVzVAWNHqWk4tY1L1RnSflaE6TAo2HAgMBAAGjIzAhMA4G A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB AQAxMxbEqwK3pr5wzp6lTPQlXn4gZjMlUFhPlecFmo+6J9SYH07tDvY5LdAhEGE/ fVES2AHTKV7KZa5AxXCRqFqShD51l68aCyg2N1w0Mrev5H3KjT0UuxVDaJ/M09Na m+VIT1VsPiEkPBdFj+inSYQP0BWsMBXhe1Mg9oJyXZCKzNDHYCMoxi5Vde3KGZ6L Si+2l7kd3zwssIzgF8LaUqwcN0WC0fl/zgH/JYo+1hz/kcMGj2K6zcOjsvXAyZ/0 hi2T4ZUL/DV+1SBb1Q9hLLs1vzUsmennxuEC940fXc3krMw3bKtRtG0R8bgcWbPb wCPIQvsx2EmSwdtSYOqof9Wv -----END CERTIFICATE----- 2021-01-24T18:11:40.046922Z info parsed scheme: "" 2021-01-24T18:11:40.046927Z info scheme "" not registered, fallback to default scheme 2021-01-24T18:11:40.046938Z info ccResolverWrapper: sending update to cc: {[{istio-pilot.istio-system.svc:15012 0 }] } 2021-01-24T18:11:40.046942Z info ClientConn switching balancer to "pick_first" 2021-01-24T18:11:40.047057Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING } 2021-01-24T18:11:40.085873Z info Starting gateway SDS 2021-01-24T18:11:40.187466Z info sds SDS gRPC server for workload UDS starts, listening on "/etc/istio/proxy/SDS" 2021-01-24T18:11:40.187542Z info sds SDS gRPC server for ingress gateway controller starts, listening on "/var/run/ingress_gateway/sds" 2021-01-24T18:11:40.187531Z info sds Start SDS grpc server 2021-01-24T18:11:40.187567Z info sds Start SDS grpc server for ingress gateway proxy 2021-01-24T18:11:40.187571Z info PilotSAN []string{"istio-pilot.istio-system.svc"} 2021-01-24T18:11:40.187621Z info Starting proxy agent 2021-01-24T18:11:40.187662Z info Opening status port 15020 2021-01-24T18:11:40.187693Z info Received new config, creating new Envoy epoch 0 2021-01-24T18:11:40.187738Z info Epoch 0 starting 2021-01-24T18:11:41.406363Z info Envoy proxy is NOT ready: Get http://127.0.0.1:15000/stats?usedonly&filter=^(cluster_manager.cds|listener_manager.lds).(update_success|update_rejected)$: dial tcp 127.0.0.1:15000: connect: connection refused 2021-01-24T18:11:42.189868Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --log-format [Envoy (Epoch 0)] [%Y-%m-%d %T.%e][%t][%l][%n] %v -l warning --component-log-level misc:error] [Envoy (Epoch 0)] [2021-01-24 18:11:42.224][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:11:42.224][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:11:43.406922Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:11:43.589][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:11:43.589][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:11:45.406869Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:11:46.011][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:11:46.011][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:11:47.406933Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:11:47.444][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:11:47.444][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:11:49.406768Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:11:51.407092Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:11:53.406907Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:11:55.406834Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:11:57.407212Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:11:59.406811Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:00.047149Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting... 2021-01-24T18:12:00.047245Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"} 2021-01-24T18:12:01.047412Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING } 2021-01-24T18:12:01.406750Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:03.406980Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:05.407071Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:12:07.274][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:12:07.274][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:12:07.406742Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:09.406889Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:11.406554Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:13.406755Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:15.406844Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:17.406537Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:19.406795Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:21.047495Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting... 2021-01-24T18:12:21.047542Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"} 2021-01-24T18:12:21.406880Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:22.813360Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING } 2021-01-24T18:12:23.406873Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:25.406651Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:27.406715Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:29.406651Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:31.406598Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:12:32.515][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:12:32.515][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:12:33.406663Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:35.407079Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:12:35.718][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:12:35.718][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:12:37.407151Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:39.406682Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:41.406570Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:12:42.660][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:12:42.660][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:12:42.813440Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting... 2021-01-24T18:12:42.813483Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"} 2021-01-24T18:12:43.406825Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:45.406872Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:45.857720Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING } 2021-01-24T18:12:47.407005Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:49.406834Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:51.406617Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:53.407196Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:55.406961Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:57.228538Z info sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default new connection 2021-01-24T18:12:57.407219Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:12:59.406660Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:00.903701Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout". Reconnecting... 2021-01-24T18:13:00.903785Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout"} 2021-01-24T18:13:00.903816Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:00.953958Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 50 millisec 2021-01-24T18:13:00.954081Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:01.054248Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 100 millisec 2021-01-24T18:13:01.054333Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:01.254467Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 200 millisec 2021-01-24T18:13:01.254537Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:01.406843Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:01.654701Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 400 millisec 2021-01-24T18:13:01.654778Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:02.454924Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 800 millisec 2021-01-24T18:13:02.455019Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:03.406835Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:04.055157Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 1600 millisec 2021-01-24T18:13:04.055244Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout" 2021-01-24T18:13:04.734418Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING } 2021-01-24T18:13:05.406606Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected [Envoy (Epoch 0)] [2021-01-24 18:13:07.019][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, no healthy upstream [Envoy (Epoch 0)] [2021-01-24 18:13:07.019][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:54] Unable to establish new stream 2021-01-24T18:13:07.255368Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istio-pilot.istio-system.svc on 10.96.0.10:53: read udp 10.244.3.3:33205->10.96.0.10:53: i/o timeout", retry in 3200 millisec 2021-01-24T18:13:07.406709Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:09.406937Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:11.406790Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:13.407097Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:15.406754Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:17.407000Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:19.406913Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:21.406575Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:23.406549Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:24.734524Z info grpc: addrConn.createTransport failed to connect to {istio-pilot.istio-system.svc:15012 0 }. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting... 2021-01-24T18:13:24.734584Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {TRANSIENT_FAILURE connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"} 2021-01-24T18:13:24.734613Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.734637Z error cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default request:67cb46cd-3ffe-4517-ae8f-de957988f870 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.734668Z error cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.734684Z error sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default Close connection. Failed to get secret for proxy "router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.734770Z info sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-1 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled [Envoy (Epoch 0)] [2021-01-24 18:13:24.734][21][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:91] gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.776637Z info sds node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default new connection 2021-01-24T18:13:24.846060Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.896173Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec 2021-01-24T18:13:24.896212Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:24.996341Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec 2021-01-24T18:13:24.996435Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:25.196564Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec 2021-01-24T18:13:25.196607Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:25.406711Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:25.596787Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec 2021-01-24T18:13:25.596889Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:26.397058Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec 2021-01-24T18:13:26.397135Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:27.406821Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:27.997277Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 1600 millisec 2021-01-24T18:13:27.997362Z error citadelclient Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" 2021-01-24T18:13:29.406690Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:31.021823Z info pickfirstBalancer: HandleSubConnStateChange: 0xc000267a80, {CONNECTING } 2021-01-24T18:13:31.197481Z warn cache node:router~10.244.3.3~istio-ingressgateway-8577c95547-v25xk.istio-system~istio-system.svc.cluster.local-2 resource:default request:0c2f9f23-5d14-486a-8449-dcfb6bfde6a3 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 3200 millisec 2021-01-24T18:13:31.406707Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected 2021-01-24T18:13:33.406815Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejectedThe text was updated successfully, but these errors were encountered: