cp-kafka-rest in Kubernetes exits with error #286

rroopreddy · 2017-07-04T00:26:50Z

Starting Kafka Rest Proxy in Kubernetes exits with

   $ kubectl logs -f kafka-rest-lr991

   + [[ -n tcp://10.0.0.95:8082 ]]
  + echo 'PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.'
  + exit 1

  PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.

Environment
- Zookeeper deployed into Kubernetes cluster using image from - https://hub.docker.com/r/elevy/zookeeper/
- Kafka deployed into Kubernetes cluster using image from - https://hub.docker.com/r/cloudtrackinc/kubernetes-kafka/
- Kafka REST Proxy deployed into Kubernetes cluster using image from - https://hub.docker.com/r/confluentinc/cp-kafka-rest/
Followed this thread in Stackoverflow (https://stackoverflow.com/questions/41868161/kafka-in-kubernetes-cluster-how-to-publish-consume-messages-from-outside-of-kub/41869856).
Following "kaka-rest-controller.yaml" file is used to start the pod in Kubernetes with required ENV variables

------------kaka-rest-controller.yaml-----------------
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    app: kafka-rest
    component: kafka-rest-endpoint
  name: kafka-rest
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: kafka-rest
        component: kafka-rest-endpoint
    spec:
      containers:
      - name: kafka-rest
        image: confluentinc/cp-kafka-rest:3.2.2
        env:
          - name: KAFKA_REST_HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: KAFKA_REST_ZOOKEEPER_CONNECT
            value: zookeeper:2181

Not sure if there are additional environmental variables missing, By default the Docker image seem to set the PORT by default and seem to check for it to exit.

Any help is appreciated

The text was updated successfully, but these errors were encountered:

rroopreddy · 2017-07-08T01:53:53Z

This is what the compete startup log output looks like. there is a default env variable from docker image "KAFKA_REST_PORT=tcp://10.0.0.14:8082" that later gets checked to EXIT

$ kubectl logs -f kafka-rest-q0kwg
===> ENV Variables ...
APT_ALLOW_UNAUTHENTICATED=false
COMPONENT=kafka-rest
CONFLUENT_DEB_VERSION=2
CONFLUENT_MAJOR_VERSION=3
CONFLUENT_MINOR_VERSION=2
CONFLUENT_PATCH_VERSION=2
CONFLUENT_VERSION=3.2.2
HOME=/root
HOSTNAME=kafka-rest-q0kwg
KAFKA_ADVERTISED_HOST_NAME=172.17.0.9
KAFKA_PORT=tcp://10.0.0.233:9092
KAFKA_PORT_9092_TCP=tcp://10.0.0.233:9092
KAFKA_PORT_9092_TCP_ADDR=10.0.0.233
KAFKA_PORT_9092_TCP_PORT=9092
KAFKA_PORT_9092_TCP_PROTO=tcp
KAFKA_REST_HOST_NAME=172.17.0.9
KAFKA_REST_PORT=tcp://10.0.0.14:8082
KAFKA_REST_PORT_8082_TCP=tcp://10.0.0.14:8082
KAFKA_REST_PORT_8082_TCP_ADDR=10.0.0.14
KAFKA_REST_PORT_8082_TCP_PORT=8082
KAFKA_REST_PORT_8082_TCP_PROTO=tcp
KAFKA_REST_SERVICE_HOST=10.0.0.14
KAFKA_REST_SERVICE_PORT=8082
KAFKA_REST_SERVICE_PORT_KAFKA_REST_PORT=8082
KAFKA_REST_ZOOKEEPER_CONNECT=zookeeper:2181
KAFKA_SERVICE_HOST=10.0.0.233
KAFKA_SERVICE_PORT=9092
KAFKA_SERVICE_PORT_KAFKA_PORT=9092
KAFKA_VERSION=0.10.2.1
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
LANG=C.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
PYTHON_PIP_VERSION=8.1.2
PYTHON_VERSION=2.7.9-1
SCALA_VERSION=2.11
SHLVL=1
ZOOKEEPER_1_PORT=tcp://10.0.0.147:2181
ZOOKEEPER_1_PORT_2181_TCP=tcp://10.0.0.147:2181
ZOOKEEPER_1_PORT_2181_TCP_ADDR=10.0.0.147
ZOOKEEPER_1_PORT_2181_TCP_PORT=2181
ZOOKEEPER_1_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_1_PORT_2888_TCP=tcp://10.0.0.147:2888
ZOOKEEPER_1_PORT_2888_TCP_ADDR=10.0.0.147
ZOOKEEPER_1_PORT_2888_TCP_PORT=2888
ZOOKEEPER_1_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_1_PORT_3888_TCP=tcp://10.0.0.147:3888
ZOOKEEPER_1_PORT_3888_TCP_ADDR=10.0.0.147
ZOOKEEPER_1_PORT_3888_TCP_PORT=3888
ZOOKEEPER_1_PORT_3888_TCP_PROTO=tcp
ZOOKEEPER_1_SERVICE_HOST=10.0.0.147
ZOOKEEPER_1_SERVICE_PORT=2181
ZOOKEEPER_1_SERVICE_PORT_CLIENT=2181
ZOOKEEPER_1_SERVICE_PORT_ELECTION=3888
ZOOKEEPER_1_SERVICE_PORT_FOLLOWERS=2888
ZOOKEEPER_2_PORT=tcp://10.0.0.134:2181
ZOOKEEPER_2_PORT_2181_TCP=tcp://10.0.0.134:2181
ZOOKEEPER_2_PORT_2181_TCP_ADDR=10.0.0.134
ZOOKEEPER_2_PORT_2181_TCP_PORT=2181
ZOOKEEPER_2_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_2_PORT_2888_TCP=tcp://10.0.0.134:2888
ZOOKEEPER_2_PORT_2888_TCP_ADDR=10.0.0.134
ZOOKEEPER_2_PORT_2888_TCP_PORT=2888
ZOOKEEPER_2_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_2_PORT_3888_TCP=tcp://10.0.0.134:3888
ZOOKEEPER_2_PORT_3888_TCP_ADDR=10.0.0.134
ZOOKEEPER_2_PORT_3888_TCP_PORT=3888
ZOOKEEPER_2_PORT_3888_TCP_PROTO=tcp
ZOOKEEPER_2_SERVICE_HOST=10.0.0.134
ZOOKEEPER_2_SERVICE_PORT=2181
ZOOKEEPER_2_SERVICE_PORT_CLIENT=2181
ZOOKEEPER_2_SERVICE_PORT_ELECTION=3888
ZOOKEEPER_2_SERVICE_PORT_FOLLOWERS=2888
ZOOKEEPER_3_PORT=tcp://10.0.0.2:2181
ZOOKEEPER_3_PORT_2181_TCP=tcp://10.0.0.2:2181
ZOOKEEPER_3_PORT_2181_TCP_ADDR=10.0.0.2
ZOOKEEPER_3_PORT_2181_TCP_PORT=2181
ZOOKEEPER_3_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_3_PORT_2888_TCP=tcp://10.0.0.2:2888
ZOOKEEPER_3_PORT_2888_TCP_ADDR=10.0.0.2
ZOOKEEPER_3_PORT_2888_TCP_PORT=2888
ZOOKEEPER_3_PORT_2888_TCP_PROTO=tcp
ZOOKEEPER_3_PORT_3888_TCP=tcp://10.0.0.2:3888
ZOOKEEPER_3_PORT_3888_TCP_ADDR=10.0.0.2
ZOOKEEPER_3_PORT_3888_TCP_PORT=3888
ZOOKEEPER_3_PORT_3888_TCP_PROTO=tcp
ZOOKEEPER_3_SERVICE_HOST=10.0.0.2
ZOOKEEPER_3_SERVICE_PORT=2181
ZOOKEEPER_3_SERVICE_PORT_CLIENT=2181
ZOOKEEPER_3_SERVICE_PORT_ELECTION=3888
ZOOKEEPER_3_SERVICE_PORT_FOLLOWERS=2888
ZOOKEEPER_PORT=tcp://10.0.0.189:2181
ZOOKEEPER_PORT_2181_TCP=tcp://10.0.0.189:2181
ZOOKEEPER_PORT_2181_TCP_ADDR=10.0.0.189
ZOOKEEPER_PORT_2181_TCP_PORT=2181
ZOOKEEPER_PORT_2181_TCP_PROTO=tcp
ZOOKEEPER_SERVICE_HOST=10.0.0.189
ZOOKEEPER_SERVICE_PORT=2181
ZOOKEEPER_SERVICE_PORT_CLIENT=2181
ZULU_OPENJDK_VERSION=8=8.17.0.3
_=/usr/bin/env
===> User

. /etc/confluent/docker/apply-mesos-overrides
+ . /etc/confluent/docker/apply-mesos-overrides

#!/usr/bin/env bash
#
# Copyright 2016 Confluent Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Mesos DC/OS docker deployments will have HOST and PORT0
# set for the proxying of the service.
#
# Use those values provide things we know we'll need.

[ -n "${HOST:-}" ] && [ -z "${KAFKA_REST_HOST_NAME:-}" ] && \
	export KAFKA_REST_HOST_NAME=$HOST || true # we don't want the setup to fail if not on Mesos
++ '[' -n '' ']'
++ true


echo "===> ENV Variables ..."
+ echo '===> ENV Variables ...'
env | sort
+ env
+ sort

echo "===> User"
+ echo '===> User'
id
+ id

echo "===> Configuring ..."
+ echo '===> Configuring ...'
/etc/confluent/docker/configure
+ /etc/confluent/docker/configure
uid=0(root) gid=0(root) groups=0(root)
===> Configuring ...

dub ensure KAFKA_REST_ZOOKEEPER_CONNECT
+ dub ensure KAFKA_REST_ZOOKEEPER_CONNECT
dub ensure KAFKA_REST_HOST_NAME
+ dub ensure KAFKA_REST_HOST_NAME

dub path /etc/"${COMPONENT}"/ writable
+ dub path /etc/kafka-rest/ writable

if [[ -n "${KAFKA_REST_PORT-}" ]]
then
  echo "PORT is deprecated. Please use KAFKA_REST_LISTENERS instead."
  exit 1
fi
+ [[ -n tcp://10.0.0.14:8082 ]]
+ echo 'PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.'
+ exit 1
PORT is deprecated. Please use KAFKA_REST_LISTENERS instead.

geofflamrock · 2017-07-20T03:26:40Z

Do you have a Kubernetes service called kafka-rest in your setup?

Environment variables are added by Kubernetes for services by default, and I found that I had to name the service something else like kafka-rest-service for it to not find that environment variable and continue on.

bojidar-bg · 2018-11-08T16:11:13Z

I managed to workaround this issue without renaming the service by unsetting the environment variable before running:

apiVersion: apps/v1
kind: StatefulSet
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: confluentinc/cp-kafka-rest:5.0.1
        # ...
        command: ["bash", "-c", "unset KAFKA_REST_PORT; /etc/confluent/docker/run"]

Aidan-Pasquale99 · 2020-02-07T15:15:10Z

I managed to workaround this issue without renaming the service by unsetting the environment variable before running:

apiVersion: apps/v1
kind: StatefulSet
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: confluentinc/cp-kafka-rest:5.0.1
        # ...
        command: ["bash", "-c", "unset KAFKA_REST_PORT; /etc/confluent/docker/run"]

This fixed the same issue when deploying the Schema Registry image, thank you!

tirelibirefe · 2020-05-20T02:33:47Z

I'm trying to install schema-registry via Helm helm install incubator/schema-registry --version 1.1.9
The problem is still exist since 2017.
Could you please give more details about the workaround the above? Where/How can I apply that workaround in helm charts?

b4nnertail · 2020-07-13T16:50:24Z

I'm trying to install schema-registry via Helm helm install incubator/schema-registry --version 1.1.9
The problem is still exist since 2017.
Could you please give more details about the workaround the above? Where/How can I apply that workaround in helm charts?

I am also using incubator/schema-registry and could resolve the issue by simply changing the release name from schema-registry to anything else, in this case sr:

helm install sr -f schema-registry_values.yaml incubator/schema-registry

tirelibirefe · 2020-07-13T16:58:57Z

I'm trying to install schema-registry via Helm helm install incubator/schema-registry --version 1.1.9
The problem is still exist since 2017.
Could you please give more details about the workaround the above? Where/How can I apply that workaround in helm charts?

I am also using incubator/schema-registry and could resolve the issue by simply changing the release name from schema-registry to anything else, in this case sr:

helm install sr -f schema-registry_values.yaml incubator/schema-registry

It was about almost 2 months ago. Currently I have a working system but I really don't remember anything about workaround.

WZ · 2020-07-31T00:19:08Z

This is because, by default, kubelet injects a set of ENVs automatically (for service discovery purpose). Another workaround is to disable this by setting enableServiceLinks: false in the pod spec (https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/#accessing-the-service)

raohammad · 2021-01-29T13:18:45Z

I managed to workaround this issue without renaming the service by unsetting the environment variable before running:

apiVersion: apps/v1
kind: StatefulSet
spec:
  # ...
  template:
    # ...
    spec:
      containers:
      - image: confluentinc/cp-kafka-rest:5.0.1
        # ...
        command: ["bash", "-c", "unset KAFKA_REST_PORT; /etc/confluent/docker/run"]

This approach worked for me. Just that in my case, I did not have KAFKA_REST_PORT rather KAFKA_PORT. Finally this is how the command looks like for my deployment template of Kafka

command: ["bash", "-c", "unset KAFKA_PORT; /etc/confluent/docker/run"]

Tarasovych · 2022-10-20T05:23:30Z

Same issue with cp-kafka (0.6.1), but there is no way to override command

fbascheper mentioned this issue Jan 25, 2018

Incorrect port appended to hostname in base_uri of response body confluentinc/kafka-rest#362

Open

ozbillwang mentioned this issue Jun 28, 2018

Docker image fails to start citing deprecated PORT but I am using LISTENERS in env confluentinc/schema-registry#689

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cp-kafka-rest in Kubernetes exits with error #286

cp-kafka-rest in Kubernetes exits with error #286

rroopreddy commented Jul 4, 2017 •

edited

Loading

rroopreddy commented Jul 8, 2017 •

edited

Loading

geofflamrock commented Jul 20, 2017

bojidar-bg commented Nov 8, 2018

Aidan-Pasquale99 commented Feb 7, 2020

tirelibirefe commented May 20, 2020

b4nnertail commented Jul 13, 2020

tirelibirefe commented Jul 13, 2020

WZ commented Jul 31, 2020

raohammad commented Jan 29, 2021

Tarasovych commented Oct 20, 2022

cp-kafka-rest in Kubernetes exits with error #286

cp-kafka-rest in Kubernetes exits with error #286

Comments

rroopreddy commented Jul 4, 2017 • edited Loading

rroopreddy commented Jul 8, 2017 • edited Loading

geofflamrock commented Jul 20, 2017

bojidar-bg commented Nov 8, 2018

Aidan-Pasquale99 commented Feb 7, 2020

tirelibirefe commented May 20, 2020

b4nnertail commented Jul 13, 2020

tirelibirefe commented Jul 13, 2020

WZ commented Jul 31, 2020

raohammad commented Jan 29, 2021

Tarasovych commented Oct 20, 2022

rroopreddy commented Jul 4, 2017 •

edited

Loading

rroopreddy commented Jul 8, 2017 •

edited

Loading