HDFS: Clients connect to standby NameNode

[rajarshp]

Hi Team,

We are still using Stackable HDFS 24.7 in our AWS Rosa env. We have used the below config to create the HDFS in our env and exposed it via Load Balancer (the listener class which is configured as Load Balancer)

Now we are want to use this Service name to connect the HDFS from our application. So we tried with the headless service but we are getting below error. so as a work around we are using <active_pod_name>.<hdfs_headless_service>

StandbyException): Operation category READ is not supported in state standby.

We tried to use the direct service name of the active server but the problem is if there is a change in the active/passive service config the application failing with above error.

so, could you please help with the config so that if we use the headless service name to connect it it can automatically redirect it to the active server or any other way to use the active server without using the direct service name ?

Helm to configure HDFS:

apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
  name: test-hdfs
  namespace: test-dev-01
spec:
  clusterConfig:
    dfsReplication: 1
    listenerClass: cluster-internal
    zookeeperConfigMapName: test-zk
  clusterOperation:
    reconciliationPaused: false
    stopped: false
  dataNodes:
    cliOverrides: {}
    config:
      affinity:
        nodeAffinity: null
        nodeSelector: null
        podAffinity: null
        podAntiAffinity: null
      listenerClass: external-stable-test
      logging:
        containers: {}
        enableVectorAgent: null
      resources:
        cpu:
          max: "2"
          min: "1"
        memory:
          limit: 4Gi
          runtimeLimits: {}
        storage: 
          data:
            capacity: 8Gi
    configOverrides: {}
    envOverrides: {}
    podOverrides: {}
    roleConfig:
      podDisruptionBudget:
        enabled: true
        maxUnavailable: null
    roleGroups:
      default:
        cliOverrides: {}
        config:
          affinity:
            nodeAffinity: null
            nodeSelector: null
            podAffinity: null
            podAntiAffinity: null
          logging:
            containers: {}
            enableVectorAgent: null
          resources:
            cpu:
              max: "1"
              min: "1"
            memory:
              limit: 4Gi
              runtimeLimits: {}
            storage:
              data:
                capacity: 8Gi
        configOverrides: {}
        envOverrides: {}
        podOverrides: {}
        replicas: 2
  image:
    productVersion: 3.3.6
    pullPolicy: Always
  journalNodes:
    cliOverrides: {}
    config:
      affinity:
        nodeAffinity: null
        nodeSelector: null
        podAffinity: null
        podAntiAffinity: null
      logging:
        containers: {}
        enableVectorAgent: null
      resources:
        cpu:
          max: "1"
          min: "1"
        memory:
          limit: 4Gi
          runtimeLimits: {}
        storage:
          data:
            capacity: 8Gi
    configOverrides: {}
    envOverrides: {}
    podOverrides: {}
    roleConfig:
      podDisruptionBudget:
        enabled: true
        maxUnavailable: null
    roleGroups:
      default:
        cliOverrides: {}
        config:
          affinity:
            nodeAffinity: null
            nodeSelector: null
            podAffinity: null
            podAntiAffinity: null
          logging:
            containers: {}
            enableVectorAgent: null
          resources:
            cpu:
              max: "1"
              min: "1"
            memory:
              limit: 4Gi
              runtimeLimits: {}
            storage:
              data:
                capacity: 8Gi
        configOverrides: {}
        envOverrides: {}
        podOverrides: {}
        replicas: 2
  nameNodes:
    cliOverrides: {}
    config:
      affinity:
        nodeAffinity: null
        nodeSelector: null
        podAffinity: null
        podAntiAffinity: null
      listenerClass: external-unstable-test
      logging:
        containers: {}
        enableVectorAgent: null
      resources:
        cpu:
          max: "1"
          min: "1"
        memory:
          limit: 4Gi
          runtimeLimits: {}
        storage:
          data:
            capacity: 8Gi
    configOverrides: {}
    envOverrides: {}
    podOverrides: {}
    roleConfig:
      podDisruptionBudget:
        enabled: true
        maxUnavailable: null
    roleGroups:
      default:
        cliOverrides: {}
        config:
          affinity:
            nodeAffinity: null
            nodeSelector: null
            podAffinity: null
            podAntiAffinity: null
          logging:
            containers: {}
            enableVectorAgent: null
          resources:
            cpu:
              max: "1"
              min: "1"
            memory:
              limit: 4Gi
              runtimeLimits: {}
            storage:
              data:
                capacity: 8Gi
        configOverrides: {}
        envOverrides: {}
        podOverrides: {}
        replicas: 2

[nightkr]

Hey!

HDFS expects every client to know the full list of namenodes, and then it is the responsibility of each client to pick and use the active NN. (See here.)

Normally we configure this for you if you use our generated discovery ConfigMaps, but otherwise you will need to configure something similar manually. If our generated configuration isn't suitable for your use case then we'd love to hear why.

Side note: your ListenerClasses look slightly backwards. Generally, you'd want to use LoadBalancer-powered external-stable listeners for namenodes (to avoid configuration churn), but NodePort-powered external-unstable listeners for datanodes (since LoadBalancer traffic tends to be more expensive and slower to provision).

[rajarshp]

thanks @nightkr your suggestion helped me to fix this issue with default configmap generate by the operator

[rajarshp]

btw @nightkr could you please let me know when we will have 24.11 certified for AWS Rosa ?

[nightkr]

OpenShift is generally @razvan's area, but as I understand it the certification finally went through this morning.