Using unicode username/password for RabbitMQ agent will result in 401 unauthorised access

[mxamin]

Note: If you have a feature request, you should contact support so the request can be properly tracked.

Output of the info page

===============
Agent (v7.21.0)
===============

  Status date: 2020-07-21 19:48:59.566785 UTC
  Agent start: 2020-07-21 19:35:07.318096 UTC
  Pid: 382
  Go Version: go1.13.11
  Python Version: 3.8.1
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: error

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: 135µs
    System UTC time: 2020-07-21 19:48:59.566785 UTC

  Host Info
  =========
    bootTime: 2020-04-05 09:54:42.000000 UTC
    kernelArch: x86_64
    kernelVersion: 4.15.0-1063-aws
    os: linux
    platform: debian
    platformFamily: debian
    platformVersion: bullseye/sid
    procs: 129
    uptime: 2577h40m29s
    virtualizationRole: guest
    virtualizationSystem: docker

  Hostnames
  =========
    hostname: localhost
    instance-id: i-09a102e5481303d4d
    socket-fqdn: df17fe1509a6
    socket-hostname: df17fe1509a6
    hostname provider: container

=========
Collector
=========

  Running Checks
  ==============

    cpu
    ---
      Instance ID: cpu [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/cpu.d/conf.yaml.default
      Total Runs: 55
      Metric Samples: Last Run: 6, Total: 324
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 0s
      Last Execution Date : 2020-07-21 19:48:47.000000 UTC
      Last Successful Execution Date : 2020-07-21 19:48:47.000000 UTC


    disk (2.10.1)
    -------------
      Instance ID: disk:2539eb9332db6987 [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/disk.yaml
      Total Runs: 56
      Metric Samples: Last Run: 188, Total: 10,528
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 0, Total: 0
      Average Execution Time : 26ms
      Last Execution Date : 2020-07-21 19:48:56.000000 UTC
      Last Successful Execution Date : 2020-07-21 19:48:56.000000 UTC


    docker
    ------
      Instance ID: docker [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/docker.d/conf.yaml.default
      Total Runs: 55
      Metric Samples: Last Run: 59, Total: 3,245
      Events: Last Run: 0, Total: 2
      Service Checks: Last Run: 1, Total: 55
      Average Execution Time : 17ms
      Last Execution Date : 2020-07-21 19:48:54.000000 UTC
      Last Successful Execution Date : 2020-07-21 19:48:54.000000 UTC


    rabbitmq (1.15.0)
    -----------------
      Instance ID: rabbitmq:2caf95082cc855fe [OK]
      Configuration Source: file:/etc/datadog-agent/conf.d/rabbitmq.yaml
      Total Runs: 55
      Metric Samples: Last Run: 514, Total: 28,301
      Events: Last Run: 0, Total: 0
      Service Checks: Last Run: 2, Total: 110
      Average Execution Time : 280ms
      Last Execution Date : 2020-07-21 19:48:55.000000 UTC
      Last Successful Execution Date : 2020-07-21 19:48:55.000000 UTC
      metadata:
        version.major: 3
        version.minor: 7
        version.patch: 12
        version.raw: 3.7.12
        version.scheme: semver

========
JMXFetch
========

  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    CheckRunsV1: 55
    Connections: 0
    Containers: 0
    Dropped: 0
    DroppedOnInput: 0
    Events: 0
    HostMetadata: 0
    IntakeV1: 8
    Metadata: 0
    Pods: 0
    Processes: 0
    RTContainers: 0
    RTProcesses: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Series: 0
    ServiceChecks: 0
    SketchSeries: 0
    Success: 118
    TimeseriesV1: 55

==========
Logs Agent
==========


  Logs Agent is not running

=========
APM Agent
=========
  Status: Running
  Pid: 381
  Uptime: 832 seconds
  Mem alloc: 7,336,976 bytes
  Hostname: queue02
  Receiver: 0.0.0.0:8126
  Endpoints:
    https://trace.agent.datadoghq.eu

  Receiver (previous minute)
  ==========================
    No traces received in the previous minute.
    Default priority sampling rate: 100.0%

  Writer (previous minute)
  ========================
    Traces: 0 payloads, 0 traces, 0 events, 0 bytes
    Stats: 0 payloads, 0 stats buckets, 0 bytes

=========
Aggregator
=========
  Checks Metric Sample: 63,153
  Dogstatsd Metric Sample: 5,643
  Event: 3
  Events Flushed: 3
  Number Of Flushes: 55
  Series Flushed: 56,042
  Service Check: 884
  Service Checks Flushed: 932

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 5,642
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 413,940
  Udp Packet Reading Errors: 0
  Udp Packets: 544
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 0

Additional environment details (Operating System, Cloud provider, etc):

Steps to reproduce the issue:

1. Provide a unicode username/passowrd for RabbitMQ instance (basic auth):

instances:
    -  rabbitmq_api_url: http://localhost:15672/api/
       username: myusername
       password: unicode_pass_l£fsS

2. Restart datadog-agent

Describe the results you received:
Although the entered username/password are correct I see following error message in logs:

HTTP access denied: user 'myusername' - invalid credentials
Error executing check: Cannot open RabbitMQ API url: http://localhost:15672/api/overview 401 Client Error: Unauthorized for url: http://localhost:15672/api/overview

Describe the results you expected:
Since the username/password are correct, I should see no error message and auth should be successful. I checked the username/password on RabbitMQ admin page and it works.

Additional information you deem important (e.g. issue happens only occasionally):
It seems that in newer versions, datadog-agent does some kind of decodings on instance values which will cause such problems.

To be exact, I think this is the PR that decode all values and convert them to native strings: #4730

[FlorianVeaux]

Hi,
I tried to reproduce but I'm facing issues with RabbitMQ simply refusing to have a user with unicode characters.
For example this command docker run -e RABBITMQ_DEFAULT_USER=user -e RABBITMQ_DEFAULT_PASS=è -p 5672:5672 -p 15672:15672 rabbitmq:3.7-management raises the following exception:

BOOT FAILED
===========

Error description:
    rabbit:start_it/1 line 505
    rabbit:boot_error/2 line 1020
    rabbit_lager:log_locations/0 line 97
    rabbit_lager:ensure_lager_configured/0 line 198
    rabbit_lager:lager_configured/0 line 206
    lager:list_all_sinks/0 line 345
    lager_config:get/2 line 71
    ets:lookup(lager_config, {'_global',handlers})
error:{badmatch,{error,{5,file_io_server,invalid_unicode}}}
Log file(s) (may contain more information):

Because the integration simply makes http requests, could you try to run the following command:

curl -i -u <USER>:<UNICODE_PASSWORD> http://localhost:15672/api/whoam

Maybe the answer is that, the rabbitmq UI converts/escapes the unicode characters?

[moodh]

Hi, my colleague just left for vacation, but this is how we added the user:

sudo docker-compose exec rabbitmq rabbitmqctl add_user monitor unicode_password_with_£

[FlorianVeaux]

Hey @moodh
I was able to create a user with a unicode password and confirmed that the rabbitmq integration was still working.

I'm thinking you may not have granted the required permissions, can you try this curl command then? Rabbitmq should give you the reason for the 401.

curl -i -u monitor:<YOUR_PASS> http://<RABBIT_MQ_HOST>:15672/api/overview

[moodh]

Hi, with a unicode password all endpoints works from curl/web browser to rabbitmq, just not through the datadog integration. We tested adding encoding correctly in the integration code which made the integration work so its definitely a problem in the integration with that version combination.

My colleague will be gone for a month now and he has a reproducible environment so I'm not able to test more scenarios without him.

[FlorianVeaux]

Hi again, took me sometime to dig into that issue but you're right non-ascii characters in the password or the username will prevent the rabbitmq integration to work correctly.

Some technical details:
The integration communicates with rabbitmq using HTTP with the help of the requests library.
With that in mind I was able to confirm that:

• curl -v -u datadog:àà http://localhost:15672/api/whoami works
• requests.get("http://localhost:15672/api/whoami", auth=('datadog', 'àà')) works on python2
• requests.get("http://localhost:15672/api/whoami", auth=('datadog', 'àà')) does not work on python3
• requests.get("http://localhost:15672/api/whoami", auth=('datadog', 'àà'.encode('utf-8'))) works on python3

My understanding is that:

• RabbitMQ server does not send any charset information to the client, so requests has no way of knowing how to encode the username and password.
• requests defaults to latin1 encoding and not utf-8, see this github issue [Python 3] Basic auth relying on utf-8 encoding by default psf/requests#4564
• The RabbitMQ server expects utf-8 encoded auth data so it sens back a 401 status code.

All in all what does it means for you is that:

• There is no way currently to have a non-ascii username or password with the rabbitmq integration on python3. Your only option is to use Agent 6 which uses python2
• We'll work on some quick patch soon (only in the rabbitmq integration) so that the auth data is encoded in utf-8.

[moodh]

Hi, great that you found the cause. Don't worry about fixing it soon for our sake, we simply changed the password when we realized that was the issue. :)