SAML Login Fails With Invalid Response #1016

newlandk · 2018-01-18T21:47:05Z

ISSUE TYPE

Bug Report

COMPONENT NAME

UI
API

SUMMARY

When logging into AWX with SAML configured the final redirect appears to be to an incorrect port that is of the container and not the web interface.

ENVIRONMENT

AWX version: 1.0.2.370
AWX install method: official image
Ansible version: 2.4.1
Operating System: Centos 7.6
Web Browser: Chrome/Safari

STEPS TO REPRODUCE

Configure AWX with a SAML Identity Provider. The SAML portion of the authentication appears to work up to the point where a redirect occurs to an incorrect port and protocol (that appears to be the port and protocol on the docker container) and not that which the web/API is hosted from.

It appears this may be a regression from the transition from social-app-django to social-auth-core and social-auth-app-django.

There has been some activity here that seems related: #119 but it looks like there have been changes in the authentication back-end since it was closed and the issue covers multiple challenges.

EXPECTED RESULTS

Redirected to the URL that includes the hostname and port of the web/API logged in.

ACTUAL RESULTS

An error after successfully logging into the IDp.

Authentication failed: SAML login failed: ['invalid_response'] (The response was received at http://awx.test.com:8052/sso/complete/saml/ instead of https://awx.test.com/sso/complete/saml/).

newlandk · 2018-01-19T02:34:53Z

I'm not too familiar with social-core or Python, but I've done my best to understand it.
It looks like social-app-django may be using the container to determine port and protocol since it is the source of the SAML request

newlandk · 2018-01-24T05:14:57Z

It turns out the issue faced was proxy related and I had to modify the settings.py configuration to allow the use of the X-Forwarded-Port. After this django provides the correct port to social-auth for the complete URL.

Good reference if you run into the same issue:
Tower Proxy Documentation

the doc has details regarding the configuration you need to set for AWX to use additional headers provided by the proxy server.
You may need to configure your proxy server to provide the port as it is was provided by default and simply being https does not assume port 443
Using the official docker images I mounted the settings.py as a volume with the following lines added
- USE_X_FORWARDED_PORT = True
- USE_X_FORWARDED_HOST = True

This issue was especially confusing since the there are multiple places in the UI that state the full URL of the server including protocol and social-auth determines these values from django instead.

BenJuan26 · 2018-04-06T20:01:11Z

Can you explain a little about how you mounted settings.py as a volume? I tried changing awx/installer/roles/image_build/files/settings.py but even after rebuilding all containers, my version of the settings was not located at /etc/tower/settings.py inside the container.

I'm getting the same error you've provided here.

newlandk · 2018-04-06T23:11:12Z

@BenJuan26 we just mount a modified settings.py outside the container and mount it in the container as a volume for the time being.

When you run your docker-compose add arguments similar to this for awx_web:

services:
  awx_web:
    volumes:
      - /src/settings.py:/etc/awx/settings.py

More information here on volumes/mounts: https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume

sudomateo · 2018-06-16T01:21:09Z

So the way I fixed this to work correctly with SAML was by building the container images locally and pushing them to the remote AWX server. This alleviates the need to mount a volume.

First I commented out the following lines in installer/inventory:

#dockerhub_base=ansible
#dockerhub_version=latest

This allows you to build the container image locally instead of pulling from ansible's dockerhub images.

Next I added the following lines to installer/roles/image_build/files/settings.py

USE_X_FORWARDED_PORT = True
USE_X_FORWARDED_HOST = True

This is what gets injected into the container images for the web and task containers.

Then I configured the nginx reverse proxy on the AWX remote host. My config looks like this:

server {
    listen 80;
    server_name awx.example.com;

    location / {
        return 301 https://awx.example.com$request_uri;
    }
}
server {  
    listen 443 ssl;
    server_name awx.example.com;

    ssl_certificate cert.pem;
    ssl_certificate_key key.pem;
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA"; 
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;

    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    add_header X-Content-Type-Options nosniff;


    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port 443;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

The important line here is proxy_set_header X-Forwarded-Port 443;. This forwards the correct port along for AWX to use with the USE_X_FORWARDED_HOST = True setting above.

Lastly, I added the HTTP_X_FORWARDED_FOR header in Settings > System > Remote Host Headers

I used https://gist.github.com/defionscode/fc21488e44d73cdd919f81ee1b43e204 as a reference for how my SAML config should look.

jeunii · 2019-02-25T22:46:42Z

Hello @sudomateo
I encountered a similar issue. Actually I have the exact same issue. My AWX setup is deployed in a kubernetes cluster. And rather than nginx as my Ingress, I have HA Proxy.
Would you by any chance know what would be the equivalent haproxy config ?

newlandk · 2019-02-25T23:12:37Z

I believe you would have configuration options like this under your frontend to overcome the challenges we faced noted above.
http-request set-header X-Forwarded-Proto https if https
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Port 443
option forwardfor

Good luck!

sudomateo · 2019-02-26T22:06:33Z

@jeunii I'm sorry I do not have an exact HAProxy configuration handy but what @newlandk commented is a good start. As long as the headers are properly set it should be fine.

svrraja · 2019-07-20T09:52:22Z

How to fix the issue if its running in kubernetes, How to configure the NGINX reverse proxy

naren-ambati · 2019-08-19T20:20:31Z

@sudomateo I'm facing the same issue, But the changes I did for enabling tls is completely different of what you used in nginx.conf.

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

    
    server {
                listen 8052 default_server;
        
        # If you have a domain name, this is where to add it
        server_name awx.project.qa;
        keepalive_timeout 65;
        ssl_certificate           /etc/nginx/certs/awx.project.qa.chain.pem;
        ssl_certificate_key       /etc/nginx/certs/awx.project.qa.key;
        ssl on;
        ssl_session_cache  builtin:1000  shared:SSL:10m;
        ssl_protocols TLSv1.2;
        ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
        ssl_prefer_server_ciphers on;
        access_log            /var/log/nginx/awx.access.log;
        ssl_session_tickets off;
        ssl_stapling on;
        ssl_stapling_verify on;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' cdn.pendo.io; img-src 'self' data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' cdn.pendo.io; img-src 'self' data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;

      }
    }
}

I'm trying to make use of your nginx configuration, but my awx_web is not working and I don't see any errors. Do you have any turn around so that I can use your configuration in the above. Any help is appriciated, Thanks.

loitho · 2019-12-27T18:22:17Z

I'm putting my reply here too.
This applies if you're running the AWX_WEB with docker-compose with the default port configuration with SSL

If the Nginx that forward its requests to the Django server (the uwsgi_pass uwsgi;) isn't running on a standard port (like here, it's running in the 8053) you can either :

Change the configuration so that Nginx listens on the 443 port (and modify the docker compose accordingly) That's the "easy" way but I tried to understand how to make it work with the default configuration.
Or you can leave the default port configuration, and just modify the nginx.conf that's in the "docker_compose_dir" option
Simply add the following line to the "location /" part :

uwsgi_param HTTP_X_FORWARDED_PORT 443;

And off you go. The error :

The response was received at https://XXXXX:8053/sso/complete/saml/ instead of https://XXXXX/sso/complete/saml/

Should be gone.
I'm attaching my nginx.conf so that you can check it out (it's the AWS 9.0.1 default file, just with the line addition)

#user awx;

worker_processes  1;

pid        /tmp/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    server_tokens off;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /dev/stdout main;

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }

    sendfile        on;
    #tcp_nopush     on;
    #gzip  on;

    upstream uwsgi {
        server 127.0.0.1:8050;
        }

    upstream daphne {
        server 127.0.0.1:8051;
    }

        server {
        listen 8052 default_server;
        server_name _;

        # Redirect all HTTP links to the matching HTTPS page
        return 301 https://$host$request_uri;
    }

    server {
#        listen 443 ssl;
        listen 8053 ssl;

        ssl_certificate /etc/nginx/awxweb.pem;
        ssl_certificate_key /etc/nginx/awxweb.pem;

        # If you have a domain name, this is where to add it
        server_name _;
        keepalive_timeout 65;

        # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
        add_header Strict-Transport-Security max-age=15768000;
        add_header Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";
        add_header X-Content-Security-Policy "default-src 'self'; connect-src 'self' ws: wss:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' *.pendo.io; img-src 'self' *.pendo.io data:; report-uri /csp-violation/";

        # Protect against click-jacking https://www.owasp.org/index.php/Testing_for_Clickjacking_(OTG-CLIENT-009)
        add_header X-Frame-Options "DENY";

        location /nginx_status {
          stub_status on;
          access_log off;
          allow 127.0.0.1;
          deny all;
        }

        location /static/ {
            alias /var/lib/awx/public/static/;
        }

        location /favicon.ico { alias /var/lib/awx/public/static/favicon.ico; }

        location /websocket {
            # Pass request to the upstream alias
            proxy_pass http://daphne;
            # Require http version 1.1 to allow for upgrade requests
            proxy_http_version 1.1;
            # We want proxy_buffering off for proxying to websockets.
            proxy_buffering off;
            # http://en.wikipedia.org/wiki/X-Forwarded-For
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            # enable this if you use HTTPS:
            proxy_set_header X-Forwarded-Proto https;
            # pass the Host: header from the client for the sake of redirects
            proxy_set_header Host $http_host;
            # We've set the Host header, so we don't need Nginx to muddle
            # about with redirects
            proxy_redirect off;
            # Depending on the request value, set the Upgrade and
            # connection headers
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

        location / {
            # Add trailing / if missing
            rewrite ^(.*)$http_host(.*[^/])$ $1$http_host$2/ permanent;
            uwsgi_read_timeout 120s;
            uwsgi_pass uwsgi;
            include /etc/nginx/uwsgi_params;

            proxy_set_header X-Forwarded-Port 443;
           # that's the important bit
            uwsgi_param HTTP_X_FORWARDED_PORT 443;
        }
    }
}

I haven't seen much ADFS SAML configuration available and since I had quite some trouble setting it up, I'm also adding my ADFS configuration for "SAML ENABLED IDENTITY PROVIDERS"

{
 "saml_ms_adfs": {
  "attr_last_name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/surname",
  "entity_id": "http://signin.server.com/adfs/services/trust",
  "x509cert": "<redacted>",
  "url": "https://signin.server.com/adfs/ls/",
  "attr_username": "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname",
  "attr_email": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress",
  "attr_first_name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname",
  "attr_user_permanent_id": "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"
 }
}

Big thanks to this thread : pallets/werkzeug#1465 (comment) for the enlightenment

Add the uwsgi_param 'HTTP_X_FORWARDED_PORT' to nginx configuration, This prevents the python-saml "invalid_response" error related issue : ansible#5570 and ansible#1016 Signed-off-by: loitho

ryanpetrello · 2020-01-03T15:10:08Z

For those following along in this thread, @loitho has contributed a patch which should alleviate this issue (which I'm merging shortly):

#5577

peacengell · 2023-05-30T12:19:16Z

Hello All,

I found a fix, tested and working. Credit goes to [ https://medium.com/@_jonas/traefik-kubernetes-ingress-and-x-forwarded-headers-82194d319b0e ] This kind man, thank you.

If you using k3s or Kubernetes and Traefik, you can edit this file

Traefix is setup using Helm Chart.
The file path is /var/lib/rancher/k3s/server/manifests/traefik.yaml

Edit it and add the three line with #< with it.

valuesContent: |-
    podAnnotations:
      prometheus.io/port: "8082"
      prometheus.io/scrape: "true"
    providers:
      kubernetesIngress:
        publishedService:
          enabled: true
    additionalArguments: #<
      - "--entryPoints.web.proxyProtocol.insecure" #<
      - "--entryPoints.web.forwardedHeaders.insecure" #<

Helm will redeploy your traefix pod, once restarted give it a 10 min.

Then check.

awxbot added state:needs_triage type:bug component:ui component:api labels Jan 18, 2018

newlandk closed this as completed Jan 24, 2018

newlandk mentioned this issue Jan 24, 2018

SAML Login fails with invalid endpoint #119

Closed

sudomateo mentioned this issue Jun 28, 2018

Job details and Job view not working #1861

Closed

tfmm mentioned this issue Feb 8, 2019

Using Okta SAML with the UI behind an NGINX proxy, CSRF fails #3194

Closed

ryanpetrello mentioned this issue Jun 21, 2019

Azure ad authentication not sending correct base URL #4120

Closed

loitho mentioned this issue Dec 24, 2019

SAML Login error invalid_response request was received at incorrect port #5570

Closed

loitho mentioned this issue Dec 31, 2019

Add a uwsgi param to prevent SAML error #5577

Merged

teddyphreak mentioned this issue Sep 24, 2020

Port 930b468 from https://github.com/ansible/awx ansible/awx-operator#28

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SAML Login Fails With Invalid Response #1016

SAML Login Fails With Invalid Response #1016

newlandk commented Jan 18, 2018

newlandk commented Jan 19, 2018 •

edited

Loading

newlandk commented Jan 24, 2018 •

edited

Loading

BenJuan26 commented Apr 6, 2018

newlandk commented Apr 6, 2018 •

edited

Loading

sudomateo commented Jun 16, 2018

jeunii commented Feb 25, 2019

newlandk commented Feb 25, 2019 •

edited

Loading

sudomateo commented Feb 26, 2019

svrraja commented Jul 20, 2019

naren-ambati commented Aug 19, 2019 •

edited

Loading

loitho commented Dec 27, 2019

ryanpetrello commented Jan 3, 2020

peacengell commented May 30, 2023 •

edited

Loading

SAML Login Fails With Invalid Response #1016

SAML Login Fails With Invalid Response #1016

Comments

newlandk commented Jan 18, 2018

ISSUE TYPE

COMPONENT NAME

SUMMARY

ENVIRONMENT

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

newlandk commented Jan 19, 2018 • edited Loading

newlandk commented Jan 24, 2018 • edited Loading

BenJuan26 commented Apr 6, 2018

newlandk commented Apr 6, 2018 • edited Loading

sudomateo commented Jun 16, 2018

jeunii commented Feb 25, 2019

newlandk commented Feb 25, 2019 • edited Loading

sudomateo commented Feb 26, 2019

svrraja commented Jul 20, 2019

naren-ambati commented Aug 19, 2019 • edited Loading

loitho commented Dec 27, 2019

ryanpetrello commented Jan 3, 2020

peacengell commented May 30, 2023 • edited Loading

newlandk commented Jan 19, 2018 •

edited

Loading

newlandk commented Jan 24, 2018 •

edited

Loading

newlandk commented Apr 6, 2018 •

edited

Loading

newlandk commented Feb 25, 2019 •

edited

Loading

naren-ambati commented Aug 19, 2019 •

edited

Loading

peacengell commented May 30, 2023 •

edited

Loading