Admin panel does not work with some 1.29.2 instances

[deajan]

Subject

On a fresh installed Vaultwarden instance, I cannot use the admin panel since the js script won't load.
In my navigator, I get the following:

Failed to load resource: the server responded with a status of 400 (Bad Request)
diagnostics:1 Refused to execute script from 'https://my.vaultwarden.instance.local/vw_static/bootstrap-native.js' because its MIME type ('application/json') is not executable, and strict MIME type checking is enabled.
admin_diagnostics.js:138 Uncaught ReferenceError: bootstrap is not defined
    at HTMLButtonElement.copyToClipboard (admin_diagnostics.js:138:5)

Deployment environment

Running Vaultwarden 1.29.2, installed via docker vaultwarden/server:latest image (actually podman on RHEL 9.2).
Using Vivaldi browser (latest), but also tried Edge and Brave (same results).
Vaultwarden is reverse proxied by HaProxy.
MySQL version: mariadb-server-10.5.16-2.el9_0.x86_64

Steps to reproduce

Open webwault /admin route, enter ADMIN TOKEN, and try to open anything in settings/users/organizations panel

Expected behaviour

^^

Actual behaviour

Interface is unresponsive since JS never got loaded

Troubleshooting data

Strangely enough, I have another Vaultwarden setup (a test one) with SQLite, same version as this one, and everything works flawlessly. I don't find any explanation for this.

Support string:

Your environment (Generated via diagnostics page)

• Vaultwarden version: v1.29.2
• Web-vault version: v2023.7.1
• OS/Arch: linux/x86_64
• Running within Docker: true (Base: Debian)
• Environment settings overridden: false
• Uses a reverse proxy: true
• IP Header check: true (X-Real-IP)
• Internet access: true
• Internet access via a proxy: false
• DNS Check: true
• Browser/Server Time Check: true
• Server/NTP Time Check: true
• Domain Configuration Check: true
• HTTPS Check: true
• Database type: MySQL
• Database version: 10.5.16-MariaDB
• Clients used:
• Reverse proxy and version:
• Other relevant information:

Config (Generated via diagnostics page)

Show Running Config

Environment settings which are overridden:

{
  "_duo_akey": null,
  "_enable_duo": true,
  "_enable_email_2fa": true,
  "_enable_smtp": true,
  "_enable_yubico": true,
  "_icon_service_csp": "",
  "_icon_service_url": "",
  "_ip_header_enabled": true,
  "_smtp_img_src": "cid:",
  "admin_ratelimit_max_burst": 3,
  "admin_ratelimit_seconds": 300,
  "admin_session_lifetime": 20,
  "admin_token": "***",
  "allowed_iframe_ancestors": "",
  "attachments_folder": "data/attachments",
  "auth_request_purge_schedule": "30 * * * * *",
  "authenticator_disable_time_drift": false,
  "data_folder": "data",
  "database_conn_init": "",
  "database_max_conns": 10,
  "database_timeout": 30,
  "database_url": "*****://*******************************************************************************",
  "db_connection_retries": 15,
  "disable_2fa_remember": false,
  "disable_admin_token": false,
  "disable_icon_download": false,
  "domain": "*****://************************",
  "domain_origin": "*****://************************",
  "domain_path": "",
  "domain_set": true,
  "duo_host": null,
  "duo_ikey": null,
  "duo_skey": null,
  "email_attempts_limit": 3,
  "email_expiration_time": 600,
  "email_token_size": 6,
  "emergency_access_allowed": true,
  "emergency_notification_reminder_schedule": "0 3 * * * *",
  "emergency_request_timeout_schedule": "0 7 * * * *",
  "enable_db_wal": true,
  "event_cleanup_schedule": "0 10 0 * * *",
  "events_days_retain": null,
  "extended_logging": true,
  "helo_name": null,
  "hibp_api_key": null,
  "icon_blacklist_non_global_ips": true,
  "icon_blacklist_regex": null,
  "icon_cache_folder": "data/icon_cache",
  "icon_cache_negttl": 259200,
  "icon_cache_ttl": 2592000,
  "icon_download_timeout": 10,
  "icon_redirect_code": 302,
  "icon_service": "internal",
  "incomplete_2fa_schedule": "30 * * * * *",
  "incomplete_2fa_time_limit": 3,
  "invitation_expiration_hours": 120,
  "invitation_org_name": "Vaultwarden",
  "invitations_allowed": true,
  "ip_header": "X-Real-IP",
  "job_poll_interval_ms": 30000,
  "log_file": "/data/vaultwarden.log",
  "log_level": "Info",
  "log_timestamp_format": "%Y-%m-%d %H:%M:%S.%3f",
  "login_ratelimit_max_burst": 10,
  "login_ratelimit_seconds": 60,
  "org_attachment_limit": 1024000,
  "org_creation_users": "*****************************",
  "org_events_enabled": true,
  "org_groups_enabled": false,
  "password_hints_allowed": true,
  "password_iterations": 600000,
  "push_enabled": false,
  "push_installation_id": "***",
  "push_installation_key": "***",
  "push_relay_uri": "https://push.bitwarden.com",
  "reload_templates": false,
  "require_device_email": false,
  "rsa_key_filename": "data/rsa_key",
  "send_purge_schedule": "0 5 * * * *",
  "sendmail_command": null,
  "sends_allowed": true,
  "sends_folder": "data/sends",
  "show_password_hint": false,
  "signups_allowed": true,
  "signups_domains_whitelist": "*********",
  "signups_verify": false,
  "signups_verify_resend_limit": 6,
  "signups_verify_resend_time": 3600,
  "smtp_accept_invalid_certs": false,
  "smtp_accept_invalid_hostnames": false,
  "smtp_auth_mechanism": null,
  "smtp_debug": false,
  "smtp_embed_images": true,
  "smtp_explicit_tls": null,
  "smtp_from": "*******************",
  "smtp_from_name": "Gestionnaire Mot de Passe",
  "smtp_host": "*********************",
  "smtp_password": null,
  "smtp_port": 25,
  "smtp_security": "starttls",
  "smtp_ssl": null,
  "smtp_timeout": 15,
  "smtp_username": null,
  "templates_folder": "/templates",
  "tmp_folder": "data/tmp",
  "trash_auto_delete_days": null,
  "trash_purge_schedule": "0 5 0 * * *",
  "use_sendmail": false,
  "use_syslog": false,
  "user_attachment_limit": 102400,
  "web_vault_enabled": true,
  "web_vault_folder": "web-vault/",
  "websocket_address": "0.0.0.0",
  "websocket_enabled": true,
  "websocket_port": 3012,
  "yubico_client_id": null,
  "yubico_secret_key": null,
  "yubico_server": null
}

[deajan]

Strangely enough, when comparing source of both of my 1.29.2 Vaultwarden setups, both on docker, both behind Haproxy, there is a small difference.

The HTML sources of the admin settings page of the one described above that doesn't work end with:

    </style>
    <script src="/vw_static/admin_settings.js"></script>

    <!-- This script needs to be at the bottom, else it will fail! -->
    <script src="/vw_static/bootstrap-native.js"></script>
</body>
</html>

Whereas the sources of my other instance ends with:

    </style>
    <script src="/vw_static/admin_settings.js"></script>

    <!-- This script needs to be at the bottom, else it will fail! -->
    <script src="/vw_static/bootstrap.bundle.js"></script>
</body>
</html>

Of course I have no idea why.
When inspecting the network request, I see that bootstrap-native.js simply doesn't exist:

{"ErrorModel":{"Message":"Static file not found: bootstrap-native.js","Object":"error"},"ExceptionMessage":null,"ExceptionStackTrace":null,"InnerExceptionMessage":null,"Message":"Static file not found: bootstrap-native.js","Object":"error","ValidationErrors":{"":["Static file not found: bootstrap-native.js"]},"error":"","error_description":""}

[stefan0xC]

bootstrap-native.js was replaced with bootstrap.bundle.js in #3730. So it should only load the latter. Maybe you have the page still cached? (Maybe check your HAProxy settings if you override the cache-control settings?)

[deajan]

This was a new and fresh setup on a separate domain. Cache cannot be involved here. Any ideas where to search ?

[stefan0xC]

Any ideas where to search ?

Do you maybe override the admin templates? Then you have to update them as well.

[BlackDex]

It's either that, or the reverse proxy/browser is caching some files.

[deajan]

Fair enough.
It was the admin templates being overrided indeed, thanks @stefan0xC
@BlackDex Sorry for the noise ;)

[stefan0xC]

No worries. I sometimes forget that you can do that.

@BlackDex Could we detect if a template is overriden?

[BlackDex]

I think we somehow could. That might be a good extra check to add.