latest nginx-alpine tag is not working properly

[fichte]

in the following example this is a small docker stack with owasp/modsecurity-crs as proxy and portainer behind owasp/modsecurity-crs

the latest tag owasp/modsecurity-crs:4.10.0-nginx-alpine-202501270601 produces weird errors

the previous tag owasp/modsecurity-crs:4.10.0-nginx-alpine-202501050801 works perfectly

it looks suspicious to me. both image are from the public docker registry. i dont have the time to debug this. i can assure that the config is exactly the same.

[theseion]

Thanks for the report @fichte. Unfortunately, I don't see any obvious issues when I run the image locally. If you could provide your configuration for your Portainer setup, I could try to replicate it.

Have you tried using the regular nginx image?

[Serverfrog]

I just had the same kind of problem for me. I was not running it on Portainer but on a Kubernetes Cluster. I was also just using the nginx tag.

The Problem was for me that, if the Response Content is above a Size (can't really tell what the minimum is, but should be above 4000 Characters), the First ~1080 Characters get "corrupted" in the sense that they will be some later content in there.

Thats result for example with a JSON response that wont start with { but with some later content and results in malformed JSON.
That also results in Content-Encoding errors with other Sources, where the Browser validates it (CSS, JS, etc).
Going back Versions fixed that for me, so it looks like this is really a problem with that build (i didn't changed anything beside the Image).

There was no difference in result if the response was from the modsecurity container itself (static file serving) or a reverse-proxy configuration.
Also its sometimes correct, so it feels a bit like a memory corruption to me. The First Request had a high chance to be correct, the exact same one the next time resulted often in the wrong/corrupt response.

[fichte]

just open the developer console and go to http://localhost

i just added portainer and removed the comments from the original docker-compose.yaml

working docker-compose.yml with previous tag

x-defaults: &default-settings
  environment:
    SERVERNAME: localhost
    PARANOIA: 1
    BLOCKING_PARANOIA: 1
    ANOMALY_INBOUND: 5
    ANOMALY_OUTBOUND: 4
    REPORTING_LEVEL: 2
  volumes:
    - ./REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf:/etc/modsecurity.d/owasp-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
    - ./RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf:/etc/modsecurity.d/owasp-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

services:
  crs-nginx:
    environment:
      - BACKEND="http://portainer:9000"
    image: owasp/modsecurity-crs:4.10.0-nginx-alpine-202501050801
    ports:
      - "80:8080"
    <<: *default-settings

  portainer:
    image: portainer/portainer-ce:sts
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

non working docker-compose.yml with latest version

x-defaults: &default-settings
  environment:
    SERVERNAME: localhost
    PARANOIA: 1
    BLOCKING_PARANOIA: 1
    ANOMALY_INBOUND: 5
    ANOMALY_OUTBOUND: 4
    REPORTING_LEVEL: 2
  volumes:
    - ./REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf:/etc/modsecurity.d/owasp-crs/rules/REQUEST-900-EXCLUSION-RULES-BEFORE-CRS.conf
    - ./RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf:/etc/modsecurity.d/owasp-crs/rules/RESPONSE-999-EXCLUSION-RULES-AFTER-CRS.conf

services:
  crs-nginx:
    environment:
      - BACKEND="http://portainer:9000"
    image: owasp/modsecurity-crs:nginx-alpine
    ports:
      - "80:8080"
    <<: *default-settings

  portainer:
    image: portainer/portainer-ce:sts
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

[leitgab]

I can confirm above mentioned behavior as described by @Serverfrog with latest nginx tagged version on Podman with podman kube play.

[theseion]

Thank you all for your reports. We'll try to reproduce the issue. The only thing i can think of at the moment that could have caused this is the QEMU update we did.

[theseion]

I can confirm the issue. It really does look like some kind of memory corruption / memory offset computation issue. I strongly suspect that we're seeing an issue with QEMU. I will upgrade to v9.2.0 and hope that that fixes the issue.

[theseion]

@leitgab @fichte which image architecture did you pull? I assume for Kubernetes it was linux/amd64?

[fichte]

i tried both linux/arm64 and linux/amd64

[fichte]

@theseion when do you expect to release a fixed version?

[theseion]

Unfortunately, I haven't yet been able to pinpoint the issue. I'm working on a fix and hope to get it done within the next couple of hours, but it might be tomorrow, depending on what I (don't) find.

[fichte]

@theseion maybe you can share some of your findings :) it looks really suspicious to me, maybe there is something bad happening within the dependencies?

[theseion]

What I can say up to now:

• both nginx and nginx-alpine images are affected
• the issue does not appear to be a change that we made (consciously)
• I can no longer build functioning nginx* images at all, even when checking out older versions of the repository
• it cannot be related to QEMU, as I have the same issue when building locally
• it does not appear to be related to buildkit, the same issue occurs with version v0.18.2 (v0.19.0 is currently stable)
• the upstream nginxinc/nginx-unprivileged images are fine

[fichte]

it is definately the modsecurity module, i deactivated it inside the running container and reloaded the nginx process. then everything works properly.

[theseion]

Thanks. I've been slowly coming to the same conclusion.

[fichte]

at compile stage i see some "fatal" stuff, dont know if they are normal at that point

20.03 libtoolize: putting auxiliary files in '.'.
20.03 libtoolize: copying file './ltmain.sh'
20.04 libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'build'.
20.04 libtoolize: copying file 'build/libtool.m4'
20.05 libtoolize: copying file 'build/ltoptions.m4'
20.05 libtoolize: copying file 'build/ltsugar.m4'
20.05 libtoolize: copying file 'build/ltversion.m4'
20.06 libtoolize: copying file 'build/lt~obsolete.m4'
21.63 fatal: No annotated tags can describe '580fe192df1694934f93e3e008058996ea479e35'.
21.63 However, there were unannotated tags: try --tags.
22.38 fatal: No annotated tags can describe '580fe192df1694934f93e3e008058996ea479e35'.
22.38 However, there were unannotated tags: try --tags.
23.70 fatal: No annotated tags can describe '580fe192df1694934f93e3e008058996ea479e35'.
23.70 However, there were unannotated tags: try --tags.
24.18 configure.ac:50: installing './ar-lib'
24.18 configure.ac:50: installing './compile'
24.19 configure.ac:168: installing './config.guess'
24.19 configure.ac:168: installing './config.sub'
24.19 configure.ac:45: installing './install-sh'
24.19 configure.ac:45: installing './missing'
24.24 parallel-tests: installing './test-driver'
24.29 examples/multiprocess_c/Makefile.am: installing './depcomp'
24.57 configure.ac: installing './ylwrap'
25.67 fatal: No annotated tags can describe '580fe192df1694934f93e3e008058996ea479e35'.
25.67 However, there were unannotated tags: try --tags.
25.90 + ./configure --with-yajl --with-ssdeep --with-lmdb --with-pcre2 --with-maxmind --enable-silent-rules

[airween]

Guys, is it possible that the image uses the latest (and unreleased) ModSecurity-nginx connector? Just saying that has a bug which produces (I risk) the same behavior. See ModSecurity-nginx/336.

If you use the unreleased version, please do not do that.

[theseion]

Yep, that's it. We're using the main branch. I'll fix that dependency tomorrow morning. Off to bed now. Thanks for the help.

[airween]

Though this is a different symptom but I'm afraid the root cause (broken Nginx connector) is the same.