Not support arm64 images

[Owenxh]

Demo images not working on arm64(m1) k8s cluster.

[TylerHelmuth]

@Owenxh please supply more details such as the helm chart version you are using and any error messages you are seeing.

[Owenxh]

Env

• k8s cluster version:
  Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/arm64"}

• open-telemetry/demo version: v1.2.1

• Linux arm64 （Virtual Machines on Mac M1）

Install opentelemetry-demo

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install my-otel-demo open-telemetry/opentelemetry-demo

Logs of some pod

k logs deployment.apps/my-otel-demo-accountingservice

exec ./accountingservice: exec format error

k logs my-otel-demo-loadgenerator-d56d9c664-ztrg7

exec /bin/sh: exec format error

Status of Pods

k get pod

NAME                                                  READY   STATUS              RESTARTS      AGE     IP              NODE          NOMINATED NODE   READINESS GATES
my-otel-demo-accountingservice-5c76df7958-slqv2       0/1     CrashLoopBackOff    6 (69s ago)   6m56s   10.234.204.47   k8s-node-03   <none>           <none>
my-otel-demo-adservice-6b646cc8cf-dcc75               0/1     ContainerCreating   0             6m55s   <none>          k8s-node-02   <none>           <none>
my-otel-demo-cartservice-5fd4dcc98f-5wc24             0/1     CrashLoopBackOff    5 (47s ago)   6m54s   10.234.254.27   k8s-node-01   <none>           <none>
my-otel-demo-checkoutservice-56ff79c9d-fw679          0/1     CrashLoopBackOff    6 (68s ago)   6m55s   10.234.16.80    k8s-node-02   <none>           <none>
my-otel-demo-currencyservice-86cff54b7d-k7nhk         0/1     ContainerCreating   0             6m54s   <none>          k8s-node-03   <none>           <none>
my-otel-demo-emailservice-677f674b4d-fztc6            0/1     ContainerCreating   0             6m54s   <none>          k8s-node-03   <none>           <none>
my-otel-demo-featureflagservice-765dbd9f7f-k8r85      0/1     CrashLoopBackOff    4 (50s ago)   6m56s   10.234.16.77    k8s-node-02   <none>           <none>
my-otel-demo-ffspostgres-68dd8cc54f-dfg6l             1/1     Running             0             6m56s   10.234.254.25   k8s-node-01   <none>           <none>
my-otel-demo-frauddetectionservice-678795cc64-55ww5   0/1     CrashLoopBackOff    6 (61s ago)   6m56s   10.234.254.24   k8s-node-01   <none>           <none>
my-otel-demo-frontend-6ddb77486c-pvvlf                0/1     ContainerCreating   0             6m54s   <none>          k8s-node-01   <none>           <none>
my-otel-demo-frontendproxy-9685b56c7-8t8n5            0/1     CrashLoopBackOff    6 (75s ago)   6m55s   10.234.254.26   k8s-node-01   <none>           <none>
my-otel-demo-grafana-5c77687d9b-cx8q2                 0/1     ContainerCreating   0             6m53s   <none>          k8s-node-01   <none>           <none>
my-otel-demo-jaeger-656d6dfbf7-hmh68                  1/1     Running             0             6m56s   10.234.254.23   k8s-node-01   <none>           <none>
my-otel-demo-kafka-777658c5cc-c48tz                   0/1     CrashLoopBackOff    6 (74s ago)   6m54s   10.234.254.29   k8s-node-01   <none>           <none>
my-otel-demo-loadgenerator-d56d9c664-ztrg7            0/1     CrashLoopBackOff    6 (71s ago)   6m55s   10.234.204.49   k8s-node-03   <none>           <none>
my-otel-demo-otelcol-74c56d7bc4-nwwdw                 0/1     ContainerCreating   0             6m54s   <none>          k8s-node-02   <none>           <none>
my-otel-demo-paymentservice-556874756-dhm49           0/1     CrashLoopBackOff    6 (70s ago)   6m56s   10.234.16.78    k8s-node-02   <none>           <none>
my-otel-demo-productcatalogservice-688dcc4c7d-bf97w   0/1     ContainerCreating   0             6m56s   <none>          k8s-node-03   <none>           <none>
my-otel-demo-prometheus-server-5b6cc54554-k8gzw       0/1     Pending             0             6m56s   <none>          <none>        <none>           <none>
my-otel-demo-quoteservice-887485db9-ss7w7             0/1     ContainerCreating   0             6m56s   <none>          k8s-node-03   <none>           <none>
my-otel-demo-recommendationservice-55fd9c9f49-xt8ng   0/1     CrashLoopBackOff    6 (62s ago)   6m55s   10.234.204.50   k8s-node-03   <none>           <none>
my-otel-demo-redis-768c7847cb-txlm8                   1/1     Running             0             6m56s   10.234.204.45   k8s-node-03   <none>           <none>
my-otel-demo-shippingservice-856cfdb489-8q5lk         0/1     CrashLoopBackOff    6 (55s ago)   6m56s   10.234.16.76    k8s-node-02   <none>           <none>
otel-agent-cx2bs                                      1/1     Running             0             23h     10.234.204.31   k8s-node-03   <none>           <none>
otel-agent-h2rxs                                      1/1     Running             0             23h     10.234.16.74    k8s-node-02   <none>           <none>
otel-agent-hlhsw                                      1/1     Running             0             23h     10.234.254.22   k8s-node-01   <none>           <none>
otel-collector-7bd8c45467-w9xrt                       1/1     Running             0             23h     10.234.16.75    k8s-node-02   <none>           <none>

[TylerHelmuth]

@Owenxh I am seeing issues with featureflagservice which historically has had trouble on M1 macs. @puckpuck @austinlparker is the demo still publishing arm64 images?

[TylerHelmuth]

If I do a docker pull it looks like it pulls an amd image:

tylerhelmuth@tylerhelmuth opentelemetry-helm-charts % docker pull ghcr.io/open-telemetry/demo:v1.2.1-currencyservice
v1.2.1-currencyservice: Pulling from open-telemetry/demo
846c0b181fff: Pull complete 
fe3f2a8ebaf4: Pull complete 
4f4fb700ef54: Pull complete 
96debf61826a: Pull complete 
43cbe0ab3e35: Pull complete 
bf23da66a2f5: Pull complete 
64133b9a85c0: Pull complete 
f18aede9083d: Pull complete 
Digest: sha256:37b79077c5d7463f64765e7d50e49149f1218645a6343fb3906e01e3be0f7ffc
Status: Downloaded newer image for ghcr.io/open-telemetry/demo:v1.2.1-currencyservice
ghcr.io/open-telemetry/demo:v1.2.1-currencyservice
tylerhelmuth@tylerhelmuth opentelemetry-helm-charts % docker inspect ghcr.io/open-telemetry/demo:v1.2.1-currencyservice
[
    {
        "Id": "sha256:527c70578fd9790e0502e65cbcf9e263435642508289b072a4ccbe320a629239",
        "RepoTags": [
            "ghcr.io/open-telemetry/demo:v1.2.1-currencyservice"
        ],
        "RepoDigests": [
            "ghcr.io/open-telemetry/demo@sha256:37b79077c5d7463f64765e7d50e49149f1218645a6343fb3906e01e3be0f7ffc"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2022-12-09T14:24:25.904801985Z",
        "Container": "",
        "ContainerConfig": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": null
        },
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive"
            ],
            "Cmd": null,
            "Image": "",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/bin/sh",
                "-c",
                "/usr/local/bin/currencyservice ${CURRENCY_SERVICE_PORT}"
            ],
            "OnBuild": null,
            "Labels": null
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 634690956,
        "VirtualSize": 634690956,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/77652d0238177f67f7876cd050d131b197700f1026d1eee9f1ea7e736a010f06/diff:/var/lib/docker/overlay2/c71bb7fe4a8799456c0be35eea35844afe40dc8ea265301796f05c8a2cef0379/diff:/var/lib/docker/overlay2/0de56092142a62f259123e64a655243bd186afbc33fc41f59fffdd69f5f7d0fa/diff:/var/lib/docker/overlay2/b67250c53bbca34f347fa75aa4ac26e61149bba0af23636f684059627de700dd/diff:/var/lib/docker/overlay2/e3f2ca6ef3b8fc0b60cb9ebaadb1d149feecfc38421003ba905da64b2c21ae92/diff:/var/lib/docker/overlay2/ba9fb8198d72ba770cef457e75fbc7352583097f933c0c1ed10dda949bd5c2ad/diff:/var/lib/docker/overlay2/0bee6c2d42117e3b62531db5b865577fc41ca9e4e689e29bb0ca9b8e3a5d1b8f/diff",
                "MergedDir": "/var/lib/docker/overlay2/27a7ccdfd2259039370600d228f92404ebfdc57c09c860f04e652814b9879ae0/merged",
                "UpperDir": "/var/lib/docker/overlay2/27a7ccdfd2259039370600d228f92404ebfdc57c09c860f04e652814b9879ae0/diff",
                "WorkDir": "/var/lib/docker/overlay2/27a7ccdfd2259039370600d228f92404ebfdc57c09c860f04e652814b9879ae0/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:0002c93bdb3704dd9e36ce5153ef637f84de253015f3ee330468dccdeacad60b",
                "sha256:2a330da6b5d2facaf706351cb99d7831492f835bafac0a2d4c24832112555b18",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:d86dffec057cab099660791c22f01bb21424e9f506a9a75f9dc845a9a4668d27",
                "sha256:7bcbb16e1074079acd26fd5357c101d0de97a53f76cacbc172cb7b81d3fb7aa4",
                "sha256:da5a6e11e44333477c0742b73ff7424d3311e2cf15bd5501376016526856c914",
                "sha256:c5b436a2fd570ae08b6d713be078a78276856db30326031c8b36c55859ce4de7",
                "sha256:9051b920b10bc7b4b43d9a81e60a90478b33fc5bcffc1fae637e5ff418dde051"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

[VtG242]

I would say that demo is working on M1. At least in my docker desktop kubernetes cluster. In my case the problem was only with image for featureflagservice and build featureflagservice before start the demo solved the problem:
docker compose build featureflagservice

The rest of CrashLoopBackOff issues in my case have been caused by OOM and setting a bigger memory limits for problematic deployments solved it.

[puckpuck]

Now that the otel demo 1.3.0 and accompanying helm chart (0.18.0) are released, can you test everything again? We believe we fixed the arm build issue, and have made some changes to memory limits.

[TylerHelmuth]

@puckpuck wasn't open-telemetry/opentelemetry-demo@ea64998 merged after 1.3.0 was published?

[eskirk]

@puckpuck did you mean 0.17.0? helm update claims there is not a 0.18.0 release.

[puckpuck]

Yes sorry Helm 0.17.0.

We still don't publish ARM builds (my mistake, that PR got merged after the last release). The 1.3.0 release does fix some issues with building locally on ARM, particularly for the featureflag service.

Part of the next demo release (1.4) is to stabilize all code/telemetry and produce multi-arch images.

[puckpuck]

The latest release of the Demo and the accompanying Helm chart update #672 added ARM support.

This is fixed now.