apm agent fails while using http apm url

[alita-moore]

Describe the bug

When using istio sidecar / mtls for securing inter-node communication, it is required that you use http instead of https. However, when the agent is loading using an http apm url, the connection fails.

To Reproduce

Steps to reproduce the behavior:

1. use this config when setting up the apm server:

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm
  namespace: elastic-system
spec:
  version: ${version}
  count: 1
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  elasticsearchRef:
    name: search
    namespace: elastic-system
  kibanaRef:
    name: kibana
    namespace: elastic-system

init the node apm client

const apm = elastic.start({
    serviceName,
    secretToken,
    serverUrl: "http://apm-apm-http.elastic-system.svc.cluster.local:8200",
    verifyServerCert: false
  });

Notice the failed connections

{"log.level":"error","@timestamp":"2025-01-01T16:01:13.133Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error: error fetching APM Server version: read ECONNRESET"}
{"log.level":"error","@timestamp":"2025-01-01T16:01:13.135Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error (ECONNRESET): read ECONNRESET"}
{"instanceID":"644eb56a-49c8-4b69-abde-211d18c866a2","level":"info","message":"Connection successfully (re)established","service":"ml-serving-core","timestamp":"2025-01-01T16:01:13.138+00:00"}
{"instanceID":"644eb56a-49c8-4b69-abde-211d18c866a2","level":"info","message":"Re-establishing bindings for exchange 'direct'","service":"ml-serving-core","timestamp":"2025-01-01T16:01:13.139+00:00"}
{"log.level":"error","@timestamp":"2025-01-01T16:01:13.180Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error (ECONNRESET): read ECONNRESET"}
{"log.level":"error","@timestamp":"2025-01-01T16:01:43.092Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","message":"APM Server transport error (ECONNRESET): read ECONNRESET"}

Expected behavior

Environment (please complete the following information)

• OS: linux 6.1.0-27-amd64
• Node.js version: nodejs 4.9.0
• APM Server version: 8.17.0
• Agent version: Elastic 4.9.0

How are you starting the agent? (please tick one of the boxes)

• Calling agent.start() directly (e.g. require('elastic-apm-node').start(...))
• Requiring elastic-apm-node/start from within the source code
• Starting node with -r elastic-apm-node/start

Additional context

Agent config options:

Click to expand

{
    serviceName,
    secretToken,
    serverUrl: "http://apm-apm-http.elastic-system.svc.cluster.local:8200",
    verifyServerCert: false
  }

package.json dependencies:

Click to expand

 "elastic-apm-node": "^4.9.0",

NOTE:
keep in mind that I verified that the apm server itself is working as expected:

curl http://apm-apm-http.elastic-system.svc.cluster.local:8200 -H "Authorization: Bearer XXXXXXX"
{
  "build_date": "2024-12-10T15:27:20Z",
  "build_sha": "8e83e22899d80430e15dc09131ec9a219fce4638",
  "publish_ready": true,
  "version": "8.17.0"
}

[trentm]

curl http://apm-apm-http.elastic-system.svc.cluster.local:8200 -H "Authorization: Bearer XXXXXXX"
...

@alita-moore Are you able to show the output of that same curl ... command, but using curl's -v argument? That will show protocol details. I'm curious if APM server is still expecting a TLS connection with that configuration. I could be wrong.