[exporter/elasticsearch] Logstash format compatibility

[JaredTan95]

Description: Logstash format compatibility. Traces or Logs data can be written into an index in logstash format.

Link to tracking Issue:
close #29624

Documentation: added some descriptions for logstash_format configurations.

1. otel-col.yaml

receivers:
  otlp:
    protocols:
      grpc:
  filelog:
    include: [ ./examples/kubernetes/varlogpods/containerd_logs-0_000011112222333344445555666677778888/logs/0.log ]
    start_at: beginning
    operators:
      # Find out which format is used by kubernetes
      - type: router
        id: get-format
        routes:
          - output: parser-docker
            expr: 'body matches "^\\{"'
          - output: parser-crio
            expr: 'body matches "^[^ Z]+ "'
          - output: parser-containerd
            expr: 'body matches "^[^ Z]+Z"'
      # Parse CRI-O format
      - type: regex_parser
        id: parser-crio
        regex: '^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout_type: gotime
          layout: '2006-01-02T15:04:05.999999999Z07:00'
      # Parse CRI-Containerd format
      - type: regex_parser
        id: parser-containerd
        regex: '^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$'
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # Parse Docker format
      - type: json_parser
        id: parser-docker
        output: extract_metadata_from_filepath
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      # Extract metadata from file path
      - type: regex_parser
        id: extract_metadata_from_filepath
        regex: '^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$'
        parse_from: attributes["log.file.path"]
        cache:
          size: 128  # default maximum amount of Pods per Node is 110
      # Update body field after finishing all parsing
      - type: move
        from: attributes.log
        to: body
      # Rename attributes
      - type: move
        from: attributes.stream
        to: attributes["log.iostream"]
      - type: move
        from: attributes.container_name
        to: resource["k8s.container.name"]
      - type: move
        from: attributes.namespace
        to: resource["k8s.namespace.name"]
      - type: move
        from: attributes.pod_name
        to: resource["k8s.pod.name"]
      - type: move
        from: attributes.restart_count
        to: resource["k8s.container.restart_count"]
      - type: move
        from: attributes.uid
        to: resource["k8s.pod.uid"]
exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      label1: value1

  elasticsearch/log:
    tls:
      insecure: false
    endpoints: [http://localhost:9200]
    logs_index: otlp-logs
    logstash_format:
      enabled: true
    timeout: 2m
    flush:
      bytes: 10485760
    retry:
      max_requests: 5
    sending_queue:
      enabled: true
  elasticsearch/traces:
    tls:
      insecure: false
    endpoints: [http://localhost:9200]
    traces_index: otlp-traces
    logstash_format:
      enabled: true
    timeout: 2m
    flush:
      bytes: 10485760
    retry:
      max_requests: 5
    sending_queue:
      enabled: true

  debug:

processors:
  batch:

extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    logs:
      receivers: [otlp,filelog]
      processors: [batch]
      exporters: [debug, elasticsearch/log]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, elasticsearch/traces]

3. es index created when otel-col write traces and logs:

4. query index data:

[JaredTan95]

@fatsheep9146 Can you help to review this?

[fatsheep9146]

elasticsearchexporter/logs_exporter.go:22: File is not gci-ed with --skip-generated -s standard -s default -s prefix(github.com/open-telemetry/opentelemetry-collector-contrib) (gci)
index string

@JaredTan95 failed lint should be fixed