elasticsearch

Elasticsearch

Creates tcpdog index

PUT tcpdog
{
  "mappings": {
    "properties": {
      "Timestamp": {
        "type":   "date",
        "format": "epoch_second"
      },
      "GeoLocation": {
        "type": "geo_point"
      }
    }
  }
}

tcpdog yaml config

config:
  urls:
    - ELASTICSEARCH_CLUSTER_ADDR
  username: HTTP_BASIC_AUTH
  password: HTTP_BASIC_AUTH
  geoField: TCP_FIELD # tcp field supposed to resolve to Geo, default: SAddr
  cloudID: CLOUD_ID # endpoint for the Elastic Service (https://elastic.co/cloud)
  apiKey: API_KEY # base64-encoded token for authorization; if set, overrides username and password
  index: INDEX_NAME # elasticsearch index name, default: tcpdog
  workers: WORKERS_NUMBER # number of marshaler workers
  flushBytes: FLUSH_BYTES # flush threshold in bytes, default: 5MB
  flushInterval: FLUSH_INTERVAL_SEC # periodic flush interval in second, default: 1 second
  tlsConfig: TLS_Config # TLS configuration, check TLS config page

TLS Config

---

Demo with docker [gRPC + Elasticsearch]

Requirement: A Linux host (kernel versions 4.16 or later) with the below:

• libbcc
• docker

In this demo, a host sends the tcp informations to itself through gRPC and you can see the result on the Kibana

Single node cluster with docker

docker run -d -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.2

Kibana with docker

docker run -d --link YOUR_ELASTICSEARCH_CONTAINER_NAME_OR_ID:elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.10.2

Open a browser and try http://localhost:5601 and go to the Dev Tools then create tcpdog index:

PUT tcpdog
{
  "mappings": {
    "properties": {
      "Timestamp": {
        "type":   "date",
        "format": "epoch_second"
      },
      "GeoLocation": {
        "type": "geo_point"
      }
    }
  }
}

Now you need to create index pattern at Stack Management >> Index patterns Call your index pattern like tcpdog*

We need to clone the repo to have the configurations and download the binaries as below:

git clone https://github.com/mehrdadrad/tcpdog.git
cd scripts/examples/grpc_elastic
curl -sL https://github.com/mehrdadrad/tcpdog/releases/latest/download/tcpdog -o tcpdog
curl -sL https://github.com/mehrdadrad/tcpdog/releases/latest/download/tcpdog-server -o tcpdog-server
chmod +x tcpdog
chmod +x tcpdog-server

Run server

./tcpdog-server -config ./server.yml

Open another terminal and run agent

./tcpdog -config ./agent.yml

If your server doesn't have any active TCP connections try the below simple curl at another terminal:

curl www.google.com

You should be able to see the ingested data at Kibana >> Discover