Skip to content

Latest commit

 

History

History
40 lines (35 loc) · 2.38 KB

README.md

File metadata and controls

40 lines (35 loc) · 2.38 KB

qumulo-analytics-elasticsearch

This script will collect analytics data from qumulo storage clusters, transform and format data, and submit the results to an elasticsearch cluster of your choice

It includes the following:

  • metrics.py: A script that queries qumulo storage clusters and submits analytics data to elasticsearch
  • credentials.json: Simple json for hostname and credential combinations for the qumulo rest client to query
  • requirements.txt: A list of python packages required for the script to run
  • qMetricsExamples.json: A quick-start template for visualizing script output in Kibana

For quick testing/development setup:

  1. Download the qumulo command line tools
    • Visit the storage cluster web interface
    • Under 'APIs & Tools' -> click 'Download command-line tools'
  2. Create Qumulo API user
    • Visit the web interfaces of the storage clusters to query, login as admin
    • Under 'Sharing' -> 'Users & Groups' create a qumulo-metrics user with API read privileges
  3. Prepare the python requirements
    • Clone this repository
    • Install requirements via 'pip install -r requirements.txt from inside the folder
  4. Prepare required files
    • Copy 'metrics.py' and 'credentials.json' to the unzipped qumulo comand line tools folder
    • Open credentials.json and fill out credentials from step 2 for each cluster
    • Add/remove json entries as needed
  5. Customize elasticsearch output endpoint(s)
    • Inside metrics.py enter the hostname of your elasticsearch endpoint(s)
    • If you use deviantony/docker-elk customize the 'docker-compose.yml' for additional JVM heap size with "ES_JAVA_OPTS: "-Xmx4g -Xms4g" and increase your docker engine to a minimum of 8GB memory and 4 cores
    • This script will default to the localhost elasticsearch cluster
  6. Start the script
    • In the converged folder execute 'python metrics.py' and monitor the output of the script to ensure the shipper is working.
  7. Visualize Outputs:
    • Visit Kibana at: http://127.0.0.1:5601
    • create default index pattern 'qperf-*' and 'create'
    • Under 'Management' -> 'Index Patterns' -> 'Create Index Pattern' add 'qcapacity-' and 'qfiles-'
    • Under 'Management' -> 'Saved Objects' -> 'Import' upload the qMetricsExamples.json file
    • Under 'Dashboard' -> 'Open' select 'MasterDashQ' and edit the time-frame as desired.

Author: Felix Russell (c) University of Washington Institute for Health Metrics and Evaluation