Merged in port to telemetry analysis framework

Formats.mkd

@@ -0,0 +1,95 @@
+File Format used for Telemetry Dashboard
+========================================
+_All format described here are internal, not for external consumption._
+
+External users should include `telemetry.js` and consume data through this
+interface. Reading the raw data is hard and these data format may change, but
+the javascript interface is designed to be reasonably stable.
+
+Analysis Tasks Output Format
+----------------------------
+The analysis task writes a single file to disk called `result.txt`.
+
+/my/dim/../ JSON
+
+  filtrerPath:
+    revision:
+    buildid:
+    histogram:
+
+
+
+Web Facing Bucket Layout
+------------------------
+
+LATEST_VERSION = v2
+v1/
+  data
+v2/
+  check-points.json
+  check-points/ (one for every week)
+    YYYYMMDDhhmmss/
+      versions.json
+      FILES_PROCESSED
+      FILES_MISSING
+      <channel>/<version>/
+        MEASURE-by-build-date.json
+        MEASURE-by-submission-date.json
+        filter-tree.json
+        histograms.json
+        revisions.json
+  latest-current.json = most recent current, contents of versions.json
+  current/
+    YYYYMMDDhhmmss/
+      versions.json
+      FILES_PROCESSED
+      FILES_MISSING
+      <channel>/<version>/
+        MEASURE-by-build-date.json
+        MEASURE-by-submission-date.json
+        filter-tree.json
+        histograms.json
+        revisions.json
+
+
+Web Facing JSON Format:
+-----------------
+
+
+/<channel>/<version>
+
+MEASURE.json
+[
+  [
+    bucket0,
+    bucket1,
+    ...,
+    bucketN,
+    sum,                # -1, if missing
+    log_sum,            # -1, if missing
+    log_sum_squares,    # -1, if missing
+    sum_squares_lo,     # -1, if missing
+    sum_squares_hi,     # -1, if missing
+    count,
+    filter_id,
+  ],
+]
+
+filters.json
+{
+  _id:    filter_id,
+  name:   "filter-name",
+  <option>: {
+      <subtree>
+    }
+}
+
+histograms.json
+
+{
+  MEASURE: {
+    description: ...
+    ...
+  }
+}
+

Makefile

@@ -1,17 +1,5 @@
-FILES= histogram_tools.py Histograms.json specs.py dashboard.zip
-all: $(FILES)
-
-Histograms.json:
-	wget -c http://hg.mozilla.org/mozilla-central/raw-file/tip/toolkit/components/telemetry/Histograms.json -O $@
-
-histogram_tools.py:
-	wget -c http://hg.mozilla.org/mozilla-central/raw-file/tip/toolkit/components/telemetry/histogram_tools.py -O $@
-
-specs.py: Histograms.json
-	python specgen.py $< > $@
-
-dashboard.zip: specs.py dashboard.py
-	zip $@ $?
+egg:
+	python setup.py bdist_egg
 
 clean:
-	rm -f $(FILES)
+	rm -rf dist build telemetry_dashboard.egg-info

README.md

@@ -1,10 +1,43 @@
-#Telemetry Dashboard
+Telemetry Dashboard
+===================
+Telemetry dashboard is an analysis job that aggregates telemetry histograms and
+simple measures, and offers an decent presentation. The default dashboard
+developed in this repository is hosted at
+(telemetry.mozilla.com)[http://telemetry.mozilla.com]. But the aggregated data
+is also available for consumption by third-party applications, so you don't need
+to do the aggregation on your own.
 
-Generate static files for a telemetry dashboard.
+Consuming Telemetry Aggregations
+--------------------------------
+Include into your code `http://telemetry.mozilla.com/js/telemetry.js` feel free
+to use the other modules too.
+Don't go about reading the raw JSON files, they are not designed for human
+consumption!
 
 
-# How to Run
-Host `html/` on local host an start hacking the files.
-Observe that `telemetry.js` is a meant as layer between UI and the raw files.
-The jquery plugin `jquery.telemetry.js` has some tricks that makes custom dashboard easy,
-and deals with the most complicated filtering and state management.
+Hacking Telemetry Dashboard
+---------------------------
+If you want to improve the user-interface for telemetry dashboard, clone this
+repository, setup a static server that hosts the `html/` folder on our localhost
+and start hacking. This is easy!
+
+If you want to add new aggregations, or improve upon existing aggregations,
+change the storage format, take a look at `Formats.mkd`. Talk to the guy who is
+maintaining telemetry dashboard.
+
+Basic flow is as follows:
+  1. An `.egg` file is generated with `make egg`
+  2. Analysis tasks are created with telemetry-server
+  3. `DashboardProcessor` from `analysis.py` aggregated telemetry submissions,
+     this process is driven by telemetry-server.
+  4. `Aggregator` from `aggregator.py` collects results from analysis tasks, by:
+    1. Downloads existing data from s3
+    2. Fetch task finished messages from SQS
+    3. Download `result.txt` files in parallel
+    4. Updates results on disk
+    5. Publishes updated results in a new subfolder of `current/` on s3, every
+       once in a while.
+    6. Check points all aggregated data to a subfolder of `check-points/` on s3,
+       every once in a while.
+    7. Repeat
+