Spike 5000 queries round deux

[epugh]

Description

Demonstrate running 5000 queries in a background process, storing the results in a Snapshot.

Ideally, we run the 5000 queries, building the snapshot. We then make the front end interact as if it was a CSV Upload of data. If that works, great. If it still doesn't scale, then we have a seperate UI. And we figure out how to run not just A scorer, but lots of scorers, and store them..

• Think about moving response_body to it's own table, it's really not part of the snapshot_queries and it appears to bloat the table a lot..
• Think about writing some code to convert the raw data from Solr into what Quepid expects. Snapshot_queries, snapshot_docs, and some scores?
• Add to snapshot picker a sense of date of the snapshot.
• Convert Solr response in snapshot_queries into snapshot_docs.
• Add logic to calculate a metric using a scorer. Can we write a scorer in Ruby?
• Read in scorer from database table, not file!
• calculate a global score !
• Add a dashboard for all the case scorers?
• Experiment with loading a Snapshot in place of the actual running of the case.
• Need to add the number_of_rows limit to reqeusting docs
• Need to add the fl parameter to limit the fields being returned to what is defined in the case.
• Tested with scorers that need access to the fields of a doc
• Need to be able to manage case scores, like deleting them if you have an odd ball one.
• Follow pattern in book and case of having a column to track the job. So either add it to case.nightly_run_job or add it to the snapshot....
• Always show the most recent snapshot. If there are no snapshots then prompt to run the queries with a big ass button in the middle.
• Only keep most recent run;'s web_requests... otherwise too much data.

Down the road, when we want multiple scorers:

• establish a single north star metric.. it shows up on the home page.
• establish a relationship between a case_scores and scorers, right now when you change scorers, we don't track that.
• Filter the graphs to only show scores that match the scorer being used.

Motivation and Context

We need moar data! This replaces an earlier attempt, #976 that kept the AngularJS app as the runner for Javascript. It just didn't work... Headless app wasn't the right way to go.

[epugh]

I'd love to get a bit more detail on how to pick a baseDate.. and if the baseDate could be "today" instead? @david-fisher

[david-fisher]

base date of today is just fine. The one in the code was the date of the frozen in time index. You can use:

baseDate = Date.now()

[epugh]

Maybe done?