K4 Beta2 to Beta3 upgrade issues

[ppf2]

Not sure if we intend to provide an upgrade path from K4 beta2 to beta3.

The following are issues after upgrading from beta2 to beta3. The upgrade is done by taking a snapshot of the .kibana index, restoring it to another cluster and then associating beta3 to it so it will automatically pick up the existing .kibana index (from beta2).

Beta3 is brought up, the saved searches show up fine (and work) from the Discover screen.

1. The default index pattern is lost after upgrading. Minor issue. Can simply set it again.

2. Saved dashboards do not render and fails with query exceptions:

KbnError@http://localhost:5601/index.js?_b=4673:38611:21
FetchFailure@http://localhost:5601/index.js?_b=4673:38661:1
fetchService/fetchThese/</</sendResponse@http://localhost:5601/index.js?_b=4673:41536:62
fetchService/fetchThese/</</<@http://localhost:5601/index.js?_b=4673:41545:15
fetchService/fetchThese/</<@http://localhost:5601/index.js?_b=4673:41540:11
qFactory/defer/deferred.promise.then/wrappedCallback@http://localhost:5601/index.js?_b=4673:20764:31
qFactory/ref/<.then/<@http://localhost:5601/index.js?_b=4673:20850:26
$RootScopeProvider/this.$get</Scope.prototype.$eval@http://localhost:5601/index.js?_b=4673:21893:16
$RootScopeProvider/this.$get</Scope.prototype.$digest@http://localhost:5601/index.js?_b=4673:21705:15
$RootScopeProvider/this.$get</Scope.prototype.$apply@http://localhost:5601/index.js?_b=4673:21997:13
done@http://localhost:5601/index.js?_b=4673:17570:34
completeRequest@http://localhost:5601/index.js?_b=4673:17784:7
createHttpBackend/</xhr.onreadystatechange@http://localhost:5601/index.js?_b=4673:17723:1

The following is the underlying error:

[2014-12-30 16:50:36,925][DEBUG][action.search.type       ] [Justin Hammer] [github][0], node[nuun_sNeTVukIxaZcZ3ySg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@38395b]
org.elasticsearch.search.SearchParseException: [github][0]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"query":{"filtered":{"query":{"query_string":{"query":"_type:pull_request"}},"filter":{"bool":{"must":[{"query_string":{"query":"*"}},{"query":{"query_string":{"query":"*"}}}],"must_not":[]}}}},"size":0,"aggs":{},"fields":["*","_source"],"script_fields":{"milestone.due_on":{"script":"if (doc[\"milestone.due_on\"].value == 0) { null } else { doc[\"milestone.due_on\"].value }"},"closed_at":{"script":"if (doc[\"closed_at\"].value == 0) { null } else { doc[\"closed_at\"].value }"},"@timestamp":{"script":"if (doc[\"@timestamp\"].value == 0) { null } else { doc[\"@timestamp\"].value }"},"updated_at":{"script":"if (doc[\"updated_at\"].value == 0) { null } else { doc[\"updated_at\"].value }"},"created_at":{"script":"if (doc[\"created_at\"].value == 0) { null } else { doc[\"created_at\"].value }"},"milestone.updated_at":{"script":"if (doc[\"milestone.updated_at\"].value == 0) { null } else { doc[\"milestone.updated_at\"].value }"},"milestone.created_at":{"script":"if (doc[\"milestone.created_at\"].value == 0) { null } else { doc[\"milestone.created_at\"].value }"},"milestone.closed_at":{"script":"if (doc[\"milestone.closed_at\"].value == 0) { null } else { doc[\"milestone.closed_at\"].value }"}}}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:350)
    at org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:333)
    at org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:330)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.query.QueryParsingException: [github] No filter registered for [query_string]
    at org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:322)
    at org.elasticsearch.index.query.BoolFilterParser.parse(BoolFilterParser.java:92)
    at org.elasticsearch.index.query.QueryParseContext.executeFilterParser(QueryParseContext.java:343)
    at org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:324)
    at org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:74)
    at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:277)
    at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
    at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
    at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
    at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
    ... 9 more

The issue here is that the query sent by K4 Beta3 has an extra filter clause (1st bool clause below) and this extra clause is invalid, missing the query clause wrapper:

"filter": {
        "bool": {
          "must": [
            {
              "query_string": {
                "query": "*"
              }
            },
            {
              "query": {
                "query_string": {
                  "query": "*"
                }
              }
            }
          ],
          "must_not": []
        }
      }

If I then stop beta3, launch beta2 (associated to the same cluster, reading from the same .kibana index without doing anything else), the dashboard starts to work properly again.

[rbjarnason]

I have problems after upgrade as well - just an empty page on http://logs:5601/#/undefined after I visit my site at http://logs:5601/

[maxhyjal]

I ran into issue as well after upgrading to beta3

Segmented Fetch: SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures

while it's working fine without any issue in kibana beta2.
I'm just glad that switching between different kibana 4 versions is simple...

[sooth]

Same problem here

[rashidkpc]

Duplicate of #2377

[tmandry]

Same here. @rashidkpc the global settings issue is just part of the problem, the other is that existing dashboards don't function anymore after the upgrade. I don't know if you want to support the upgrade or not.

Did anyone find a way to edit the dashboard objects manually to fix this problem?

[tmandry]

Okay it's fairly easy to fix manually, just find the saved dashboard object (under Settings -> Objects) and in the kibanaSavedObjectMeta... field, turn the "filter" object into an array with the existing object as its only element.

If that doesn't make sense, just create a new dashboard and look at how that field is structured.

[rashidkpc]

Yeah, we definitely don't guarantee object schemas across the betas, we will support upgrades after GA though.

[x-cray]

@tmandry Thanks for the hint, works like a charm.

[rashidkpc]

I also created this #2620