Error loading aggregate data, averages not available #288

jf87 · 2019-01-11T11:15:31Z

I get that error every time when the app is open for the first time and after it has been killed.
I replaced the hard-coded e-mission url in metrics.js with my own server, but it does not seem to help.
What data is this supposed to get?

Thanks :-)

shankari · 2019-01-13T07:59:12Z

@jf87 great question!

The answer is multi-part and I know the answer for some parts and not for others :)

Although the diary tab is selected when the error appears, it is not actually generated from the diary screen. This is because the default tab to load is the "Dashboard" screen, which tries to load aggregate data for comparison with individual data, in order to apply peer pressure. While entering the metrics screen, we check to see if there is an ongoing trip and then switch the current tab

https://github.com/e-mission/e-mission-phone/blob/master/www/js/metrics.js#L29

As for the actual error, I have occasionally seen it while testing against a development server running on my laptop, but not on the main server. I haven't debugged it, assuming that it is probably caused by too little data. I am pretty sure that the error is coming from the server; have you checked the progress of the http:///result/metrics/timestamp call on the server?

Probably good to look at webserver.log, webserver-errors.log and stdout/stdin. If you see a stacktrace in the logs, or the trace of the call just ends unexpectedly, please post the log snippets (after redacting locations) here.

happy to take a quick look since I am pretty familiar with the e-mission code and can there' s a good chance that I can quickly estimate the error location.

shankari · 2019-01-17T17:20:19Z

@jf87 Are you still seeing this error? Or does it disappear once you get more data? I haven't seen it once we have sufficient data on the server, so maybe it went away?...

jf87 · 2019-01-22T12:28:58Z

Sorry for the delayed reply. I am working tomorrow again on this and I'll let you know. (but i think it's still there)

jf87 · 2019-02-12T08:57:53Z

This is the issue I think.
The following post request returns just empty results.

{"aggregate_metrics": [[], [], [], []]}

curl -XPOST --header "Content-Type: application/json" --data '{"freq":"D","start_time":1548633600,"end_time":1549756800,"metric_list":["duration","median_speed","count","distance"],"is_return_aggregate":true}' https://e-mission.eecs.berkeley.edu/result/metrics/timestamp -w "\n%{time_connect} : %{time_starttransfer} : %{time_total}"
Plus there is an issue for your server. This request takes 110s to complete.
Any idea why it is empty and why it takes so long?

shankari · 2019-02-13T00:54:39Z

so I just created a new server (actually multiple new servers) while testing e-mission/e-mission-server#637 and e-mission/e-mission-docker#3) and I can't reproduce this even with only one user. I looked at the code, and the popup only occurs when the server returns an error.

      getAggMetricsFromServer().then(function(results) {
          ...              <- HTTP 200 OK, even with empty arrays
      })
      .catch(function(error) {  <- HTTP != 200
        $ionicLoading.hide();
        $scope.carbonData.aggrCarbon = "Unknown";
        $scope.caloriesData.aggrCalories = "Unknown...";
        $ionicPopup.alert({
          title: "Error loading aggregate data, averages not available",
          template: JSON.stringify(error)
        });
        console.log(error);
      });
   };

shankari · 2019-02-13T01:02:03Z

I also connected a debugger to the client and inserted a breakpoint in at the first line in the success block ($scope.fillAggregateValues(results.data.aggregate_metrics);). The breakpoint was hit, even though the results were blank

comparing this to the error reported in #288 (comment)

it looks like the main difference is that:

in the working case: data: {aggregate_metrics: [[], [], [], []]}
in the error case: data: null

shankari · 2019-02-13T01:02:37Z

can you look at the server logs and see if there is anything there?

shankari · 2019-02-13T01:04:07Z

I also compared the status of the response.

In the working case

status: 200
statusText: "OK"

in the error popup above

status: -1
statusText: ""

shankari · 2019-02-13T01:18:53Z

since the call is through the angular $http library,

      var getMetricsResult = $http.post(
        "https://e-mission.eecs.berkeley.edu/result/metrics/timestamp",
        clonedData)
      return getMetricsResult;

we can look at the $http implementation
https://github.com/angular/angular.js/blob/6706353a71e3c11c56c0b19be03600dac57aafe1/src/ng/http.js#L907

according to the comments there,

 *   in the error callback being called.
 *   Also, status codes less than -1 are normalized to zero. -1 usually means
 *   the request was aborted, e.g. using a `config.timeout`. More information
 *   about the status might be available in the `xhrStatus` property.
 *

So could it just be that the call timed out while connecting to the server? Can you check the server logs to see whether it received a request at around the time of the error, and if so, what happened to it?

shankari · 2019-02-13T01:21:40Z

The error popup #288 (comment) does not have an xhrStatus property ☹️ so "more information on the status" is not available

jf87 · 2019-02-13T09:49:33Z

Server shows this:
START 2019-02-13 09:48:29.837809 POST /result/metrics/timestamp END 2019-02-13 09:48:29.861343 POST /result/metrics/timestamp a21840aa-6687-4ab5-9ed1-xxxxxxxxx 0.02340316772460937

jf87 · 2019-02-13T13:29:17Z

Ok. I found the reason:

Refused to connect to https://myownserver.com because it violates the following Content Security Policy directive...

It is necessary to adapt www/index.html of the cordova app in case of a separate server instance.

<meta http-equiv="Content-Security-Policy" content="default-src 'self' data: gap: https://ssl.gstatic.com http://nominatim.openstreetmap.org https://myownserver.com https://api.ionic.io/push/tokens emission: 'unsafe-eval'; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; img-src 'self' data: http://*.tile.openstreetmap.org http://tile.stamen.com https://*.tile.stamen.com http://*.tile.stamen.com 'unsafe-inline'">

I guess https://api.ionic.io/push/tokens emission also needs to be changed in case I enable iOS push notifications?

It might be good to collect these places somewhere in the docs.
Or maybe git grep with some keywords will already help to figure out where to make adaptations?

shankari · 2019-02-14T00:37:33Z

aha! that's why I have only seen this periodically - most of the other groups that used their own servers turned off the aggregate code. TripAware might have been the only one that kept it.

It might be good to collect these places somewhere in the docs.

It is in the docs, although maybe not in a very organized place
https://github.com/e-mission/e-mission-docs/blob/master/docs/e-mission-phone/create_a_new_custom_client.md

If you can add this information to the doc (and potentially rename it for clarity), it would be great!

shankari · 2019-02-14T00:38:59Z

I guess https://api.ionic.io/push/tokens emission also needs to be changed in case I enable iOS push notifications?

Ionic is actually javascript framework on top of angular. We used to use them for push notifications, but now we use firebase, and the mapping happens all on the server side. So we should be able to remove https://api.ionic.io/push/tokens from that list.

shankari · 2019-02-14T01:46:37Z

Also, I have a longer-term question.

Background: The reason that the aggregate call is affected by this while the user call is not is because $http is a javascript call and the user call goes through a native plugin. The native plugin is configured through the connectionConfig.json and so it can, by default, only call back to the server associated with the phone app.

And the reason that we used $http instead of the native plugin (among many reasons) is that the native plugin always sends the user token, and my original vision was that the aggregate data would always be public.

But now, we know that is not a good model for anything other than heatmaps, and even for heatmaps, you need to be careful. The new vision is to have an aggregator, and queries that authenticate themselves with the aggregator.

Question: (@jf87 @PatGendre) So should we start going towards this model by requiring authentication for the aggregate data as well? The effect would be that the aggregate data would only be visible to the community of contributors. So if I submit data, I can see aggregate data, otherwise I can't. This would not be a hard change, and it would also allow us to use the native plugin for all server calls, which would resolve this issue.

Thoughts?

shankari · 2019-02-14T01:47:52Z

@deepalics0044 I think you ran into this when you set up your own server initially?

jf87 · 2019-02-14T10:57:38Z

Yes. I think the better solution would be to use the native plugin.
Plus you can easily extend it to more fine-grained access models having authentication.
Access control should happen on the server side and the individual e-mission deployer should be able to decide if aggregate data is available publicly, only to participants or even only to specific users/on a specific granularity.

shankari · 2019-02-14T17:07:53Z

Access control should happen on the server side and the individual e-mission deployer should be able to decide if aggregate data is available publicly, only to participants or even only to specific users/on a specific granularity.

Will probably only implement the first two options (aggregate data is available publicly v/s only to participants) initially to avoid over-engineering. Will try to get to this after merging the GIS based mode detection.

jf87 · 2019-02-15T13:02:56Z

Will probably only implement the first two options (aggregate data is available publicly v/s only to participants) initially to avoid over-engineering. Will try to get to this after merging the GIS based mode detection.

Yes sure. I just wanted to hint that this option is flexible enough to implement various schemes :-)

PatGendre · 2019-02-22T15:42:31Z

I agree with Jonathan "Access control should happen on the server side and the individual e-mission deployer should be able to decide [how] aggregate data is available."
Also, aggregate data may be used in a completely separate app than personal data, e.g. it is not necessarily useful to request and display aggregate data for the smartphone app, it could be used in a desktop browser web GIS app.

shankari · 2019-04-21T13:08:14Z

I'm making some changes to the web interface, so decided to tackle some parts of this at the same time.

Quick response to @PatGendre wrt

Also, aggregate data may be used in a completely separate app than personal data, e.g. it is not necessarily useful to request and display aggregate data for the smartphone app, it could be used in a desktop browser web GIS app.

This is already supported. Check out the "Analysis" or "Metrics" tabs on the existing web UI. And because the phone app uses HTML + javascript, we are able to largely re-use the same code on the phone and web apps. IIRC, the main differences are in the HTML, mainly because I am not very good at responsive UX.

In fact, the initial and current version (before my pending changes) was that user specific information would only be available through the phone app, and the aggregate information would be publicly available through both the phone and web apps.

PatGendre · 2019-04-23T07:40:55Z

@shankari : thanks! I'll be watching the improvements you are bringing to the web interface. The discussion above with Jonathan gets clearer to me now.

PatGendre · 2019-05-27T08:01:18Z

@shankari
Hi , I still have this "Error loading aggregate data, averages not available" error on the dashboard.
We I do a the request corresponding to this error in the phone log curl -XPOST --header "Content-Type: application/json" --data '{"freq":"D","start_time":1557705600,"end_time":1558828800,"metric_list":["duration","median_speed","count","distance"],"is_return_aggregate":true}' https://fabmob.grfmap.com:8080/result/metrics/timestamp -w "\n%{time_connect} : %{time_starttransfer} : %{time_total}\n"
The returned data is neither null nor {aggregate_metrics: [[], [], [], []]}.

PatGendre · 2019-05-28T14:47:36Z

As discussed yesterday our problem is not related to the need to adapt the www/index.html of the cordova app in case of a separate server instance. (in <meta http-equiv="Content-Security-Policy" ).
It was related to a webserver conf, that we solved.
By the way this issue can be closed ? As it has been solved and described in the docs.

shankari · 2019-05-28T14:53:26Z

Sure. I left it open for the long-term issue, but I guess we can file a new issue for that.

The switch to WKWebView forced the use of CORS. There is no workaround. https://ionicframework.com/docs/v3/wkwebview/#cors We didn't encounter this in covid-19 repo since we made all calls through the built-in native plugin that automatically adds the user token and sends out the call. In emission, though, we need make aggregate calls that should not have a user token. We had been using XHR for this (through the angular `$http` server) and all those calls broke. While we could make the appropriate change on the server side, that would not be consistent with our long term goal (e-mission/e-mission-docs#408, e-mission/e-mission-docs#288) So for now, I use the alternative documented here https://ionicframework.com/docs/v3/wkwebview/#i-cant-implement-cors Concretely: - add the native plugin - add a new method `CommHelper.getAggregateData` that wraps it in a promise - change all instances of `$http.post` -> `CommHelper.getAggregateData` - use the configured connectUrl instead of hardcoding - remove the hardcoded URL from index.html Bonus fix: - Dealt with error in the heatmap if there was no data and thus, no bounds by checking to see if the bounds were valid Testing done: Navigated through the app screens until all the aggregate calls were invoked (see below). No errors in the console. ``` [Log] getting aggregate data without user authentication from http://localhost:8080/result/metrics/timestamp with arguments {"freq":"D","start_time":1586131200,"end_time":1587254400,"metric_list":["duration","median_speed","count","distance"],"is_return_aggregate":true} (cordova.js, line 1540) [Log] getting aggregate data without user authentication from http://localhost:8080/result/heatmap/incidents/timestamp with arguments {"start_time":1587187062,"end_time":1587359861,"sel_region":null} (cordova.js, line 1540) [Log] getting aggregate data without user authentication from http://localhost:8080/result/heatmap/pop.route/local_date with arguments {"modes":null,"from_local_date":{"year":2020,"month":4,"day":17,"hour":22},"to_local_date":{"year":2020,"month":4,"day":18,"hour":22},"sel_region":null} (cordova.js, line 1540) [Log] getting aggregate data without user authentication from http://localhost:8080/result/heatmap/incidents/local_date with arguments {"modes":null,"from_local_date":{"year":2020,"month":4,"day":17,"hour":22},"to_local_date":{"year":2020,"month":4,"day":18,"hour":22},"sel_region":null} (cordova.js, line 1540) ```

shankari transferred this issue from e-mission/e-mission-phone Feb 11, 2019

jf87 mentioned this issue Feb 14, 2019

Note to replace some berkeley.edu urls with own server url #317

Merged

shankari mentioned this issue May 22, 2019

Carbon calculations on phone #396

Closed

shankari closed this as completed May 28, 2019

shankari mentioned this issue May 28, 2019

Support additional configuration options for aggregate data #408

Closed

tms-martins mentioned this issue Jun 3, 2019

Update carbon calculations for GIS server (richer modes) e-mission/e-mission-phone#576

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error loading aggregate data, averages not available #288

Error loading aggregate data, averages not available #288

jf87 commented Jan 11, 2019

shankari commented Jan 13, 2019 •

edited

Loading

shankari commented Jan 17, 2019

jf87 commented Jan 22, 2019

jf87 commented Feb 12, 2019

shankari commented Feb 13, 2019 •

edited

Loading

shankari commented Feb 13, 2019

shankari commented Feb 13, 2019

shankari commented Feb 13, 2019 •

edited

Loading

shankari commented Feb 13, 2019

shankari commented Feb 13, 2019

jf87 commented Feb 13, 2019 •

edited

Loading

jf87 commented Feb 13, 2019

shankari commented Feb 14, 2019

shankari commented Feb 14, 2019

shankari commented Feb 14, 2019 •

edited

Loading

shankari commented Feb 14, 2019

jf87 commented Feb 14, 2019

shankari commented Feb 14, 2019

jf87 commented Feb 15, 2019

PatGendre commented Feb 22, 2019

shankari commented Apr 21, 2019

PatGendre commented Apr 23, 2019

PatGendre commented May 27, 2019

PatGendre commented May 28, 2019

shankari commented May 28, 2019

Error loading aggregate data, averages not available #288

Error loading aggregate data, averages not available #288

Comments

jf87 commented Jan 11, 2019

shankari commented Jan 13, 2019 • edited Loading

shankari commented Jan 17, 2019

jf87 commented Jan 22, 2019

jf87 commented Feb 12, 2019

shankari commented Feb 13, 2019 • edited Loading

shankari commented Feb 13, 2019

shankari commented Feb 13, 2019

shankari commented Feb 13, 2019 • edited Loading

shankari commented Feb 13, 2019

shankari commented Feb 13, 2019

jf87 commented Feb 13, 2019 • edited Loading

jf87 commented Feb 13, 2019

shankari commented Feb 14, 2019

shankari commented Feb 14, 2019

shankari commented Feb 14, 2019 • edited Loading

shankari commented Feb 14, 2019

jf87 commented Feb 14, 2019

shankari commented Feb 14, 2019

jf87 commented Feb 15, 2019

PatGendre commented Feb 22, 2019

shankari commented Apr 21, 2019

PatGendre commented Apr 23, 2019

PatGendre commented May 27, 2019

PatGendre commented May 28, 2019

shankari commented May 28, 2019

shankari commented Jan 13, 2019 •

edited

Loading

shankari commented Feb 13, 2019 •

edited

Loading

shankari commented Feb 13, 2019 •

edited

Loading

jf87 commented Feb 13, 2019 •

edited

Loading

shankari commented Feb 14, 2019 •

edited

Loading