Add Mozilla Location Services (MLS) support

[E3V3A]

Issue: This is not an issue, but would be a great improvement to the flexibility of our BTS data sources.

We are currently depending on the OpenCellID (OCID) DB to obtain the BTS info we need for our detection. This data is used to populate the DBe table. However, Mozilla Location Services (MLS) is now mature, detailed and large enough for us to benefit from their carefully curated data. It is maintained on GitHub as ichnaea.

We already have a very long issue thread in issue #71. It is too long and not enough specific, which is why this issue is created. We already have much of the framework for getting data, so we need to add a setting as in this comment:

External BTS data source(s): 
( ) OCID only
( ) MLS only
(o) both OCID and MLS

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[ph0t0n]

Question-1: Will we need any kind of ranking algorithm between OCID vs MLS data, or just use both data sources and combine them if the user requests?

Question-2: How can we detect maliciously cells that are purposefully added to these databases? is there any technical way until the cells do something suspicious?

[Edited for readability]

[SecUpwN]

Q. will we need any kind of ranking algorithm between OCID vs MLS data, or just use both data sources and combine them if the user requests?

@ph0t0n, this is actually something I'd like to know as well since with #467 we already have support for Local-GSM-Backend, which is basically OCID and MLS combined. @E3V3A, please clear this up for us.

harder related Q. how can we detect maliciously cells that are purposefully added to these databases? is there any technical way until the cells do something suspicious?

Still to be solved with #411. Seems like the OCID team is working hard on this as well.

[E3V3A]

Q-1: No, yes, exactly as in OP.

Q-2: We can't which is the reason for above. But we can detect in several ways by implementing #230.

[SecUpwN]

Ask a proper question and you shall receive a proper answer... ;)

Deal. Do we need to add anything else other than already existing support for Local-GSM-Backend?

[E3V3A]

This issue is not about LGB support, but about MLS. (open new issue for that.) So yes, we need to add MLS and fix the stuff that was broken due to faulty LGB implementation, i.e. #470.

[DJaeger]

LGB does NOT support querying MLS.
It is an local only (offline) location backend.

But an backend for (online) querying MLS also exists:
https://github.com/microg/IchnaeaNlpBackend

If we are able to switch to request the Cell IDs from LGB over any type of UnifiedNlp API we could use the same API for the MLS backend. In any other case we need a new and different solution to request the Cell IDs from MLS, as it does not have any database file.

[E3V3A]

Cleaned up this thread.

We need work on implementing the MLS support for the BTS downloader.

Adding BUG label so other dev's can have a look. I also removed DATABASE since our DB now should support all relevant data from MLS.

[agilob]

In spare time I was contributing to n76/Local-GSM-Backend, if you want I can add intent-based API to retrieve and provide information about locally saved cells.

[SecUpwN]

Re-opening this to see what @hannosch has to say about how to best implement support for MLS.

[hannosch]

I'm not sure what exactly you want to do with MLS (and what has been done already). If you want to store the MLS data locally and query it offline, the files are available at https://location.services.mozilla.com/downloads in the same format as OCID. Though uncompressed these are above one gigabyte in size.

If you want to use the online query API, the API documentation lives at https://mozilla.github.io/ichnaea/api/geolocate.html these days. I'm pretty sure I gave you an API key in the past.

If you want to contribute data back to the service, there's also the submission API at https://mozilla.github.io/ichnaea/api/geosubmit2.html

With libstumbler and the microG unifiedNLP library (https://wiki.mozilla.org/CloudServices/Location/Software#Android) there are also two existing libraries that do the work for you.

[rugk]

Nice idea. Any progress on this?

Also note that MLS already includes opencellid so adding both would be redundant.

[f3ndot]

We should definitely consider looking at MLS, maybe even as a replacement, since as @rugk says it contains OCID data. My suggestion is because OCID's API is slow, times out, and goes down sometimes.

I haven't been able to download from OCID's Web API today

[SecUpwN]

We should definitely consider looking at MLS, maybe even as a replacement, since as @rugk says it contains OCID data. My suggestion is because OCID's API is slow, times out, and goes down sometimes.

@f3ndot, thanks for the proposal. If we replace OpenCellID with MLS, will all data collected be also contributed back to OpenCellID as well? If so, it sounds like a win-win situation: We get a more reliable download and both services double-verified data. Shall I assign you to this Issue? We'd then need to:

• Make the real switch to MLS (maybe the source of their official app MozStumbler helps)
• Update all code to say MLS or MozillaLocationService instead of OpenCellID
• Update all translations (I can tackle this when the previous things are done)

Before we change to MLS, I'd like to hear @He3556 and @larsgrefer on what they think. We also should to find out if there are any "benefits" from contributing back (faster downloads or our app being listed on their leaderboard (which may generate some traffic to our repo and attract more developers.

[rugk]

will all data collected be also contributed back to OpenCellID as well?

I don't know. At least they could easily do it as the data is public domain.
From their stats you could think that such an import happened this month as the number of unique cell networks in OpenCellID rose much and it is now higher than the MLS number.
However obviously I don't know whether this assumption is true or what caused this, but you may ask Mozilla or opencellid.

[hannosch]

Hanno from MLS here: As of last week OpenCellID now also includes the cell data from MLS. We have bidirectional sharing of the aggregate cell data finally working.

The two projects do differ in their handling of raw observation data. OpenCellID releases those publicly, while we think this is too much of a privacy / tracking problem on the MLS side. It should be up to each user to decide which of those two (or both) projects they want to contribute to.

[hannosch]

We also should to find out if there are any "benefits" from contributing back (faster downloads or our app being listed on their leaderboard (which may generate some traffic to our repo and attract more developers.

The MLS leaderboard in its current form is being retired and replaced by a new one which is going to be specific to our own Stumbler application. It will require signing up with real accounts (Firefox Accounts). While there is an API and other apps could use the new leaderboard, it focuses on individual users, not promoting apps.

If your software uses MLS, we'll put it on the list at https://wiki.mozilla.org/CloudServices/Location/Software

[SecUpwN]

While there is an API and other apps could use the new leaderboard, it focuses on individual users, not promoting apps.

@hannosch, since our app has had so many troubles with OpenCellID and your MLS really makes a mature impression, we're now considering the complete move from OpenCellID to MLS. Here's the catch: We will likely need a few more developers helping us hardcore to properly make the switch. Do you know of any developers you could point to our repository in order to fully replace OpencellID with MLS?

[SecUpwN]

this is why someone left the 'good luck' virus

@j4s0nmchr1st0s, what do you mean? Please clarify.

[hannosch]

@SecUpwN I'm more than happy for your app to switch to MLS. Unfortunately I don't know of any Android developers to help in such an effort. I can answer some questions, but since I'm not an Android or even Java developer myself, I can't help with the code directly.

[SecUpwN]

The progress is made then lost like an antimatter containment breach.

That is the thing with ALPHA apps. Heavy lifting with too few contributors and too many complaints.

For example the new version of aimsicd has less capability than older versions

True. But that is the case with any other WIP project as well. We're currently also mving to yet another type of database so that developers will have an easier life. And making them enjoy the project is key.

and there we are debugging everything again.

We? Who is "we"? This is your second post on our repository. Please help us instead of moaning. ;-)

[agilob]

@hannosch sorry, but I'm not sure if we're on the same page. AIMSICD doesn't need to know location of user after the request. What it needs is data about cell, let's say I send a request about cell id 12345, expected response that would be helpful in the project is:
cell_id:12345
lac:123
mcc:12
mnc:1
carrier:orange
lat (BTS location):cd.ab
lon (BTS location:ab.cd
first seen:today
last seen:0
PSC:999

I just checked MLS api and all it provides, is lat/lon of BTS after quite detailed request (not providing signal strength results in "invalid request"
{"error":{"code":400,"message":"Parse Error","errors":[{"domain":"global","message":"Parse Error","reason":"parseError"}]}}
My device doesn't provide strength signal for a cell, so the idea of using MLS seems to be missing the point. AIMSICD doesn't need my location, it needs location of BTS and info about it.

[hannosch]

@agilob

I'm not sure how you got the parse error, if I do a "simple" request for a single cell network:

curl -H "Content-Type: application/json" \
https://location.services.mozilla.com/v1/geolocate?key=test \
-d '{"radioType": "wcdma", "cellTowers": [{"mobileCountryCode": 262, "mobileNetworkCode": 1, "locationAreaCode": 1822, "cellId": 936763}], "fallbacks": {"ipf": false, "lacf": false}}'

I get back:

{"location": {"lat": 53.1229293, "lng": 8.9884341}, "accuracy": 1000.1}

For cell requests all fields should be optional, except for the five fields to uniquely identify a cell network.

Now the result this gives back is where we think a user is, when they see this cell network. That's subtly different from where the actual cell tower is. For the most part OCID and MLS do the same thing here and use (weighted) averages over all observations to calculate the cell networks estimated position. I didn't comment on this, as both projects give you the same thing. Neither claims to know where the cell tower or antenna is. The only exception to that is a minority of the OCID dataset, which is marked in their download file with the "changeable" column set to 0.

On the MLS side we don't have an online API to access more metadata about any of the cell networks. If you absolutely require this, you can use our CSV file downloads of the entire data set. Those include creation and modification times as well as sample numbers. The format for this is the same as that used by OCID (https://mozilla.github.io/ichnaea/import_export.html).

[agilob]

I'm not sure how you got the parse error, if I do a "simple" request for a single cell network:

I just wanted to get you example request... and it worked. Looks like my bad, sorry about that.

Regarding the metadata, in such situation MLS becomes irrelevant here, the only thing it "could be useful" would be just to confirm if CID exists at all. But as MLS and OCID contain the same data, AIMSICD already has code to check if CID exists in OCID data. I think this issue can be closed (and "bug" label removed, as it's not a bug).