-
Notifications
You must be signed in to change notification settings - Fork 967
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check CID via Mozilla Location Service #71
Comments
thanks for your info! i was checking the MozStumbler a few days ago. but pls help me - what does "accurancy : 1200.4" mean? 1200.4 meters or feet? if so, maybe they should call it inaccurancy :) |
Smile, i think range or inacurracy sounds better. :-) The latitude and longitude are numbers, with seven decimal places of actual precision. The coordinate reference system is WGS 84. The accuracy is an integer measured in meters and defines a circle around the location. If you are inside the circle with the given accuracy this is the right place where the cellid is "at home". http://winfwiki.wi-fom.de/index.php/Bild:Coo.png That sounds good, Mozilla Location Service wants to release their cellular data so we can use them offline.
|
@He3556 @Gitschubser @SecUpwN @xLaMbChOpSx |
I would love to take this part. But since I do not have a working E-Mail address at this moment, would you please takle this, @E3V3A? I'll be talking to developers as soon as I have a new mail provider. Promised! |
Here is the wiki from opencellid: Maybe for any question you can find an answer. :-) I use TowerCollector to collect measurements because i think for me it is the best app. In the API you can find more interesting answers in example about the collected measurements and which data are included there. http://wiki.opencellid.org/wiki/API Opencellid calculates the middle of latitude/longitude, not the real place from the base station, using all measurements from a unique cellid. I think to check the position of a cellid we should use more than one source because
Triangulation is only possible when your mobile phone gives you the neighbour cells. |
@SecUpwN @Gitschubser So there we have our answer:
This means that most city tower locations are way off and wrong, since most cell towers are directional, so that you get a "cell tower" centered to where there are most users, such as in shops, cafees and restaurants. |
Means, we'd have to collect all data by ourselves, @E3V3A? |
@SecUpwN That is what this thread is about. If you recall, we have recently discussed this in #13 and other threads (issues). Basically we can use Google (which you hate, but which has the best and easiest API) or that provided by Mozilla. If you can find any other alternatives, that would be just great. We don't need to collect ALL data, we can use OCI to get an idea about the cell neighborhood and then use the above mentioned APIs to get their exact locations, in the region and for the cells we're interested in. |
The most city tower locations are not wrong they are not very accurate if there are not much measurements available. The most measurements exists on streets where people drive with car or walk. See map Mozilla Location Service. For triangulation there is used the middle from all measurements and the range (see patent google). Triangulation with the real position from the base station i think is not possible. If there is only one measurement from the position of the cellid you can not calculate a accurate position from the base station (middle lat/lon, 360 degree radius, exact range). All projects live from the people that help to support with measurements. If you want to use the accurate position you need expensive equipment which measures the run time and calculates the exact position and the direction too. I think we should use existing projects and look how we can use the existing data for the project or build a own database with the real locations like http://senderliste.de/ or http://gsm.yz.to/. Data which we can use for free and that exist are the best data at this time. Help to collect data for this projects and the positions would be better and let us use them how it is possible at the moment. ;-) |
@Gitschubser Perhaps I misunderstand some of what you say, but just to reiterate, we can not build our own database with all cell towers, because that is an entirely different project, in addition it is useless, since we're only interested in towers in the immediate rural neighborhood (neighboring cells) and the history of those. Also I don't see how the links you provide can help us? Are you related to those sites? We have to use the simplest tools/APIs available to the best of our abilities, and keep things as simple and on track as possible. We're not going anywhere by expanding this project into unmanageable realms. Also normal triangulation is obviously (since we do not have direction) not possible without additional accurate variables such as one or more of: signal strength, exact current location, TA, TX/RX power. Also, the average of peoples GPS locations, give you exactly that. The center where people are, which is very different from that of the BTS tower. |
Well i like the stuff @Gitschubser is talking about. I think he is talking about supporting Services like OpenCellID with the data we are collecting. Maybe too early to think about that, ok. But I have a CellID App and sent data a few month ago, but now there are no towers in my area :( But something else, maybe we could set options in the menu if people want to use Google, Mozilla or CellID. I think this is better than kicking out all the code if we think we want to use another service. |
i just got our App working on my HTC Wildfire :) and i see there are options for different maps - realy cool! and i have a list with neighboring cells. yeah! |
Congrats, @He3556! Keep recommending our App to others and test it thoroughly! 👍 |
@He3556 I'm definitely not throwing out any ideas, I think OCI is great, but that our App cannot rely on that data alone. When we get this app going, we should definitely upload our better and corrected data to OCI's DBs. |
I'm sceptical of location as an indicator; carriers move their stuff about, |
@AlexHarrowell You can be as skeptical you want, but unless you can come up with, and present a better and simpler solution, that's how we will do it. |
Hi, my name is Markus. I´m the maintainer of opencellid.org. I was asked by E:V:A to join this discussion. Regarding the precision of the cell tower positions the above made statements are correct: we just average the GPS positions of all measurements of a cell and this is not necessarily giving a good estimation of the real position of the cell tower, but the best what we can provide based on the currently existing information. There is one option which is not obvious: approx. 3% of all known cell towers in the OCID database have a precise position, for ex. most of the towers in Poland. This is the case because we got donations of some data that provided such information. Some of the guys active on senderliste.de also provided us with precise cell tower information, some of them unfortunately refused to do so mainly because the OpenCellID license allows to use the collected data commercially. As you might know I took over the maintainer-ship of the OpenCellID project approx. 1 year ago after the project was a bit sleepy for a while. The first thing we did was rebuilding the backend which took approx. 1 year full-time development. This was necessary because the amount of data is huge: currently we have 845 million records in our database and approx. 2 million records are added every single day. This initially most important task is done now and the MongoDB cluster is up and running providing the fastest OpenCellID UI that ever existed. Now the next step is cleaning up the database: It holds some information which is obviously wrong, e.g. a cell with MCC 262 (Germany) in Africa. This will take another 3 months or so because of the huge computation that must be executed and the pretty complex algorithm to be implemented. As soon as this is done, we are ready for the future.
are the three most promising things to be implemented for better accuracy in many cases. |
@msemm Welcome Markus!
Also, I think "guaranteed" is a grave overstatement, in regard to the BTS signal from all 4 directions. I can say for sure that will never happen in an urban area. BTS antennas are highly directional and only the RF engineers are able to tell how they have been multiplexed together, to get their beam profile. But I think you already know this far better than me. So I think the only way to improve the BTS positioning average procedure, is to also use TA and signal strength as a weight, when performing the average. (I.e. a 2D-average embedded in a 3D space :) |
Thanks @msemm for joining this discussion. Like @E3V3A asked in no. 2 "Is there any quality assessment of the GPS positions maintained in the DB as well?" This could be provided by a value how many degree around the BTS are scanned by the App user. For example, if the BTS was seen from all 4 sides (N-W-S-E) the value is 360°. But that would be lots of calculation for 2 million new records a day. But as long as there are enough fingerprint measurements from each Cell (also the distance), a fake BTS would have a totally different fingerprint (location & signal strength). So we should try to collect as many data we can. I wish there was a App for the iPhone to do this. |
EDIT: EVA for readability Answers:
Signal strength : This is not as easy as it sounds: Imagine you are in front of a concrete building or behind. Both positions are just e.g. 10m away from each other. But in case the base station is behind the house, then the signal strength in front of the house might be very different from the signal strength behind the house. This is why we believe that fingerprinting is a much better approach than just computing the distance of the handset to the cell tower based on the signal strength, because in this case the big difference in signal strength outlined in the example above is an advantage instead of a disadvantage because it allows to find out if you are in front of the house or behind. The same is true for TA: Due to reflections these values might also deviate from the TA value the direct line of sight would give you. We have implemented such a fingerprinting software for WiFi, then equipped our offices with a WiFi repeater in each corner of the building. Then we recorded the fingerprints and later used these fingerprints to compare them with the actual signal strength of each repeater. This allowed us to achieve a 2-3m accuracy inside our office. EDIT: E:V:A removed email footer |
@He3556 Degree around the BTS: Can be considered. Please discuss this and potential other ideas regarding data quality and come back with a list of ideas. May be we can then even organize a Skype chat for summarizing the next steps (if any). Servers: Don´t worry about that issue, we´ll handle it. Currently we have 6 servers for OpenCellID, each with 128 GB RAM, 4TB SSD, Raid and 2x8 cores, so there is some room for additional computation especially as long as it can be parallelized. Let´s focus on the quality of the data:
Unfortunately I cannot be as active on this mailing list as it might be appropriate due to other obligations: We are currently launching ginstr which feeds my family and pays for the OpenCellID infrastructure as well and therefore has priority. Sorry for that. I´ll try to follow the discussion here as good as I can because I like your project idea! [ EDIT: E:V:A removed email footer] |
@msemm Thanks for quick response. I'll join the Wiki discussions and add my input. But here is my response fo the current discussion.
Thanks again for taking your time. |
@He3556 Have you figured out what is that Accuracy yet? |
It is a pleasure for me contributing to this very friendly and professional discussion. [EDIT: E:V:A removed email footer] |
@msemm First of all, don't trust Stackoverflow answers. But, I also see the dilemma with WP. Technically it is clear that WP also have to provide for full RIL support and everything you would need, but it has been very well hidden from the API in WP7-8. All WP phones are based on the Qualcomm chip-sets:
I've seen code in that firmware, so it's just a matter of "talking nicely to the modem in the right way". Not that it is going to be easy without an API, but certainly possible. I'm sure they have all this functionality in Service Mode application and elsewhere. The lack of instructions may also have something to do with MS WP App Market certification. |
This post was moved from issue #87 to here, and have been duplicated in a new issue #88. @E3V3A @SecUpwN @He3556
We know that Google has more accurate information about cell towers in comparison to OpenCellID. I did some research on their non-public api. I downloaded Gmaps.apk and decompiled it to smali. There I found a url "http://www.google.com/glm/mmap", which I googled. Here info how to use this api (redundant): http://www.open-electronics.org/how-to-find-the-location-with-gsm-cells/ (later I realized that it was already posted here). So what we can do is simply use their hidden api to check if they have a particular cell in their database and if they do we can get GPS coordinates of the cell + submit it to OpenCellID. We can also get the coordinates of Neighbour-Cells and calculate a more or less precise location based on signal strength of the cells. The question is how reliable is Googles mobile network info? If we have a cell which is not in Google's database, it could be an indicator that it is an IMSI-Catcher. It would be better if we could download all mobile network info in an area. I'm interested which data is stored in Google's offline maps and if it is possible to access this data somehow. I tried to intercept offline maps data but couldn't bypass SSL encryption (Certificate pinning and other problems). But I found these tools so I'm sure there is a way to bypass SSL or attach a debugger to GMaps. |
moved from issue #87 very interesting observations. Not only to save energy - but to get important info about the location of the cell towers. To your question how reliable the google data is: I am still thinking about a way to get the exact lon/lat. of the BTS directly, without scanning the whole area around the BTS. Google Server for assisted GPS (SUPL_HOST=supl.google.com or also Nokia SUPL_HOST=supl.nokia.com) sends data about the satellite positions to the mobile. (So there is no need to receive it from the satellite directly.) But the satellites are seen from the location of the BTS. So we could calculate the lan/lat of the BTS "backwards". There is only one point i am not sure about. They also know the location of the mobile (triangulation) and could correct the lon/lat towards the mobile.
UPDATE: no German Telecom cell towers found in the query, so i can't check my local towers So i would try to check the coordinates from different sources against each other.
UPDATE: didn't find any proved/official geolocations of towers, so i can not check agains the db. UPDATE: Still nothing found about Tower coordinates when providing Provider Network AGPS. |
@Gitschubser I was just checking: But:
|
@E3V3A Hi, Hanno from Mozilla here, I'm in charge of running the technical side of the Mozilla Location Service (MLS).
In general many of the things @msemm said about OCI also apply to MLS. We calculate averages over all reports as the aggregate estimate. This means the position estimate is more likely the center of the covered cell area, and certainly not the position of the tower / antenna. If you send us multiple cells, and we have data for them, we'll to trilateration over all cells. We also use WiFi, cell, cell location areas and GeoIP to provide a position estimate. So you might get accuracies up to several thousands of kilometers in the worst case, where we only know you are either in China or Russia based on GeoIP data. If you have specific questions on MLS, feel free to reach out or mention me directly in a github issue. |
@hannosch @msemm |
@andr3jx That would be against their terms of service. You aren't allowed to use their API and then store the result and use it in your own service or distribute it. These companies rely on keeping their databases private today. Both our projects try to change that and make sure these datasets become a free, public resource. |
@andr3jx Please don't violate the terms of these companies. Doing so doesn't help anyone. If you submitted stolen data to one of the open projects, you'd just force us to deal with law suits and waste a lot of our time and potentially money. I want to get to an open dataset, but we need to do so based on a community effort from the ground up, not by stealing data. Openstreetmap has shown that this works, it just takes a while longer. |
@hannosch Don't worry I won't, It was just a thought. I hope Mozilla can build a big community for this task. |
@hannosch Thank you so much for getting in contact with us, it is very much appreciated. I'll send you a private email, while we wait for the DB. Then we'll have to figure out which DB to use, if not both OCI and MLS. |
OpenCellID (OCID) and Mozilla Location Services (MLS/Ichnaea) will now support sharing of data as well as using the same data formats. mozilla/ichnaea#282 and mozilla/ichnaea#283 |
A map that shows the cellids from MLS - MLS Cell Network Map: https://carto.rudloff.pro/gsm |
@hannosch and @banjaxbanjo, I would like to fix this Issue to also solve #349. Here is my idea: We already have this button |
@SecUpwN I won't mind if you add such a feature to your app and download the MLS cell data. But both the OCID cell data and MLS data are about 170mb each in compressed form. It's probably rather inefficient to let every app user download and parse this data for the apps purposes. You might want to stand up some service that gets the data from both sources, combines it and offers it in a more suitable form, maybe make it possible to only get the data for a certain country or even a limited number of operators (mnc's). |
I see little point in doing both since MLS is based on -- or using OCID data. Either one or the other, but not both. |
It depends on what you do. If you do an online query using the MLS API, you'll get results from both data sources. But if you download the dataset yourself, then MLS only publishes its own dataset and OCID publishes theirs. Each of them is about 7 million cells these days, last I checked the overlap is only around one million between those. We do have to keep the datasets separate on the storage level, as they have different license terms attached to them. OCID has historically inherited CC-BY-SA 3.0 for their project, whereas MLS has started out under public domain (CC-0). |
Thanks for clarifying @hannosch. I've been following your wetting procedures closely, and would probably trust your data set more, since it is transparent to what you do. In either case I would like to see a possibility to download your data separately from within our app. For example by using a setting of:
|
sorry we have to close a few issues - i this still needs clarification pls let me know. |
This is an enhancement. |
Do a request via Mozilla Location Service and check if the cellid is in the area.
Example result (you get latitude, longitude and the accuracy):
This could be interesting as fallback to opencellid.
Maybe there are people here that would be help to collect cellids for the project.
Apk for android:
https://github.com/mozilla/MozStumbler/releases
API:
https://mozilla-ichnaea.readthedocs.org/en/latest/api/index.html
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.
The text was updated successfully, but these errors were encountered: