Issue downloading database

[Kaicastledine]

Thank you for your feedback! Please add below details so that we can help you a bit faster.

---

Agreements

Please check these boxes to ensure you've actually read and understood these important rules!

• I have reviewed and accepted the guidelines for contributing to this project.
• I have searched the open Issues and made sure I am not filing a duplicate.
• I have read the notes on debugging and will properly maintain this Issue.

---

Overview

Please add a short and easy to understand description of the Issue you've experienced here.

• AIMSICD version (0.T.42-alpha-normal):
• Xperia Z2 - CM12.1-2016041-nightly-sirrus:
• Link to logfile without identifyable data:

D/roidIMSICatcherDetector(24132): BaseTask detach:com.secupwn.aimsicd.ui.activities.MainActivity
W/SQLiteConnectionPool(24132): A SQLiteConnection object for database '/data/data/com.SecUpwN.AIMSICD/databases/aimsicd.db' was leaked!  Please fix your application to end transactions in progress properly and to close the database when it is no longer needed.
W/RequestTask(24132): Trying to talk to OCID timed out after 60 seconds. API is slammed? Throttled?
W/SQLiteConnectionPool(24132): A SQLiteConnection object for database '/data/data/com.SecUpwN.AIMSICD/databases/aimsicd.db' was leaked!  Please fix your application to end transactions in progress properly and to close the database when it is no longer needed.

---

Reproduction

When ever you try to download database I get the error contacting OpencellID.
After update of APP was given DEV API key, WIll enter in my normal API key next to test
Already did a manual install of the database(now old) but did not work.

References

If your Issue is related to any other existing Issues or pull requests, please link them here.

---

Screenshots

If you experienced visual glitches, please add a screenshot without metadata here.

[Nordlenning]

Hi @Kaicastledine
I have noticed that OICD servers have been extremely slow last couple of weeks (via my Lap), if working at all. Sometimes you get OK sometimes timeout so beware of this. I have managed to get up & download ok. Notice issue #837.

[Kaicastledine]

I've been trying on and off over the last week but no luck so far even when I've downloaded and put them in manually.

Wanted to try rule out it being the DEV API key or even an issue with the app/phone

Will check that now

[Kaicastledine]

The API key was quick after the new app update yesterday (API request from app)
I was able to upload cells within minutes but never download the database.

[Nordlenning]

Yes, that is usually the case, I take copy of the db file when download & keep it stored & updated weekly on my SD. I also take a copy of the ocid key & store it as a txt file on my SD. Just in case.
I have made it a habit to always do a clean install, deleting the old aimsicd catalog first to avoid any possible conflicts. It works for me.

[Nordlenning]

Pssst @Kaicastledine It can be vice to close this issue since your CID is displayed in the log, that's against the rules ..I guess @SecUpwN will delete this log when he arrives.

[Kaicastledine]

@Nordlenning Cough its gone

Must of missed that one

[mimi89999]

@Nordlenning What files do download from OCID and where do you put them?

[Nordlenning]

According to filedate & yesterdays download & viewing them in database, it is the file:
files/OpenceddID/opencellid.csv Located on my /storage/sdcard0/android/data/com.setupnv/aimsicd
I copy/overwrite this catalog to my SD1 before new installations as a safety rule.

[mimi89999]

But which file from the website?

[mimi89999]

cell_towers or measurements?

[Nordlenning]

The file is opencellid.csv That is the file that varies in size & gets updated when you got download ok.
I have not logged the actual transaction to see the exact file-transaction.

[Nordlenning]

cell_towers or measurements?

They appear in the db as imported ocid db files

[Kaicastledine]

Interestingly mine data path is
android/data/com.SecUpwN.AIMSICD/Files/OpenCellID/aimsicd-ocid-data.csv

Just checking and I do not actually have a database folder located under OpenCellIDs data folder

[Kaicastledine]

I've just gone a put in the new CSV and renamed it to match my current filename aimsicd-ocid-data.csv

Still wont view the database

[Kaicastledine]

How can I change the database location ?

I've just noticed its referencing /storage/emulated/0/android/data/com.SecUpwN.AIMDCID/files/OpenCellID/opencellid.csv

I need it to be storage/sdcard0/android/data/com.SecUpwN.AIMDCID/files/OpenCellID/opencellid.csv

Emulated only has a "Legacy" Folder

[Kaicastledine]

Just tried to export the DB and under /com.SecUpwN.AIMDCID/files/
There is an error.txt

States that it cannot find /storage/emulated/0/android/data/com.SecUpwN.AIMDCID/files/aimsicd_dump.db

[mimi89999]

I reinstalled AIMSICD, entered my API key, but I can't download OCID data.

[Nordlenning]

@mimi89999
Nothing appears in the Imported OCID database in db-view ?
No opencellid.csv file in the OpenCellID catalog ?

[Nordlenning]

This is output from imported OCID data yesterday,

[Nordlenning]

Went over to my laptop & found that OCID server is up & running, but I can not use it online.
Not possible to view measurements or uploaded bts on the website. Have been like this a long time.
So something is wrong there.
Ping 13 packets transmitted, 13 received, 0% packet loss, time 12016ms

[Kaicastledine]

I get /OpenCellID/aimsicd-ocid-data.csv

I think its an issue with the app pointing to an old storage location on android

Its using the old legacy locations based on older versions of android. Newer versions have moved the SDcard names and locations

12016 ms ? ..... Thats some serious latency

[Nordlenning]

@mimi89999 Today i gave this some attention again, after uninstalling first .Clean install.
App creates 2 dirs during installation 1 dir on both SD cards sd0 & sd1 under Android.
No OCID dir exists after installation in theese dirs.

Then I entered my OCID key into requester.
Then I Downloaded OCID db from OCID servert. OK :) <<< from the menu.
Then I noticed that OpenCellID dir had been created on sd0.
Containing 1 file opencellid.csv

Then I repeated the whole procedure & downloaded db from Map View .OK <<<

D/roidIMSICatcherDetector(27038): BaseTask addTask activity:com.secupwn.aimsicd.ui.activities.MainActivity
I/RequestTask(27038): DBE_DOWNLOAD_REQUEST write to: /storage/sdcard0/Android/data/com.SecUpwN.AIMSICD/files/OpenCellID/opencellid.csv
D/RequestTask(27038): doInBackground DBE_DOWNLOAD_REQUEST total not returned!
I/AIMSICDDbAdapter(27038): UpdateOpenCellID: OCID CSV size (lines): 1001
D/AIMSICDDbAdapter(27038): PopulateDBeImport(): inserted 1001 cells.
D/AIMSICDDbAdapter(27038): CheckDBe() Attempting to delete bad import data from DBe_import table...
I/AIMSICDDbAdapter(27038): CheckDBe() Deleted BTS entries from DBe_import table with bad LAC/CID...

[SecUpwN]

It can be vice to close this issue since your CID is displayed in the log, that's against the rules

Thanks for reminding him, @Nordlenning. We just want to avoid users to compromise their own privacy.

I have noticed that OICD servers have been extremely slow last couple of weeks (via my Lap), if working at all. Sometimes you get OK sometimes timeout so beware of this. I have managed to get up & download ok.

@CellularPrivacy/database, I am honest when I say that I have a headache with OpenCellID now and I guess it is time to completely move to Mozilla Location Services in #446, which not only provides the OpenCellID data, but also their own - and even sharing that to OpenCellID. Please join the discussion.

[Kaicastledine]

I've done a clean install and I still have the same issues.
All the backups fail also due to this mix up with the storage locations.

Also the CID was removed shortly after and the issues are still present .

[Kaicastledine]

Will need to check it but editing one line of code for a storage location could cause more issues.

[SecUpwN]

Will need to check it but editing one line of code for a storage location could cause more issues.

If you find a better way for our app to detect and use the storage location, please submit a pull request!

[Kaicastledine]

Indeed, From what I know I think It was to do with a change made in android ages ago to do with where they store data to solve the Internal SD and ExT SD issue .

How ever I have this installed on CM12.1 Based on android 5.1. Would have to check to see if this storage situation is the same on both.

What other Versions of android do you know have this issue? could create a check on install for the X location based on X version.

[Nordlenning]

Downloaded Latest v0.1.43-alfa from Git-hub.
Removed my old installation & clean installed v0.1.43-alfa.
connected to my WiFi.

Vent up on the menu-bar/preferences & pushed download API key.
My downloaded Key was immediately displayed in the Key Requester.
Pushed on the Main screen outside the requester to close it.
Vent up to the menu-bar again to double-check that the OCID Key was in place (displayed) & it was.
Pushed on the Main screen outside the requester to close it.

Vent to main menu & pushed download OCID Data. Download Success. +1200 BTS imported.

Vent to Imported OCID data in DB-view & scrolled 1200 + Imported BTS. (See pic, New Look).
Then I tried to download from Map-View & got this standard 604 HTTP/1.1 500 Internal Server Error from OCID. I have success rate of 1/30 or less with OCID server.

So on my 1'st try this was not an issue. Then I run into some other issues reported in issue #870
Hope this is of any help. None of the data in the pic is related to me..

[SecUpwN]

@Nordlenning, thanks for xyour extensive tests regarding this Issue. Even though I am a bit speechless as of why so few people are participating to solve bugs like these, I hope @CellularPrivacy/database will have a look at this. The key will be to find out if the main cause is OpenCellID itself. And if it is, we need to find out if the move to MLS (see and particpate in #446) would solve these Issues all at once.

[Nordlenning]

This have been a repeated issue many times for a long time.

This download problem is again in my opinion not AIMSICD fault at the current moment.
I have given pic & logs. The problem is on OCID server side. You cannot use OCID on they're own site. Try watch recently uploaded or new discovered bts on they're own site & you will see. If you watch the traffic with e.g Wireshark you will see that its only errors & retransmissions etc.

This issue on OCID servers have lasted a long time & I seriously don't understand why they are not informing of this or fixing this. I have tested OCID site with different browser & Os, all with the same result. I would like to here from the OCID team on this before I would spend a second on fixing this in AIMSICD. When OCID site is working on my laptop then i will look at this in AIMSICD again.

Upload Routine: This is where i think many users makes a big mistake I have noticed.
It is clearly stated in many excellent posts by @E3V3A, a years ago. AIMSICD don't contribute to OCID. Many users reports upload to OCID OK, not knowing they're upload is wasted.
This upload OK msg should in my opinion state "Not implemented yet" making the user aware of this.
If they (like me) want to contribute to OCID they must run 2 apps, I use OCID inViu. (I have tested stumbler)

I tend to follow @agilob in his thaugts about this migrating question at the moment, but I would like to know how long this problem with OCID servers is expected to exist... Hope this is of any use.

[Nordlenning]

Note to new users:
If you cannot download OCID data you will get the YELLOW notification Icon.
This will turn GREEN When you get OCID db downloaded,
Same when you discover new bts, AIMSICD will notify you with YELLOW notification.
Don't worry, just keep on collecting data...

[Kaicastledine]

Hello all

Sorry for not posting back sooner.

Just updated and lost all my local databases and backups ? not sure what happened there but have removed and re added the app anyway.

However went to request a new API key to test but denied as I requested one before updating so have to wait.

Went to use my old key and have got a API key request timeout .. Might be due to it not being used as a Dev key was used for a while.

Will try again tomorrow and see what happens when downloading the database.

[Nordlenning]

It is not possible to download from OCID server any more after the takeover & some modifications from Unwired Labs. The problem is not related to AIMSICD. Sorry about that.