Changed default path for custom navit.xml

[mvglasow]

This came out of discussion in #904:

#877 changes the default path in which Navit looks for a custom navit.xml on Android.

Prior to #877, one could place a custom navit.xml into /sdcard/navit/ and Navit would use that; this was also the path where Navit would look for maps and store downloaded maps.

#877 changes that: the default path for navit.xml and maps is now /data/data/org.navitproject.navit, the app’s private data dir, not accessible from a connected PC. However, there is now a preference to change that path—one can easily change that to /sdcard/navit (or anything else), which takes a one-time manual step.

TL;DR: does everyone agree that we should keep it that way, or should we tackle this differently?

The rationale behind this is that /sdcard/navit requires special permission to access, which the user needs to grant manually on later versions of Android. The new default eliminates that need for users who just download maps and are not customizing navit.xml.

@jandegr furthermore writes:

On the most recent SDK's the path mnt/sdcard can't work anymore, but before #877 that path did not work on those devices either.

I’d like to put that up for discussion. Essentially, we have three options:

Option 1: Shared storage (usually /sdcard/navit)

This is the previous behavior.

Pros:

• On devices with a separate /sdcard device or partition, this is usually the largest partition (important as map files for bigger areas are quite large)
• Can be accessed from a PC connection or file manager apps

Cons:

• On older devices (where /sdcard is FAT-formatted), file size is limited to 2 GB (and map files can easily get bigger than that)
• Requires storage permission to access

Pros or cons (depending on your point of view):

• Uninstalling the app will leave the data intact

Option 2: External storage (usually /sdcard/Android/data/org.navitproject.navit)

Pros:

• On devices with a separate /sdcard device or partition, this is usually the largest partition
• Can be accessed from a PC connection or file manager apps
• Does not require storage permission to access

Cons:

• On older devices (where /sdcard is FAT-formatted), file size is limited to 2 GB (and map files can easily get bigger than that)

Pros or cons (depending on your point of view):

• Uninstalling the app will remove all data

Option 3: Internal storage (usually /data/data/org.navitproject.navit)

Pros:

• On older devices (where /sdcard is FAT-formatted), file size is not limited to 2 GB (though the whole partition might not be that large)
• Does not require storage permission to access

Cons:

• On devices with a separate /sdcard device or partition, this is usually the smaller partition and might not be big enough for map files
• Can only be accessed by the app itself or from a root shell, not from a PC connection or by file manager apps

Pros or cons (depending on your point of view):

• Uninstalling the app will remove all data

Variations of the above:

• Different default paths: one for navit.xml, another one for map data
• Multiple paths for navit.xml (ordered list of the above options): if nothing is found in one path, look in the next

@jandegr, what did you mean by:

On the most recent SDK's the path mnt/sdcard can't work anymore

—that it might not work? Or that it definitely won’t? (I have not yet seen a device where /sdcard is not accessible even for apps with shared storage permission, but I certainly may have missed something.)

Currently I am gravitating towards option 2, as it makes data manipulation from a PC easier. We might add an automatic fallback to /sdcard/navit/ for navit.xml if it is not found in the first path, to avoid breaking compatibility with older setups. (Right now, upgrading from pre-#877 to post-#877 will render Navit useless until the user figures out they have to set the path manually).

[lains]

We might add an automatic fallback to /sdcard/navit/ for navit.xml if it is not found in the first path, to avoid breaking compatibility with older setups. (Right now, upgrading from pre-#877 to post-#877 will render Navit useless until the user figures out they have to set the path manually).

I agree that a smooth upgrade path from the current folders would be a good thing. Otherwise, existing users may struggle to find the (easy) solution to recover a working map at startup.
This could be done the way you suggest, or even the other way around also: if /sdcard/navit/navit.xml exists, then this path is used by default the application, otherwise, it switches to the new behaviour. What's your opinion?

[lains]

Also, from my tests, it seems that an automatic check of /sdcard/navit is really required because each time I re-install a test application built on my machine, I loose the manually setup map location and I have to do it again after launching the new compiled app.

[mvglasow]

As @jandegr pointed out in #890, Android 10 (Q; API 29) introduces scoped storage. In a nutshell:

• Accessing files via File with a conventional filesystem path will be restricted to app-specific directories (private and external storage).
• Other files on external storage can only be accessed using the MediaStore API or the Storage Access Framework.
• The path returned by Environment.getExternalStorageDirectory() is no longer directly accessible to apps.
• For the moment, apps can circumvent this by either targeting API 28 (Android 9) or lower, or by requesting legacy shared storage in their manifest. However, this will be discontinued with the next major platform update (presumably API 30/Android R/Android 11).

This means that shared external storage and portability do not go well together. Then again, Navit could easily move things to private storage (internal or external) and continue addressing files via filesystem paths.

We can now do the following:

• Opt out of shared storage for now (afaik Google Play will require us to target the latest API)—this way automatic migration will still work for Android 10 users (otherwise they would only work for Android 9 or lower).
• When falling back to shared external storage (aka /sdcard/navit), test if it is accessible before using it
• If we determine the user is still using shared storage but we cannot access it, display an error message explaining the user how to resolve the situation
• If we find out data still resides in shared external storage and it is accessible, offer to move it into private external storage (prompt the user). Being a move on the same volume, that should be a fairly quick operation even with several gigabytes of map data.

Further reading:

• https://developer.android.com/training/data-storage/files/external

[jandegr]

Hi,

The only long-term solution I could think of (and briefly tested) is to select a path with Intent(ACTION_OPEN_DOCUMENT_TREE). This can take care of permisson as well.
Just making the native code work with uri's would be a whole lot of work, in java no problem at all.

So what would work is to copy files into the private storage folder, but that can already be done with android's own file explorer.

[jkoan]

The only long-term solution I could think of (and briefly tested) is to select a path with Intent(ACTION_OPEN_DOCUMENT_TREE). This can take care of permisson as well.
Just making the native code work with uri's would be a whole lot of work, in java no problem at all.

@jandegr Could we make this work for devices down to our current minsdk version?

[mvglasow]

After updating to Android 10 and running into this pitfall, I would recommend the following course of action:

• Point the file browser (launched from the Set map location) menu item to the app’s external storage directory (usually $SDCARD/Android/data/org.navitproject.navit/files, but use the API call to find out) by default (no other dir is set), or when the selected dir is not accessible
• If we determine the selected dir (or the one we fall back to) is not underneath the app’s external (or private) storage and the API supports app-external storage (need to figure out the min API version for that), display a warning that these files need to be moved in order for Navit to still be usable on Android 11+ (see below for Android 10). If the user selects Yes, move files and set the path in preferences; else store a flag that the user declined (we might wish to provide a way for the user to return to this dialog later in that case)
• Opt out of scoped storage so the migration can still be performed on Android 10 (Android 11+ will no longer honor the opt-out)

@jandegr:

copy files into the private storage folder, but that can already be done with android's own file explorer

sure it can be done that way, but that’s a bit like shorting your car to start it, rather than using the ignition key… not exactly the UX I would expect.

[mvglasow]

According to information on the net, Android 11 reversed the previous decision of locking apps out of the file system, see https://stackoverflow.com/questions/60360368/android-11-r-file-path-access and https://stackoverflow.com/questions/62782648/android-11-scoped-storage-permissions/66366102#66366102. A new permission may need to be requested, as far as I can see it.

After upgrading to Android 11, however, Navit is no longer seeing any of my local files (navit.xml or maps), which are in /sdcard/Android/data/org.navitproject.navit/files. Haven’t quite figured out why this isn’t working.

[jkoan]

According to information on the net, Android 11 reversed the previous decision of locking apps out of the file system, see https://stackoverflow.com/questions/60360368/android-11-r-file-path-access and https://stackoverflow.com/questions/62782648/android-11-scoped-storage-permissions/66366102#66366102. A new permission may need to be requested, as far as I can see it.

After upgrading to Android 11, however, Navit is no longer seeing any of my local files (navit.xml or maps), which are in /sdcard/Android/data/org.navitproject.navit/files. Haven’t quite figured out why this isn’t working.

i got similar user issues via mail, also couldn't figure it out so far

[mvglasow]

I’ve opened a new issue #1117 with everything I have been able to collect so far.