Routing no longer possible on WinCE?

[clarified]

This is really a continuation of the old trac report: https://trac.navit-project.org/ticket/1158

Op system: Win CE core 6.0. on Mstar semiconductor, 99MB ram. Cortex-A7-MSB2531 which is a 7" satnav believed to be using MediaTek MT3351C, 128MB flash, 800 * 480 display.

I have been using it with recent versions of navit for several years with pretty good results.
But after downloading navit-30-10-19.zip, that is the navit.zip with a time stamp around 30/10/19, it takes an absurdly long time to calculate very short routes and causes a WinCE out of memory
notice & crash on even routes around 20 miles in an area which only has moderate OSM data density. So it is no longer useable on WinCE, it seems.

I have tried some of the suggestions in the trac ticket including renaming the xpm directory
and set the cache limit to 512000 (cache_size="512000" on the <config xmlns:xi line ).
This has made no discernable difference.

I did a git clone and had a very brief scan with 'git log' but no commit stood out as a likely cause.

Is navit checked on WinCE with limited memory? I imagine that most navit installations are on other operating systems with more RAM these days.

The satnav is very limited with no diagnostic tools, so I can't provide much useful information. And I have next to zero knowledge of WinCE. Sorry.

Any suggestions? Will WinCE support continue? Some of the commits looked quite radical: were they tested on WinCE?

[gefin]

check how your device draw the map without routing. If it has delays check issue #894 (huge polygons memory usage ).

Also check the route_depth setting. With TT i use "4:15%,6:1%,8:10000,18:2000" in large city. May be you try something between the standard setting and this real restricted sample.

[clarified]

Quick preliminary reply:

The map drawing without routing is a bit sluggish, but probably about the same as it was with the working versions. I am travelling for a day or two, so won't be able to investigate further immediately.

[clarified]

I used navit on a 200 mile journey, and can now confirm as before that routing crashes. Without routing, the map updates are so slow that the position displayed is wrong, and in most cases the updates are many seconds apart. That said, at one point, the position was updated reasonably quickly, but it was not obvious why that happened. This was with a small cache size set.

navit.log included nothing usual: but it did include
error:navit:file_init:Maps larger than 2GB are not supported by this binary, sizeof(off_t)=zu

However earlier working versions of navit also emitted similar messages into navit.log, although I am not sure they were flagged as errors. The actual maps sizes were:
mainmap.bin 978M and contourmap.bin 82M.
Larger maps worked with earlier versions.

This is just an update. I need to experiment further with various settings. But navit has gone from working well on WinCE to being completely broken :-(

I need to review my navit.xml: the current version worked well but I see that it is somewhat old, so I need to rewrite and check the entries.

[clarified]

The navit.log error message

error:navit:file_init:Maps larger than 2GB are not supported by this binary, sizeof(off_t)=zu

seems to come from the last few lines of file.c, and, unless I misunderstand the code (it is many years since I wrote any C), it is not an error: rather an informational message.

[metalstrolch]

Hmm, assuming a pre 20-10-19 navit still performs better, there are two major upgrades to navit that are candidates having caused this:

1. Polygon with holes support, aka multipolygon support.

• This can easily be checked by using a "old" map not containing any polygons with holes. As the changes only require more memory / performance on drawing if such features are in the map, this can simply be checked and confirmed / outruled. So please try with a old map once worked for you.

2. Traffic distortion handling.
   This was a bigger change in routing strategy, i think. Albeit definitly not working on WinCE yet, I Don't know the routing core changes done for that. Unfortunately I don't know much about this topic to help here.

What was the last version known to work or, how old was it?

[clarified]

Update:

I have rewritten my custom navit.xml to reduce memory usage as much as possible and can now route for short distances, at least so far as I can test. This is using 0.5.4. But the time taken to calculate routes of more than a few 10's of km is ridiculously long: many minutes, and can be around 20 minutes. Trying to route for distances of around 100km usually results in a crash sometimes with wince reporting a memory problem. Furthermore restarting navit does not clear
the memory: it requires a power cycle - so some sort of global memory leak.

I have added new vehicle_profiles with a variety of route_depth specifications, so I can select
the lighter ones when trying to route over longer distances.

As for testing, I have only just discovered the demo option which seems to have little documentation. Otherwise, I was testing in a limited way with the real gps signal. Both are very time consuming, so my results are preliminary.

I have also tried to use lighter layouts like T@H and car-simple in the hope that might use less memory.

In summary, things are a little better, but for distances more than, say 50km, the navit on this satnav is not of much use. I realize that the OSM maps have become more detailed, and as an OSM contributor, I am partly to blame. I cannot tell whether that is at the root of this drop in performance.

I have skimmed some of the code, but I would need a ridiculous amount of time to get to grips with it.

[clarified]

Yes, I ought to try with an old map, if I have kept a copy somewhere.

As for which version last worked, I am a little unsure. I normally download a "navit.zip" which does not seem to have a simple text file included with the version number. I can look once running on wince, but then I have no printer, screen grab or any easy way to record the version short of looking on screen and writing it down :-( The best that I can do is note the date of most of the file in the navit.zip, but I doubt if that helps you much.

I will try to experiment further and report.

[clarified]

Progress!. It seems to be the map. So presumably multipolygon support.

I found an old map which seems to be dated March, 2018. With that map, navit 05.4 works brilliantly.

So should there be some way of turning off Multipolygon support for such devices? Or it it new buggy code that will eventually work on limited satnavs? Would that be a switch to maptool perhaps?

[metalstrolch]

If it's with the map, then most probably it isn't multipolygon support as such, nor is it drawing of polygons with holes. It's most probably the sheer number of map elements inside the map. By supporting multipolygons (basically converting them to traditional binfile polygons) we add tons of elements to the map. Some of them are quite big in terms of node count. There are traditional polygons with high node count as well, but not as many as per multipolygon. On searching the map, the device needs to uncompress the tiles. With many elements inside them they tend to get bigger requiring more ram to uncompress.

Proper solution could be:
Have one map file with only the things required for routing, throwing away all polygons not required like landuses, nature reserves and many more that are not routable but quite big. The biggest I know of is lake Nasser, Egypt with almost 25 MiB size one polygon.

And another map with the rest. So one could disable this if his device isn't powerful enough.
While this setup wouldn't require many changes to Navit if any at all, it would require changes in infrastructure mainly maptool. It would require two sets of configuration for maptool. But the worst thing is that this would require to process the map twice. It's a long run to do this on world. And it wouldn't work well with current planet extractor. And it would bother all users with more powerful devices.
If you want to try, just alter maptools attrmap (navit/maptool/osm.c) throwing out all things you want to remove from the map. You can even feed that as a parameter to precompiled maptool.
Maybe it already helps to convert your own map instead of downloading it from planet extractor, as this won't contain all the things from high tiles way off your region.

I'll double check the polygon code for memory leaks, but I don't think this is the reason. I'll even try to find my old winCE device with some more ram than yours and check results there. At least the polygon with holes code was reported to work on wince back when I wrote it.

[clarified]

Thanks for the reply.
I had a quick look at osm.c and may try experimenting with that when I have time.

"You can even feed that as a parameter to precompiled maptool." Is that the -r option?
I couldn't find any documentation on the format for the rule-file, and it wasn't obvious from maptool.c .

I have a copy of maptool on a debian testing system. It is hard to tell which version it is since there is no --version switch. But the dpkg -s has version 0.5.3+dfsg.1-2+b1. I wonder whether that has multipolygon support, and if not, whether it may give a temporary quick solution for my immediate
problem.

[metalstrolch]

Yes that's the -r switch. But to be honest I never tried it. You can implicitly see if the maptool has multipolygon support. If it offers the -3 (--32bit) switch next to the -6 switch, it should be OK. We changed the default to generate 64bit zip files. So we added a -3 option to force 32 bit zip files. That happened around the same time multipolygon was implemented.

[clarified]

Thanks. It looks as if the Debian version has the -6 switch so maybe too recent. However I found a post that suggested how to use the -r switch: it basically takes the attmap strings in a file. But first I tried a simple test of maptool :

maptool --protobuf -i /ssdback1/ael/mapping/mkgmap/great-britain-latest.osm.pbf test.bin

but that SEGFAULTED almost immediately after writing some empty *.tmp files. gdb
shows

Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x000055c63a68cf30 in osmpbf.blob_header.init ()
Backtrace showed:
(gdb) bt
#0 0x000055c63a68cf30 in osmpbf.blob_header.init ()
#1 0x000055c63a6917da in protobuf_c_message_unpack ()
#2 0x000055c63a68a7ec in ?? ()
#3 0x000055c63a68abb8 in map_collect_data_osm_protobuf ()
#4 0x000055c63a679e73 in main ()

Well, I think a segfault is always a bug, but that aside it looks as if it can't decode the pbf file.

Maybe I misunderstand how to use maptool. I better try and read some more documentation and maybe skim some code.

So perhaps some progress once I can understand how to get maptool to work.

[clarified]

Just a correction. I meant to say that the -3 switch was missing from the Debian maptool, so
perhaps it might work even without needing the -r input. But first I have to get it stop segfaulting...

[clarified]

I have just compiled maptool from my local git repository and it now seems to be working. So it would seem that the debian version is somehow broken. When I have time, I will try to remember the bug report that to Debian.

[clarified]

Update after a time away mapping for OSM. And I used my wince 7" satnav with navit and an old map. It really is excellent: the satnav seems to be a repurposed 7" tablet, I believe manufactured by MediaTek themselves. So loss of support for wince on navit would be a disaster.

Meanwhile, I have been experimenting further with maptool. I mentioned some of that in issue #975.

I am now attempting to understand how to use the -r (rule file) option.
I extracted the rules from osm.c and edited into the attached rule.txt.

Trying that with maptool -r rule.txt .... gave errors like:
no id found for 'addr:housenumber=* house_number'

I had a similar error for every line in rule.tex. So it looks as if I am missing a field in the entries.
Can anyone suggest what is needed, before I try to fathom the code? The rule file format seems to be undocumented.

Files:
So clearly I am using the wrong format for the rule.txt file

[I seem to be unable to attach files for some reason, maybe a github problem with a palemoon browser?]

[clarified]

After a quick skim of osm.c, it seems that there can be no leading whitespace on lines in rules.txt

I am running maptool again as I write this with a corrected rule.txt matching the attrmap in osm.c, s I expect it to generate a standard map. After that I can try experimenting with a reduced number of rules removing landuse and the like to see how that helps.

[clarified]

I seem to have success. At least I generate a map.bin using -r rule.txt which is very close to the
one generated without using -r. For this experiment, the rule.txt contained all the entries in attrmap, so that is to be expected.

I now need to edit out some of the lines in rule.txt to see whether that solves the wince routing problem. If this works as expected, then all that is needed to solve the wince problem is to provide
a set of rules files for limited devices. With no code changes. I realise that doesn't solve the planet extractor problem of multiple runs, but at least wince users could compile their own maps locally.
I little extra documentation for that would help.

But I mustn't get ahead of myself. I now need to understand which items are routable, and which polygons/multipolylgons are high cost. I hope that will be obvious from navit.xml or maybe code.
Any pointers/

As in #975, I am getting some worrying warnings (mainly about almost all towns "missing"), but that's a secondary problem for now.

[clarified]

I am surprised to report that I have just tried the maps that I compiled with maptool, without removing any of the rules, on my wince satnav and the routing problems are solved.

This is in contrast to the maps that I collected from the planet extractor which could only route over minimal distances.

As above, my source was great-britain-latest.osm.pbf which I suspect I collected from https://download.geofabrik.de/europe/great-britain-latest.osm.pbf

So perhaps there is some "problem" with the planet extractor.

Just now I am on a limited internet connection, so I may not be able to try with a later pbf for a while.

[clarified]

Earlier I found that I could not attach files here: it turns out that noscript was blocking a github-production-repository on amazonaws. So I now attach those missing files here.

no_rule.log.gz
rule.txt

[metalstrolch]

This isn't a surprise at all. This is a limitation on how the navit binfile works. See https://navit.readthedocs.io/en/latest/maps.html#binfile

The world is divided into four tiles. Every tile is divided into 4 subtiles and so on. Map elements crossing a tile border are shifted one tile up. So things crossing the equator move to the highest tile.

If extracting a map from the world, then you have to take ALL tiles covering your area. So you'll always get the equator tile for example.

If processing only a portion of the world, then there are no elements crossing the equator ant the top level tile is small Same for a lot of the higher level tiles.
Just compare your new processed binfile with the one downloaded from the extractor. You'll find that the downloaded one is significantly bigger,

So in the end this confirms that the number of map elements is indeed the problem.
-> For constrained devices reduce the amount of elements. Processing only a portion of the worls helps. But this won't work with planet extractor.

There is a proposal at #918 that may improve this situation as it would allow to cut elements in pieces tiling them more down. I already partly did this in #894. Seems your device is even more constrained.

[clarified]

OK. Thanks for that. I am looking at the documentation on the bin format now.

One problem with navit is that the documentation is scattered over so many places: I keep discovering more. I realise that the readthedocs site is an attempt to put much of it in a single place.

It has taken this long thread to discover the problem with some wince devices. Can I help by adding a note to, well that's the problem, which navit site? I suggest
https://wiki.navit-project.org/index.php/WinCE
is the obvious place for now.

My note would say essentially, don't use the Planet Extractor if you have a low end device, or find that a map from the Planet Extractor is too slow. But maybe #918 will render that obsolete?