enhancement:maptool:improve processing speed and fix some memory holes #901
Conversation
|
How many threads did you assigned on your tests and what would you suggest as a optimum if running with ryzen 2600 and NVMe ssd? |
|
I used the now default of 8 threads. I wouldn't use many more because, with every thread, you split the resulting hash. I used 7GiB of slice size on a 16GiB RAM machine. Remember to do My machine is a core I7 NUC with a spinning USB attached disc. So I expect yours to be way faster. Unfortunately currently the disk speed doesn't help that much. Some explanation: On creating the hashes, you basically read all items in a common queue, and every thread hashes the next item it gets into its own private hash. As hashing scales bad with number of elements, this helps a lot because a: more processing power, and b: smaller hashes per cpu. BUT: Additionally using your SSD, the smaller fix contained will help a lot, I think. instead of reading the coordinates (several billions in world) one by one from disk I added a buffer reading them several MiB at once and then serving them from the buffer. This helps most on turn restriction processing. |
|
Somebody planning to test this or is already testing this? Works for me. If no objections: Planning to merge on Friday 18th October 2019. |
|
Sorry i cant test (map size>>device size) |
Maptool running against pbf of Germany ;) Will show a comparison when finished |
|
Cool. @jkoan . I'll await your result, Expect the order of some elements in the binfile to be different, because the threading shuffles the processing order a bit. Shouldn't matter much, as osm files are usually ordered by OSM ID, which is purely random in terms of map functionality as well. @gefin Unfortunately I don't have a clue when this will hit planet extractor after merge, as I don't know any way to see maptool version used there. Hopefully we will never notice this pull request there. |
|
Does anybody have a ready-made tool to extract regions out of a world binfile? While the algorithm is basically described in https://navit.readthedocs.io/en/latest/maps.html#binfile , it would stupid to reimplement something that seems to exist at least for planet extractor. Would help a lot to check #906 |
i don't know a tool like this, but we plan to rewrite planet extractor anyway since its closed source and not maintainable. But this is something for another issue, not this PR |
|
Bummer. Seems to be the reason why I never found the source for it. |
Runs number 1&2 did run after each other. Run number 3 and 4 run at the same time (different dirs). |
|
Merging this albeit it seems to be system dependent or dependent on the size of map file how much speed is gained. At least on my setup this saves around 20% processing time. At least some memory holes get fixed. |
|
Will keep the branch for a while for reference. |
navit-gps#901) * Convert turn relation processing to multi threaded * Enhance item read function to block read the items for faster overall reading speed. * Fix some memory holes found by valgrind. There are still many left. Specially in coastline and Country border code.
#901) * Convert turn relation processing to multi threaded * Enhance item read function to block read the items for faster overall reading speed. * Fix some memory holes found by valgrind. There are still many left. Specially in coastline and Country border code.
#901) * Convert turn relation processing to multi threaded * Enhance item read function to block read the items for faster overall reading speed. * Fix some memory holes found by valgrind. There are still many left. Specially in coastline and Country border code.
navit-gps#901) * Convert turn relation processing to multi threaded * Enhance item read function to block read the items for faster overall reading speed. * Fix some memory holes found by valgrind. There are still many left. Specially in coastline and Country border code.
This pull request increases processing speed of maptool on large areas (world) by partly parallelizing the turn restriction processing as well as by introducing buffered read of nodes on relation processing.
Time saved on my box: -1day / ~20%