Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancement:maptool:slice big polygons into smaller parts #906

Merged
merged 14 commits into from
Oct 28, 2019

Conversation

metalstrolch
Copy link
Contributor

This commit allows to slice all polygons (except poly_water_sliced) into
squares so that they fit into smaller map tiles. Holes if present are
correctly sliced as well.

This has the advantage that they do not need to be processed albeit not
being displayed, and navit-planet-extractor can filter them out.

This has the disadvantage that the big polygons are displayed checkered
if the corresponding itemgra in config features an outline.

TODO: Slice already on polygon creation. Currently the slicing has to be
performed twice, because of tile preprocessing and saving is done in
seperate steps from the tempfiles.

TODO: Make the trigger for slicing configurable. Right now it slices
everything smaller than tile level 4.

Hopefully fixes #894

This commit allows to slice all polygons (except poly_water_sliced) into
squares so that they fit into smaller map tiles. Holes if present are
correctly sliced as well.

This has the advantage that they do not need to be processed albeit not
being displayed, and navit-planet-extractor can filter them out.

This has the disadvantage that the big polygons are displayed checkered
if the corresponding itemgra in config features an outline.

TODO: Slice already on polygon creation. Currently the slicing has to be
performed twice, because of tile preprocessing and saving is done in
seperate steps from the tempfiles.

TODO: Make the trigger for slicing configurable. Right now it slices
everything smaller than tile level 4.
@metalstrolch
Copy link
Contributor Author

If the question arises, why i didn't reuse the coastline code which somehow does something quite similiar, the answer is simply: I don't dare.
I do not fully understand the coastline code yet and I don't want to screw it because my changes / additions. Maybe in the future when I understand maptool better I'm able to merge the itembin slicer with the coastline code saving some doubling.

Same is somehow true for the place I added the slicing. I know this is processed twice, but as I do not fully understand the tile selection code as well, this seemed to be the less intrusive way of doing this.

And for exactly this reason I will not care for CodeFactor reporting about code doubling. (my version uses double, while the geom.c version uses int which overflows when used on map coordinates instead of display pixels. And float is OK for the maptool. Definitely.)

metalstrolch and others added 4 commits October 14, 2019 10:29
This commit treats multipolygon members with empty role as "outer". This
rescues a lot of ancient or wrong created multipolygon relations into the
binfile.
Remove some stars to fix doxygen doc.
@metalstrolch
Copy link
Contributor Author

Seems to be OK and do what is wanted. Currently world is processing. Will try to post the sliced elements here. Not so many at all, but some are really huge.
Will try to generate world extracts like planet extractor does to cross check the results.

@jkoan
Copy link
Member

jkoan commented Oct 18, 2019

Seems to be OK and do what is wanted. Currently world is processing. Will try to post the sliced elements here. Not so many at all, but some are really huge.
Will try to generate world extracts like planet extractor does to cross check the results.

For a quick solution just unpack the planet and repack it into a zipfile which only contains the wanted pices. The empty files should not ne necessary any more.

@metalstrolch
Copy link
Contributor Author

@jkoan Unfortunately it's not that easy. The 64bit zip file support of Navit is quite crippled and based on a very old zip file specification, but required for any map bigger than 4GiB. So your nowadays "zip" tool creates a zip file that's valid but will happily cause Navit to segfault. I'm in progress of writing a small application allowing to extract things creating Navit compatible files. And in the long run we should update / fix Navits zip file support. But that's another story.

@jkoan
Copy link
Member

jkoan commented Oct 21, 2019

Couldent try it out myself, but probably this could help as well, in our repo deep deep in the source i found this https://github.com/navit-gps/navit/blob/trunk/navit/script/map_index.php it looks loke this is some kind of old state of the map extraction server. As well ther seems to be a cli option too https://github.com/navit-gps/navit/blob/trunk/navit/script/mapextract.php

@metalstrolch
Copy link
Contributor Author

@jkoan Ah, didn't know that, but did never search for php. My "C" version of an extractor is basically running and allowed me to find the problem here in 52d1f53.
I can steal the mercator conversion function from the PHP script at least which is still missing from my version.

Use overlapping for slicing big polygons. Neccesary because Navit's tile
system for tiles without overlap and those with is crippled.

This produces ugly results, but sorts the slices to correct tiles.
@metalstrolch
Copy link
Contributor Author

The tool to extract portions from navit bin file is here: https://github.com/metalstrolch/navit_binfile_extractor
The parameters are in the same order as the web based planet extractor puts them into the resulting file name.
Its working, but may lack error handling (but there are no errors eh?)

@jkoan The PHP version in trunk will fail, as it will calculate tile bboxes wrong. See #918

@metalstrolch
Copy link
Contributor Author

World processed without errors. Seems to do what it is expected to do, including the somehow suboptimal look of the sliced polygons. But there are not so many at all. For those being curious, there's the actual list:

OSM Info:http://www.openstreetmap.org/relation/167043 slice down 4 steps from acb to (acbaaaa - accaaaa)
OSM Info:http://www.openstreetmap.org/relation/1536358 slice down 4 steps from abd to (abdaaaa - acaaaaa)
OSM Info:http://www.openstreetmap.org/relation/2687974 slice down 4 steps from bab to (babaaaa - bacaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823200 slice down 4 steps from dca to (dcaaaaa - dcbaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823697 slice down 7 steps from  to (aaaaaaa - aaaaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823707 slice down 5 steps from dc to (dcaaaaa - ddaaaaa)
OSM Info:http://www.openstreetmap.org/relation/3003587 slice down 4 steps from dcb to (dcbaaaa - dccaaaa)
OSM Info:http://www.openstreetmap.org/relation/3878946 slice down 5 steps from ba to (baaaaaa - bbaaaaa)
OSM Info:http://www.openstreetmap.org/relation/4641113 slice down 4 steps from bcb to (bcbaaaa - bccaaaa)
OSM Info:http://www.openstreetmap.org/relation/6818398 slice down 6 steps from a to (aaaaaaa - baaaaaa)
OSM Info:http://www.openstreetmap.org/relation/9350856 slice down 6 steps from a to (aaaaaaa - baaaaaa)
OSM Info:http://www.openstreetmap.org/relation/253949 slice down 4 steps from bda to (bdaaaaa - bdbaaaa)
OSM Info:http://www.openstreetmap.org/relation/3987743 slice down 5 steps from ad to (adaaaaa - baaaaaa)
OSM Info:http://www.openstreetmap.org/relation/4039486 slice down 6 steps from b to (baaaaaa - caaaaaa)
OSM Info:http://www.openstreetmap.org/relation/5576397 slice down 4 steps from aca to (acaaaaa - acbaaaa)
OSM Info:http://www.openstreetmap.org/relation/6282022 slice down 4 steps from bda to (bdaaaaa - bdbaaaa)
OSM Info:http://www.openstreetmap.org/relation/8186637 slice down 4 steps from add to (addaaaa - baaaaaa)
OSM Info:http://www.openstreetmap.org/relation/207292 slice down 4 steps from bda to (bdaaaaa - bdbaaaa)
OSM Info:http://www.openstreetmap.org/relation/337014 slice down 4 steps from cbb to (cbbaaaa - cbcaaaa)
OSM Info:http://www.openstreetmap.org/relation/1536355 slice down 4 steps from abd to (abdaaaa - acaaaaa)
OSM Info:http://www.openstreetmap.org/relation/2687976 slice down 4 steps from baa to (baaaaaa - babaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823201 slice down 5 steps from dd to (ddaaaaa - aaaaaaa)
OSM Info:http://www.openstreetmap.org/relation/5491350 slice down 4 steps from bab to (babaaaa - bacaaaa)
OSM Info:http://www.openstreetmap.org/relation/166275 slice down 4 steps from acb to (acbaaaa - accaaaa)
OSM Info:http://www.openstreetmap.org/relation/1268072 slice down 4 steps from ada to (adaaaaa - adbaaaa)
OSM Info:http://www.openstreetmap.org/relation/1279614 slice down 6 steps from b to (baaaaaa - caaaaaa)
OSM Info:http://www.openstreetmap.org/relation/1952895 slice down 4 steps from bcb to (bcbaaaa - bccaaaa)
OSM Info:http://www.openstreetmap.org/relation/1964696 slice down 4 steps from acd to (acdaaaa - adaaaaa)
OSM Info:http://www.openstreetmap.org/relation/2532525 slice down 4 steps from abc to (abcaaaa - abdaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823670 slice down 4 steps from ccb to (ccbaaaa - cccaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823696 slice down 5 steps from dd to (ddaaaaa - aaaaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823719 slice down 6 steps from c to (caaaaaa - daaaaaa)
OSM Info:http://www.openstreetmap.org/relation/3623948 slice down 4 steps from acb to (acbaaaa - accaaaa)
OSM Info:http://www.openstreetmap.org/relation/9706230 slice down 4 steps from bcb to (bcbaaaa - bccaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823675 slice down 4 steps from dda to (ddaaaaa - ddbaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823708 slice down 4 steps from cdb to (cdbaaaa - cdcaaaa)
OSM Info:http://www.openstreetmap.org/relation/4555382 slice down 4 steps from bcb to (bcbaaaa - bccaaaa)
OSM Info:http://www.openstreetmap.org/relation/8864760 slice down 5 steps from ad to (adaaaaa - baaaaaa)
OSM Info:http://www.openstreetmap.org/relation/9845896 slice down 4 steps from dab to (dabaaaa - dacaaaa)
OSM Info:http://www.openstreetmap.org/relation/1760170 slice down 4 steps from ada to (adaaaaa - adbaaaa)
OSM Info:http://www.openstreetmap.org/relation/2740046 slice down 4 steps from bda to (bdaaaaa - bdbaaaa)
OSM Info:http://www.openstreetmap.org/relation/2791738 slice down 4 steps from bda to (bdaaaaa - bdbaaaa)
OSM Info:http://www.openstreetmap.org/relation/1834172 slice down 4 steps from bda to (bdaaaaa - bdbaaaa)
OSM Info:http://www.openstreetmap.org/relation/2823691 slice down 5 steps from cc to (ccaaaaa - cdaaaaa)
OSM Info:http://www.openstreetmap.org/relation/3004618 slice down 4 steps from dcb to (dcbaaaa - dccaaaa)
OSM Info:http://www.openstreetmap.org/relation/3137761 slice down 4 steps from cca to (ccaaaaa - ccbaaaa)
OSM Info:http://www.openstreetmap.org/relation/3304008 slice down 4 steps from bcb to (bcbaaaa - bccaaaa)
OSM Info:http://www.openstreetmap.org/relation/23227 slice down 4 steps from bca to (bcaaaaa - bcbaaaa)
OSM Info:http://www.openstreetmap.org/relation/2791372 slice down 6 steps from b to (baaaaaa - caaaaaa)
OSM Info:http://www.openstreetmap.org/relation/2859405 slice down 4 steps from abc to (abcaaaa - abdaaaa)
OSM Info:http://www.openstreetmap.org/relation/3817092 slice down 4 steps from dab to (dabaaaa - dacaaaa)
OSM Info:http://www.openstreetmap.org/relation/8590456 slice down 6 steps from a to (aaaaaaa - baaaaaa)
OSM Info:http://www.openstreetmap.org/relation/9466755 slice down 4 steps from acb to (acbaaaa - accaaaa)
OSM Info:http://www.openstreetmap.org/way/49001537 slice down 4 steps from aad to (aadaaaa - abaaaaa)
OSM Info:http://www.openstreetmap.org/way/219219881 slice down 4 steps from abc to (abcaaaa - abdaaaa)
OSM Info:http://www.openstreetmap.org/way/231394351 slice down 4 steps from abd to (abdaaaa - acaaaaa)
OSM Info:http://www.openstreetmap.org/way/321693450 slice down 4 steps from bcd to (bcdaaaa - bdaaaaa)
OSM Info:http://www.openstreetmap.org/way/372353383 slice down 4 steps from abb to (abbaaaa - abcaaaa)
OSM Info:http://www.openstreetmap.org/way/432066068 slice down 4 steps from bcc to (bccaaaa - bcdaaaa)
OSM Info:http://www.openstreetmap.org/way/436098551 slice down 5 steps from ca to (caaaaaa - cbaaaaa)
OSM Info:http://www.openstreetmap.org/way/436690530 slice down 5 steps from ca to (caaaaaa - cbaaaaa)
OSM Info:http://www.openstreetmap.org/way/447195251 slice down 4 steps from acb to (acbaaaa - accaaaa)
OSM Info:http://www.openstreetmap.org/way/544274312 slice down 5 steps from ca to (caaaaaa - cbaaaaa)
OSM Info:http://www.openstreetmap.org/way/571486923 slice down 4 steps from daa to (daaaaaa - dabaaaa)
OSM Info:http://www.openstreetmap.org/way/662627631 slice down 4 steps from adc to (adcaaaa - addaaaa)
OSM Info:http://www.openstreetmap.org/way/663224610 slice down 4 steps from aca to (acaaaaa - acbaaaa)
OSM Info:http://www.openstreetmap.org/way/680080430 slice down 4 steps from cbb to (cbbaaaa - cbcaaaa)

@metalstrolch
Copy link
Contributor Author

Will wait for result on #894 before proceeding.

@metalstrolch
Copy link
Contributor Author

Confirmed to do what it should. Preparing to merge.

@metalstrolch metalstrolch merged commit b99187e into trunk Oct 28, 2019
@metalstrolch metalstrolch deleted the tile_cutter branch October 28, 2019 12:28
hoehnp pushed a commit that referenced this pull request Nov 4, 2019
This commit allows to slice all polygons (except poly_water_sliced) into
squares so that they fit into smaller map tiles. Holes if present are
correctly sliced as well.

This has the advantage that they do not need to be processed albeit not
being displayed, and navit-planet-extractor can filter them out.

This has the disadvantage that the big polygons are displayed checkered
if the corresponding itemgra in config features an outline.

TODO: Slice already on polygon creation. Currently the slicing has to be
performed twice, because of tile preprocessing and saving is done in
seperate steps from the tempfiles.

TODO: Make the trigger for slicing configurable. Right now it slices
everything smaller than tile level 4.
@metalstrolch metalstrolch mentioned this pull request Aug 31, 2020
viktorgino pushed a commit that referenced this pull request Sep 22, 2020
This commit allows to slice all polygons (except poly_water_sliced) into
squares so that they fit into smaller map tiles. Holes if present are
correctly sliced as well.

This has the advantage that they do not need to be processed albeit not
being displayed, and navit-planet-extractor can filter them out.

This has the disadvantage that the big polygons are displayed checkered
if the corresponding itemgra in config features an outline.

TODO: Slice already on polygon creation. Currently the slicing has to be
performed twice, because of tile preprocessing and saving is done in
seperate steps from the tempfiles.

TODO: Make the trigger for slicing configurable. Right now it slices
everything smaller than tile level 4.
jkoan pushed a commit to jkoan/navit that referenced this pull request Jun 30, 2021
)

This commit allows to slice all polygons (except poly_water_sliced) into
squares so that they fit into smaller map tiles. Holes if present are
correctly sliced as well.

This has the advantage that they do not need to be processed albeit not
being displayed, and navit-planet-extractor can filter them out.

This has the disadvantage that the big polygons are displayed checkered
if the corresponding itemgra in config features an outline.

TODO: Slice already on polygon creation. Currently the slicing has to be
performed twice, because of tile preprocessing and saving is done in
seperate steps from the tempfiles.

TODO: Make the trigger for slicing configurable. Right now it slices
everything smaller than tile level 4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Maps from planetextractor render slow on weak devices.
2 participants