Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkgimages: storage of link_ids should be compressed #48218

Closed
timholy opened this issue Jan 10, 2023 · 0 comments · Fixed by #48673
Closed

pkgimages: storage of link_ids should be compressed #48218

timholy opened this issue Jan 10, 2023 · 0 comments · Fixed by #48673
Assignees
Labels

Comments

@timholy
Copy link
Member

timholy commented Jan 10, 2023

Towards the end of writing out pkgimages, we write a section of link_ids:

julia/src/staticdata.c

Lines 2477 to 2484 in 1e5fdb2

write_uint32(f, jl_array_len(s.link_ids_gctags));
ios_write(f, (char*)jl_array_data(s.link_ids_gctags), jl_array_len(s.link_ids_gctags)*sizeof(uint64_t));
write_uint32(f, jl_array_len(s.link_ids_relocs));
ios_write(f, (char*)jl_array_data(s.link_ids_relocs), jl_array_len(s.link_ids_relocs)*sizeof(uint64_t));
write_uint32(f, jl_array_len(s.link_ids_gvars));
ios_write(f, (char*)jl_array_data(s.link_ids_gvars), jl_array_len(s.link_ids_gvars)*sizeof(uint64_t));
write_uint32(f, jl_array_len(s.link_ids_external_fnvars));
ios_write(f, (char*)jl_array_data(s.link_ids_external_fnvars), jl_array_len(s.link_ids_external_fnvars)*sizeof(uint64_t));

These store the buildid of the module for an external linkage; they act as an invariant name for a module we depend on, and are used for pointer relocation when we deserialize.

It turns out these are quite repetitious, and the large majority of such linkages are against the sysimage with buildid 0. Consequently we should be able to store them with some form of compression. This will only modestly shrink the size of pkgimages (maybe 5%ish), but it seems worth having.

@timholy timholy self-assigned this Jan 10, 2023
timholy added a commit that referenced this issue Jan 14, 2023
This reduces the size of our precompile cache files, using
run-length encoding (RLE) to represent the module of external
linkages. Most linkages seem to be against the sysimg itself,
and RLE allows long stretches of such linkages to be encoded
compactly.

Closes #48218
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants