Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

turn Method.specializations into a simpler table+list like the TypeName.cache #35099

Merged
merged 3 commits into from
Mar 30, 2020

Conversation

vtjnash
Copy link
Member

@vtjnash vtjnash commented Mar 13, 2020

Some numbers:

Not a huge affect on speed, afaik, but pretty good on system image saving space:
126 MB => 123 MB measured

30_086 methods in the system (Base + stdlib)

the tuple pair here is (count of hash table, count of linear list)
mean number of specializations: (1.70, 0.39)
total number of specializations: (51_284, 11_743) => 81% hashable
max number of specializations: (1_166, 304)

julia> histogram(filter(!iszero, r), bins=40, xscale=log10)
                    ┌                                        ┐ 
   [   0.0,   50.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 9492   
   [  50.0,  100.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 85                       
   [ 100.0,  150.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇ 30                          
   [ 150.0,  200.0) ┤▇▇▇▇▇▇▇▇▇▇▇ 17                            
   [ 200.0,  250.0) ┤▇▇▇▇▇▇▇ 6                                 
   [ 250.0,  300.0) ┤▇▇▇▇▇▇▇ 6                                 
   [ 300.0,  350.0) ┤▇▇▇▇▇ 4                                   
   [ 350.0,  400.0) ┤▇▇▇▇ 3                                    
   [ 400.0,  450.0) ┤ 0                                        
   [ 450.0,  500.0) ┤ 0                                        
   [ 500.0,  550.0) ┤▇▇▇ 2                                     
   [ 550.0,  600.0) ┤ 0                                        
   [ 600.0,  650.0) ┤ 1                                        
   [ 650.0,  700.0) ┤ 0                                        
   [ 700.0,  750.0) ┤ 0                                        
   [ 750.0,  800.0) ┤ 1                                        
   [ 800.0,  850.0) ┤ 0                                        
   [ 850.0,  900.0) ┤ 1                                        
   [ 900.0,  950.0) ┤ 0                                        
   [ 950.0, 1000.0) ┤ 1                                        
   [1000.0, 1050.0) ┤ 0                                        
   [1050.0, 1100.0) ┤ 0                                        
   [1100.0, 1150.0) ┤ 0                                        
   [1150.0, 1200.0) ┤ 1                                        
                    └                                        ┘ 
                                Frequency [log10]
julia> histogram(filter(!iszero, l), bins=10, xscale=log10)
                  ┌                                        ┐ 
   [  0.0,  50.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 2349   
   [ 50.0, 100.0) ┤▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 22                         
   [100.0, 150.0) ┤▇▇▇▇▇▇▇▇ 6                                
   [150.0, 200.0) ┤▇▇▇ 2                                     
   [200.0, 250.0) ┤▇▇▇ 2                                     
   [250.0, 300.0) ┤ 0                                        
   [300.0, 350.0) ┤ 1                                        
                  └                                        ┘ 
                              Frequency [log10]

Most frequently specialized methods in table (over 100), with counts:

102 push!(a::Array{T,1}, item) where T in Base at array.jl:911
102 throw_boundserror(A, I) in Base at abstractarray.jl:537
105 rehash!(h::Dict{K,V}, newsz) where {K, V} in Base at dict.jl:175
106 getindex(t::Tuple, i::Int64) in Core.Compiler at tuple.jl:24
106 promote_type(::Type{T}, ::Type{T}) where T in Base at promotion.jl:212
107 (::Type{LinearIndices})(A::Union{Core.SimpleVector, AbstractArray}) in Base at indices.jl:453
107 isslotempty(h::Dict, i::Int64) in Base at dict.jl:170
107 isslotmissing(h::Dict, i::Int64) in Base at dict.jl:172
108 _array_for(::Type{T}, itr, ::Base.HasShape{N}) where {T, N} in Base at array.jl:656
108 isslotfilled(h::Dict, i::Int64) in Base at dict.jl:171
108 promote_type(::Type{T}, ::Type{Union{}}) where T in Base at promotion.jl:213
108 resize!(a::Array{T,1} where T, nl::Integer) in Base at array.jl:1060
110 !=(x, y) in Base at operators.jl:193
112 _deleteend!(a::Array{T,1} where T, delta::Integer) in Base at array.jl:878
115 (::Type{Dict{K,V}})() where {K, V} in Base at dict.jl:89
118 (::Type{Array{T,1}})(::UndefInitializer, d::Tuple{Int64}) where T in Core at boot.jl:414
125 #sprint#352(context, sizehint::Integer, ::typeof(sprint), f::Function, args...) in Base at strings/io.jl:101
125 eachindex(::IndexLinear, A::AbstractArray{T,1} where T) in Base at abstractarray.jl:267
125 eltype(x) in Base at array.jl:125
126 collect_to!(dest::AbstractArray{T,N} where N, itr, offs, st) where T in Base at array.jl:707
129 (::Type{Base.Broadcast.Broadcasted{Style,Axes,F,Args} where Args<:Tuple where F where Axes})(f::F, args::Args, axes) where {Style, F, Args<:Tuple} in Base.Broadcast at broadcast.jl:179
132 (::Type{Base.Broadcast.Broadcasted{Style,Axes,F,Args}})(f, args, axes) where {Style<:Union{Nothing, Base.Broadcast.BroadcastStyle}, Axes, F, Args<:Tuple} in Base.Broadcast at broadcast.jl:170
132 collect_to_with_first!(dest::AbstractArray, v1, itr, st) in Base at array.jl:686
132 ht_keyindex2!(h::Dict{K,V}, key) where {K, V} in Base at dict.jl:304
133 convert(::Type{Any}, x) in Core at boot.jl:393
137 first(itr) in Base at abstractarray.jl:341
140 axes1(A::AbstractArray) in Base at abstractarray.jl:95
142 ht_keyindex(h::Dict{K,V}, key) where {K, V} in Base at dict.jl:279
146 _growend!(a::Array{T,1} where T, delta::Integer) in Base at array.jl:869
146 isequal(x, y) in Base at operators.jl:123
152 Base.IteratorSize(x) in Base at generator.jl:90
152 convert(::Type{T}, a::AbstractArray) where T<:Array in Base at array.jl:532
155 size(a::Array{T,1} where T) in Base at array.jl:155
160 promote_rule(::Type{var"#s69"} where var"#s69", ::Type{var"#s68"} where var"#s68") in Base at promotion.jl:235
172 convert(::Type{T}, x::T) where T<:Tuple{Any,Vararg{Any,N} where N} in Base at essentials.jl:309
174 convert(::Type{Any}, x) in Base at essentials.jl:170
175 axes(A) in Base at abstractarray.jl:74
179 print(io::IO, xs...) in Base at strings/io.jl:43
180 (::Type{Pair})(a, b) in Base at pair.jl:15
180 BoundsError(a, i) in Core at boot.jl:244
191 ==(x, y) in Base at operators.jl:83
192 getindex(A::Array, i1::Int64) in Base at array.jl:786
193 _setindex!(h::Dict, v, key, index) in Base at dict.jl:354
193 length(a::Array) in Base at array.jl:221
193 promote_result(::Type{var"#s69"} where var"#s69", ::Type{var"#s68"} where var"#s68", ::Type{T}, ::Type{S}) where {T, S} in Base at promotion.jl:237
198 (::Type{Base.Generator{I,F}})(f, iter) where {I, F} in Base at generator.jl:32
198 getproperty(x::Type, f::Symbol) in Base at Base.jl:28
200 tail(x::Tuple) in Base at essentials.jl:205
207 setindex!(h::Dict{K,V}, v0, key::K) where {K, V} in Base at dict.jl:380
208 (::Type{Base.Generator})(f::F, iter::I) where {I, F} in Base at generator.jl:32
213 argtail(x, rest...) in Base at essentials.jl:189
234 (::Type{Array{T,1}})(::UndefInitializer, m::Int64) where T in Core at boot.jl:405
242 promote_type(::Type{T}, ::Type{S}) where {T, S} in Base at promotion.jl:217
276 isempty(x::Tuple) in Base at tuple.jl:383
283 indexed_iterate(t::Tuple, i::Int64) in Base at tuple.jl:81
283 indexed_iterate(t::Tuple, i::Int64, state) in Base at tuple.jl:81
290 getindex(t::NamedTuple, i::Symbol) in Base at namedtuple.jl:97
295 haskey(nt::NamedTuple, key::Union{Integer, Symbol}) in Base at namedtuple.jl:272
299 string(xs...) in Base at strings/io.jl:174
300 print_to_string(xs...) in Base at strings/io.jl:125
301 (::Type{NamedTuple{names,T}})(args::T) where {names, T<:Tuple} in Core at boot.jl:550
321 structdiff(a::NamedTuple{an,T} where T<:Tuple, b::Union{Type{NamedTuple{bn,T} where T<:Tuple}, NamedTuple{bn,T} where T<:Tuple}) where {an, bn} in Base at namedtuple.jl:294
342 (::Type{NamedTuple{names,T} where T<:Tuple})(args::Tuple) where names in Core at boot.jl:546
354 iterate(g::Base.Generator, s...) in Base at generator.jl:43
355 setindex!(A::Array{T,N} where N, x, i1::Int64) where T in Base at array.jl:824
387 (::Type{Pair{A,B}})(a, b) where {A, B} in Base at pair.jl:7
543 iterate(t::Tuple) in Base at tuple.jl:60
543 iterate(t::Tuple, i::Int64) in Base at tuple.jl:60
632 length(t::Tuple) in Base at tuple.jl:19
779 convert(::Type{T}, x::T) where T in Base at essentials.jl:171
859 setproperty!(x, f::Symbol, v) in Base at Base.jl:34
973 getindex(t::Tuple, i::Int64) in Base at tuple.jl:24
1166 getproperty(x, f::Symbol) in Base at Base.jl:33

Most frequently specialized methods in linear list (over 50), with counts:

50 (::Type{T})(x::Tuple) where T<:Tuple in Base at tuple.jl:225
52 (::Type{Base.Broadcast.Broadcasted{Style,Axes,F,Args}})(f, args, axes) where {Style<:Union{Nothing, Base.Broadcast.BroadcastStyle}, Axes, F, Args<:Tuple} in Base.Broadcast at broadcast.jl:170
53 (::Type{Base.Generator})(f::F, iter::I) where {I, F} in Base at generator.jl:32
54 setindex!(h::Dict{K,V}, v0, key::K) where {K, V} in Base at dict.jl:380
59 collect_to_with_first!(dest::AbstractArray, v1, itr, st) in Base at array.jl:686
62 isempty(x::Tuple) in Base at tuple.jl:383
63 argtail(x, rest...) in Base at essentials.jl:189
63 collect_to!(dest::AbstractArray{T,N} where N, itr, offs, st) where T in Base at array.jl:707
66 structdiff(a::NamedTuple{an,T} where T<:Tuple, b::Union{Type{NamedTuple{bn,T} where T<:Tuple}, NamedTuple{bn,T} where T<:Tuple}) where {an, bn} in Base at namedtuple.jl:294
67 tail(x::Tuple) in Base at essentials.jl:205
69 (::Type{Base.Generator{I,F}})(f, iter) where {I, F} in Base at generator.jl:32
70 checkbounds(A::AbstractArray, I...) in Base at abstractarray.jl:501
70 throw_boundserror(A, I) in Base at abstractarray.jl:537
74 (::Type{NamedTuple{names,T}})(args::T) where {names, T<:Tuple} in Core at boot.jl:550
74 (::Type{NamedTuple{names,T}})(args::Tuple) where {names, T<:Tuple} in Base at namedtuple.jl:69
75 iterate(g::Base.Generator, s...) in Base at generator.jl:43
77 to_indices(A, inds, I::Tuple{Any,Vararg{Any,N} where N}) in Base at indices.jl:324
79 BoundsError(a, i) in Core at boot.jl:244
80 string(xs...) in Base at strings/io.jl:174
84 print(io::IO, xs...) in Base at strings/io.jl:43
86 setindex!(A::Array{T,N} where N, x, i1::Int64) where T in Base at array.jl:824
92 print_to_string(xs...) in Base at strings/io.jl:125
106 promote_rule(::Type{var"#s69"} where var"#s69", ::Type{var"#s68"} where var"#s68") in Base at promotion.jl:235
115 promote_result(::Type{T}, ::Type{S}, ::Type{Union{}}, ::Type{Union{}}) where {T, S} in Base at promotion.jl:240
118 promote_type(::Type{T}, ::Type{S}) where {T, S} in Base at promotion.jl:217
118 setproperty!(x, f::Symbol, v) in Base at Base.jl:34
137 promote_result(::Type{var"#s69"} where var"#s69", ::Type{var"#s68"} where var"#s68", ::Type{T}, ::Type{S}) where {T, S} in Base at promotion.jl:237
144 promote_typeof(x, xs...) in Base at promotion.jl:262
182 iterate(t::Tuple) in Base at tuple.jl:60
182 iterate(t::Tuple, i::Int64) in Base at tuple.jl:60
211 getproperty(x, f::Symbol) in Base at Base.jl:33
236 length(t::Tuple) in Base at tuple.jl:19
304 getindex(t::Tuple, i::Int64) in Base at tuple.jl:24

via:

julia> Base.visit(f, @nospecialize x) = nothing
julia> function Base.visit(f, m::Module)
                         for x in names(m, all=true)
                           isdefined(m, x) || continue
                           x = Base.unwrap_unionall(getfield(m, x))
                           x isa Module && x !== m && parentmodule(x) === m && Base.visit(f, x)
                           x isa DataType  || continue
                           x.name.module === m || continue
                           (!isdefined(x.name, :mt) || x.name.mt === DataType.name.mt || x.name.mt === Symbol.name.mt) && continue
                           Base.visit(f, x.name.mt.defs)
                         end
                         m === Core && Base.visit(f, DataType.name.mt.defs)
                         m === Core && Base.visit(f, Symbol.name.mt.defs)
                     end
julia> ms = [:Base64, :CRC32c, :Dates, :DelimitedFiles, :Distributed, :FileWatching, :Future, :InteractiveUtils, :LibGit2, :Libdl, :LinearAlgebra, :Logging, :Markdown, :Mmap, :Pkg, :Printf, :Profile, :REPL, :Random, :SHA, :Serialization, :SharedArrays, :Sockets, :SparseArrays, :Statistics, :SuiteSparse, :Test, :UUIDs, :Unicode, :Pkg, :Base, :Core]
julia> for m in ms
              @eval import $m
              Base.visit(getfield(Main, m)) do l::Method
       a = sum(i -> isassigned(l.specializations, i), 0:length(l.specializations))
       b = sum(i -> isassigned(l.linearspecializations, i), 0:length(l.linearspecializations))
       println(a, '\t', b)
                nothing
              end
              end

@vtjnash vtjnash requested a review from JeffBezanson March 13, 2020 04:19
@vtjnash vtjnash force-pushed the jn/method-specs-list branch 2 times, most recently from 05686c7 to e583e4e Compare March 13, 2020 15:39
@vtjnash vtjnash force-pushed the jn/method-specs-list branch 7 times, most recently from be8c8f5 to 3149a58 Compare March 27, 2020 17:34
vtjnash added 3 commits March 30, 2020 09:49
Type{Union[}} == typeof(Union{}) is an unusual special case
The Type objects are not normalized by insertion into the cache, unlike
all other types, so we need to be careful not to assign them a stable
hash (which pre-supposes they will be hash-normalized).
@vtjnash vtjnash force-pushed the jn/method-specs-list branch from 3149a58 to 0241c5f Compare March 30, 2020 13:51
@vtjnash vtjnash merged commit 58034ad into master Mar 30, 2020
@vtjnash vtjnash deleted the jn/method-specs-list branch March 30, 2020 17:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants