Go 1.21 update list

[roman-khimov]

It's an experimental format, usually I've created an issue for specific language update related topic, but we have huge number of Go repositories and replicating these issues into all of them isn't practical, replicating one issue is more feasible. And most of the time they're solved all at once.

So, when we're to drop support for 1.20, the following things should be done:

• use min()/max() where appropriate
  It's not easy to identify all cases where it's applicable, but there are likely some and we can search a bit for them. We also have some time before 1.20 is out, so leave comments if you notice something in the code
• use clear() to reset maps
  We certainly have cases where a new map is created and old one is forgotten, because dropping elements one by one is no fun. Now we can clear() and reuse old maps which is supposed to be more efficient. Also add a comment if there are some known places in the code for it.
• replace slice.Clean() with clear()
• replace slice.Reverse() with slices.Reverse()
• drop util/slice completely
• use slices.BinarySearch() and slices.BinarySearchFunc()
  We have a number of places with sort.Search(), likely they can be improved
• use slices.Compact() where applicable
  PublicKeys.Unique() or other deduplication cases.
• use slices.Sort() instead of sort.Slice where applicable
• use other slices functions where applicable
• use maps package where applicable
  This doesn't seem to be as useful as slices, but some use cases can probably be found
• use bytes.AvailableBuffer()
  Some serialization code likely can be improved with it.
• check that we don't use any deprecated crypto/elliptic things
• consider using errors.ErrUnsupported where applicable
• use sync.OnceFunc (we have cases like that) and OnceValue(s) (maybe)

[roman-khimov]

clear() -> (mp *Pool) RemoveStale

[roman-khimov]

clear() -> (t *Trie) Collapse

[roman-khimov]

slices.BinarySearch is limited to https://pkg.go.dev/cmp#Ordered, likely not very useful for us now.

[carpawell]

BTW, some new built-in funcs may have their experimental analogues like golang.org/x/exp/maps -> clear/maps.clear. I used some of them while was not able to use not released yet funcs. Just a reminder for an ansigneee.

[AnnaShaleva]

We need to update github.com/consensys/gnark from 0.9.1 to 0.10.0 along with this change since v0.10.0 requires at least Go 1.21.

[roman-khimov]

FYI,

 func BenchmarkSlicez(b *testing.B) {
         arr := slices.Repeat([]byte{1, 2, 3, 4}, 32)
         for i := 0; i < b.N; i++ {
-                _ = append([]byte{1, 2, 3}, arr...)
+                _ = slices.Concat([]byte{1, 2, 3}, arr)
         }
 }

yields:

cpu: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics  
          │ append-concat.old │          append-concat.new          │
          │      sec/op       │   sec/op     vs base                │
Slicez-16         47.89n ± 1%   60.92n ± 2%  +27.21% (p=0.000 n=10)

[roman-khimov]

type twoInts struct {
        a int
        b int
}

func BenchmarkSortStructs(b *testing.B) {
        var (
                random = rand.New(rand.NewPCG(seed1, seed2)) // static seeds for reproducible results                                                                                                                                                                                                                                                                                        
                arr    = make([]twoInts, numElements)
                arrcp  = slices.Clone(arr)
        )
        for i := range arr {
                arr[i].a = i
        }
        random.Shuffle(len(arr), func(i, j int) { arr[i], arr[j] = arr[j], arr[i] })
        b.Run("sort.Slice", func(b *testing.B) {
                for range b.N {
                        b.StopTimer()
                        copy(arrcp, arr)
                        b.StartTimer()
                        sort.Slice(arrcp, func(i, j int) bool {
                                return arrcp[i].a < arrcp[j].a
                        })
                }
        })
        b.Run("slices.SortFunc", func(b *testing.B) {
                for range b.N {
                        b.StopTimer()
                        copy(arrcp, arr)
                        b.StartTimer()
                        slices.SortFunc(arrcp, func(a, b twoInts) int {
                                return a.a - b.a
                        })
                }
        })
}

func BenchmarkSortStructPointers(b *testing.B) {
        var (
                random = rand.New(rand.NewPCG(seed1, seed2)) // static seeds for reproducible results                                                                                                                                                                                                                                                                                        
                arr    = make([]*twoInts, numElements)
                arrcp  = slices.Clone(arr)
        )
        for i := range arr {
                arr[i] = &twoInts{a: i}
        }
        random.Shuffle(len(arr), func(i, j int) { arr[i], arr[j] = arr[j], arr[i] })
        b.Run("sort.Slice", func(b *testing.B) {
                for range b.N {
                        b.StopTimer()
                        copy(arrcp, arr)
                        b.StartTimer()
                        sort.Slice(arrcp, func(i, j int) bool {
                                return arrcp[i].a < arrcp[j].a
                        })
                }
        })
        b.Run("slices.SortFunc", func(b *testing.B) {
                for range b.N {
                        b.StopTimer()
                        copy(arrcp, arr)
                        b.StartTimer()
                        slices.SortFunc(arrcp, func(a, b *twoInts) int {
                                return a.a - b.a
                        })
                }
        })
}

For 100k:

goos: linux
goarch: amd64
pkg: github.com/nspcc-dev/neo-go/config
cpu: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics  
                                      │ sort.structs │
                                      │    sec/op    │
SortStructs/sort.Slice-16                12.44m ± 1%
SortStructs/slices.SortFunc-16           6.799m ± 2%
SortStructPointers/sort.Slice-16         9.314m ± 2%
SortStructPointers/slices.SortFunc-16    7.581m ± 3%
geomean                                  8.790m

                                      │ sort.structs │
                                      │     B/op     │
SortStructs/sort.Slice-16               88.00 ± 0%
SortStructs/slices.SortFunc-16          0.000 ± 0%
SortStructPointers/sort.Slice-16        56.00 ± 0%
SortStructPointers/slices.SortFunc-16   0.000 ± 0%
geomean                                            ¹
¹ summaries must be >0 to compute geomean

                                      │ sort.structs │
                                      │  allocs/op   │
SortStructs/sort.Slice-16               3.000 ± 0%
SortStructs/slices.SortFunc-16          0.000 ± 0%
SortStructPointers/sort.Slice-16        2.000 ± 0%
SortStructPointers/slices.SortFunc-16   0.000 ± 0%
geomean                                            ¹
¹ summaries must be >0 to compute geomean

[roman-khimov]

func BenchmarkBinarySearchSimple(b *testing.B) {
        var (
                random = rand.New(rand.NewPCG(seed1, seed2)) // static seeds for reproducible results                                                                                                                                                                                                                                                                                        
                arr    = make([]int, numElements)
                tgts   = random.Perm(numElements)
        )
        for i := range arr {
                arr[i] = i
        }
        b.Run("sort.Search", func(b *testing.B) {
                for range b.N {
                        for _, tgt := range tgts {
                                sort.Search(len(arr), func(i int) bool {
                                        return arr[i] >= tgt
                                })
                        }
                }
        })
        b.Run("slices.BinarySearch", func(b *testing.B) {
                for range b.N {
                        for _, tgt := range tgts {
                                slices.BinarySearch(arr, tgt)
                        }
                }
        })
}

func BenchmarkBinarySearchFunc(b *testing.B) {
        var (
                random = rand.New(rand.NewPCG(seed1, seed2)) // static seeds for reproducible results                                                                                                                                                                                                                                                                                        
                arr    = make([]twoInts, numElements)
                tgts   = make([]twoInts, numElements)
        )
        for i := range arr {
                arr[i].a = i
        }
        copy(tgts, arr)
        random.Shuffle(len(tgts), func(i, j int) { arr[i], arr[j] = arr[j], arr[i] })
        b.Run("sort.Search", func(b *testing.B) {
                for range b.N {
                        for _, tgt := range tgts {
                                sort.Search(len(arr), func(i int) bool {
                                        return arr[i].a >= tgt.a
                                })
                        }
                }
        })
        b.Run("slices.BinarySearchFunc", func(b *testing.B) {
                for range b.N {
                        for _, tgt := range tgts {
                                slices.BinarySearchFunc(arr, tgt, func(e, t twoInts) int {
                                        return e.a - t.a
                                })
                        }
                }
        })
}

Yields this:

goos: linux
goarch: amd64
pkg: github.com/nspcc-dev/neo-go/config
cpu: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics  
                                            │   binary    │
                                            │   sec/op    │
BinarySearchSimple/sort.Search-16             8.516m ± 1%
BinarySearchSimple/slices.BinarySearch-16     5.947m ± 1%
BinarySearchFunc/sort.Search-16               2.689m ± 4%
BinarySearchFunc/slices.BinarySearchFunc-16   2.621m ± 3%
geomean                                       4.347m

                                            │    binary    │
                                            │     B/op     │
BinarySearchSimple/sort.Search-16             0.000 ± 0%
BinarySearchSimple/slices.BinarySearch-16     0.000 ± 0%
BinarySearchFunc/sort.Search-16               0.000 ± 0%
BinarySearchFunc/slices.BinarySearchFunc-16   0.000 ± 0%
geomean                                                  ¹
¹ summaries must be >0 to compute geomean

                                            │    binary    │
                                            │  allocs/op   │
BinarySearchSimple/sort.Search-16             0.000 ± 0%
BinarySearchSimple/slices.BinarySearch-16     0.000 ± 0%
BinarySearchFunc/sort.Search-16               0.000 ± 0%
BinarySearchFunc/slices.BinarySearchFunc-16   0.000 ± 0%
geomean                                                  ¹
¹ summaries must be >0 to compute geomean