Skip to content

Commit

Permalink
Update benchmark results (#148)
Browse files Browse the repository at this point in the history
  • Loading branch information
@sebastienros
sebastienros authored Aug 17, 2024
1 parent 887b4c1 commit acff73d
Showing 1 changed file with 67 additions and 64 deletions.
131 changes: 67 additions & 64 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -110,72 +110,75 @@ In this benchmark, Parlot Fluent is more than 10 times faster than Pidgin, and P
When compiled, the Parlot grammar shows even better results, without losing its simplicity.

```
BenchmarkDotNet v0.13.10, Windows 11 (10.0.22631.3155/23H2/2023Update/SunValley3)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.200
[Host] : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
ShortRun : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
BenchmarkDotNet v0.14.0, Debian GNU/Linux 12 (bookworm) (container)
Intel Xeon E-2336 CPU 2.90GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.303
[Host] : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
ShortRun : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|-------------------- |------------:|------------:|----------:|------:|--------:|-------:|----------:|------------:|
| ParlotRawSmall | 176.1 ns | 41.63 ns | 2.28 ns | 0.60 | 0.00 | 0.0322 | 304 B | 0.46 |
| ParlotCompiledSmall | 292.4 ns | 43.69 ns | 2.39 ns | 1.00 | 0.00 | 0.0696 | 656 B | 1.00 |
| ParlotFluentSmall | 360.3 ns | 44.18 ns | 2.42 ns | 1.23 | 0.01 | 0.0696 | 656 B | 1.00 |
| PidginSmall | 5,237.3 ns | 2,162.97 ns | 118.56 ns | 17.91 | 0.33 | 0.0839 | 832 B | 1.27 |
| | | | | | | | | |
| ParlotRawBig | 846.2 ns | 106.47 ns | 5.84 ns | 0.55 | 0.00 | 0.1268 | 1200 B | 0.42 |
| ParlotCompiledBig | 1,551.9 ns | 274.97 ns | 15.07 ns | 1.00 | 0.00 | 0.3052 | 2888 B | 1.00 |
| ParlotFluentBig | 1,921.7 ns | 116.32 ns | 6.38 ns | 1.24 | 0.01 | 0.3052 | 2888 B | 1.00 |
| PidginBig | 26,802.7 ns | 6,253.70 ns | 342.79 ns | 17.27 | 0.09 | 0.4272 | 4152 B | 1.44 |
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Gen1 | Allocated | Alloc Ratio |
|--------------------------------|------------------|----------------|---------------|-------|---------|----------|----------|-----------|-------------|
| ParlotRawSmall | 245.526 ns | 9.8448 ns | 0.5396 ns | 0.45 | 0.00 | 0.0482 | - | 304 B | 0.44 |
| ParlotCompiledSmall | 541.858 ns | 12.6762 ns | 0.6948 ns | 1.00 | 0.00 | 0.1097 | - | 688 B | 1.00 |
| ParlotFluentSmall | 631.942 ns | 16.6197 ns | 0.9110 ns | 1.17 | 0.00 | 0.1097 | - | 688 B | 1.00 |
| PidginSmall | 6,232.913 ns | 455.9577 ns | 24.9926 ns | 11.50 | 0.04 | 0.1297 | - | 832 B | 1.21 |
| | | | | | | | | | |
| ParlotRawBig | 1,253.877 ns | 96.9166 ns | 5.3123 ns | 0.41 | 0.00 | 0.1907 | - | 1200 B | 0.39 |
| ParlotCompiledBig | 3,084.008 ns | 94.1393 ns | 5.1601 ns | 1.00 | 0.00 | 0.4883 | - | 3080 B | 1.00 |
| ParlotFluentBig | 3,447.370 ns | 93.6097 ns | 5.1311 ns | 1.12 | 0.00 | 0.4883 | - | 3080 B | 1.00 |
| PidginBig | 31,896.523 ns | 1,898.4107 ns | 104.0583 ns | 10.34 | 0.03 | 0.6104 | - | 4152 B | 1.35 |
```

### JSON Benchmarks

This benchmark was taken from the Pidgin repository and demonstrates how to perform simple JSON document parsing. It exercises the parsers with different kinds of documents. Pidgin, Sprache, Superpower and Parlot are compared. The programming models are all based on parser combinator.
For reference, Newtonsoft.Json is also added to show the differences with a dedicated parser.
The results show that Sprache and Superpower are the slowest and most allocating ones. Parlot provides the best performance in all scenarios, being at least 2 times faster than the second fastest. The allocations of Parlot are also better or equivalent to the ones of Pidgin. This simple implementation is also faster than Newtonsoft, though it is far from being as rigorous.
The results show that Sprache and Superpower are the slowest and most allocating ones. Parlot provides the best performance in all scenarios, being 2 times faster than the second fastest. The allocations of Parlot are also better or equivalent to the ones of Pidgin. This simple implementation is also faster than Newtonsoft, though it is far from being as rigorous. The best JSON parser is by far System.Text.Json, don't build your own!

```
BenchmarkDotNet v0.13.10, Windows 11 (10.0.22631.3155/23H2/2023Update/SunValley3)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.200
[Host] : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
ShortRun : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
BenchmarkDotNet v0.14.0, Debian GNU/Linux 12 (bookworm) (container)
Intel Xeon E-2336 CPU 2.90GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.303
[Host] : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
ShortRun : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Gen1 | Allocated | Alloc Ratio |
|------------------------ |------------:|-----------:|----------:|------:|--------:|---------:|---------:|-----------:|------------:|
| BigJson_ParlotCompiled | 49.79 us | 16.541 us | 0.907 us | 1.00 | 0.00 | 9.9487 | 1.7700 | 91.79 KB | 1.00 |
| BigJson_Parlot | 52.81 us | 7.892 us | 0.433 us | 1.06 | 0.03 | 9.9487 | 1.3428 | 91.79 KB | 1.00 |
| BigJson_Pidgin | 124.43 us | 15.320 us | 0.840 us | 2.50 | 0.03 | 9.7656 | 1.7090 | 91.7 KB | 1.00 |
| BigJson_Newtonsoft | 82.05 us | 18.702 us | 1.025 us | 1.65 | 0.05 | 22.0947 | 5.8594 | 203.1 KB | 2.21 |
| BigJson_Sprache | 1,137.37 us | 259.708 us | 14.235 us | 22.84 | 0.14 | 572.2656 | 113.2813 | 5271.8 KB | 57.43 |
| BigJson_Superpower | 810.11 us | 185.500 us | 10.168 us | 16.27 | 0.16 | 97.6563 | 13.6719 | 905.93 KB | 9.87 |
| | | | | | | | | | |
| DeepJson_ParlotCompiled | 29.51 us | 2.907 us | 0.159 us | 1.00 | 0.00 | 10.6812 | 2.0142 | 98.33 KB | 1.00 |
| DeepJson_Parlot | 34.88 us | 8.163 us | 0.447 us | 1.18 | 0.02 | 10.6812 | 1.1597 | 98.33 KB | 1.00 |
| DeepJson_Pidgin | 233.67 us | 614.276 us | 33.671 us | 7.92 | 1.15 | 10.7422 | 2.1973 | 98.79 KB | 1.00 |
| DeepJson_Newtonsoft | 49.00 us | 22.889 us | 1.255 us | 1.66 | 0.03 | 19.4702 | 5.7373 | 179.13 KB | 1.82 |
| DeepJson_Sprache | 978.92 us | 667.766 us | 36.603 us | 33.17 | 1.07 | 316.4063 | 110.3516 | 2914.45 KB | 29.64 |
| | | | | | | | | | |
| LongJson_ParlotCompiled | 43.54 us | 75.614 us | 4.145 us | 1.00 | 0.00 | 13.0615 | 2.6245 | 120.34 KB | 1.00 |
| LongJson_Parlot | 51.76 us | 27.313 us | 1.497 us | 1.19 | 0.08 | 13.0615 | 2.6245 | 120.34 KB | 1.00 |
| LongJson_Pidgin | 135.62 us | 263.370 us | 14.436 us | 3.15 | 0.60 | 13.0615 | 2.5635 | 120.25 KB | 1.00 |
| LongJson_Newtonsoft | 77.94 us | 48.599 us | 2.664 us | 1.80 | 0.13 | 21.9727 | 7.2021 | 202.68 KB | 1.68 |
| LongJson_Sprache | 1,215.53 us | 427.688 us | 23.443 us | 28.06 | 2.31 | 462.8906 | 97.6563 | 4261.31 KB | 35.41 |
| LongJson_Superpower | 681.67 us | 996.045 us | 54.597 us | 15.80 | 2.52 | 78.1250 | 15.6250 | 726.79 KB | 6.04 |
| | | | | | | | | | |
| WideJson_ParlotCompiled | 27.79 us | 21.605 us | 1.184 us | 1.00 | 0.00 | 4.3945 | 0.4883 | 40.56 KB | 1.00 |
| WideJson_Parlot | 27.09 us | 19.498 us | 1.069 us | 0.98 | 0.02 | 4.3945 | 0.3967 | 40.56 KB | 1.00 |
| WideJson_Pidgin | 59.53 us | 31.875 us | 1.747 us | 2.14 | 0.05 | 4.3945 | 0.3662 | 40.48 KB | 1.00 |
| WideJson_Newtonsoft | 53.96 us | 5.562 us | 0.305 us | 1.94 | 0.09 | 11.5967 | 2.5635 | 106.72 KB | 2.63 |
| WideJson_Sprache | 588.36 us | 467.479 us | 25.624 us | 21.21 | 1.45 | 300.7813 | 38.0859 | 2766.87 KB | 68.21 |
| WideJson_Superpower | 456.39 us | 219.195 us | 12.015 us | 16.44 | 0.74 | 48.8281 | 4.3945 | 451.81 KB | 11.14 |
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Gen1 | Allocated | Alloc Ratio |
|--------------------------------|------------------|----------------|---------------|-------|---------|----------|----------|-----------|-------------|
| BigJson_ParlotCompiled | 99,206.961 ns | 425.9311 ns | 23.3467 ns | 1.00 | 0.00 | 14.8926 | 2.3193 | 93936 B | 1.00 |
| BigJson_Parlot | 97,746.352 ns | 1,682.2353 ns | 92.2090 ns | 0.99 | 0.00 | 14.8926 | 2.3193 | 93936 B | 1.00 |
| BigJson_Pidgin | 176,281.977 ns | 43,607.4055 ns | 2,390.2690 ns | 1.78 | 0.02 | 14.8926 | 2.1973 | 93904 B | 1.00 |
| BigJson_Newtonsoft | 118,091.588 ns | 2,407.3099 ns | 131.9528 ns | 1.19 | 0.00 | 33.0811 | 10.7422 | 207976 B | 2.21 |
| BigJson_SystemTextJson | 22,621.221 ns | 145.1274 ns | 7.9549 ns | 0.23 | 0.00 | 3.9063 | 0.3052 | 24696 B | 0.26 |
| BigJson_Sprache | 1,572,933.219 ns | 51,532.2187 ns | 2,824.6548 ns | 15.86 | 0.02 | 859.3750 | 171.8750 | 5398324 B | 57.47 |
| BigJson_Superpower | 1,016,317.802 ns | 30,384.2385 ns | 1,665.4626 ns | 10.24 | 0.01 | 146.4844 | 23.4375 | 927676 B | 9.88 |
| | | | | | | | | | |
| DeepJson_ParlotCompiled | 55,788.357 ns | 807.2790 ns | 44.2497 ns | 1.00 | 0.00 | 15.9912 | 1.7090 | 100656 B | 1.00 |
| DeepJson_Parlot | 56,326.295 ns | 541.5577 ns | 29.6846 ns | 1.01 | 0.00 | 15.9912 | 1.7090 | 100656 B | 1.00 |
| DeepJson_Pidgin | 235,759.393 ns | 17,787.4510 ns | 974.9902 ns | 4.23 | 0.02 | 21.7285 | 5.6152 | 137000 B | 1.36 |
| DeepJson_Newtonsoft | 69,953.638 ns | 1,579.8252 ns | 86.5955 ns | 1.25 | 0.00 | 29.1748 | 8.6670 | 183432 B | 1.82 |
| DeepJson_SystemTextJson | NA | NA | NA | ? | ? | NA | NA | NA | ? |
| DeepJson_Sprache | 1,344,336.604 ns | 36,171.7543 ns | 1,982.6959 ns | 24.10 | 0.03 | 474.6094 | 173.8281 | 2984396 B | 29.65 |
| | | | | | | | | | |
| LongJson_ParlotCompiled | 78,447.519 ns | 3,279.8496 ns | 179.7796 ns | 1.00 | 0.00 | 19.2871 | 4.0283 | 121152 B | 1.00 |
| LongJson_Parlot | 73,648.078 ns | 788.6781 ns | 43.2301 ns | 0.94 | 0.00 | 19.2871 | 4.0283 | 121152 B | 1.00 |
| LongJson_Pidgin | 172,058.354 ns | 7,881.7898 ns | 432.0275 ns | 2.19 | 0.01 | 19.5313 | 3.9063 | 123136 B | 1.02 |
| LongJson_Newtonsoft | 94,847.667 ns | 4,123.7818 ns | 226.0384 ns | 1.21 | 0.00 | 33.0811 | 9.5215 | 207544 B | 1.71 |
| LongJson_SystemTextJson | 17,431.022 ns | 460.8791 ns | 25.2623 ns | 0.22 | 0.00 | 3.9063 | 0.3052 | 24696 B | 0.20 |
| LongJson_Sprache | 1,313,562.949 ns | 65,279.5707 ns | 3,578.1936 ns | 16.74 | 0.05 | 695.3125 | 150.3906 | 4363588 B | 36.02 |
| LongJson_Superpower | 837,530.596 ns | 16,634.4617 ns | 911.7910 ns | 10.68 | 0.02 | 118.1641 | 23.4375 | 744234 B | 6.14 |
| | | | | | | | | | |
| WideJson_ParlotCompiled | 42,290.622 ns | 565.5451 ns | 30.9994 ns | 1.00 | 0.00 | 6.5918 | 0.5493 | 41504 B | 1.00 |
| WideJson_Parlot | 40,642.455 ns | 2,317.0918 ns | 127.0076 ns | 0.96 | 0.00 | 6.5918 | 0.5493 | 41504 B | 1.00 |
| WideJson_Pidgin | 69,621.323 ns | 1,316.3023 ns | 72.1510 ns | 1.65 | 0.00 | 6.5918 | 0.4883 | 41448 B | 1.00 |
| WideJson_Newtonsoft | 58,908.398 ns | 1,693.4597 ns | 92.8242 ns | 1.39 | 0.00 | 17.3950 | 3.4790 | 109280 B | 2.63 |
| WideJson_Sprache | 715,086.114 ns | 39,534.8179 ns | 2,167.0368 ns | 16.91 | 0.05 | 451.1719 | 57.6172 | 2833274 B | 68.27 |
| WideJson_Superpower | 493,485.514 ns | 28,264.4806 ns | 1,549.2716 ns | 11.67 | 0.03 | 73.7305 | 6.8359 | 462657 B | 11.15 |
```

### Regular Expressions
Expand All @@ -184,30 +187,30 @@ Regular expressions can also be replaced by more formal parser definitions. The
an email with the pattern `[\w\.+-]+@[\w-]+\.[\w\.-]+`. Note that in the case of pattern matching Parlot can use the pattern matching mode and do fewer allocations.

```
BenchmarkDotNet v0.13.10, Windows 11 (10.0.22631.3155/23H2/2023Update/SunValley3)
12th Gen Intel Core i7-1260P, 1 CPU, 16 logical and 12 physical cores
.NET SDK 8.0.200
[Host] : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
ShortRun : .NET 8.0.2 (8.0.224.6711), X64 RyuJIT AVX2
BenchmarkDotNet v0.14.0, Debian GNU/Linux 12 (bookworm) (container)
Intel Xeon E-2336 CPU 2.90GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK 8.0.303
[Host] : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
ShortRun : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
Job=ShortRun IterationCount=3 LaunchCount=1
WarmupCount=3
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Allocated | Alloc Ratio |
|-------------------- |----------:|----------:|----------:|------:|--------:|-------:|----------:|------------:|
| RegexEmailCompiled | 75.72 ns | 138.05 ns | 7.567 ns | 1.00 | 0.00 | 0.0221 | 208 B | 1.00 |
| RegexEmail | 143.11 ns | 203.91 ns | 11.177 ns | 1.90 | 0.17 | 0.0219 | 208 B | 1.00 |
| ParlotEmailCompiled | 125.19 ns | 79.34 ns | 4.349 ns | 1.66 | 0.15 | 0.0136 | 128 B | 0.62 |
| ParlotEmail | 176.86 ns | 38.78 ns | 2.126 ns | 2.35 | 0.28 | 0.0339 | 320 B | 1.54 |
| Method | Mean | Error | StdDev | Ratio | RatioSD | Gen0 | Gen1 | Allocated | Alloc Ratio |
|---------------------|------------|-----------|-----------|-------|---------|--------|------|-----------|-------------|
| RegexEmailCompiled | 85.561 ns | 1.8927 ns | 0.1037 ns | 1.00 | 0.00 | 0.0331 | - | 208 B | 1.00 |
| RegexEmail | 183.975 ns | 7.1863 ns | 0.3939 ns | 2.15 | 0.00 | 0.0331 | - | 208 B | 1.00 |
| ParlotEmailCompiled | 145.651 ns | 1.9186 ns | 0.1052 ns | 1.70 | 0.00 | 0.0355 | - | 224 B | 1.08 |
| ParlotEmail | 253.400 ns | 5.2169 ns | 0.2860 ns | 2.96 | 0.00 | 0.0505 | - | 320 B | 1.54 |
```

### Versions

The benchmarks were executed with the following versions:

- Parlot 0.0.19
- Pidgin 3.2.2
- Sprache 2.3.1
- Parlot 1.0.2
- Pidgin 3.3.0
- Sprache 3.0.0-develop-00049
- Superpower 3.0.0
- Newtonsoft.Json 13.0.3

Expand Down

0 comments on commit acff73d

Please sign in to comment.