L2 L3 MEM traffic on Intel Skylake SP CascadeLake SP
Intel introduced with Intel Skylake SP (and the successor Cascadelake SP) a change in the cache hierarchy. The sizes of each layer are changed (L2 larger, L3 smaller) and the L3 is now a victim cache. This results also in a different approach to measure the traffic between L2, L3 and memory.
On all architectures before Intel Skylake SP, the caches are inclusive. This means that all cache lines that are currently in L1 are contained in L2 and L3 cache as well (same for all lines in L2 that are also present in L3). With Intel Skylake SP, the L3 cache became a victim cache (non-inclusive) while L1 and L2 continue being inclusive.
If a core requests data from memory, it is directly loaded into L2 (and then in L1) bypassing the L3 cache. If a cache lines need to be evicted from L2, the current line state is checked and, based on some heuristics, the cache line is either dropped (makes sense for clean cache lines), evicted to L3 (makes sense for modified and shared cache lines) or even evicted directly to memory. The exact heuristics are not published by Intel.
For the CPU architectures before Intel Skylake SP, LIKWID uses two events for loaded (L2_LINES_IN_ALL, rf107) and evicted (L2_TRANS_L2_WB, r40f0) cache lines. This was enough to achieve a high accuracy because all data coming from memory and going to memory has to flow through L2 and L3. With Intel Skylake SP the situation changed and the event L2_LINES_IN_ALL is the sum of loads from L3 and memory (simply all cache lines coming into L2 independent of the source). The same is true for the L2_TRANS_L2_WB event. There is no differentiation between evicts to L3 and evicts to memory and also no event for counting dropped cache lines.
In order to show the difference at hardware level, I took some measurements (LIKWID version 4.3.4) with the current L3 group using a single core on an Intel Broadwell EP (Intel(R) Xeon(R) CPU E5-2697 v4 @ 2.30GHz, CPU clock fixed to 2.30 GHz) and an Intel Skylake SP (Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz, CPU clock fixed to 2.40 GHz). As benchmark application I used likwid-bench with the benchmarks load, store, copy, stream and triad. All benchmarks work on double precision floating-point arrays using scalar operations.
For BDX, the behavior is as expected:
- As soon as the data sizes fill the L2 cache almost completely, the cache lines loaded from L3 (
L2_LINES_IN_ALL*64) raises until it reaches the 64 Byte per iteration. - The benchmark does not evict any data, hence the
L2_TRANS_L2_WB*64event stays zero, - When the data sizes come closer to the L3 size (11 MByte as Cluster-on-Die is enabled), the data fetched from memory
SUM(CAS_COUNT_RD)*64increases until 64 Byte.
For SKX, the behavior is different:
- When the L2 cache is almost full, cache lines are loaded from L3 (
L2_LINES_IN_ALL*64) until we read 64 Byte per iteration. Same as for BDX. - But also
L2_TRANS_L2_WB*64raises. To the same extent as (L2_LINES_IN_ALL*64). This is because the cache lines are evicted to L3 and fetched again because the SKX L2 cache marks them as they are likely to be reused shortly. - No data is written back to memory, as
SUM(CAS_COUNT_WR)*64stays zero. - When the data sizes come closer to the L3 size (full 28 MByte usable although SNC is enabled), the data fetched from memory
SUM(CAS_COUNT_RD)*64increases until 64 Byte.
| BDX | SKX |
|---|---|
As soon as the data sizes fill the L2 cache almost completely, the cache lines loaded from L3 (L2_LINES_IN_ALL*64) raises until it reaches the 64 Byte per iteration. |
When the L2 cache is almost full, cache lines are loaded from L3 (L2_LINES_IN_ALL*64) until we read 64 Byte per iteration. Same as for BDX. |
The benchmark does not evict any data, hence the L2_TRANS_L2_WB*64 event stays zero |
L2_TRANS_L2_WB*64 raises similar to (L2_LINES_IN_ALL*64). This is because the cache lines are evicted to L3 and fetched again because the SKX L2 cache marks them as they are likely to be reused shortly. |
SUM(CAS_COUNT_RD)*64 stays zero |
SUM(CAS_COUNT_RD)*64 stays zero |
When the data sizes come closer to the L3 size (11 MByte as Cluster-on-Die is enabled), the data fetched from memory SUM(CAS_COUNT_RD)*64 increases until 64 Byte |
When the data sizes come closer to the L3 size (full 28 MByte usable although SNC is enabled), the data fetched from memory SUM(CAS_COUNT_RD)*64 increases until 64 Byte. |
I posted the question in the Intel developer forums and it's not only me/LIKWID having that problem. At the current state, the events for traffic in and out of L2 do not allow a differentiation of the source resp. destination.
The memory traffic can be measured properly and with high accuracy assuming 64B for each read and write operation to memory. But the memory controllers are located in the Uncore part of the CPU and thus the counts reflect the traffic to/from all cores of a socket (+ intersocket traffic)
It is probably possible to use L3 events (part of the Uncore) to retrieve the counts for data flowing into L3, data being loaded by the L2 and the evictions to memory. But the Uncore is socket-specific and consequently does not allow the attribution of a single cores' data consumption.
In a meeting with Intel, we got a list of events:
- MEM_INST_RETIRED.ALL_LOADS
- MEM_INST_RETIRED.ALL_STORES
- MEM_LOAD_L3_HIT_RETIRED.XSNP_HIT
- MEM_LOAD_L3_HIT_RETIRED.XSNP_HITM
- MEM_LOAD_L3_HIT_RETIRED.XSNP_MISS
- MEM_LOAD_L3_HIT_RETIRED.XSNP_NONE
- MEM_LOAD_L3_MISS_RETIRED.LOCAL_DRAM
- MEM_LOAD_L3_MISS_RETIRED.REMOTE_DRAM
- MEM_LOAD_L3_MISS_RETIRED.REMOTE_L4 (**)
- MEM_LOAD_MISC_RETIRED.NON_DRAM (**)
- MEM_LOAD_MISC_RETIRED.UC
- MEM_LOAD_MISC_RETIRED.UNKNOWN_SOURCE (**)
All events marked with (**) are not published and consequently not usable by LIKWID. We tried the other events but for some it was clear that it wouldn't work. E.g. the MEM_INST_RETIRED.ALL_* events count the number of loads resp. stores that are issued, executed and retired (completed) by the core, hence some units away from L2, L3 and memory. Moreover, there are cases where an instruction triggers data movement in the background (e.g. read-for-ownership for stores where the destination cache line is not present in the L1 cache).
The L3 performance group for Intel Skylake still uses the two events mentioned above. So, keep in mind that L2_LINES_IN_ALL contains loads from L3 and memory and L2_TRANS_L2_WB contains writebacks to L3 (and memory).
When releasing the Intel Cascadelake SP/AP chips, Intel published two new events IDI_MISC.WB_UPGRADE (Counts number of cache lines that are allocated and written back to L3 with the intention that they are more likely to be reused shortly) and IDI_MISC.WB_DOWNGRADE (Counts number of cache lines that are dropped and not written back to L3 as they are deemed to be less likely to be reused shortly). Whole list of Cascadelake SP/AP events here. One of the problems is, that these events are already mentioned in the errata section of the specification update document for Intel Cascadelake SP/AP:
CLX3. IDI_MISC Performance Monitoring Events May be Inaccurate
Problem: The IDI_MISC.WB_UPGRADE and IDI_MISC.WB_DOWNGRADE performance monitoring events (Event FEH; UMask 02H and 04H) counts cache lines evicted fromthe L2 cache. Due to this erratum, the per logical processor count may be incorrect when both logical processors on the same physical core are active. The aggregate count of both logical processors is not affected by this erratum.
Implication: IDI_MISC performance monitoring events may be inaccurate.
Workaround: None identified.
Status:No fix.
The Intel Cascadelake SP micro architecture is the follow-up to Intel Skylake SP. As an experiment I added the two events to the Intel Skylake SP event file and ran some benchmarks using a single core to get accurate results. Same configuration as in the above comparison between Intel Broadwell EP and Intel Skylake SP.