smdk_v2.2

v2.2: CXL Hot Monitoring Unit / CXL-CLI / Bug fix

1. CXL Hotness Monitoring Unit Support (CXL 3.2.8.2.8)

• The CHMU (CXL Hotness Monitoring Unit) is a new CXL feature designed to operate across both a CXL host and device.
  It allows a CXL Type 3 device to identify its "hot" memory units by tracking memory access counts.
  Software on a CXL host connected to the CXL device can access the hotlist through interfaces defined in the CXL Specification.
• Extend the CXL device driver to support CHMU
• Extend the QEMU to enable CHMU emulation

2. Update NDCTL(CXL-CLI): v79 → v80

3. Bug Fix

• Issue no: #31
  Fix for a memory leak issue that occurs when the 'use_auto_arena_scaling' allocator option is set to 'false.'
  When 'use_auto_arena_scaling' is set to 'false,' the number of arenas is limited to one per pool.
  When set to 'true,' arenas are created in proportion to the number of CPUs, as before.

Full Changelog: smdk_v2.1...smdk_v2.2

Signed-off-by: KyungsanKim ks0204.kim@samsung.com
Signed-off-by: JunhyeokIm junhyeok.im@samsung.com
Signed-off-by: WonjaeLee wj28.lee@samsung.com
Signed-off-by: SeungjunHa seungjun.ha@samsung.com
Signed-off-by: JehoonPark jehoon.park@samsung.com
Signed-off-by: HojinNam hj96.nam@samsung.com
Signed-off-by: YoungshinPark yshin0.park@samsung.com
Signed-off-by: HeesooKim habil.kim@samsung.com

smdk v2.1

v2.1: Intelligent Tiering Engine / Userspace CLI Tool

1. Intelligent Tiering Engine

• BW-Aware tiering: Weighted page allocation

Inspired by CAPTION - https://arxiv.org/abs/2303.15375
The weight ratio has been changed to be adjusted in more finely.
~v2.0: Set weight to zero for saturated node, and max bandwidth of node for others.
v2.1: Set weight to as much as available bandwidth based on max bandwidth.

2. Kernel

• Baseline v6.4 -> v6.9

3. CXL-CLI

• Baseline v78 -> v79

set-alert-config command has been integrated into the ndctl upstream since v79.
Please refer to ndctl v79 release note for more details.

4. Documentation

• Updated

Installation Guide
Plugin Guide

Signed-off-by: JunhyeokIm junhyeok.im@samsung.com
Signed-off-by: WonjaeLee wj28.lee@samsung.com
Signed-off-by: SeungjunHa seungjun.ha@samsung.com
Signed-off-by: JehoonPark jehoon.park@samsung.com
Signed-off-by: HojinNam hj96.nam@samsung.com
Signed-off-by: YoungshinPark yshin0.park@samsung.com

smdk v2.0

v2.0: Intelligent Tiering Engine / CMM-DC Device Support / Vanilla Linux Integration / Userspace CLI Tool

SMDK v2.0 expands the functionality of the Reference SDM solution for CXL Compute Pool(DDR/CMM-D/CMM-DC) and focuses on integration
with developing CXL SWs. The main contents are

1. Intelligent Tiering Engine

• Provide experimental memory tiering/pooling features to explore and lead to CXL usecases
  
• BW/Capacity-Aware tiering: BW order allocation, BW order fallback, Weighted page allocation
  

BW-Aware tiering(v1.5~), Capacity-Aware tiering(v1.1~), Memory-Node SW Interleaving/Grouping(v1.2~), OS Level Swap/Cache(v1.3~)

2. CMM-DC Device Support

• DDR/CMM-D(a.k.a CXL MXP)/CMM-DC(a.k.a CXL PNM) Resource Allocator API and Testcases

3. Vanilla Linux Integration

• Mitigate changes in Linux MM and DD based on kernel v6.6
  
• Using vanilla memory zoning and syscall, less dependency on CXL/DAX driver

4. CXL-CLI

• Support QoS CMDs and expand Region CMDs to configure SW interleaving/grouping

5. Use of CXL SW

• Use OneMCC PNM Library and Linux RFC by Gregory Price

6. Testcase and Documentation

• Apply v2.0 changes above and refactoring
  

SMDK Architecture

Installation Guide

Kernel Guide

Plugin Guide

• Introduce the new experimental result that shows the benefit of DDR(hot)/CXL(cold) tiering on GPT application using SMDK API
  

Experiment Results

Signed-off-by: KyungsanKim ks0204.kim@samsung.com
Signed-off-by: WonjaeLee wj28.lee@samsung.com
Signed-off-by: JunhyeokImjunhyeok.im@samsung.com
Signed-off-by: SeungjunHa seungjun.ha@samsung.com
Signed-off-by: JehoonPark jehoon.park@samsung.com
Signed-off-by: HojinNam hj96.nam@samsung.com
Signed-off-by: JeongtaePark jtp.park@samsung.com
Signed-off-by: YoungshinPark yshin0.park@samsung.com
Signed-off-by: HanDeokLee handeok.lee@samsung.com

smdk v1.5.1

v1.5.1: SMDK on QEMU

Support SMDK on QEMU

• QEMU supports CXL type3 volatile device since about v8.1.0.
  Reflecting the requirement out of CXL industry researchers, SMDK supports QEMU emulation since v1.5.1.
  The version allows using userspace plugins(library, cli, BM, testcases) and OS interfaces(swap and cache) on the QEMU v8.1.50 included.
  Please refer to https://github.com/OpenMPDK/SMDK/wiki/4.-Kernel#427-qemu to understand how to emulate CXL memory and SMDK on top of the QEMU.

• Limitation
  group-noop is only supported on QEMU among three grouping functions - group-zone, group-node, and group-noop.
  MLC BW tool and PMU related SW are not working on QEMU due to the CPU emulation.

1. Kernel

• Patch on v6.4

2. QEMU

• QEMU v8.1.50 and launcher scripts

3. Documentation

• Kernel Guide (updated) : https://github.com/OpenMPDK/SMDK/wiki/4.-Kernel#427-qemu
  

smdk v1.5

v1.5: Adaptive Interleaving / Userspace CLI Tool / Kernel (update)

1. Adaptive Interleaving

• SMDK v1.5 supports a new DDR/CXL memory tiering scenario, adaptive interleaving, by expanding the Intelligent Tiering Engine.
  This is a DDR/CXL memory interleaving in a software manner to lead to an effective bandwidth aggregation of DDR/CXL memories.
  While it works, adaptive interleaving maintains the maximum bandwidth map of online numa-nodes, keeping track of in-use bandwidth of a node.
  When the bandwidth saturation of a node is detected, it handles incoming allocation request from other memory nodes in an autonomous way.
  This is geared to mitigate imbalanced memory utilization of DDR/CXL memory without an user intervention.

2. CXL-CLI

• CXL Spec commands

Added: get shutdown state, set shutdown state, get scan media capabilities, scan media, get scan media results, sanitize

Deprecated: get poison list

3. Kernel

• Baseline v6.0-rc6 -> v6.4

4. Documentation

• Updated

SMDK Architecture: Intelligent Tiering Engine - https://github.com/OpenMPDK/SMDK/wiki/2.-SMDK-Architecture
Installation Guide: Adaptive Interleaving - https://github.com/OpenMPDK/SMDK/wiki/3.-Installation
Plugin Guide: Adaptive Interleaving and New CXL spec commands - https://github.com/OpenMPDK/SMDK/wiki/5.-Plugin
Experiment Result : Adaptive Interleaving - https://github.com/OpenMPDK/SMDK/wiki/7.-Experiment-Results

smdk v1.4

v1.4: CXL-Cache / Userspace CLI Tool (update)

This update reflects the voices of Industry partners who cooperate with us.

1. CXL Cache

• In addition to the usecase that uses CXL device as System RAM and Swap interfaces, SMDK v1.4 allows another usecase that uses CXL device as OS-Level Cache interface.
• CXL Cache is the 2nd-level page cache with pluggable and page-granularity attributes that stores clean file-backed pages.
  Upon CXL Cache, a file-backed page is traversed in following memory order - pagecache(near), cxlcache(far), disk(farthest).

2. CXL-CLI

• CXL Spec commands added

identify, get-health-info, set-alert-config, get-alert-config, get-firmware-info, transfer-firmware, activate-firmware

• CXL Cache control commands added

enable-cxlcache, disable-cxlcache, flush-cxlcache, check-cxlcache

3. Documentation

• Update user guide documents

SMDK Architecture: CXL Cache - https://github.com/OpenMPDK/SMDK/wiki/2.-SMDK-Architecture

Kernel Guide: CXL Cache usage - https://github.com/OpenMPDK/SMDK/wiki/4.-Kernel

Plugin Guide: New CXL spec commands and CXL Cache control commands - https://github.com/OpenMPDK/SMDK/wiki/5.-Plugin

smdk v1.3

v1.3: CXL-Swap / CXL Composability / Userspace CLI Tool

This update reflects the voices of Industry partners who cooperate with us.

1. CXL Swap

• In addition to the usecase that uses CXL device as System RAM interface,
  SMDK v1.3 allows another usecase that uses CXL device as Swap interface. CXL Swap implements Linux frontswap and is delivered as kernel module likewise zSwap.
• However, it does not perform (de)compression while page swapping, thus, leads to CPU saving and latency QoS on swap handling. We believe it is more close to the philosophy of CXL memory expansion.

2. CXL Composability

• Composability is also a primary matter of CXL philosophy. By adopting previous Linux patches and algorithms, we enhanced compatibility and coverage of Linux VMM belows to better support of CXL composability.
• Page migration - manage CXL DRAM as movable page attribute
• Memory node on/offline - allow memory-node on/offline function for CXL DRAM
• Fragmentation avoidance - apply the algorithm for avoiding DRAM fragmentation issue to CXL DRAM

3. CXL-CLI

• CXL Swap control commands

enable_cxlswap, disable_cxlswap, flush_cxlswap, check_cxlswap

• CXL Device performance reporter

get-latency-matrix

4. Documentation

• Update and Refactoring user guide documents

SMDK Architecture - https://github.com/OpenMPDK/SMDK/wiki/2.-SMDK-Architecture

Installation Guide - https://github.com/OpenMPDK/SMDK/wiki/3.-Installation

Kernel Guide - https://github.com/OpenMPDK/SMDK/wiki/4.-Kernel

Plugin Guide - https://github.com/OpenMPDK/SMDK/wiki/5.-Plugin

Full Changelog: smdk_v1.2.2...smdk_v1.3

smdk v1.2

v1.2: N-way Grouping / Userspace CLI Tool / Application Compatibility

This update reflects the voices of Industry partners who cooperate with us.

1. N-way grouping (a.k.a memory partition)

• This is geared for those who want to assemble multiple CXL devices as they wish
  • Related Usecase : Memory Interleaving, Isolation, Virtualization
• CXL-CLI - group/node management
• SMDK Allocator - updated compatible / optimization library
• SMDK Kernel - primitive interfaces to provide online and static CXL device information, and control

2. CXL-CLI

• Grouping commands

N-way group : group-zone / group-node / group-noop / group-dax / group-add / group-remove
Listing group : group-list

• CXL Spec commands

Poison : inject-poison / get-poison / clear-poison
Timestamp : set-timestamp / get-timestamp
Event : get-event-record / clear-event-record

• Basic CXL-CLI commands

SMDK CXL-CLI is an expansion of Intel CXL-CLI

3. Application Compatibility

• A multi-node traversal application is now able to run normally. e.g.) MLC

4. Miscellaneous

• Error/Exception handling on real-world testbeds
• More testcases

5. Issue

• fix issue : kernel build failure when CONFIG_LOCKDEP=y or CONFIG_EXMEM=n (v1.2.1, thanks to Ravi Shankar)
• functionality coverage : Even though a version of BIOS and/or CXL device does not provide DVSEC ID info, the patch allows listing CXL device information and node grouping. (v1.2.2, thanks to Wu Chanco)

Documentation

• SMDK Architecture - https://github.com/OpenMPDK/SMDK/wiki/2.-SMDK-Architecture
• User Guide - https://github.com/OpenMPDK/SMDK/wiki/3.-User-Guide
• Test and Tools - https://github.com/OpenMPDK/SMDK/wiki/4.-Test-and-Tools

smdk v1.1

v1.1.0: Coverage/Reliability on Compatible/Optimization API + Memory Partition

This update reinforces coverage and reliability of v1.0.0 compatible/optimization API and memory partition features.

SMDK Allocator

• Compatible API

1. BW Aggregation / Isolation
2. Easy-configuration

• Optimization API

1. Python Binding
2. Node Specific Allocation/deallocation
3. Statistic Reporting

SMDK Kernel

1. Co-existence with DAX interface
2. CXL Memory Registration using SRAT/CEDT/DVSEC
3. Multi-socket Support
4. Update version 5.17-rc5 -> 5.18-rc3

Miscellaneous

1. Error/Exception Handling
2. Testcases

Documentation (https://github.com/OpenMPDK/SMDK/wiki)

• User Guide
• Test and Tools

smdk v1.0

v1.0.0: Compatible/Optimization API + Memory Partition

The initial OSS release of the Scalable Memory Development Kit (SMDK).

Software (https://github.com/OpenMPDK/SMDK.git)

• SMDK Plugin (allocator library, numactl extension)
• SMDK Kernel (memory partition)
• Application (IMDB, ML/AI)
• BM Tool (stream, mlc)
• Container (Application/BM Tool container)
• TC (unittests)

Documentation (https://github.com/OpenMPDK/SMDK/wiki)

1. Design Principle and Objective
2. SMDK Architecture
3. User Guide
4. Test and Tools
5. Experiment Results