- 1 Introduction
- 2 Step by step instructions on RT-Profiling tool
- 3 Roadmap
- 4 License
In industrial controller scenarios, the real-time performance indicators like jitter & cycle time are the two key criteria to evaluate the performance of RT control system. However, how to troubleshoot and tune real-time performance is always a challenge to developers. There are many factors like IRQs, network traffic could affecting the overall performance of RT control system.
This document provides a step-by-step guidance for RT control system developers/analyzers on how to setup, collect and analyze real-time control related jitter/cycle parameters as well as other system characters of the real-time control system like CPU utilization, disks accessing, network transferring, IRQs using customized nmon/nmonchart toolset.
The RT profiling tool should be able to run on all the latest Linux distributions theoretically. We will use Linux Ubuntu 16.04 x64 LTS as the default environment throughout this guide in order to minimize clutters caused by various Linux distributions.
Please install gcc, make, git utilities and ncurses dev lib on the target system if any of these packages is unavailable on the system.
-
Connect to the internet.
-
Open a command prompt terminal window.
-
Type following instruction in the terminal, input root password as prompt. Please adjust the command accordingly to install the corresponding dependency if you use Fedora or other Linux distributions.
sudo apt install gcc libncurses5-dev make gitGet the latest source code of RT-Profiling tool from the git server or from package file offline.
- Get the source code from the git server.
git clone https://github.com/intel/rt-perf-profiling-toolType following commands in the terminal to build latency and cyclictest utilities.
- Apply the patch. Where xxx stands for the absolute path of patch 0001-capture-jitter-performance-in-share-memory.patch under collection/latency folder you got at step 2.
cd testsuite/latency/
git apply --reject xxx- Build latency utility.
make- Get rt-tests source code v1.0.
git clone https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git
cd rt-tests/
git checkout stable/v1.0- Apply the patch. Where yyy stands for the absolute path of patch 0001-capture-latency-in-share-memory.patch under collection/cyclictest folder you got at step 2.
git apply --reject yyy- Build cyclictest utility.
makeBuild nmon utility, and then make nmon utility and nmonchart script executable at any place.
- Move to the nmon directory.
cd rt_profiling-tool/agent/nmon- Execute following command to build nmon binary. Where arch stands for the CPU architecture of the system under test, os stands for the target operating system nmon will be executed. We'll use nmon_AMD64_ubuntu1604 as an example here. Please specify the designated configuration combination value as the parameter of the make according to the real condition.
make <nmon_arch_os>- Setup the symbolic link of nmon utility. Here we use nmon_AMD64_ubuntu1604 as the sample, please adjust it for your self.
sudo ln -s rt_profiling-tool/agent/nmon/nmon_AMD64_ubuntu1604 /usr/local/bin/nmon- Setup the symbolic link of nmonchart script.
sudo ln -s rt_profiling-tool/analysis/nmonchart/nmonchart /usr/local/bin/nmonchartWe can use latency or cyclictest utility to generate RT performance data.
You can launch latency in terminal to generate RT performance data.
sudo ./latency -r-r: update jitter performance in share memory
You can launch cyclictest in terminal to generate RT performance data.
sudo ./cyclictest -N -t 4 -e-N: show results in nanoseconds
-t 4: start 4 threads to measure latency
-e: update latency performance in share memory
Nmon can gives you a huge amount of important performance information in one go. It can output the data on the screen in interactive mode or save the data to a comma separated file for analysis in data collector mode.
You can observe the Jitter time, Cycle time as well as CPU, memory, network, disks etc. data directly on the screen and updated every second if you launch the nmon without arguments.
- Open a terminal, type nmon. Nmon with help info page will pop up.
- You can press the corresponding keys to toggle on or off statistics. Say, you can press "ecCUd" to show RT info, CPU Utilization, CPU Utilization Stats, CPU Utilization Wide View and Disk I/O info.
- You can press "+" or "-" key to increase or decrease nmon sampling intervals in real time. By default, the screen refresh time is two seconds. The minimum refresh time is one second.
- Open a terminal, launch nmon utility with parameters specified. The .nmon data files it captured will be saved under YYMMDD_HHMM folder(where YYMMDD and HHMM are the beginning time of the collection).
nmon -feiTU -s 1 -c 36000Notes: Following parameters are supported when nmon is launched.
-f: Must be the first option on the line (switches off interactive mode). Saves data to a CSV Spreadsheet format .nmon file in the local directory
-e: Specify RT Performance info to be collected
-i: Specify IRQ info to be collected.
-t: Collect Top Process info also
-T: Collect command line info also
-U: Collect CPU Utils info als；
-s : Time between data snapshots
-c : Of snapshots before exiting。
Eg: -feiTU -s 1 -c 36000 stands for collect the data for 36000 times with time interval for one second. IRQ and Top process info are collected also. The .nmon file will be saved on current folder for further investigation by default.
Open a terminal, move to the folder where nmon utilities are saved. Execute nmonchart script to generate html files.
nmonchart HHMMDD__HHMMMHHMMDD_HHMMM is the folder name where .nmon data files are stored.
The webpage files xxxx.html will be generated under HHMMDD_HHMMM folder.
Double click the xxxx.html file generated by nmonchart to open the dynamic webpage as Figure 2.5.1.
Figure 2.5.1 Homepage of nmonchart
You can click Configuration, Top Summary, Top Command and Top Disk buttons separately on the first row to get the overall information, overall CPU consumptions and disk usage.
You can click on CPU util. CPU Use and CPU All Utils, button to monitor CPU related criteria.
Figure 2.5.2 nmonchart
- Nmon based data agent
- Support cyclic data source
- Support latency data source
- Support real-time data metrics
- Support disk/mem/net/interrupt etc.. Data metrics
- Web based visualization
- Charting for misc. data metrics
- Real-time data metrics showing with defined & selected other metrics
- Web based visualization
- Change code base to echarts based chart
- Dual-Y axis to show data in different granularity
- Nmon based data agent
- support msi_latency data source
- Web based visualization
- Charting for misc. data metrics
- Real-time data metrics showing with defined & selected other metrics
- jitter statistics analysis
- Scoring
- normal distribution
- Sorting
- Plugin based value-adding analysis module framework & sample code
- Correlation
- Abnormal
- Value added modules
The source code is licensed under GNU General Public License v3.0 or later. See LICENSE file for details. It includes software developed from 3rd-party, See NOTICE file for details.