• Stars
    star
    224
  • Rank 177,792 (Top 4 %)
  • Language
    C
  • License
    Other
  • Created over 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

LinuxKI Toolset (Trace-based performance analysis tool)

LinuxKI Toolset v7.6

The LinuxKI Toolset (or LinuxKI for short) is an opensourced advanced mission critical performance troubleshooting tool for Linux. It is designed to identify performance issues beyond the typical performance metrics and results in faster root cause for many performance issues. LinuxKI is a kernel tracing toolkit designed to answer two primary questions about the system:

  • If it's running, what's it doing?
  • If it's waiting, what's it waiting on?

LinuxKI analyzes the kernel trace data in different and often unique ways to help performance specialist drill down on often complex performance issues. The following output is an example of data displayed for a specific task:

PID:  133343 /home/mcr/bin/iotest8

RunTime    :  3.730009  SysTime   :  3.650014   UserTime   :  0.078149   StealTime  :  0.000000
SleepTime  :  5.654813  Sleep Cnt :    161700   Wakeup Cnt :     85898
RunQTime   :  0.615273  Switch Cnt:    161708   PreemptCnt :         8
Last CPU   :        57  CPU Migrs :        25   NODE Migrs :         0
Policy     : SCHED_NORMAL     vss :      1074          rss :       159

   LLC_ref   LLC_hits  LLC_hit%     Instrs     Cycles      CPI   Avg_MHz  SMI_cnt
   148482k    143389k    96.57%    4884.9m   11920.2m     2.44   3200.00       33

------------------------- Top Hardclock Functions ---------------------------
   Count     Pct  State  Function
     115  23.71%  SYS    rwsem_wake
      76  15.67%  SYS    __blk_run_queue
      47   9.69%  SYS    __schedule
      32   6.60%  SYS    blk_queue_bio
      23   4.74%  SYS    blk_finish_plug

---------------------------- Top Wait Functions -----------------------------
   Count     SlpTime  Msec/Slp  MaxMsecs  Func
   79806    5.288981     0.066   579.234  io_schedule_timeout
   47151    0.217324     0.005     0.412  rwsem_down_read_failed
   34743    0.148508     0.004     0.409  rwsem_down_write_failed

------------------------ Total I/O ------------------------- ------------------- Write I/O ------------------- -------------------- Read I/O -------------------
Device        IO/s    MB/s  AvIOsz AvInFlt   Avwait   Avserv    IO/s    MB/s  AvIOsz AvInFlt   Avwait   Avserv    IO/s    MB/s  AvIOsz AvInFlt   Avwait   Avserv
All           7980      62       8    8.57     0.00     0.08    7980      62       8    8.57     0.00     0.08       0       0       8    0.00     0.00     0.04
sdb           7980      62       8    8.57     0.00     0.08    7980      62       8    8.57     0.00     0.08       0       0       8    0.00     0.00     0.04

----------------------------- Top System Calls ------------------------------
System Call Name     Count     Rate     ElpTime        Avg        Max    Errs    AvSz     KB/s
write                79807   7980.6    9.892351   0.000124   0.579275       0    8192  63844.7  
   SLEEP            161700  16169.8    5.654813   0.000035
      Sleep Func     79806             5.288981   0.000066  io_schedule_timeout
      Sleep Func     47151             0.217324   0.000005  rwsem_down_read_failed
      Sleep Func     34743             0.148508   0.000004  rwsem_down_write_failed
   RUNQ                                0.615269
   CPU                                 3.622297
lseek                79808   7980.7    0.029498   0.000000   0.000012       0

LinuxKI is designed to be easy to install, collect data, and generate reports. It runs on Linux kernels 2.6.32 or later running on x86_64, arm64 and ppc64 platforms. It collects low-level detailed traces reflecting the behavior of the running workload, which can be analyzed online or bundled along with configuration and other performance data into a single gzip file suitable for transfer and offline analysis.

LinuxKI only enables key tracepoints in the performance paths, such as scheduler events, system call events, block I/O events, interrupt events, and CPU profiling events. It has the ability to collect system and user stack traces during sched_switch events and CPU profiling events, and much more. LinuxKI can analyze the detailed trace data in many different ways - per-PID, per-device, per-HBA path, per-CPU, per-LDOM, per-interrupt, per-Docker container, and much more.

LinuxKI can collect its trace data from either of two sources: the Linux ftrace facility, or the LiKI DLKM supplied as part of this toolset. The default tracing mechanism used is LiKI, however you may chose to use ftrace if desired. Here are some considerations:

  • LiKI is an opensource kernel tracing facility written by HPE specifically for large mission-critical servers. It collects significantly more detail than ftrace allowing more precise analysis, while imposing less overhead. However LiKI is a kernel module that is loaded into the kernel when collection begins, and removed immediately afterwards. Installation of the LiKI DLKM may taint some Linux kernels and is not supported by the Linux providers.

  • ftrace is a Linux kernel tracing facility that is built into most versions of the Linux kernels (2.6.32 and later), such as RHEL 6 and SLES 11 SP2. The runki script will use the ftrace data source if the LiKI DLKM fails to load or if the "-f" option is passed to the runki script. Only a select set of tracepoints are enabled, keeping the overhead reasonably low. The LinuxKI Toolset will do all the work of enabling the trace points and collecting the data from its in-kernel ring buffers. While the ftrace code is included in more recent Linux kernels, it is generally disabled, so HPE cannot comment on how well it has been tested by the Linux vendor (RedHat, Novel).

Using the LiKI tracing module is the preferred method, but ftrace may be used in cases where where the LiKI DLKM cannot be installed, either due to missing dependencies, or inherent risks of installing a DLKM.

Disclaimer of Warranty and Limitation of Liability Notices

Refer to the COPYING.liki, COPYING.kiinfo, LICENSE.txt files for additional information.

Overview

The following documentation provides a brief overview for downloading and installing the LinuxKI Toolset, as well as data collection and report generation. Be sure to refer to the LinuxKI MasterClass, LinuxKI Quick Reference Guide, and LinuxKI FAQ for complete documentation:

LinuxKI Quick Reference Guide
LinuxKI Frequently Asked Questions
LinuxKI MasterClass

Download

Pre-packaged RPM and DEB files are available on the Releases Page.

Prerequisites

There are no mandatory pre-requisites. LinuxKI should install and run on most Linux systems from 2.6.32 through 5.14.21

However, if you would like to use the LiKI tracing mechanism (perferred method), you will need the following packages installed to compile the LiKI module from source code:

  • kernel-devel
  • kernel-headers
  • gcc
  • make
  • elfutils-libelf-devel (4.14.12 kernels and RHEL 8.X and later)

If LiKI fails to compile, you can resolve the dependency issue and execute the module_prep script to manually build the LiKI DLKM tracing module. You can also continue to use the LinuxKI Toolset using the ftrace tracing mechanism.

Installation

The LinuxKI Toolset is provided in an RPM Package and can be installed as follows:

# rpm --install --nodeps linuxki.<version>.noarch.rpm

Or for Debian-related kernels, the toolset can be installed using the dpkg command:

# dpkg --install linuxki.<version>_all.deb

The files are installed in the /opt/linuxki directory. You should add this directory to the PATH of the root user for the duration of the data collection session.

You can also use your favorite package manager, such as yum.

# yum localinstall linuxki.<version>.noarch.rpm

Verifying the LinuxKI toolset version

You can verify the version of the LinuxKI toolset using rpm or dpkg as follows:

 $ rpm --query linuxki
 linuxki-7.5-1.noarch

 $ dpkg --status linuxki | grep Version
 Version: 7.5-1

Removing the LinuxKI toolset

You can remove the LinuxKI toolset using rpm or dpkg as follows:

 # rpm --erase linuxki

 # dpkg --remove linuxki
 # dpkg --purge linuxki

Data collection

When the system is experiencing performance problems, the runki script can be executed to collect data. By default 20 seconds of trace data will be collected, and then runki will spend some time gathering other configuration and supplemental data, and then bundle this into a single gzip archive. It might take several minutes in all to complete. Root/superuser privilege is required to collect the trace data. The data is stored in the current working directory, and may require several hundred megabytes or gigabytes of space per collection run, depending on the size of the system and amount of trace data generated. The filesystem on which data is stored should be enabled to use the filesystem cache; directIO is not recommended. If sufficient memory is available, the current working directory can be changed to /dev/shm and the runki script can collect the data in-memory and then copied to persistent storage later.

After installing the LinuxKI Toolset, a 20-second trace dump can easily be obtained as follows:

$ export PATH=$PATH:/opt/linuxki 
$ cd /dev/shm     # optional, to collect data in memory
$ runki -R        # use LiKI tracing mechanism

or

$ runki -f         # use ftrace tracing mechanism

When the data collection is complete, you will see a message similar to the following...

=== Trace completed and archived as ki_all.localhost.0613_1310.tgz

Trace analysis

After data collection, a default set of reports can be generated as follows:

$ kiall -r         # -r option creates a nodename/timestamp directory structure

Most of the generated reports are text based and can be viewed with a standard editor or text viewer.

More Repositories

1

netperf

Netperf is a benchmark that can be used to measure the performance of many different types of networking. It provides tests for both unidirectional throughput, and end-to-end latency.
C
725
star
2

swarm-learning

A simplified library for decentralized, privacy preserving machine learning
Python
331
star
3

cacti

An integrated cache and memory access time, cycle time, area, leakage, and dynamic power model
C++
301
star
4

squest

Service request portal on top of Ansible Tower/AWX
Python
294
star
5

quartz

Quartz: A DRAM-based performance emulator for NVM
C
158
star
6

dlcookbook-dlbs

Deep Learning Benchmarking Suite
Python
130
star
7

POSH-HPEOneView

PowerShell language bindings library for HPE OneView.
PowerShell
125
star
8

mcpat

An integrated power, area, and timing modeling framework for multicore and manycore architectures
C++
122
star
9

PacketRusher

High performance 5G UE/gNB Simulator and CP/UP load tester.
Go
109
star
10

oneview-ansible

This project is no longer being developed and has limited support. Please use the newer Ansible Collection project: https://github.com/HewlettPackard/oneview-ansible-collection
Python
104
star
11

yoda

GitHub extension for agile project management using the issues subsystem.
JavaScript
90
star
12

python-hpOneView

DEPRECATED - no longer actively maintained. New repository: https://github.com/HewlettPackard/oneview-python
Python
87
star
13

python-ilorest-library-old

Python library for iLO RESTful API
Python
85
star
14

wireless-tools

Wireless Tools for Linux
C
85
star
15

PowerShell-ProLiant-SDK

PowerShell sample scripts for managing HPE servers
PowerShell
85
star
16

kraal

Enables the use of Kotlin coroutines and GraalVM native-image together
Kotlin
84
star
17

monkeyble

End-to-end testing framework for Ansible
Python
78
star
18

Atlas

Atlas: Programming for Persistent Memory
C++
74
star
19

foedus

FOEDUS: Fast Optimistic Engine for Data Unification Services
65
star
20

foedus_code

FOEDUS main source code repository
C++
64
star
21

structex

Go structure annotations that supports encoding and decoding; similar to C-style bitfields. Supports bitfield packing, self-describing layout parameters, and alignment.
Go
58
star
22

ansible-ilorest-role

Ansible role for installing the Python iLOrest library and showcasing a few examples.
Python
52
star
23

terraform-provider-oneview

Automates the provisioning of physical infrastructure from a private cloud using templates from HPE OneView with Terraform
Go
49
star
24

reconbf

Recon system hardening scanner
Python
47
star
25

sandpiper

Implementation of the Loopy Belief Propagation algorithm for Apache Spark
Scala
42
star
26

sparkle

C++
38
star
27

ilo-ansible-collection

Ansible Collection and Sample Playbooks for HPE iLO
Python
38
star
28

dockerfile-parser-rs

a Rust library for parsing, validating, and modifying Dockerfiles
Rust
37
star
29

jupyterhub-samlauthenticator

jupyterhub-samlauthenticator
Python
34
star
30

cloudformation-plus

A tool that adds features to AWS CloudFormation that reduce the amount of code you must write to deploy non-trivial applications.
Python
31
star
31

cmf

CMF library helps to collect and store information associated with ML pipelines. It tracks the lineages for artifacts and executions of distributed AI pipelines. It provides API's to record and query the metadata associated with ML pipelines. The framework adopts a data first approach and all artifacts recorded in the framework are versioned and identified by the content hash.
Python
30
star
32

lustre_exporter

Prometheus exporter for use with the Lustre parallel filesystem
Go
29
star
33

oneview-python

Python library for HPE OneView
Python
28
star
34

nvthreads

C
27
star
35

oneview-golang

Golang bindings for OneView api's
Go
27
star
36

nagios-plugins-hpilo

Nagios plug-in for iLO Agentless Management
Shell
26
star
37

mds

Managed Data Structures
Java
26
star
38

dpp

Device Provisioning Protocol is a Wi-Fi Alliance program to securely provision devices to obtain network access.
C
25
star
39

mdc-toolkit

24
star
40

Docker-SimpliVity

Ansible deployment playbooks with detailed deployment guides to rapidly provision a Docker dev and Docker ops environment in less than 30 minutes
Shell
24
star
41

woodchipper

An interactive command-line log processor
Rust
23
star
42

oneview-ansible-collection

Ansible Collection and Sample Playbooks for HPE OneView
Python
22
star
43

hpe-solutions-openshift

This GitHub site contains deployment guides and resources for deploying Red Hat OpenShift on HPE platforms.
Python
22
star
44

chef-provisioning-oneview

Chef Provisioning Driver for HPE OneView
Ruby
21
star
45

lustre-csi-driver

A Lustre container storage interface that allows Kubernetes to mount/unmount provisioned Lustre filesystems into containers.
Go
20
star
46

supersim

A flexible event-driven cycle-accurate network simulator
C++
19
star
47

hpe3par_ansible_module

HPE Alletra 9000 and HPE Primera and HPE 3PAR ansible module to configure, provision and manage storage systems and resources
Python
18
star
48

galadriel

SPIFFE Federation the easy way
Go
18
star
49

csa-ce

This repo contains scripts which start HPE Cloud Service Automation Community Edition 4.7 (HPE CSA CE) in Docker.
Shell
17
star
50

ezdemo

HPE Ezmeral Deployment tool for demos
Shell
17
star
51

ilo-chef

iLO Resource Provider for Chef
Ruby
17
star
52

oneview-chef

This project is no longer being developed and has limited support. In the near future this repository will be fully deprecated. Please consider using other OneView projects, such as Terraform and Ansible Collection
Ruby
17
star
53

oneview-osdeployment

Sample HPE OneView OS deployment integration
Python
16
star
54

oneview-sdk-java

Java SDK for HPE OneView
Java
16
star
55

ilo-sdk-ruby

iLO Software Development Kit for Ruby Programmers.
Ruby
15
star
56

criu-pmem

C
15
star
57

oneview-redfish-toolkit

HPE OneView Redfish Toolkit provides a REST service to answer DMTF's Redfish compliant requests by querying HPE OneView
Python
15
star
58

javascript-ilorest-library

JavaScript
15
star
59

mpgc

Multi-Process Garbage Collector
C++
15
star
60

logger

A log parsing engine written in Java for high performance. Additionally, logger is also highly configurable.
Java
14
star
61

pcp_exporter

Exporter for Performance CoPilot Metrics
Go
14
star
62

Aruba-FlaskwithNetworking

Python
14
star
63

zabbix-plugins-hpeilo

Shell
13
star
64

hpe-oneview-hubot

HPE OneView Chatbot Integration
JavaScript
13
star
65

osfci

Go
13
star
66

prometheus-parser-rs

a Rust library for parsing and validating Prometheus query expressions
Rust
12
star
67

nagios-hpeilo-restful-extension

Nagios Plug-in for iLO RESTful Extension
Shell
12
star
68

c-spiffe

C
12
star
69

LSGI

Large-Scale Graph Inference
HTML
12
star
70

oneview-powershell-samples

HPE OneView PowerShell sample scripts
PowerShell
12
star
71

oneview-sdk-ruby

This project is no longer being developed and has limited support. In the near future this repository will be fully deprecated. Please consider using other OneView projects, such as Golang and Python.
Ruby
12
star
72

zing-stats

Tool for generating summary stat reports and graphs from Gerrit (https://www.gerritcodereview.com/) and GitHub Enterprise review and pull requests data using https://plot.ly/.
Python
12
star
73

py-spiffe

Python library for SPIFFE support
Python
11
star
74

Jenkins-stats

Tool for gathering job data from the Jenkins CI system (https://jenkins-ci.org/) and generating summary stat reports and graphs. These reports cover metrics such as success/failure rates and job duration.
Python
11
star
75

image-streamer-tools

PowerShell
10
star
76

OpenShift-on-SimpliVity

10
star
77

lsrrb

Linux Software Raid Redundant Boot
Python
10
star
78

Docker-Synergy

Shell
10
star
79

shoveller

Rust
10
star
80

hpecli

Command-line interface for HPE products and services
Go
10
star
81

hpe3par_pstoolkit

The HPE Alletra 9000 and Primera and 3PAR PowerShell Toolkit supports cmdlets, which are wrappers around the native HPE Alletra 9000 or HPE Primera or HPE 3PAR storage CLI commands and Web Services API (WSAPI).
PowerShell
10
star
82

image-streamer-esxi

9
star
83

chef-ilorest-cookbook

Chef cookbook for installing the Python iLOrest library and showcasing a few examples.
Python
9
star
84

GlobalDashboardPS

A Powershell module for working with HPE OneView Global Dashboard
PowerShell
9
star
85

hpe-notebooks

Jupyter Notebook
9
star
86

hpe3par_python_sdk

HPE Alletra 9000 and HPE Primera and HPE 3PAR Software Development Kit for Python
Python
9
star
87

RiscVEdk2

C
9
star
88

HPEDSCC-PowerShell-Toolkit

A PowerShell Toolkit that enables management of a HPE GreenLake DSCC (Data Storage Cloud Console) environment via easy to use PowerShell commands which interface with the publically available RestAPI.
PowerShell
9
star
89

gull

a multi-node fabric-attached memory manager that provides simple abstractions for accessing and allocating NVM from fabric-attached memory
C++
9
star
90

iLOAmpPack-Redfish-API-Docs

iLO Amplifier Pack Redfish API Documentation
JavaScript
8
star
91

python-opsramp

Python binding for the OpsRamp API
Python
8
star
92

ironic-driver-oneview

HPE OneView driver for OpenStack Ironic bare-metal machine provisioning
8
star
93

simplivity-ansible

Ansible Modules for HPE SimpliVity
Python
8
star
94

hpe-nvm

Documentation and utilities for NVDIMM-N device management and support
8
star
95

oneview-puppet

This project is no longer being developed and has limited support. In the near future this repository will be fully deprecated. Please consider using other OneView projects, such as Terraform and Ansible Collection
Ruby
8
star
96

SHARP

Python
7
star
97

devid-provisioning-tool

Go
7
star
98

oneview-ansible-samples

A collection of sample code for oneview-ansible
7
star
99

dc-rl

HTML
7
star
100

simplivity-python

This library provides a Python interface to the HPE SimpliVity REST APIs
Python
7
star