• Stars
    star
    117
  • Rank 301,828 (Top 6 %)
  • Language
    C
  • Created about 13 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

KVM driver for illumos

illumos-kvm: KVM for illumos

KVM is the kernel virtual machine, a framework for the in-kernel acceleration of QEMU. illumos-kvm is a port of KVM to illumos, taking advantage of illumos-specific constructs like DTrace, cyclics, mdb, kstat, OS virtualization, network virtualization, ZFS, etc. It is derived from the KVM source for Linux 2.6.34, the longterm source for which may be found here:

git://git.kernel.org/pub/scm/linux/kernel/git/longterm/linux-2.6.34.y.git

To date, this implementation has been verified with a wide range of guest operating systems including illumos itself (both SmartOS and OpenIndiana distributions), FreeBSD, Plan 9, QNX, ChromeOS, HaikuOS, Microsoft Windows and Linux.

The design center for this work is to use the virtualization features made available in the microprocessor -- and in particular, Intel's VMX. As such, behavior on microprocessors that do not support VMX -- and more specifically, the extended page tables (EPT) found in second generation VMX support -- should be graceful failure, not degraded operation.

Divergences from KVM

Divergences from KVM fall into several broad categories: some functionality has been removed or not implemented because it is obviated by features of illumos (e.g., the custom tracing facility built into KVM); some functionality has been removed because it is only relevant to hardware that lacks virtualization support (e.g., older x86 hardware) or on hardware for which illumos lacks support (e.g., PPC, s390); and some functionality has been removed because the implementation complexity was simply too great relative to its value.

Of this latter category, three areas of divergence merit special note. First, there is no support for pageable guest memory (that is, guest memory is locked down). While this is an opinionated decision at some level (in our experience, memory oversell leads to unacceptable pathologies in all but the idlest of workloads), we would welcome the work to integrate the KVM MMU notifier support into illumos-kvm.

Second (and relatedly), illumos itself has no support for kernel same-page mapping (KSM) as found in Linux. While illumos could in principle add such support, it is our experience that the memory that accrues from this is not sufficiently significant to pay for the increase in implementation and operator complexity.

Finally, there is no support currently for AMD SVM. This is not a value judgement of AMD's technology, but rather a reflection of limited engineering and testing resources. (In the spirit of full disclosure, it should be said that the sponsor of illumos-kvm, Joyent, is an Intel-funded company -- but the lack of AMD support reflects only engineering prioritization and lack of testing infrastructure; AMD SVM support would be most welcome should someone in the community be so motivated as to port and test it.)

Building illumos-kvm

Preparation

Edit the Makefile and appropriately set the path for the KERNEL_SOURCE directory to point to the root of a checked out and built illumos directory. Building illumos KVM requires several recent additions to illumos, so be sure your illumos is up to date.

Verify that you have gcc 4.4.4 installed that is used to build illumos. If you are not building this with SmartOS, you may need to modify the Makefile such that CC is pointing to the correct gcc.

Verify that you either have SUNWmake or GNU make installed.

Building

To build, simply use the default make target:

$ make

To check style, header files, and other various nits:

$ make check

Installing illumos-kvm

System requirements

To run illumos-kvm, you will need an illumos that has the fix for issue 1347 (integrated on 2011-08-11). Further, your machine will need to support VMX. To see if your machine supports VMX, run isainfo -v and look for vmx, e.g.:

  % isainfo -v
  64-bit amd64 applications
        vmx sse4.2 sse4.1 ssse3 popcnt tscp cx16 sse3 sse2 sse fxsr mmx 
        cmov amd_sysc cx8 tsc fpu 
  32-bit i386 applications
        vmx sse4.2 sse4.1 ssse3 popcnt tscp ahf cx16 sse3 sse2 sse fxsr mmx 
        cmov sep cx8 tsc fpu 

If you do not see vmx in this output, the kvm driver will be unable to attach.

Required binaries

There are two mandatory artifacts to install, and two optional component:

  • kvm is the driver itself
  • kvm.conf is the driver configuration file
  • kvm.so is the mdb module
  • JOY_kvm_link.so is the devfsadm plugin

On the target machine, place kvm in /kernel/drv/amd64 and kvm.conf in /kernel/drv. Place JOY_kvm_link.so in /usr/lib/devfsadm/linkmod then:

# add_drv kvm

You can verify that the driver installed and attached properly by checking for its presence in /dev.

# ls -l /dev/kvm

Running illumos-kvm

To run KVM, you will need the build product of the illumos-kvm-cmd repo: qemu-system-x86_64; please follow the instructions in the illumos-kvm-cmd repo to execute QEMU such that KVM is enabled.

Monitoring illumos-kvm

Once one or more VMs are running, there is a variety of tooling to help understand the operating characteristics of the system.

kvmstat

The kvmstat command, found in the illumos repository, can be used to monitor VMs. For example, here is one second of kvmstat output from a machine running two VMs (one 2 VCPU instance running Linux; another 4 VCPU instance running the illumos-derived SmartOS):

   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
  4668    0 |     23 :      6      0      0      1      0 |      6     16      0
  4668    1 |     25 :      6      1      0      1      0 |      6     16      0
  5026    0 |  17833 :    223   2946    707    106      0 |   3379  13315      0
  5026    1 |  18687 :    244   2761    512      0      0 |   3085  14803      0
  5026    2 |  15696 :    194   3452    542      0      0 |   3568  11230      0
  5026    3 |  16822 :    244   2817    487      0      0 |   3100  12963      0

As for the meaning of the columns, they are explained with kvmstat -h:

  # kvmstat -h
  Usage: kvmstat [interval [count]]

    Displays statistics for running kernel virtual machines, with one line
    per virtual CPU.  All statistics are reported as per-second rates.

    The columns are as follows:

      pid    =>  identifier of process controlling the virtual CPU
      vcpu   =>  virtual CPU identifier relative to its virtual machine
      exits  =>  virtual machine exits for the virtual CPU
      haltx  =>  virtual machine exits due to the HLT instruction
      irqx   =>  virtual machine exits due to a pending external interrupt
      irqwx  =>  virtual machine exits due to an open interrupt window
      iox    =>  virtual machine exits due to an I/O instruction
      mmiox  =>  virtual machine exits due to memory mapped I/O 
      irqs   =>  interrupts injected into the virtual CPU
      emul   =>  instructions emulated in the kernel
      eptv   =>  extended page table violations

kstat

As one might expect, kvmstat is implemented in terms of kstat. You can use kstat(1) to browse the kstats from the kvm module:

   # kstat -m kvm
   ...
   module: kvm                      instance: 0     
   name:   vcpu-4                   class:    misc
    crtime                          4407.142410068
    exits                           5367443
    fpu-reload                      57302
    halt-exits                      317275
    halt-wakeup                     8991
    host-state-reload               503920
    hypercalls                      0
    insn-emulation                  3043881
    inst-emulation-fail             0
    invlpg                          0
    io-exits                        237191
    irq-exits                       1668
    irq-injections                  320339
    irq-window-exits                1635
    mmio-exits                      617
    nmi-injections                  0
    nmi-window-exits                0
    pf-fixed                        163629
    pf-guest                        0
    pid                             3949
    request-irq-exits               0
    signal-exits                    460
    snaptime                        43219.723435123
    zonename                        global
   
   module: kvm                      instance: 4     
   name:   vm                       class:    misc
    crtime                          4407.1241134
    lpages                          0
    mmu-cache-miss                  950
    mmu-flooded                     0
    mmu-pte-updated                 0
    mmu-pte-write                   56360
    mmu-pte-zapped                  0
    mmu-recycled                    0
    mmu-unsync-page                 0
    pid                             3949
    remote-tlb-flush                1511
    snaptime                        43219.723875091
    zonename                        global

DTrace

While there is not currently a stable KVM provider, there are many SDT probes in KVM; dtrace -l -m sdt:kvm to list these.

Of these, of particular note are the kvm-guest-entry and kvm-guest-exit probes, which fire upon entry to and exit from a guest virtual machine. To determine context, one can use the vmregs variable present in illumos.

For example, here's a simple script that shows histograms of time spent in VM guests on a per-PID and per-VCPU basis:

    #pragma D option quiet
    
    kvm-guest-entry
    {
            self->entry = timestamp;
    }
    
    kvm-guest-exit
    /self->entry/
    {
            @[pid, vmregs[VMX_VIRTUAL_PROCESSOR_ID]] =
                quantize(timestamp - self->entry);
    }
    
    END
    {
            printa("pid %d, vcpu %d: %@d\n", @);
    }

Here's what the output of running the above might look like:

  pid 3949, vcpu 1: 
       value  ------------- Distribution ------------- count    
         512 |                                         0        
        1024 |@@@@@@@@@@@@@                            26805    
        2048 |@@@@@                                    11641    
        4096 |@@@@@@@                                  14187    
        8192 |@                                        1559     
       16384 |@                                        2931     
       32768 |@@@                                      5653     
       65536 |@@@@                                     8385     
      131072 |@@@                                      6926     
      262144 |@@@                                      6639     
      524288 |                                         785      
     1048576 |                                         0        

There are many other ways in which DTrace can be used to understand either host or guest behavior; see the tools subdirectory from some sample D scripts.

mdb

The kvm.so build product is an mdb module that contains several useful commands, including a kvm walker to iterate over all struct kvm structures.

Contributing to illumos-kvm

Unless and until its volume dictate that it be elsewhere, illumos KVM discussion should be on the illumos-developer mailing list. Contributions are happily accepted; please send patches to illumos-developer.

More Repositories

1

libuv

Go to
C
3,271
star
2

smartos-live

For more information, please see http://smartos.org/ For any questions that aren't answered there, please join the SmartOS discussion list: http://smartos.org/smartos-mailing-list/
C
1,437
star
3

triton

Joyent Triton DataCenter: a cloud management platform with first class support for containers.
Shell
1,202
star
4

node-verror

Rich JavaScript errors
JavaScript
1,125
star
5

containerpilot

A service for autodiscovery and configuration of applications running in containers
Go
1,104
star
6

manta

Manta is a scalable HTTP-based object store
Makefile
565
star
7

node-workflow

Task orchestration, creation and running using NodeJS
JavaScript
445
star
8

node-http-signature

Reference implementation of Joyent's HTTP Signature Scheme
JavaScript
392
star
9

node-stackvis

Stacktrace visualization tools
JavaScript
340
star
10

node-vasync

utilities for observable asynchronous control flow
JavaScript
315
star
11

v8plus

Node.js native add-ons in C
C++
265
star
12

rfd

Requests for Discussion
Roff
251
star
13

mdb_v8

postmortem debugging for Node.js and other V8-based programs
C
235
star
14

manatee

Automated fault monitoring and leader-election system for strongly-consistent, highly-available writes to PostgreSQL (Joyent SDC, Manta).
JavaScript
228
star
15

statemap

Software for rendering statemaps
Rust
219
star
16

restdown

Pretty REST API docs authored in Markdown
Python
203
star
17

sdc-docker

Docker Engine for Triton
JavaScript
182
star
18

triton-kubernetes

Kubernetes on Triton
Go
174
star
19

node-sshpk

Parse, convert, fingerprint and use SSH keys in pure node.js
JavaScript
159
star
20

nodejs-advisory-board

Meeting Minutes and Working Group Discussions
158
star
21

nhttpsnoop

Trace Node.js HTTP server activity
Shell
138
star
22

pgsqlstat

report top-level postgres stats
Shell
129
star
23

node-panic

Postmortem debugging facility for Node.js
JavaScript
120
star
24

node-assert-plus

Extra assertions on top of node's assert module
JavaScript
119
star
25

node-snmpjs

SNMP toolkit for Node.js
JavaScript
111
star
26

node-ctype

Read and write binary structures with node
JavaScript
89
star
27

node-manta

Node.js SDK for Manta
JavaScript
75
star
28

node-bunyan-syslog

Syslog Stream for node-bunyan
JavaScript
68
star
29

illumos-kvm-cmd

qemu-kvm for illumos-kvm
C
65
star
30

node-watershed

Simple WebSockets Client/Server (RFC6455)
Makefile
65
star
31

node-smartdc

Client SDK and CLI for the Joyent SmartDataCenter API
JavaScript
63
star
32

mi-centos-7

Shell
63
star
33

node-asn1

Contains parsers and serializers for ASN.1 (currently BER only)
AGS Script
61
star
34

smartos_cookbooks

Chef Cookbooks for managing the SmartOS Global Zone
JavaScript
58
star
35

moray

Moray, the highly-available key/value store (Joyent Triton, Manta)
JavaScript
58
star
36

node-vstream

instrumented streams
JavaScript
56
star
37

node-triton

Triton client tool and node.js library
JavaScript
55
star
38

node-docker-registry-client

node.js client for the docker registry
JavaScript
55
star
39

kang

Introspection for distributed systems
JavaScript
49
star
40

smfgen

Generate SMF manifests from a JSON description
JavaScript
49
star
41

jsstyle

cstyle-based JavaScript style checker
Perl
49
star
42

node-debug-school

nodeschool curriculum for debugging Node.js
JavaScript
49
star
43

node-getopt

POSIX-style getopt() for Node.js
JavaScript
47
star
44

dtruss-osx

Shell
43
star
45

node-ip6addr

IPv6/IPv4 address parsing and manipulation for node.js
JavaScript
43
star
46

pg_prefaulter

Faults pages into PostgreSQL shared_buffers or filesystem caches in advance of WAL apply
Go
43
star
47

node-camp

Asynchronous IO ...camp
JavaScript
43
star
48

manatee-state-machine

design ideas for manatee
JavaScript
42
star
49

node-docker-file-parser

Parses a dockerfile contents string and returns the array of docker commands
JavaScript
42
star
50

smartos-vmtools

Shell
40
star
51

illumos-extra

Extra non-ON software required for Illumos
C
39
star
52

sdc-nfs

user-level NFS server written in node.js
JavaScript
35
star
53

node-extsprintf

Extended POSIX-style sprintf
JavaScript
34
star
54

node-kstat

A node.js addon for reading illumos kstats
Perl
32
star
55

node-jsprim

utilities for primitive JavaScript types
JavaScript
32
star
56

knife-joyent

Opscode Chef knife plug-in for Joyent CloudAPI
Ruby
32
star
57

eng

Joyent Engineering Guide
JavaScript
31
star
58

pkgsrc-joyent

Various pkgsrc packages used by Joyent, not committed upstream yet
Makefile
31
star
59

smartos-overlay

Overlay directory specific to open-source SmartOS
30
star
60

node-fast

streaming JSON RPC over TCP
JavaScript
29
star
61

convertvm

convert OVF vm packages to smartos compatible images
JavaScript
29
star
62

minecrab

Minecraft on Joyent's Cloud & Manta on Demand
Shell
28
star
63

cloud-perf-labs

Student labs for Cloud Performance training
C
28
star
64

node-consulite

Tiny consul Node.js module for client discovery
JavaScript
28
star
65

node-piloted

Service discovery in node using ContainerPilot
JavaScript
27
star
66

node-in-the-industry

This is the script that used to generate fresh "node in the industry" content. It is no longer being maintained. See: https://github.com/nodejs/nodejs.org.
HTML
27
star
67

mi-freebsd-10

Custom FreeBSD 10 ISO builder
Shell
26
star
68

javascriptlint

JavaScript Lint
C
25
star
69

binder

Triton/Manta DNS server over Apache Zookeeper
JavaScript
25
star
70

node-tracing

User definable tracing API
JavaScript
25
star
71

python-manta

Python SDK for Manta (community maintained)
Python
24
star
72

manufacturing

Manufacturing specifications
Python
24
star
73

pglockanalyze

analyze postgres locking behavior
Makefile
23
star
74

sdcboot

SDC FDUM environment
C
23
star
75

pkgsrc-wip

Conversion of the pkgsrc-wip CVS project
Makefile
23
star
76

conch-api

Datacenter build and management service
Perl
22
star
77

node-tab

Unix-style tables for command-line utilities
Makefile
22
star
78

triton-go

Go SDK for Joyent Triton (Compute) and Triton Object Storage (Manta)
Go
21
star
79

node-spawn-async

spawn child processes asynchronously
JavaScript
19
star
80

smartmachine_cookbooks

Chef Cookbooks for managing SmartOS SmartMachines
19
star
81

syslinux

replica of syslinux repo from git://git.kernel.org/pub/scm/boot/syslinux/syslinux.git
C
19
star
82

manta-nfs

NFSv3 Manta Storage Server Gateway
JavaScript
19
star
83

daggr

filter and aggregate numeric data in plaintext or json form
JavaScript
18
star
84

mod_usdt

DTrace provider for Apache
D
18
star
85

freebsd-vpc

Control plane for `projects/VPC` branch of `joyent/freebsd`
Go
18
star
86

mibe

Machine Image Build Environment
PHP
17
star
87

node-zfs

Node.js library to interface with ZFS utilities
JavaScript
17
star
88

ruby-manta

Ruby interface for Joyent's Manta service
Ruby
17
star
89

pgstatsmon

Node.js service for shoveling Postgres stats into Prometheus
JavaScript
17
star
90

tsg-infrastructure

Shell
17
star
91

java-manta

Java Manta Client SDK
Java
16
star
92

manta-thoth

Thoth is a Manta-based system for core and crash dump management
JavaScript
16
star
93

node-nfs

Node.js SDK for writing Portmap/Mount/NFS (v3) servers
JavaScript
16
star
94

triton-terraform

16
star
95

java-http-signature

Library for performing RSA signed HTTP requests in Java
Java
16
star
96

summit-workshop

Node.js Summit - Day Zero Workshop
JavaScript
16
star
97

sdc-adminui

Operator portal for SmartDataCenter
JavaScript
15
star
98

sdc-headnode

Responsible for building and setting up the Triton (formerly SmartDataCenter) headnode.
JavaScript
15
star
99

openbsd-kvm-image-builder

Scripts to create a custom OpenBSD install ISO and a KVM image for use in SmartOS and Triton.
Shell
15
star
100

ipxe

C
14
star