ucb-bar/riscv-torture

===========================================================================
RISC-V Torture Test Generator
===========================================================================
# Author: Yunsup Lee and Henry Cook
# Date: January 29th, 2012
# Version: (under version control)

This is the RISC-V torture test generator and framework. This repository
contains three sub-projects that build upon one another. The first, 
[generator], is used to create a single random torture test. The second,
[testrun], is used to run a particular test on particular simulators,
diffing the resulting signature with the ISA simulator and optionally
creating a derivative test subset that pinpoints the divergence. The third,
[overnight], wraps testrun, allowing tests to be run repeatedly for a given
duration or until a failure count.

 
---------------------------------------------------------------------------
Instructions
---------------------------------------------------------------------------

Modify "config/default.config" to set the parameters desired for building tests
(e.g., setting which instructions to use and in which ratio).

Modify "Makefile" as desired to execute the C simulator or RTL simulator of
your choice, and to set the other parameters as you require.

To build a single test and test it on Spike:

$ make igentest

To build single test and run it on the C simulator or RTL simulator, use 
"make cgentest" or "make rgentest".

To run overnight tests, you can use "make cnight" and "make rnight".
   
---------------------------------------------------------------------------
Signatures
---------------------------------------------------------------------------

Torture works by dumping the register state out to memory at the end of the
test program execution. This output is then compared against the output from
the Spike ISA simulator.

The torture program writes the register state to the memory address specified
by "xreg_output_data", which is located in the memory section 
".global begin_signature". The Spike ISA simulator will write out the data
found in the "begin_signature" section on exit if provided with the
"+signature=" argument:

$ spike +signature=my_spike_signature.txt test_binary

The Rocket-chip infrastructure uses the "riscv-fesvr" program to control the
execution of the C and RTL simulators. The "riscv-fesvr" also accepts the
+signature argument too.

$ ./csim-rocket-chip +signature=my_rocket_signature.txt test_binary

A simple diff between the Spike and chip simulator signatures will tell you if
any errors have occurred. 

$ diff my_spike_signature.txt my_rocket_signature.txt


**PORTING TORTURE TO YOUR OWN RISC-V PROCESSOR:**

If you would like to use riscv-torture with your own RISC-V processor, you will
need to provide a way to dump the "begin_signature" section to a file.

---------------------------------------------------------------------------
Low-level Usage
---------------------------------------------------------------------------
  
Some basic use cases are illustrated here (note the Makefile abstracts this for
you).

Make a single test:
% ./sbt generator/run
% cd output
% make
% spike +signature=test.sig test

Take an existing test and diff the signatures of ISA and C simulators:
% ./sbt 'testrun/run -a output/test.S -c /path/to/reference-chip/emulator/emulator'

*** Currently, due to the limiation of scala process library, you cannot
torture the RTL simulator ***
# Generate a random test and diff the signatures of ISA and RTL simulators:
# % ./sbt 'testrun/run -r /path/to/reference-chip/vlsi/build/vcs-sim-rtl/simv'

Run tests for 30 minutes, email hcook when done, and save failures to dir:
% ./sbt 'overnight/run -m 30 -e [email protected] -p dir'

---------------------------------------------------------------------------
Installing
---------------------------------------------------------------------------

% git submodule update --init


---------------------------------------------------------------------------
Overnight Overview
---------------------------------------------------------------------------

This framework utilizes both the test runner and test generator to perform
a long terms serach for failing test cases. It takes the following command
line arguments:

Usage: overnight/run [options] 
  -C <file> | --config <file>
        config file
  -p <dir> | --permdir <dir>
        dir to store failing tests
  -c <file> | --csim <file>
        C simulator
  -r <file> | --rtlsim <file>
        RTL simulator
  -e <address> | --email <address>
        email to report to
  -t <count> | --threshold <count>
        number of failures to trigger email
  -m <int> | --minutes <int>
        number of minutes to run tests

You can only generate tests with one instruction mix at a time (based on
the setting in the config file). It doesn't matter what simulator you use
with the -r and -c flags, they just determines the name used to describe
whose diff failed.

---------------------------------------------------------------------------
Testrun Overview
---------------------------------------------------------------------------

This utility compares the signatures generated by passing the -testsig flag
to the specified simulators. If it encounters a difference, it subdivides
the test into many subtests and searches for which exact program segment
reveals the failure. It takes the following command line arguments:

Usage: testrun/run [options] 
 -C <file> | --config <file>
        config file
  -a <file> | --asm <file>
        input ASM file
  -c <file> | --csim <file>
        C simulator
  -r <file> | --rtlsim <file>
        RTL simulator
  -s <boolean> | --seek <boolean>
        Seek for failing pseg
  -d <boolean> | --dump <boolean>
        Dump mismatched signatures

If you don't specify a asm file, a random one will be generated for you.
You can only generate tests with one instruction mix at a time (based on
the setting in the config file). It doesn't matter what simulator you use
with the -r and -c flags, they just determines the name used to describe
whose diff failed. By default, a failed diff will result in the subtest 
sweep occuring, but this search can be diasbled. Note that the pseg ID
reported is actually the pseg following the pseg containing the error.
You can optionally dump mistmatched signatures to the dir containing the 
asm file under test.

---------------------------------------------------------------------------
Generator Overview
---------------------------------------------------------------------------

To generate a random test, the torture test generator randomly generates
many test sequences from a set of test sequences that are written by hand,
performs a random register allocation for all test sequences, and finally
randomly interleaves instructions from these test sequences. To extend the
set of tests or coverage, the programmer needs to write new test sequences.
It takes the following command line arguments:

Usage: generator/run [options] 
  -o <filename> | --output <filename>
        output filename
  -C <file> | --config <file>
        config file

The following sections describe adding new functionality to the generator.

---------------------------------------------------------------------------
Test sequence example
---------------------------------------------------------------------------

Before we talk about how to write a test sequence, let's look at a very
simple example. The following example is a test sequence, which emits an
add instruction.

class SeqADD extends Seq
{
  val src1 = reg_read_any()
  val src2 = reg_read_any()
  val dest = reg_write(src1, src2)
  insts += ADD(dest, src1, src2)
}

As I hinted in the overview that the test generator will do register
allocation you don't write a string of instructions with architectural
registers. You request a virtual registers (i.e., registers that are yet
tied down to architectural registers) when you need them, save them in
scala values, and use them when you need to (e.g., in an instruction).

---------------------------------------------------------------------------
Types of virtual registers
---------------------------------------------------------------------------

 - Hidden (position dependent registers): Registers that will have
   different values when the code is positioned at a different address. An
   example is registers that hold addresses. Registers that are hidden should
   be excluded from the output signature.

 - Visible (position independent registers): Registers that are not hidden,
   therefore will have the same values when the code is positioned at
   a different address. These registers should be included as part of the
   output signature.

---------------------------------------------------------------------------
How to write a sequence
---------------------------------------------------------------------------

Use the following functions to request a register, and generate a string of
instructions (look at Inst.scala to see what instructions are available)
that uses these virtual registers, and add them to the insts array.

 - reg_read_zero(): returns register x0
 - reg_read_any(): returns any type of register (hidden or visible)
 - reg_read_visible(): returns a visible register
 - reg_write_ra(): returns register ra for write
 - reg_write_visible(): returns a visible register for write
 - reg_write_hidden(): returns a hidden register for write
 - reg_write(regs: Reg*): returns a register that matches the type of regs
   (if any reg in regs are hidden, the output type is hidden)

Note that the torture test framework is written in scala, you can use any
scala functionality to generate instructions. Look at SeqALU.scala,
SeqMem.scala, and SeqBranch.scala to get inspired.

---------------------------------------------------------------------------
Future TODO
---------------------------------------------------------------------------

 - provide support for loops

 - generate statistics of a test to get a sense of coverage
   + statistics should include instruction count of each type
   + statistics should include register usage

 - complete floating point tests
   + add floating point memory move tests
   + improve floating point init randomization
   + add rounding modes tests

 - complete vector tests
   + better randomization
   + add SeqVOnly: Tests special vf-only instructions

 - code refactoring
   + consolidate RegPool logic
   + detect and suppress unallocatable sequences