SpvGenTwo

SpvGenTwo is a SPIR-V building and parsing library written in C++17, no other dependencies! No STL or other 3rd-Party library needed. The library comes with its own set of SPIR-V definitions generated from the machine readable grammar and therefore does not require any vulkan or spirv-headers includes. The generator can be found here: rustspvgen.

I built this library as a 'slim' backend for runtime material/shader-editors (like Proto) to avoid introducing enormous libraries like DXC (including LLVM and Frontend) or SPIRV-Tools to the codebase.

SpvGenTwo is still under development, many parts are still missing, but all the building blocks are there. SpvGenTwo is designed to be extensible and customizable, it is fairly easy to implement new SPIR-V instructions and extensions, use custom allocators and define own type inference rules. Note that it is still possible to generate invalid SPIR-V modules because not all inputs are checked yet. Use the SPIR-V validator spirv-val from the SDK and have the specification at hand while using this library.

I mainly focused on Shader capabilities, so the Kernel and OpenCL side is a bit under-developed. Any community contributions in that regard are very welcome!

Overview:

• Examples
  • Test Project
• Building
• Tools
  • SPIR-V Disassembler
  • SPIR-V Reflector
  • SPIR-V Linker
• Documentation
• Contributing
• Copyright and Licensing
• Roadmap
• Gallery
• Coverage
  • Core
  • GLSL.std.450
  • OpenCl.std.100

Examples

ConsoleLogger log;
HeapAllocator alloc; // custom user allocator

// create a new spir-v module
Module module(&alloc, &log);

// configure capabilities and extensions
module.addCapability(spv::Capability::Shader);
module.addCapability(spv::Capability::VulkanMemoryModelKHR);
module.addExtension(spv::Extension::SPV_KHR_vulkan_memory_model);
Instruction* ext = module.getExtensionInstructionImport(u8"GLSL.std.450");
module.setMemoryModel(spv::AddressingModel::Logical, spv::MemoryModel::VulkanKHR);

// global variables
Instruction* uniformVar = module.uniform<vector_t<float, 3>>(u8"u_Position");

// float add(float x, float y)
Function& funcAdd = module.addFunction<float, float, float>(u8"add", spv::FunctionControlMask::Const);
{
    BasicBlock& bb = *funcAdd; // get entry block to this function

    Instruction* x = funcAdd.getParameter(0);
    Instruction* y = funcAdd.getParameter(1);

    Instruction* z = bb.Add(x, y);
    bb.returnValue(z);
}

// void entryPoint();
{
    EntryPoint& entry = module.addEntryPoint(spv::ExecutionModel::Fragment, u8"main");
    entry.addExecutionMode(spv::ExecutionMode::OriginUpperLeft);
    BasicBlock& bb = *entry; // get entry block to this function

    Instruction* uniVec = bb->opLoad(uniformVar);
    Instruction* cross = bb.ext<GLSL>()->opCross(uniVec, uniVec); // use GLSL.std.450 extension
    Instruction* s = bb->opDot(cross, uniVec);
    entry->call(&funcAdd, s, s); // call add(s, s)
    entry->opReturn();
}

// custom spir-v binary serializer:
BinaryFileWriter writer("test.spv");
module.finalizeAndWrite(writer);

The resulting SPIR-V binary when disassembled using spirv-dis:

; SPIR-V
; Version: 1.0
; Generator: SpvGenTwo SPIR-V IR Tools(30); 0
; Bound: 20
; Schema: 0
               OpCapability Shader
               OpCapability VulkanMemoryModelKHR
               OpExtension "SPV_KHR_vulkan_memory_model"
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel Logical VulkanKHR
               OpEntryPoint Fragment %main "main" %u_Position
               OpExecutionMode %main OriginUpperLeft
               OpName %u_Position "u_Position"
               OpName %add "add"
               OpName %main "main"
      %float = OpTypeFloat 32
    %v3float = OpTypeVector %float 3
%_ptr_Uniform_v3float = OpTypePointer Uniform %v3float
          %9 = OpTypeFunction %float %float %float
       %void = OpTypeVoid
          %3 = OpTypeFunction %void
 %u_Position = OpVariable %_ptr_Uniform_v3float Uniform
        %add = OpFunction %float Const %9
         %11 = OpFunctionParameter %float
         %12 = OpFunctionParameter %float
         %13 = OpLabel
         %14 = OpFAdd %float %11 %12
               OpReturnValue %14
               OpFunctionEnd
       %main = OpFunction %void None %3
         %15 = OpLabel
         %16 = OpLoad %v3float %u_Position None
         %17 = OpExtInst %v3float %1 Cross %16 %16
         %18 = OpDot %float %17 %16
         %19 = OpFunctionCall %float %add %18 %18
               OpReturn
               OpFunctionEnd

Test Project

Set CMake option SPVGENTWO_BUILD_TESTS to TRUE to build included examples:

• Types
• Constants
• ControlFlow
• FunctionCall
• Extensions
• ExpressionGraph
• Linkage
• FragmentShader
• GeometryShader
• ComputeShader

Project Structure

SpvGenTwo is split into 5 folders:

• lib contains the foundation to generate SPIR-V code. SpvGenTwo makes excessive use of its allocator interface, no memory is allocated from the heap. SpvGenTwo comes with its on set of container classes: List, Vector, String and HashMap. Those are not built for performance, but they shouldn't be much worse than standard implementations (okay maybe my HashMap is not as fast as unordered_map, build times are quite nice though :). Everything within this folders is pure C++17, no other dependencies (given that SPVGENTWO_REPLACE_PLACEMENTNEW and SPVGENTWO_REPLACE_TRAITS are used).
• common contains some convenience implementations of abstract interfaces: HeapAllocator uses C malloc and free, BindaryFileWriter uses fopen, ConsoleLogger uses vprintf, ModulePrinter uses snprintf. It also has some additional classes like Callable (std::function replacement), Graph, ControlFlowGraph, Expression and ExprGraph, they follow the same design principles and might sooner or later be moved to lib if needed.
• test contains small, self-contained code snippets that each generate a SPIR-V module to show some of the fundamental mechanics and APIs of SpvGenTwo.
• dis is a SPIR-V disassembler tool like spirv-dis to print assembly language text.
• refl is a SPIR-V reflection tool like SPIRV-Reflect to extract descriptor bindings and other relevant info from SPIR-V binary modules.
• link is a SPIR-V linker like spirv-link for merging symbols of multiple modules into a new output module.
• test contains Catch2 unit tests. When using Visual Studio, Test Adapter for Catch2 can be used with the test/catch2.runsettings config file.

Building

Use the supplied CMakeLists.txt to generate project files for your build system. SpvGenTwo allows the user to use standard library headers (<type_traits>, <new> etc) instead of my hand-made replacements (see stdreplament.h).

• SPVGENTWO_BUILD_TESTS is set to FALSE by default. If TRUE, an executable with sources from the 'test' folder will be built.
  • Note that the SpvGenTwoTest executable project depends on SPIR-V Tools which requires python to be present for its tests (even when building without tests facepalm)
• SPVGENTWO_BUILD_DISASSEMBLER is set to FALSE by default. If TRUE, an executable with sources from the 'dis' folder will be built.
• SPVGENTWO_BUILD_REFLECT is set to FALSE by default. If TRUE, an executable with sources from the 'refl' folder will be built.
• SPVGENTWO_BUILD_LINKER is set to FALSE by default. If TRUE, an executable with sources from the 'link' folder will be built.
• SPVGENTWO_REPLACE_PLACEMENTNEW is set to TRUE by default. If FALSE, placement-new will be included from <new> header.
• SPVGENTWO_REPLACE_TRAITS is set to TRUE by default. If FALSE, <type_traits> and <utility> header will be included under spvgentwo::stdrep namespace.
• SPVGENTWO_LOGGING is set to TRUE by default, calls to module.log() will have not effect if FALSE.
• SPVGENTWO_ENABLE_WARNINGS is set to TRUE by default and will enable most pedantic warnings-as-errors for all targets except the tests.
• SPVGENTWO_ENABLE_OPERANDVALIDATION is set to TRUE by default and enables an additional validation step for every makeOp/opXZY call which is not necessary for non-development builds.

Note that I mainly develop on Windows using Clang and MSVC but I'll also try to support GCC/linux. I don't have any Apple hardware so I can't debug any issues there, but you are welcome to contribute fixes for this platform.

Tools

SpvGenTwo includes a couple of CLI tools to explore and test the libraries capabilities.

SPIR-V Disassembler

SpvGenTwoDisassembler source can be found at dis/source/dis.cpp, it is just a single source file demonstrating the basic parsing and IR walking capabilities of SpvGenTwo.

CLI: SpvGenTwoDisassembler [file] <option> <option> ...

Options

• -assignids re-assigns instruction result IDs starting from 1. Some SPIR-V compilers emit IDs in a very high range, making it hard to read and trace data flow in assembly language text, assignIDs helps with that.
• -serialize writes the parsed SPIR-V program to a serialized.spv file in the working directory (this is a debug feature).
• -noinstrnames don't replace result IDs with OpNames
• -noopnames don't replace operand IDs with OpNames
• -nopreamble don't print SPIR-V preamble
• -colors use ANSI color codes
• -tabs " " example: use 4 spaces instead of 2 tabs

SPIR-V Reflector

SpvGenTwoReflect source can be found at refl/source/refl.cpp

CLI: SpvGenTwoReflect [file] <option> <option> ...

Options

• -var name select variable by name (if OpVariable was annotated by OpName) for DescriptorType & Decoration printing (name has to be a UTF-8 string)
  • -var MyBuffer
• -deco decoration select decoration to query for in the module
  • -deco print all decorations in the module
  • -deco DescriptorSet to print only DescriptorSets
  • -deco DescriptorSet -deco Binding to print only DescriptorSets & Bindings
• -funcs list functions names in the module
• -vars list global variables in the module (StorageClass != Function)
• -types list types and constatns in the module
• -id Id print SPIR-V assembly text for the instruction with result Id
  • -id 24
• -colors use ANSI color codes

SPIR-V Linker

SPIR-V Shader library linker and patcher. See SpvGenTwoLinker for detailed description. Source can be found at refl/source/link.cpp.

Documentation

Please read the documentation for more detailed information on how to use SpvGenTwo and some reasoning about my design choices.

Contributing

Please follow the "fork-and-pull" Git workflow:

• Fork the repo on GitHub
• Clone the project to your own machine
• Commit changes to your own branch
• Push your work back up to your fork
• Submit a Pull request so that we can review your changes

Make sure to merge the latest from "upstream" before making a pull request.

Copyright and Licensing

The SpvGenTwo open source project is licensed under MIT license. Any contribution you make to this original repository shall be licensed under same license. You are still free to distribute your contributions (in your own fork) under the license you desire.

Roadmap

A list of short and long term goals for this library:

• Implement more Instructions, at least 90% of Shader capabilities
• Improve validation / logging / error reporting
• Write more unit tests
• Implement SPIRV-Tools like helper tools
• Implement some front-end or DSL like SPEAR

Gallery

SpvGenTwo is used in:

• Proto

• SHADERed

• TanTien-Engine (shadertrans)

Leave PR with your software if you want it to be added here :)

Coverage

SPIR-V IR generation progress, parsing is independent and auto generated. This table indicates whether the operation is already implemented in the Instruction class or can be generated by other facilities. This is not a complete list, if an entry is missing, assume it has not been implemented yet. You can file an Issue to request its implementation or just use Instruction::makeOp(...) directly.

Core

GLSL.std.450

OpenCl.std.100

Not implemented yet