keijiro/UnityMLStableDiffusion

Stars
469
Rank 93,595 (Top 2 %)
Language
C#
License
Other
Created over 1 year ago
Updated over 1 year ago

keijiro/UnityMLStableDiffusion

keijiro

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Core ML Stable Diffusion on Unity

Unity Core ML Stable Diffusion Plugin

Stable Diffusion plugin for Unity, based on Apple's Core ML port. You can run the model on-editor and at-runtime without needing any extra components.

System Requirements

Unity 2023.1 or later
Apple Silicon Mac (editor/runtime support) with macOS 13.1 or later
iPad Pro with Apple silicon (runtime support) with iOS 16.2 or later

Although the plugin supports iOS, it requires huge amount of memory to run the model, so it only supports memory-rich iPad models.

How To Try

Before running the sample project, you must put the model files in the Assets/StreamingAssets directory.

Clone or download the pre-converted Stable Diffusion 2 model repository.
Copy the split_einsum/compiled directory into Assets/StreamingAssets.
Rename the directory to StableDiffusion.

It takes a long time (a few minutes) for the first run. After this initialization step, it only takes a few tens of seconds to generate an image.

Performance Considerations

You can change which processing unit it uses by switching the "Compute Units" property in the Tester component.

It depends on the device model to choose the best option:

M1/M2 Mac and iOS: GPUs in those devices are not powerful enough, so you should select "CPU and NE" (Neural Engine) or "All".
M1/M2 Pro/Max Mac: GPUs in those devices have enough processing power compared to NE, so "CPU and GPU" can be a better option.

When using "CPU and GPU" mode, you should use the "original" model instead of the "split_einsum" model. Please overwrite the StreamingAssets/StableDiffusion directory with the original/compiled directory.

Sample Projects

Flipbook3: Running the image-to-image pipeline with a real-time 3D scene.

AICommand

ChatGPT integration with Unity Editor

Skinner

Special Effects with Skinned Mesh in Unity

KinoGlitch

Video glitch effects for Unity

AIShader

ChatGPT-powered shader generator for Unity

Klak

Creative coding library for Unity

Kino

A collection of custom post processing effects for Unity

Voxelman

Unity ECS + C# Job System example

Lasp

Low-latency Audio Signal Processing plugin for Unity

Pcx

Point cloud importer & renderer for Unity

TestbedHDRP

Testbed project for Unity HDRP (High Definition Render Pipeline)

KvantSpray

Object instancing/particle animation system for Unity

ShaderGraphExamples

Simple examples of Unity shader graphs.

NoiseShader

Noise shader library for Unity

Pix2Pix

Real-time pix2pix implementation with Unity

Reaktion

Audio reactive animation toolkit for Unity

Smrvfx

Skinned mesh sampling with VFX Graph

KinoBloom

Bloom effect for Unity

StableFluids

A straightforward GPU implementation of Jos Stam's "Stable Fluids" on Unity.

FFmpegOut

Video capture plugin for Unity with FFmpeg.

KinoFog

Global fog effect for Unity

PuppetTest

An experiment in procedural dance animation

SdfVfxSamples

Unity VFX Graph samples with SDF (signed distance field)

HdrpVatExample

VAT (Vertex Animation Texture) with Unity Shader Graph and Visual Effect Graph

Retro3D

A shader that imitates low grade 3D graphics.

Rsvfx

An example that shows how to connect RealSense depth camera to Unity VFX Graph

KlakNDI

NewTek NDI™ plugin for Unity

MidiJack

MIDI input plugin for Unity

Swarm

An example of use of compute shaders and procedural instancing.

KinoMotion

Motion blur post-processing effect for Unity

ShaderSketches

KodeLife shader sketches I wrote while commuting

KlakSpout

Spout plugin for Unity

Firefly

Unity ECS example for special effects

LaspVfx

Audio reactive Unity VFX with LASP

Akvfx

Azure Kinect plugin for Unity VFX Graph

KvantWig

Non-realistic hair simulation in Unity

UnitySkyboxShaders

Custom skybox shaders

KvantGrass

Animating grass shader for Unity

TimelineParticleControl

An example of controlling particle system from timeline.

MiniEngineAO

SSAO image effect from Microsoft MiniEngine, ported to Unity.

Minis

Minis: MIDI Input for New Input System -- A plugin that adds MIDI input support to Unity's new Input System

SkeletalGeometricEffects

Experiments on geometry shader instancing with skeletal animations

BibcamVfx

AR VFX samples with Bibcam (burnt-in barcode metadata camera)

RippleEffect

Water surface ripple effect for Unity

KinoContour

Contour line filter for Unity

Flipbook

Flip book effect example for Unity

CloudSkybox

Skybox shader with volumetric clouds

OscJack

Lightweight C# implementation of OSC server/client

DepthInverseProjection

An example showing how to inverse-project depth samples into the view/world space in Unity.

HandPoseBarracuda

Hand and finger tracking solution (MediaPipe Hands) for Unity Barracuda

Bibcam

Burnt-in barcode metadata camera

FaceMeshBarracuda

Lightweight facial capture package for Unity

KinoAqua

Watercolor effect for Unity URP/HDRP

Retro3DPipeline

A minimal example of a custom render pipeline with the Retro3D shader.

unity-dither4444

AssetPostProcessor for making high-quality 16-bit color textures in Unity.

SonarFx

Sonar effect for Unity

MidiAnimationTrack

SMF (.mid) file importer for Unity Timeline

SkyboxPlus

Basic skybox extensions for Unity

CosineGradient

Cosine gradient generator for Unity

KinoBokeh

Depth of field effect for Unity

ProceduralMotionTrack

Simple procedural motion with Unity Timeline.

Rcam2

Remote depth camera with iPad Pro with LiDAR sensor

AsyncCaptureTest

Non-blocking screen capture example with asynchronous GPU readback

KvantSwarm

Flowing line renderer for Unity

ShadowDrawer

Shadow matte shader for Unity

StandardGeometryShader

An example of a geometry shader with Unity's standard lighting model support.

KvantStream

A simple GPU particle system for Unity

PerlinNoise

1D/2D/3D Perlin noise function for Unity

GeoVfx

Examples of geographical data visualization with Unity VFX Graph

ComputeMarchingCubes

[Unity] GPU-optimized marching cubes isosurface reconstruction

Metamesh

Unity primitive mesh generator as an asset importer

Ngx

Ngx - Neural network based visual generator and mixer

KvantWall

GPU accelerated object array animator for Unity

VfxGraphTestbed

My testbed for Unity VFX Graph

KlakHap

HAP video player plugin for Unity

StandardTriplanar

Unity's standard shader with triplanar mapping

WfcMaze

WFC (Wave Function Collapse) with Unity

RDSystem

Reaction-diffusion system with CustomRenderTexture.

NoiseBall

Mesh deforming shader example (Unity)

Prisma

Pepper's ghost live visuals for VRDG #4

KvantLattice

A GPU accelerated fractal terrain renderer for Unity

ChromaPack

ChromaPack -- A lossy texture compression plugin for Unity.

Cloner

An example of use of procedural instancing.

VideoPlayerEffects

Experimental special effects for VideoPlayer (Unity 5.6 new feature)

NoiseBall2

A small example of procedural modeling with compute shaders.

PostProcessingUtilities

Utility scripts for the post processing stack

Metatex

Metadata-only texture importer for Unity

ContactShadows

Experimental implementation of contact shadows for Unity.

ColorSuite

Multifunctional color adjustment plugin for Unity.

UnityDxrTest

A testbed project for Unity real-time ray tracing features

NNCam

Virtual background with Unity Barracuda

VfxPyro

Fireworks particle effect with Unity VFX Graph

Dkvfx

An example that shows how to use recorded Depthkit footage with Unity VFX Graph

CrossFadingLod

(Unity) Cross-fading LOD shader example

Boids

Flocking behavior simulation (Unity)

MoonAndEarth

Moon and Earth assets for Unity

KinoBinary

Binary image effect for Unity

UnityAnime4K

Anime4K upscaler for Unity

VideolabTest

OP-Z videolab examples

GVoxelizer

KlakSyphon

Syphon plugin for Unity