VTuber-MomoseHiyori
Recently, Kennard did some work on Deep Learning and Computer Vision. At the same time, he realized that he could make a Live2D VTuber model by Cubism and Unity which could simulate his facial expression. After watching some tutorials, Kennard successfully made his first Live2D model Momose Hiyori!
VTuber Demo
Face Behavior Test : Head shaking, Nodding, Head rotation, Eye blinking, Eye half-opening, Eyeball rotation, Mouth opening.
Development Environment
Description | Specification |
---|---|
System | Windows 10 |
Camera | Integrated Webcam |
Algorithm Language | Python 3.7 (Anaconda) |
IDE | PyCharm 2019.2.5 |
Related Libraries | opencv, dlib, pytorch |
Model Tool | Live2D Cubism Editor 4.0 |
Unity Engine | Unity 2019.4.1f1 LTS |
Script Language | C# |
Folder Specification
- Recognition : Packed algorithm for face recognition.
- UnityAssets : Unity materials for those who want to make Live2D VTuber by themselves. Here is the tutorial.
Usage
Step 0 : Preparation
- Prepare Python IDE (recommend Pycharm) and install Python 3.7 (recommend Anaconda).
- Download ckpts model, unzip and place it as
Recognition\face_alignment\ckpts
. - Download VTuber_MomoseHiyori application folder.
- Clone the repository by
git clone https://github.com/KennardWang/VTuber-MomoseHiyori.git
. - Enter the root by
cd Recognition
.
Step 1 : Test Camera
There are 2 types of running environments, please choose the correct one based on individual conditions. At the beginning, please install related dependencies by pip install -r requirements.txt
.
-
CPU env
- Install
dlib v19.22.0
byconda install -c conda-forge dlib
. - Finally, run
python main.py --debug --cpu
to test. - If it runs normally, you can see your face, and press
q
to end up.
- Install
-
GPU env
- Download and install CUDA v10.2 & CUDNN v8.3.1.
- Install pytorch by
pip3 install torch==1.10.2+cu102 torchvision==0.11.3+cu102 torchaudio===0.10.2+cu102 -f https://download.pytorch.org/whl/cu102/torch_stable.html
. - Finally, run
python main.py --debug
to test. - If it runs normally, you can see your face, and press
q
to end up.
CPU env (left) has the higher FPS but the lower accuracy. (It cannot recognize well if some parts of the face are covered.)
GPU env (right) has the higher accuracy, the better fluency and consecutiveness but the lower FPS by virtue of the busier data IO.
Step 2 : Connect Unity
- Click
VTuber_MomoseHiyori.exe
to run. - In CPU env, run
python main.py --debug --cpu --connect
. - In GPU env, run
python main.py --debug --connect
.
Tips
The following tips may help to improve the effect:
- Use spotlight : Try to make your face look brighter, the spotlight probably shows better effect than the natural light.
- Adjust the face position : The debug mode of the camera may help you to know the position of your face. Try to make the green frame larger and central but not over the boundary.
- Do not wear glasses : Wearing glasses may influence the accuracy of eye recognition.
- Show your forehead : If your hair is so long that covers your eyes, it will have side effects on the accuracy of eye recognition.
Optimization
- Use the Live2D model.
- Add two eye events : Eye Half-opening and Eyeball Rotation.
- Optimize some parameters to improve accuracy.
- The multi-functional of window display is more convenient and feasible for live streaming.
References
-
Algorithms
Project Author head-pose-estimation Yin Guobing face-alignment Adrian Bulat GazeTracking Antoine LamΓ© VTuber_Unity AIθ΅
License
Author
- Kennard Wang ( 2020.6.27 )