• Stars
    star
    2,181
  • Rank 21,148 (Top 0.5 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created about 4 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

AlphaPlayer is a video animation engine.

AlphaPlayer

Powered by ByteDance Live.

AlphaPlayer是直播中台使用的一个视频动画特效SDK,可以通过制作Alpha通道分离的视频素材,再在客户端上通过OpenGL ES重新实现Alpha通道和RGB通道的混合,从而实现在端上播放带透明通道的视频。

这套方案对设计师而言明显降低了特效的制作成本,对于客户端而言有着更可靠的性能和稳定性,且相比cocos2d引擎有着更低的入门门槛和维护成本,为复杂动画的实现提供了一种全新的方式,新的复杂动画开发将会变得更加简单高效。

背景

在直播项目的原有礼物动画实现效果是通过cocos引擎实现的,大部分动画都是通过一系列的旋转平移缩放组合而成,能实现的动画效果较简单且开发成本较高。为了丰富动画的表现形式,降低开发成本,我们引入了AlphaPlayer的动画实现方案。

方案对比

目前较常见的动画实现方案有原生动画、帧动画、gif/webp、lottie/SVGA、cocos引擎,对于复杂动画特效的实现做个简单对比

方案 实现成本 上手成本 还原程度 接入成本
原生动画 复杂动画实现成本高
帧动画 实现成本低,但资源消耗大
gif/webp 实现成本低,但资源消耗大 只支持8位颜色
Lottie/SVGA 实现成本低,部分复杂特效不支持 部分复杂特效不支持
cocos2d引擎 实现成本高 较高 较高
AlphaPlayer 开发无任何实现成本,一次接入永久使用

而在复杂动画特效高效实现的场景中,我们的备选方案会更少一些,可以将讨论集中在Cocos2d、Lottie、Webp和本文的AlphaPlayer上。

Lottie

Lottie是非常优选的多平台动画效果解决方案,其简单原理是将AE动画导出的json文件解析成每个layer层级对象,再绘制成对应的Drawable,最后显示在View上。在不涉及mask和mattes等特性时性能非常优选,主要耗时基本集中在Canvas#draw()上而已,json解析时通过流读取的方式避免一次性加载全部json数据带来的OOM问题。

但是也存在部分不足:

  1. 如果动画涉及到mask或mattes等特性时,需要生成2~3个临时bitmap实现动画效果,容易引起内存抖动,且涉及的canvas#saveLayer()和canvas#drawBitmap()会带来额外的耗时;
  2. 如果动画中还直接使用了图片,在ImageAssetManager首次对图片解码是在主线程进行的(据了解在iOS的版本上,使用图片导致内存和CPU的性能消耗会更大);
  3. 主要耗时还是在draw()上,绘制区域越大耗时越长;
  4. 目前支持的AE特性还有待完善,此外有一些特性在低版本上可能还会没有效果,设计资源时需要规避。(Supported After Effect Features

实际使用过程中,最常见的首启全屏引导动画基本都会包含上面提到的前三点不足,这种情况下性能其实是大幅退化的。因此对于直播场景的复杂特效动画而言,Lottie就不是一个很合适的实现方案了。

Cocos2d-x

Cocos2d-x支持非常多的游戏功能,诸如精灵、动作、动画、粒子特效、骨骼动画等等,可以供开发者自由实现各种姿势的动画效果,再加上自身具备跨平台能力和轻量级,同时支持Lua作为开发语言,可以说是非常适合植入移动端作为动画效果实现方案的游戏引擎。

但实际使用维护中会面临很多问题:

  1. 体积大,即使经过裁剪也还有2M左右的大小,如果不是核心场景需要基本很难允许接入;
  2. 对开发者的技术栈有较高要求;
  3. 无法满足快速迭代;
  4. 维护难度大,尤其是在Android机型兼容的问题上。

Webp

Webp相比PNG和JPEG格式体积可以减少25%,在移动端的平台支持上也很全面,支持24位RGB色;缺点是资源体积比较大,而且使用的软解效率低下,低端机上有明显卡顿问题。

AlphaPlayer

相比于上面提到的几个方案,AlphaPlayer的接入体积极小(只有40KB左右),而且对动画资源的还原程度极高,资源制作时不用考虑特效是否支持的问题,对开发者和设计师都非常友好。通过接入ExoPlayer或者自研播放器可以解决系统播放器在部分机型上可能存在的兼容性问题,且能带来更好的解码性能。

运行效果

demo

基本原理

主要有两个核心,一个是IMediaPlayer,负责视频解码,支持外部自行实现;另一个是VideoRenderer,负责将解析出来的每一帧画面进行透明度混合,再输出到GLTextureView或者GLSurfaceView上。

大致的混合过程可以看下图示例

introduction

原素材的画面中左边部分使用RGB通道存储了原透明视频的Alpha值,右边部分使用RGB通道存储了原透明视频的RGB值,然后在端上通过OpenGL重新将每个像素点的Alpha值和RGB值进行组合,重新得到ARGB视频画面,实现透明视频的动画效果。

混合过程的详细代码可以见 frag.shvertex.sh

快速接入

iOS

添加依赖

pod 'BDAlphaPlayer'

初始化View
BDAlphaPlayerMetalView *metalView = [[BDAlphaPlayerMetalView alloc] initWithDelegate:self];
[self.view addSubview:metalView];
播放动画视频
BDAlphaPlayerMetalConfiguration *configuration = [BDAlphaPlayerMetalConfiguration defaultConfiguration];
NSString *testResourcePath = [[[NSBundle mainBundle] bundlePath] stringByAppendingPathComponent:@"TestResource"];
NSString *directory = [testResourcePath stringByAppendingPathComponent:@"heartbeats"];
configuration.directory = directory;
configuration.renderSuperViewFrame = self.view.frame;
configuration.orientation = BDAlphaPlayerOrientationPortrait;

[self.metalView playWithMetalConfiguration:configuration];

Android

添加依赖
allprojects {
    repositories {
        ...
        maven { url 'https://jitpack.io' }
    }
}

dependencies {
    implementation 'com.github.bytedance:AlphaPlayer:1.0.4'
}
初始化PlayerController
val config = Configuration(context, lifecycleOwner)
// 支持GLSurfaceView&GLTextureView, 默认使用GLSurfaceView
config.alphaVideoViewType = AlphaVideoViewType.GL_TEXTURE_VIEW
// 也可以设置自行实现的Player, demo中提供了基于ExoPlayer的实现
val playerController = PlayerController.get(config, DefaultSystemPlayer())	
playerController.setPlayerAction(object: IPlayerAction {
  override fun onVideoSizeChanged(videoWidth: Int, videoHeight: Int, scaleType: ScaleType) {
  }
  override fun startAction() {
  }
  override fun endAction() {
  }
})
playController.setMonitor(object: IMonitor {
  override fun monitor(result: Boolean, playType: String, what: Int, extra: Int, errorInfo: String) {
  }
}) 
将PlayerController绑定到ViewGroup
playerController.attachAlphaView(mVideoContainer)
播放动画视频
fun startVideoAnimation() {
  val baseDir = "your video file base dir"
  val portraitFileName = "portrait.mp4"
  val portraitScaleType = 2
  val landscapeFileName = "landscape.mp4"
  val landscapeScaleType = 2
  val dataSource = DataSource().setBaseDir(baseDir)
    .setPortraitPath(portraitFileName, portraitScaleType)
    .setLandscapePath(landscapeFileName, landscapeScaleType)
  	.setLooping(false)	// 可设置该视频是否循环播放
  if (dataSource.isValid()) {
    playerController.start(dataSource)
  }
}
资源释放
fun releasePlayerController() {
  playerController.detachAlphaView(mVideoContainer)
  playerController.release()
}

GLSurfaceView & GLTextureView

SurfaceView和TextureView都是用来显示视频画面的,主要差异在于性能和层级,SurfaceView的性能要优于TextureView,但是层级限制在最顶层,TextureView则没有层级限制。可以通过如下方式指定alphaVideoViewType来设置。

val config = Configuration(context, lifecycleOwner)
// 支持GLSurfaceView&GLTextureView, 默认使用GLSurfaceView
config.alphaVideoViewType = AlphaVideoViewType.GL_TEXTURE_VIEW
val playerController = PlayerController.get(config, DefaultSystemPlayer())

高级特性

动画对齐方式

为了解决不同屏幕尺寸的兼容问题和支持半屏动画视频的指定位置播放,我们提供了多种视频裁剪对齐方式,详细可见ScaleType.kt/BDAlphaPlayerDefine.h

对齐模式 描述
ScaleToFill 拉伸铺满全屏
ScaleAspectFitCenter 等比例缩放对齐全屏,居中,屏幕多余部分留空
ScaleAspectFill 等比例缩放铺满全屏,居中,裁剪视频多余部分
TopFill 等比例缩放铺满全屏,顶部对齐
BottomFill 等比例缩放铺满全屏,底部对齐
LeftFill 等比例缩放铺满全屏,左边对齐
RightFill 等比例缩放铺满全屏,右边对齐
TopFit 等比例缩放至屏幕宽度,顶部对齐,底部留空
BottomFit 等比例缩放至屏幕宽度,底部对齐,顶部留空
LeftFit 等比例缩放至屏幕高度,左边对齐,右边留空
RightFit 等比例缩放至屏幕高度,右边对齐,左边留空

提供多种动画对齐方式的目的有二:一是需要对不同屏幕尺寸的设备进行兼容;二是希望尽量减少屏幕中视频动画的渲染区域(这对GPU功耗有线性收益),所以如果局部渲染可以满足动画表现需求,建议尽量使用局部渲染,即减少mVideoContainer的布局大小。

Alpha通道压缩方案

为了进一步减少视频动画文件的体积,我们做了很多方向的尝试,包括透明画面像素点冗余channel的复用和整体尺寸压缩,可以期待后续更新。

GLSurfaceView & GLTextureView

SurfaceView和TextureView都是用来显示视频画面的,主要差异在于性能和层级,SurfaceView的性能要优于TextureView,但是层级限制在最顶层,TextureView则没有层级限制。可以通过如下方式指定alphaVideoViewType来设置。

val config = Configuration(context, lifecycleOwner)
// 支持GLSurfaceView&GLTextureView, 默认使用GLSurfaceView
config.alphaVideoViewType = AlphaVideoViewType.GL_TEXTURE_VIEW
val playerController = PlayerController.get(config, DefaultSystemPlayer())

素材制作工具

素材制作的方式有两种:

一种是直接使用AE导出成品素材,大致流程就是后期在AE上完成动画效果后,分离出Alpha通道视频,然后在同一个AE合成里左边带Alpha通道后边带正常动画,一起渲染导出。如果还是不理解,还是让设计师去代劳吧,专业的人做专业的事。

第二种方式,在AE上完成动画后期效果后,直接输出视频序列帧,然后使用我们提供的素材制作脚本 convertAlphaVideo.py 进行处理也可以直接得出成品素材视频,脚本的大致原理如下:

tools

素材制作工具

素材制作的方式有两种:

一种是直接使用AE导出成品素材,大致流程就是后期在AE上完成动画效果后,分离出Alpha通道视频,然后在同一个AE合成里左边带Alpha通道后边带正常动画,一起渲染导出。如果还是不理解,还是让设计师去代劳吧,专业的人做专业的事。

第二种方式,在AE上完成动画后期效果后,直接输出视频序列帧,然后使用我们提供的素材制作脚本 convertAlphaVideo.py 进行处理也可以直接得出成品素材视频,脚本的大致原理如下:

tools

可以看到通道分离和画面拼接是基于ffmpeg和ImageMagick两套工具实现的,所以运行前需要先配置ffmpeg和ImageMagick的环境。

执行下面命令,等待成品素材生成。

python convertAlphaVideo.py --dir 'your pictures parent file path'

已知接入方

douyin douyin douyin douyin
抖音 抖音火山版 西瓜小视频 今日头条

联系我们

如果你有任何关于AlphaPlayer的问题或建议,可以发邮件到邮箱:[email protected], 在邮件中详细描述你的问题。

License

Apache 2.0

招聘位

再插播个招聘广告~ 抖音直播团队持续招聘客户端研发,Base涵盖北京、杭州、深圳三地,简历可直接发送到邮箱:[email protected],如想交流咨询可加wx:ZHp5LTMxOA==

More Repositories

1

IconPark

🍎Transform an SVG icon into multiple themes, and generate React icons,Vue icons,svg icons
TypeScript
8,298
star
2

xgplayer

A HTML5 video player with a parser that saves traffic
JavaScript
8,260
star
3

sonic

A blazingly fast JSON serializing & deserializing library
Assembly
6,870
star
4

monoio

Rust async runtime based on io-uring.
Rust
3,864
star
5

byteps

A high performance and generic framework for distributed DNN training
Python
3,603
star
6

lightseq

LightSeq: A High Performance Library for Sequence Processing and Generation
C++
3,193
star
7

ByteX

ByteX is a bytecode plugin platform based on Android Gradle Transform API and ASM. 字节码插件开发平台
Java
2,865
star
8

Elkeid

Elkeid is an open source solution that can meet the security requirements of various workloads such as hosts, containers and K8s, and serverless. It is derived from ByteDance's internal best practices.
Go
2,226
star
9

scene

Android Single Activity Framework compatible with Fragment.
Java
2,097
star
10

bhook

🔥 ByteHook is an Android PLT hook library which supports armeabi-v7a, arm64-v8a, x86 and x86_64.
C
2,073
star
11

flutter_ume

UME is an in-app debug kits platform for Flutter. Produced by Flutter Infra team of ByteDance
Dart
2,053
star
12

terarkdb

A RocksDB compatible KV storage engine with better performance
C++
2,044
star
13

btrace

🔥🔥 btrace(AKA RheaTrace) is a high performance Android trace tool which is based on Perfetto, it support to define custom events automatically during building apk and using bhook to provider more native events like Render/Binder/IO etc.
Kotlin
1,913
star
14

gopkg

Universal Utilities for Go
Go
1,704
star
15

android-inline-hook

🔥 ShadowHook is an Android inline hook library which supports thumb, arm32 and arm64.
C
1,660
star
16

bitsail

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
Java
1,627
star
17

go-tagexpr

An interesting go struct tag expression syntax for field validation, etc.
Go
1,470
star
18

GiantMIDI-Piano

Python
1,431
star
19

appshark

Appshark is a static taint analysis platform to scan vulnerabilities in an Android app.
Kotlin
1,363
star
20

AabResGuard

The tool of obfuscated aab resources.(Android app bundle资源混淆工具)
Java
1,307
star
21

piano_transcription

Python
1,247
star
22

CodeLocator

Kotlin
1,163
star
23

BoostMultiDex

BoostMultiDex is a solution for quickly loading multiple dex files on low Android version devices (4.X and below, SDK <21).
Java
1,106
star
24

music_source_separation

Python
1,039
star
25

Fastbot_Android

Fastbot(2.0) is a model-based testing tool for modeling GUI transitions to discover app stability problems
C++
1,031
star
26

SALMONN

SALMONN: Speech Audio Language Music Open Neural Network
Python
1,000
star
27

memory-leak-detector

C
919
star
28

fedlearner

A multi-party collaborative machine learning framework
Python
892
star
29

monolith

ByteDance's Recommendation System
Python
844
star
30

sonic-cpp

A fast JSON serializing & deserializing library, accelerated by SIMD.
C++
811
star
31

godlp

sensitive information protection toolkit
Go
770
star
32

MVDream

Multi-view Diffusion for 3D Generation
Python
744
star
33

res-adapter

Official implementation of "ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models".
Python
724
star
34

bytemd

ByteMD v1 repository
TypeScript
679
star
35

tailor

C
669
star
36

ibot

iBOT 🤖: Image BERT Pre-Training with Online Tokenizer (ICLR 2022)
Jupyter Notebook
663
star
37

RealRichText

A Tricky Solution for Implementing Inline-Image-In-Text Feature in Flutter.
Dart
658
star
38

guide

A new feature guide component by react 🧭
TypeScript
651
star
39

mockey

a simple and easy-to-use golang mock library
Go
622
star
40

magic-microservices

Make Web Components easier and powerful!😘
TypeScript
570
star
41

Fastbot_iOS

About Fastbot(2.0) is a model-based testing tool for modeling GUI transitions to discover app stability problems
Objective-C
553
star
42

flow-builder

A highly customizable streaming flow builder.
TypeScript
526
star
43

MVDream-threestudio

3D generation code for MVDream
Python
473
star
44

effective_transformer

Running BERT without Padding
C++
457
star
45

ByteTransformer

optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052
C++
449
star
46

Next-ViT

Python
426
star
47

matxscript

A high-performance, extensible Python AOT compiler.
C++
408
star
48

byteir

A model compilation solution for various hardware
MLIR
362
star
49

syllepsis

Syllepsis is an out-of-the-box rich text editor.
TypeScript
355
star
50

uss

This is the PyTorch implementation of the Universal Source Separation with Weakly labelled Data.
Python
324
star
51

OMGD

Online Multi-Granularity Distillation for GAN Compression (ICCV2021)
Python
323
star
52

neurst

Neural end-to-end Speech Translation Toolkit
Python
298
star
53

danmu.js

HTML5 danmu (danmaku) plugin for any DOM element
JavaScript
292
star
54

vArmor

vArmor is a cloud native container sandbox system based on AppArmor/BPF/Seccomp. It also includes multiple built-in protection rules that are ready to use out of the box.
Go
263
star
55

particle-sfm

ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild. ECCV 2022.
C++
263
star
56

CloudShuffleService

Cloud Shuffle Service(CSS) is a general purpose remote shuffle solution for compute engines, including Spark/Flink/MapReduce.
Java
245
star
57

lynx-llm

paper: https://arxiv.org/abs/2307.02469 page: https://lynx-llm.github.io/
Python
227
star
58

g3

Enterprise-oriented Generic Proxy Solutions
Rust
227
star
59

xgplayer-vue

Vue component for xgplayer, a HTML5 video player with a parser that saves traffic
JavaScript
219
star
60

DEADiff

[CVPR 2024] Official implementation of "DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representations"
Python
209
star
61

flux

A fast communication-overlapping library for tensor parallelism on GPUs.
C++
201
star
62

trace-irqoff

Interrupts-off or softirqs-off latency tracer
C
195
star
63

ParaGen

ParaGen is a PyTorch deep learning framework for parallel sequence generation.
Python
186
star
64

ByteMLPerf

AI Accelerator Benchmark focuses on evaluating AI Accelerators from a practical production perspective, including the ease of use and versatility of software and hardware.
Python
181
star
65

MoMA

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation
Jupyter Notebook
177
star
66

AWERTL

An non-invasive iOS framework for quickly adapting Right-To-Left style UI
Objective-C
175
star
67

Bytedance-UnionAD

Ruby
170
star
68

keyhouse

Keyhouse is a skeleton of general-purpose Key Management System written in Rust.
Rust
163
star
69

react-model

The next generation state management library for React
TypeScript
162
star
70

LargeBatchCTR

Large batch training of CTR models based on DeepCTR with CowClip.
Python
162
star
71

ic_flow_platform

IFP (ic flow platform) is an integrated circuit design flow platform, mainly used for IC process specification management and data flow contral.
Python
154
star
72

DanmakuRenderEngine

DanmakuRenderEngine is a lightweight and scalable Android danmaku library. 轻量级高扩展安卓弹幕渲染引擎
Kotlin
149
star
73

primus

Java
148
star
74

diat

A CLI tool to help with diagnosing Node.js processes basing on inspector.
JavaScript
146
star
75

coconut_cvpr2024

Jupyter Notebook
143
star
76

Hammer

An efficient toolkit for training deep models.
Python
138
star
77

ns-x

An easy-to-use, flexible network simulator library in Go.
Go
116
star
78

pv3d

Python
113
star
79

fc-clip

This repo contains the code for our paper Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
Python
109
star
80

RLFN

Winner of runtime track in NTIRE 2022 challenge on Efficient Super-Resolution
Python
106
star
81

DCFrame

DCFrame is a Swift UI collection framework, which can easily create complex UI.
Swift
100
star
82

trace-noschedule

Trace noschedule thread
C
99
star
83

decoupleQ

A quantization algorithm for LLM
Cuda
99
star
84

tar-wasm

A faster experimental wasm-based tar implementation for browsers.
Rust
95
star
85

TWIST

Official codes: Self-Supervised Learning by Estimating Twin Class Distribution
Python
95
star
86

magic-portal

⚡ A blazing fast micro-component and micro-frontend solution uses web-components under the hood.
TypeScript
91
star
87

xgplayer-react

React component for xgplayer, a HTML5 video player with a parser that saves traffic
JavaScript
84
star
88

fe-foundation

UI Foundation for React Hooks and Vue Composition Api
TypeScript
80
star
89

nnproxy

Scalable NameNode RPC Proxy for HDFS Federation
Java
79
star
90

dbatman

Go
74
star
91

Elkeid-HUB

Elkeid HUB is a rule/event processing engine maintained by the Elkeid Team that supports streaming/offline (not yet supported by the community edition) data processing. The original intention is to solve complex data/event processing and external system linkage requirements through standardized rules.
Python
74
star
92

FreeSeg

Python
69
star
93

pull_to_refresh

Flutter pull_to_refresh widget
Dart
67
star
94

Jeddak-DPSQL

DPSQL (Privacy Protection SQL Query Service) - This project is a microservice Middleware located between the database engine ( Hive , Clickhouse , etc.) and the application system. It provides transparent SQL query result desensitization capabilities.
Python
62
star
95

terark-zip

A data structure and algorithm library built for TerarkDB
C++
62
star
96

trace-runqlat

C
61
star
97

ipmb

An interprocess message bus system built in Rust.
Rust
60
star
98

X-Portrait

Source code for the SIGGRAPH 2024 paper "X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attention"
Python
59
star
99

kernel

ByteDance kernel for use on cloud.
C
57
star
100

scroll_kit

Dart
56
star