• Stars
    star
    2,030
  • Rank 22,803 (Top 0.5 %)
  • Language
    C++
  • License
    Other
  • Created over 4 years ago
  • Updated over 3 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

基于图像的游戏AI自动化框架

aitest 平台

aitest平台 是一个基于游戏图像来开发游戏AI的开源工具包。工具包主要完成了UI检测,游戏内元素识别,AI算法(DQN,IM)等功能。可以基于这个工具包完成游戏自动化测试。目前已支持的游戏类型有跑酷类,吃鸡类,射击类,MOBA类等。详情见官网

工具包主要包括AI SDK,AIClient,SDKTool(配置文件工具)这几部分。

SDKTool帮助我们生成跟具体游戏相关的AI SDK所需要配置文件。如游戏运行过程中,整个流程中需要动作交互的UI,需要识别的游戏元素等。AI开发过程中用到的信息可来自于游戏图像(AIClient获取)或其他数据。aitest 能够自动化完成游戏UI操作并进入游戏场景,像玩家一样操作手机玩游戏。

1561697374409

​ 图 1 aitest 模块图

其中AI SDK部分主要包括AI Client模块, MC模块,IO模块, UI模块, GameReg模块,AI模块。

  • AI Client主要与手机端交互,获取游戏图像,并执行UI或AI输出的动作

  • IO模块与AI Client交互,作为AI SDK的数据输入输出控制

  • MC模块与IO模块交互,将数据分发至AI SDK后端的UI和AI模块,并做一些业务逻辑的管理控制

  • UI模块根据游戏图像,识别出需要进行的UI操作

  • GameReg模块根据游戏图像,识别出游戏场景内的数字、血条、目标物等元素

  • AI模块根据识别到的游戏元素,根据AI算法执行业务逻辑

1561697556490

​ 图 2 aitest 架构图

1 环境部署

aitest 可以部署在本地,也可以部署在云端。如果需要在本地运行,除了下载工具包,AIClient外,还需要安装工具包的依赖。如果是在云端运行,只需要下载AIClient。不论采取哪种运行方式,本地PC机都需要连接手机(或手机模拟器),通过AIClient实时采集手机(或手机模拟器)的游戏画面,并传给本地或是远端的AI服务去处理。并把AI服务处理的动作结果反馈到连接PC的手机上,完成动作的交互。

img

图3 aitest 交互图

1.1镜像部署

用户可在电脑上安装docker,使用已经搭建好game_ai_sdk所需环境的ubuntu16.04 docker镜像系统。

win10用户详细内容请参考win10-docker镜像安装

mac用户详细内容请参考mac-docker镜像安装

1.2本地部署

本地部署主要是安装AI SDK, SDKTool(配置文件生成工具), AIClient 的环境依赖。

  • 部署AI SDK

    Ubuntu:

    目前SDK工具包运行在14.04或是16.04的64位 Ubuntu 系统上,使用的深度学习框架为tensorflow ,所以使用前需要安装ubuntu系统的依赖和相应的第三方工具包的依赖,如tensorflow、opencv、protobuf等。现有两种方法部署整个环境:借助自动部署工具和依次安装依赖项。

    方法1: 自动部署工具。

    自动部署工具是一个shell脚本,让用户能够自动化部署SDK的环境,减少人力以及错误率。具体内容请详见SDK环境自动部署工具说明

    方法2:如果想了解环境的依赖项也可以通过选择依次一步一步的依次安装依赖项。

    环境主要依赖nvdia-cuda 9.0, cudnn 7.0, opencv 3.4.2, protobuf 3.2.0, android adb,poster工具集等依赖包。具体请参考依次安装依赖项说明

    Windows:

    用户下载源码后,需要安装python的依赖。有部分模块是基于c++语言实现的,所以需要用户下载vs2017,并编译源码,具体请参考 python的依赖安装安装vs2017并编译源码

  • 部署 SDKTool

    AI服务进程所需要的跟具体游戏相关的配置文件和图像标注可通过SDKTool去配置,SDKTool的安装,请见文档SDKTool部署说明

2 快速开始

aitest 怎么自动玩游戏的呢?平台内置了 <天天酷跑>的例子,让我们一睹为快。

randomai

​ 图 4 左边为未训练运行效果,右边为训练后运行效果

2.1 本地运行

本地运行主要分为以下几步:

docker镜像:
win10用户可在电脑上安装docker,使用已经搭建好game_ai_sdk所需环境的ubuntu16.04 docker镜像系统。详细内容请参考快速上手

Ubuntu:

  • Step1:下载工具包

    下载AISDK,详细内容请参考文档AISDK工具包下载

  • Step2:编译AISDK

    进入game_ai_sdk目录,编译GPU或CPU对应SDK版本,详细内容请参考文档编译AISDK

  • Step3:安装apk

    PC机连接手机(安卓系统)后,从应用商店或TTKP官网下载即可。

  • Step4:启动

    启动AIClient 和 AISDK服务。详细内容请参考本地启动

Windows:

  • Step1:下载工具包

    下载AISDK,详细内容请参考文档AISDK工具包下载

  • Step2:编译SDK

    安装vs2017,设置环境变量,编译SDK,详细内容请参考文档windows下编译SDK

  • Step3:安装apk

    PC机连接手机(安卓系统)后,从应用商店或TTKP官网下载即可。

  • Step4:启动

    启动AIClient 和 AISDK服务。详细内容请参考本地启动

3 配置文件工具SDK Tool

在第2章我们介绍了如何运行一个内置的游戏AI,那么我们如何从零开始接入一个新的项目呢?首先我们需要生成跟具体游戏相关的配置文件。如控制游戏运行流程的UI界面(UI配置),进入游戏场景后,AI所需要识别的游戏场景元素(场景识别配置)等。工具包AI SDK Tool可以协助用户去生成这些配置文件。工具的下载和安装方法请参考第一章部署SDKTool部分。

SDKTool

图 5 SDKTool界面介绍

  • UI配置

    游戏运行过程中,有些需要玩家交互点击Button,配置好流程中的这些UI界面和需要点击的位置后,AI即可以像玩家一样在相应的界面点击相应的位置。方便用户的自动化测试或是其他需要自动化进入游戏的需求。详细内容,请见 UI配置文件说明

  • 场景识别配置

    根据所选择的游戏场景,设置所需要的识别任务。如QQ飞车中的数字,左下角的道具,向左转向的按钮,向右转向的按钮等。场景配置文件可配置的识别类型主要包括固定位置的物体识别(fix object),形变物体识别(deform object),数字识别(number),卡住判断(stuck),基于像素值的筛选(pixel),固定血条的识别(fix blood),不固定血条的识别(deform blood)等。具体参数说明,请见任务配置文件说明。图像识别进程根据配置项做处理后,把结果返回给Agent,Agent可以根据游戏元素的识别结果编写相应的AI逻辑。识别类型配置说明以及返回结果示例请见文档图像识别进程识别类型介绍

4 项目接入

通过第三章我们了解了配置文件工具的功能,那么我们完整接入一个新的项目,需要哪些步骤呢?以<天天酷跑>为例,我们介绍接入新项目的具体步骤。

  • 执行项目的整个逻辑流程如下,配置项目的时候需要根据这个流程来配置。

img

  • Step1: 配置游戏过程UI

    详细内容请见项目接入-UI配置

  • Step2: 采集样本,训练识别模型(可变物体的识别算法需求,此步骤为可选步骤,<天天酷跑>不需要此步骤。)

    有些识别任务(如可变物体的识别)需要训练网络模型。目前平台内置了YOLO识别算法。我们需要先采集图像样本,并对这些样本进行标注,然后基于这些标注样本去训练YOLO模型。详细过程请参考图像标注和YOLO模型训练说明

  • Step3: 配置场景识别任务

    详细内容请见项目接入-配置task任务

  • Step4: 选择AI算法

    ​ 平台内置的AI算法有DQN,IM(模仿学习),用户可以根据需求选择AI算法

  • Step5: 运行SDK

    准备工作就绪后,运行SDK,看看我们的AI是怎么玩新游戏的吧。具体内容请见本地运行

5 自动探索

UI自动探索是自动遍历和记录Android手机游戏内所有UI的功能,以<天天酷跑>为例,我们介绍UI自动探索的具体步骤。

​详细内容请见UI自动探索使用手册

  • Step1: 配置样本图片

    可以从本地导入图片,也可以在工具上实时生成样本图片。

  • Step2: 标注样本

    先自动标注样本,再人工检查并调整部分标注。

  • Step3: 训练模型

    根据需要和电脑配置调整训练参数,微调次数越大模型的效果越好,但是训练时间越长

  • Step4: 执行自动探索

    根据需要设置本次执行的点击次数。执行完指定的点击次数后自动结束。

  • Step5: 查看自动探索结果

    执行结束后可查看执行过程的图分析,场景和按钮覆盖率,以及每个样本图片的详细覆盖情况

6 二次开发

如果有更高阶的需求,比如现有的AI算法不能满足需求,如何加入自己的AI呢?想增加新的检测或识别算法,该怎么加入到整体框架上? 那么您可以增加和修改源码,并欢迎上传到git。

7 调试技巧

在用工具的时候,可能会遇到一些问题,如图像匹配有偏差等问题,请详见文档 调试技巧说明

8 常见问题及处理方法

搭建环境或者运行过程出现问题,查看FAQ

More Repositories

1

weui

A UI library by WeChat official design team, includes the most useful widgets/modules in mobile web applications.
Less
27,140
star
2

wepy

小程序组件化开发框架
JavaScript
22,491
star
3

ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform
C++
19,861
star
4

mars

Mars is a cross-platform network component developed by WeChat.
C++
17,294
star
5

tinker

Tinker is a hot-fix solution library for Android, it supports dex, library and resources update without reinstall apk.
Java
17,164
star
6

MMKV

An efficient, small mobile key-value storage framework developed by WeChat. Works on Android, iOS, macOS, Windows, and POSIX.
C++
17,138
star
7

APIJSON

🏆 实时 零代码、全功能、强安全 ORM 库 🚀 后端接口和文档零代码,前端(客户端) 定制返回 JSON 的数据和结构 🏆 Real-Time coding-free, powerful and secure ORM 🚀 providing APIs and Docs without coding by Backend, and the returned JSON of API can be customized by Frontend(Client) users
Java
17,052
star
8

vConsole

A lightweight, extendable front-end developer tool for mobile web page.
TypeScript
16,716
star
9

weui-wxss

A UI library by WeChat official design team, includes the most useful widgets/modules.
Less
15,070
star
10

QMUI_Android

提高 Android UI 开发效率的 UI 库
Java
14,423
star
11

rapidjson

A fast JSON parser/generator for C++ with both SAX/DOM style API
C++
14,163
star
12

secguide

面向开发人员梳理的代码安全指南
13,203
star
13

omi

Web Components Framework - Web组件框架
TypeScript
13,048
star
14

VasSonic

VasSonic is a lightweight and high-performance Hybrid framework developed by tencent VAS team, which is intended to speed up the first screen of websites working on Android and iOS platform.
Java
11,801
star
15

matrix

Matrix is a plugin style, non-invasive APM system developed by WeChat.
Java
11,623
star
16

wcdb

WCDB is a cross-platform database framework developed by WeChat.
C
10,793
star
17

xLua

xLua is a lua programming solution for C# ( Unity, .Net, Mono) , it supports android, ios, windows, linux, osx, etc.
C
9,296
star
18

libco

libco is a coroutine library which is widely used in wechat back-end service. It has been running on tens of thousands of machines since 2013.
C++
8,223
star
19

Hippy

Hippy is designed to easily build cross-platform dynamic apps. 👏
C++
7,983
star
20

Shadow

零反射全动态Android插件框架
Java
7,382
star
21

QMUI_iOS

QMUI iOS——致力于提高项目 UI 开发效率的解决方案
Objective-C
7,084
star
22

lemon-cleaner

腾讯柠檬清理是针对macOS系统专属制定的清理工具。主要功能包括重复文件和相似照片的识别、软件的定制化垃圾扫描、可视化的全盘空间分析、内存释放、浏览器隐私清理以及设备实时状态的监控等。重点聚焦清理功能,对上百款软件提供定制化的清理方案,提供专业的清理建议,帮助用户轻松完成一键式清理。
Objective-C
5,421
star
23

MLeaksFinder

Find memory leaks in your iOS app at develop time.
Objective-C
5,419
star
24

libpag

The official rendering library for PAG (Portable Animated Graphics) files that renders After Effects animations natively across multiple platforms.
C++
4,943
star
25

puerts

PUER(普洱) Typescript. Let's write your game in UE or Unity with TypeScript.
C++
4,902
star
26

kbone

一个致力于微信小程序和 Web 端同构的解决方案
JavaScript
4,772
star
27

TNN

TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
C++
4,388
star
28

GT

GT (Great Tit) is a portable debugging tool for bug hunting and performance tuning on smartphones anytime and anywhere just as listening music with Walkman. GT can act as the Integrated Debug Environment by directly running on smartphones.
Java
4,387
star
29

westore

小程序项目分层架构
JavaScript
4,244
star
30

tmagic-editor

TypeScript
4,190
star
31

wujie

极致的微前端框架
TypeScript
4,023
star
32

vap

VAP是企鹅电竞开发,用于播放特效动画的实现方案。具有高压缩率、硬件解码等优点。同时支持 iOS,Android,Web 平台。
Objective-C
3,898
star
33

cherry-markdown

✨ A Markdown Editor
JavaScript
3,505
star
34

HunyuanDiT

Hunyuan-DiT : A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding
Python
3,358
star
35

phxpaxos

The Paxos library implemented in C++ that has been used in the WeChat production environment.
C++
3,351
star
36

WeFlow

A web developer workflow tool by WeChat team based on tmt-workflow, with cross-platform supported and environment ready.
JavaScript
3,224
star
37

spring-cloud-tencent

Spring Cloud Tencent is a Spring Cloud based Service Governance Framework provided by Tencent.
Java
3,171
star
38

weui.js

A lightweight javascript library for WeUI.
JavaScript
3,167
star
39

tdesign

Enterprise Design System
Vue
3,156
star
40

VasDolly

Android V1 and V2 Signature Channel Package Plugin
Java
3,061
star
41

tencent-ml-images

Largest multi-label image database; ResNet-101 model; 80.73% top-1 acc on ImageNet
Python
3,050
star
42

Tendis

Tendis is a high-performance distributed storage system fully compatible with the Redis protocol.
C++
2,934
star
43

FaceDetection-DSFD

腾讯优图高精度双分支人脸检测器
Python
2,885
star
44

PhoenixGo

Go AI program which implements the AlphaGo Zero paper
C++
2,871
star
45

behaviac

behaviac is a framework of the game AI development, and it also can be used as a rapid game prototype design tool. behaviac supports the behavior tree, finite state machine and hierarchical task network(BT, FSM, HTN)
C#
2,829
star
46

PocketFlow

An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
Python
2,783
star
47

MSEC

Mass Service Engine in Cluster(MSEC) is opened source by QQ team from Tencent. It is a backend DEV &OPS engine, including RPC,name finding,load balance,monitoring,release and capacity management.
Java
2,746
star
48

phxsql

A high availability MySQL cluster that guarantees data consistency between a master and slaves.
C++
2,470
star
49

OOMDetector

OOMDetector is a memory monitoring component for iOS which provides you with OOM monitoring, memory allocation monitoring, memory leak detection and other functions.
Objective-C++
2,315
star
50

tsf

coroutine and Swoole based php server framework in tencent
PHP
2,179
star
51

tmt-workflow

A web developer workflow used by WeChat team based on Gulp, with cross-platform supported and solutions prepared.
CSS
2,173
star
52

UnLua

A feature-rich, easy-learning and highly optimized Lua scripting plugin for UE.
C++
2,169
star
53

Hardcoder

Hardcoder is a solution which allows Android APP and Android System to communicate with each other directly, solving the problem that Android APP could only use system standard API rather than the hardware resource of system.
C++
2,155
star
54

LKImageKit

A high-performance image framework, including a series of capabilities such as image views, image downloader, memory caches, disk caches, image decoders and image processors.
Objective-C
2,082
star
55

TubeMQ

TubeMQ has been donated to the Apache Software Foundation and renamed to InLong, please visit the new Apache repository: https://github.com/apache/incubator-inlong
2,022
star
56

phxrpc

A simple C++ based RPC framework.
C++
1,974
star
57

TscanCode

A static code analyzer for C++, C#, Lua
C++
1,972
star
58

ObjectDetection-OneStageDet

单阶段通用目标检测器
Python
1,966
star
59

InjectFix

InjectFix is a hot-fix solution library for Unity
C#
1,961
star
60

cloudbase-framework

腾讯云开发云原生一体化部署工具 🚀 CloudBase Framework:一键部署,不限框架语言,云端一体化开发,基于Serverless 架构。A front-end and back-end integrated deployment tool. One-click deploy to serverless architecture. https://docs.cloudbase.net/framework/index
JavaScript
1,937
star
61

soter

A secure and quick biometric authentication standard and platform in Android held by Tencent.
Java
1,928
star
62

phxqueue

A high-availability, high-throughput and highly reliable distributed queue based on the Paxos algorithm.
C++
1,899
star
63

plato

腾讯高性能分布式图计算框架Plato
C++
1,895
star
64

MedicalNet

Many studies have shown that the performance on deep learning is significantly affected by volume of training data. The MedicalNet project provides a series of 3D-ResNet pre-trained models and relative code.
Python
1,888
star
65

NeuralNLP-NeuralClassifier

An Open-source Neural Hierarchical Multi-label Text Classification Toolkit
Python
1,834
star
66

TSW

Tencent Server Web
TypeScript
1,804
star
67

sluaunreal

lua dev plugin for unreal engine 4 or 5
C++
1,780
star
68

QMUI_Web

An efficient front-end framework for developers building UI on the web.
JavaScript
1,719
star
69

Metis

Metis is a learnware platform in the field of AIOps.
Python
1,690
star
70

Biny

Biny is a tiny, high-performance PHP framework for web applications
PHP
1,687
star
71

paxosstore

PaxosStore has been deployed in WeChat production for more than two years, providing storage services for the core businesses of WeChat backend. Now PaxosStore is running on thousands of machines, and is able to afford billions of peak TPS.
C++
1,675
star
72

CodeAnalysis

Static Code Analysis - 静态代码分析
Python
1,639
star
73

MimicMotion

High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Python
1,475
star
74

TurboTransformers

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
C++
1,464
star
75

tdesign-vue-next

A Vue3.x UI components lib for TDesign.
TypeScript
1,434
star
76

nohost

基于 Whistle 实现的多账号多环境远程配置及抓包调试平台
JavaScript
1,423
star
77

TencentOS-kernel

腾讯针对云的场景研发的服务器操作系统
1,408
star
78

TBase

TBase is an enterprise-level distributed HTAP database. Through a single database cluster to provide users with highly consistent distributed database services and high-performance data warehouse services, a set of integrated enterprise-level solutions is formed.
C
1,383
star
79

WeDemo

WeDemo为微信团队开源项目,用于帮助微信开发者完成微信登录、微信分享等功能的接入和开发。开发者可参考源代码完成开发,也可以直接将代码应用到自己的App开发中,安全、便捷地在App中实现微信分享、微信登录功能。
Objective-C
1,374
star
80

feflow

🚀 A command line tool aims to improve front-end engineer workflow and standard, powered by TypeScript.
TypeScript
1,360
star
81

GAutomator

Automation for mobile games
Objective-C
1,331
star
82

flare

Flare是广泛投产于腾讯广告后台的现代化C++开发框架,包含了基础库、RPC、各种客户端等。主要特点为易用性强、长尾延迟低。
C++
1,308
star
83

TFace

A trusty face analysis research platform developed by Tencent Youtu Lab
Python
1,306
star
84

LuaPanda

lua debug and code tools for VS Code
Lua
1,235
star
85

FeatherCNN

FeatherCNN is a high performance inference engine for convolutional neural networks.
C++
1,208
star
86

tdesign-miniprogram

A Wechat MiniProgram UI components lib for TDesign.
HTML
1,204
star
87

tquic

A high-performance, lightweight, and cross-platform QUIC library
Rust
1,086
star
88

tgfx

A lightweight 2D graphics library for rendering texts, geometries, and images with high-performance APIs that work across various platforms.
C++
1,057
star
89

TencentPretrain

Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo
Python
1,005
star
90

RapidView

RapidView is an android ui and lightapp development framework
Java
979
star
91

hel

A module federation SDK which is unrelated to tool chain for module consumer. 工具链无关的运行时模块联邦sdk.
JavaScript
959
star
92

TencentKona-8

Tencent Kona is a no-cost, production-ready distribution of the Open Java Development Kit (OpenJDK), Long-term support(LTS) with quarterly updates. Tencent Kona serves as the default JDK internally at Tencent Cloud for cloud computing and other Java applications.
Java
942
star
93

FAutoTest

A UI automated testing framework for H5 and applets
Python
932
star
94

tdesign-vue

A Vue.js UI components lib for TDesign.
TypeScript
914
star
95

Pebble

Pebble分布式开发框架
C++
866
star
96

mxflutter

使用 TypeScript/JavaScript 来开发 Flutter 应用的框架。
Dart
857
star
97

Face2FaceTranslator

面对面翻译小程序是微信团队针对面对面沟通的场景开发的流式语音翻译小程序,通过微信同声传译插件提供了语音识别,文本翻译等功能。
JavaScript
836
star
98

tdesign-react

A React UI components lib for TDesign.
TypeScript
821
star
99

LightDiffusionFlow

This extension is developed for AUTOMATIC1111's Stable Diffusion web UI that provides import/export options for parameters.
JavaScript
798
star
100

Real-SR

Real-World Super-Resolution via Kernel Estimation and Noise Injection
Python
769
star