• This repository has been archived on 24/Oct/2019
  • Stars
    star
    2,749
  • Rank 15,759 (Top 0.4 %)
  • Language
    Java
  • License
    Other
  • Created over 7 years ago
  • Updated over 4 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Mass Service Engine in Cluster(MSEC) is opened source by QQ team from Tencent. It is a backend DEV &OPS engine, including RPC,name finding,load balance,monitoring,release and capacity management.

项目目前是存档状态,感谢您对腾讯开源项目的关注!您可以继续fork后更新迭代,感谢理解和支持;如果您有其他疑问,建议请发送邮件:[email protected] 与我们联系


1、概述

这里是毫秒服务引擎的代码,分好几个模块,开发语言主要是java和c/c++。每个模块需要单独编译。 同时提供编译好的docker镜像可以快速部署,强烈推荐。

编译环境要求

  • gcc version >= 4.1.2
  • autoconf version >= 2.59
  • JDK version > = 1.6
  • Maven version > = 3.2.5
  • Linux内核版本>=2.6.18
  • CPU架构:x86_64

关于用到的外部库

下载到源代码后,在能够连接外网的linux服务器上手工执行get_third_party.sh脚本,会下载各模块用到的外部库,保存在third_party目录下; srpc_java和logsys是maven组织的java项目,mvn通过pom.xml配置自动下载用到的外部库。

2、web console

2.1 简介

Web console的代码在msec_console子目录,是一个典型的Java Web Application。目录msec_console下的src子目录是java代码,web子目录是静态页面、css、图片等web资源文件。

2.2 用到的外部库

  • commons-compress-1.8.jar
  • commons-fileupload-1.2.2.jar
  • commons-io-2.4.jar
  • ini4j-0.5.4.jar
  • jackson-all-1.6.0.jar
  • javax.servlet-api-3.1.0.jar
  • jcommon-1.0.21.jar
  • jfreechart-1.0.18.jar
  • junit-4.12.jar
  • log4j-1.2.17.jar
  • mysql-connector-java-5.1.38-bin.jar
  • org.json.jar
  • protobuf-java-2.6.1.jar
  • scf4j-props-1.0.1.jar
  • slf4j-api-1.7.18.jar
  • zookeeper-3.4.8.jar

几乎所有上面的jar都能在 http://mvnrepository.com/ 下载到。

#3、remote shell#

3.1 简介

子目录remote_shell是console服务器用来对业务运营机进行文件传输、远程命令执行的系统的代码。典型的应用场景是:发布的时候利用remote_shell系统传输发布文件、执行发布命令。

子目录remote_shell下又有三个子目录:

  • server是部署在console服务器上发出命令的程序,用c语言开发;
  • agent是部署在业务运营机上的程序,它接收server的命令,用java开发的;
  • InteractiveTool是和server部署在一起的一个命令行工具,通过它可以手工发起命令,用java开发的。

3.2 用到的外部库

server用到:

  • bcprov-jdk16-1.46.jar
  • jackson-all-1.6.0.jar
  • junit-4.12.jar
  • org.json.jar

agent用到:

  • frozen, 轻量的json解析C语言库

4、monitor

4.1 简介

子目录monitor是一套集中式的监控服务,该监控服务既可作为msec的集中式监控服务供msec业务上报,也可单独部署并使用独立的standalone_console进行管理。典型的应用场景是:业务上报监控到监控agent,监控agent汇总定时上报给监控服务,开发运营人员通过msec的web_console页面或独立部署的standalone_console页面查看业务监控汇总视图。

子目录monitor下又有三个子目录:

  • lib是供agent/server用的公共库,用c++语言开发;
  • server是监控服务,用c++语言开发;
  • agent是部署在业务运营机上的程序,它汇总业务模块的监控数据,定时上报给监控服务,用c++开发的;

standalone_console是一个典型的Java Web Application。子目录standalone_console下的src子目录是java代码,web子目录是静态页面、css、图片等web资源文件。

4.2 用到的外部库

server用到:

  • protobuf库(版本2.5.0)
  • mysqlclient库(版本>=5.5.0)
  • zlib库(版本>=1.2.0)

standalone_console用到:

  • commons-compress-1.8.jar
  • commons-fileupload-1.2.2.jar
  • commons-io-2.4.jar
  • jackson-all-1.6.0.jar
  • javax.servlet-api-3.1.0.jar
  • jcommon-1.0.21.jar
  • jfreechart-1.0.18.jar
  • junit-4.12.jar
  • log4j-1.2.17.jar
  • mysql-connector-java-5.1.38-bin.jar
  • org.json.jar
  • protobuf-java-2.6.1.jar
  • scf4j-props-1.0.1.jar
  • slf4j-api-1.7.18.jar
  • zookeeper-3.4.8.jar

5、redis

5.1 简介

子目录redis_console是一套基于redis cluster的KV运维管理平台。Web侧的代码在redis_console子目录,是一个典型的Java Web Application。子目录redis_console下的src子目录是java代码,web子目录是静态页面、css、图片等web资源文件。子目录monitor_server提供了一套专用于redis服务的监控系统,和msec的monitor服务的代码实现有些微的差别,但代码路径格式是一致的。

5.2 用到的外部库

monitor_server/server用到:

  • protobuf库(版本2.5.0)
  • mysqlclient库(版本>=5.5.0)
  • zlib库(版本>=1.2.0)

redis_console用到:

  • commons-compress-1.8.jar
  • commons-fileupload-1.2.2.jar
  • commons-io-2.4.jar
  • jackson-all-1.6.0.jar
  • javax.servlet-api-3.1.0.jar
  • jcommon-1.0.21.jar
  • jedis-2.8.1.jar
  • jfreechart-1.0.18.jar
  • junit-4.12.jar
  • log4j-1.2.17.jar
  • mysql-connector-java-5.1.38-bin.jar
  • org.json.jar
  • protobuf-java-2.6.1.jar
  • scf4j-props-1.0.1.jar
  • slf4j-api-1.7.18.jar
  • zookeeper-3.4.8.jar

特别注明: 如果用于部署,redis_console子目录中的web\resources\redis.tgz里需要加上redis的二进制服务,包含redis-server和redis-cli。

6、nlb

6.1 简介

nlb是一套网络负载均衡寻址系统。源码在子目录nlb下,使用纯C语言开发。src目录为所有代码,代码主要包含agent和api两部分,配置通过zookeeper集群下发,由web console导入配置到zookeeper集群。

源码目录介绍:

  • agent agent源码
  • api api源码
  • comm 通用代码
  • tools 工具源码:包含get_route和show_servers工具

6.2 用到的外部库

  • zookeeper-3.4.8
  • jansson-2.9

7、srpc

7.1 简介

srpc是一个逻辑层rpc框架。子目录spp_rpc是srpc的源码,使用C++作为主开发语言,支持C++/Python/PHP语言,C++支持微线程。

源码主要目录介绍:

  • sync_frame 微线程源码
  • controller controller进程源码
  • proxy proxy进程源码
  • worker worker进程源码
  • rpc rpc代码
  • module/rpc/template C++自动生成代码源码
  • module/rpc/php_template PHP自动生成代码源码
  • module/rpc/py_template Python自动生成代码源码

7.2 用到的外部库

  • protobuf-2.5.0
  • http-parser
  • jansson
  • json2pb
  • libbacktrace
  • libunwind
  • php
  • r3c

8、srpc_java

8.1 简介

srpc_java是一个逻辑层的java框架, 对应的子目录是msec_srpc_java。

源码主要目录介绍:

  • srpc 框架代码
  • src 业务示例代码
  • bin 框架执行脚本
  • lib 构建之后,框架依赖的库
  • pom.xml 框架通过maven构建

8.2 用到的外部库

  • commons-beanutils-1.7.0.jar
  • commons-cli-1.2.jar
  • commons-collections-3.1.jar
  • commons-lang-2.5.jar
  • commons-logging-1.1.1.jar
  • commons-logging.jar
  • ini4j-0.5.1.jar
  • junit-3.8.1.jar
  • log4j-1.2.17.jar
  • netty-3.2.10.Final.jar
  • protobuf-java-2.5.0.jar
  • protobuf-java-format-1.2.jar

9、Elasticsearch集群

9.1 简介

子目录elsticsearch是一套基于Elasticsearch的日志集群管理平台。

Web侧的代码在es_console子目录,是一个典型的Java Web Application。目录下的src子目录是java代码,web子目录是静态页面、css、图片等web资源文件。

业务侧需要部署flume agent,并通过日志系统提供的api将数据通过agent集中发往Elasticsearch集群,关键的代码在logsys子目录。该目录下的目录介绍如下:

  • api 日志系统提供的API
  • flume-ng-mysql-sink apache flume的插件,使日志数据写入mysql, 通过maven命令行构建
  • flume-protobuf-source apache flume的插件,读取protobuf协议格式的日志数据, 通过maven命令行构建

9.2 用到的外部库

es_console用到的库:

  • Elasticsearch API v5.2 (通过pom.xml获取)
  • commons-compress-1.8.jar
  • commons-fileupload-1.2.2.jar
  • commons-io-2.4.jar
  • jackson-all-1.6.0.jar
  • javax.servlet-api-3.1.0.jar
  • jcommon-1.0.21.jar
  • jfreechart-1.0.18.jar
  • junit-4.12.jar
  • log4j-1.2.17.jar
  • mysql-connector-java-5.1.38-bin.jar
  • org.json.jar
  • protobuf-java-2.6.1.jar
  • scf4j-props-1.0.1.jar
  • slf4j-api-1.7.18.jar

logsys用到的库:

  • protobuf-2.5.0
  • commons-beanutils-1.7.0.jar
  • commons-cli-1.2.jar
  • commons-collections-3.1.jar
  • commons-lang-2.5.jar
  • commons-logging-1.1.1.jar
  • ezmorph-1.0.6.jar
  • jackson-core-asl-1.9.4.jar
  • jackson-mapper-asl-1.9.4.jar
  • json-lib-2.4-jdk15.jar
  • junit-3.8.1.jar
  • log4j-1.2.17.jar
  • mysql-connector-java-5.1.25.jar

More Repositories

1

weui

A UI library by WeChat official design team, includes the most useful widgets/modules in mobile web applications.
Less
27,045
star
2

wepy

小程序组件化开发框架
JavaScript
22,396
star
3

ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform
C++
18,267
star
4

mars

Mars is a cross-platform network component developed by WeChat.
C++
17,089
star
5

tinker

Tinker is a hot-fix solution library for Android, it supports dex, library and resources update without reinstall apk.
Java
17,013
star
6

MMKV

An efficient, small mobile key-value storage framework developed by WeChat. Works on Android, iOS, macOS, Windows, and POSIX.
C++
16,576
star
7

APIJSON

🏆 零代码、全功能、强安全 ORM 库 🚀 后端接口和文档零代码,前端(客户端) 定制返回 JSON 的数据和结构。 🏆 A JSON Transmission Protocol and an ORM Library 🚀 provides APIs and Docs without writing any code.
Java
16,474
star
8

vConsole

A lightweight, extendable front-end developer tool for mobile web page.
TypeScript
16,379
star
9

weui-wxss

A UI library by WeChat official design team, includes the most useful widgets/modules.
Less
14,962
star
10

QMUI_Android

提高 Android UI 开发效率的 UI 库
Java
14,298
star
11

rapidjson

A fast JSON parser/generator for C++ with both SAX/DOM style API
C++
13,742
star
12

secguide

面向开发人员梳理的代码安全指南
13,014
star
13

omi

Web Components Framework - Web组件框架
TypeScript
12,869
star
14

VasSonic

VasSonic is a lightweight and high-performance Hybrid framework developed by tencent VAS team, which is intended to speed up the first screen of websites working on Android and iOS platform.
Java
11,745
star
15

matrix

Matrix is a plugin style, non-invasive APM system developed by WeChat.
Java
11,335
star
16

wcdb

WCDB is a cross-platform database framework developed by WeChat.
C
9,963
star
17

xLua

xLua is a lua programming solution for C# ( Unity, .Net, Mono) , it supports android, ios, windows, linux, osx, etc.
C
8,995
star
18

libco

libco is a coroutine library which is widely used in wechat back-end service. It has been running on tens of thousands of machines since 2013.
C++
7,998
star
19

Hippy

Hippy is designed to easily build cross-platform dynamic apps. 👏
Java
7,776
star
20

Shadow

零反射全动态Android插件框架
Java
7,167
star
21

QMUI_iOS

QMUI iOS——致力于提高项目 UI 开发效率的解决方案
Objective-C
7,003
star
22

MLeaksFinder

Find memory leaks in your iOS app at develop time.
Objective-C
5,384
star
23

lemon-cleaner

腾讯柠檬清理是针对macOS系统专属制定的清理工具。主要功能包括重复文件和相似照片的识别、软件的定制化垃圾扫描、可视化的全盘空间分析、内存释放、浏览器隐私清理以及设备实时状态的监控等。重点聚焦清理功能,对上百款软件提供定制化的清理方案,提供专业的清理建议,帮助用户轻松完成一键式清理。
Objective-C
5,119
star
24

kbone

一个致力于微信小程序和 Web 端同构的解决方案
JavaScript
4,711
star
25

libpag

The official rendering library for PAG (Portable Animated Graphics) files that renders After Effects animations natively across multiple platforms.
C++
4,618
star
26

puerts

PUER(普洱) Typescript. Let's write your game in UE or Unity with TypeScript.
C++
4,543
star
27

GT

GT (Great Tit) is a portable debugging tool for bug hunting and performance tuning on smartphones anytime and anywhere just as listening music with Walkman. GT can act as the Integrated Debug Environment by directly running on smartphones.
Java
4,383
star
28

TNN

TNN: developed by Tencent Youtu Lab and Guangying Lab, a uniform deep learning inference framework for mobile、desktop and server. TNN is distinguished by several outstanding features, including its cross-platform capability, high performance, model compression and code pruning. Based on ncnn and Rapidnet, TNN further strengthens the support and performance optimization for mobile devices, and also draws on the advantages of good extensibility and high performance from existed open source efforts. TNN has been deployed in multiple Apps from Tencent, such as Mobile QQ, Weishi, Pitu, etc. Contributions are welcome to work in collaborative with us and make TNN a better framework.
C++
4,247
star
29

westore

小程序项目分层架构
JavaScript
4,200
star
30

tmagic-editor

TypeScript
3,972
star
31

vap

VAP是企鹅电竞开发,用于播放特效动画的实现方案。具有高压缩率、硬件解码等优点。同时支持 iOS,Android,Web 平台。
Objective-C
3,763
star
32

wujie

极致的微前端框架
TypeScript
3,663
star
33

phxpaxos

The Paxos library implemented in C++ that has been used in the WeChat production environment.
C++
3,299
star
34

WeFlow

A web developer workflow tool by WeChat team based on tmt-workflow, with cross-platform supported and environment ready.
JavaScript
3,224
star
35

weui.js

A lightweight javascript library for WeUI.
JavaScript
3,150
star
36

cherry-markdown

✨ A Markdown Editor
JavaScript
3,122
star
37

spring-cloud-tencent

Spring Cloud Tencent is a Spring Cloud based Service Governance Framework provided by Tencent.
Java
3,111
star
38

tencent-ml-images

Largest multi-label image database; ResNet-101 model; 80.73% top-1 acc on ImageNet
Python
3,043
star
39

tdesign

Enterprise Design System
Vue
2,956
star
40

VasDolly

Android V1 and V2 Signature Channel Package Plugin
Java
2,950
star
41

PhoenixGo

Go AI program which implements the AlphaGo Zero paper
C++
2,853
star
42

FaceDetection-DSFD

腾讯优图高精度双分支人脸检测器
Python
2,845
star
43

PocketFlow

An Automatic Model Compression (AutoMC) framework for developing smaller and faster AI applications.
Python
2,774
star
44

Tendis

Tendis is a high-performance distributed storage system fully compatible with the Redis protocol.
C++
2,753
star
45

behaviac

behaviac is a framework of the game AI development, and it also can be used as a rapid game prototype design tool. behaviac supports the behavior tree, finite state machine and hierarchical task network(BT, FSM, HTN)
C#
2,739
star
46

phxsql

A high availability MySQL cluster that guarantees data consistency between a master and slaves.
C++
2,463
star
47

OOMDetector

OOMDetector is a memory monitoring component for iOS which provides you with OOM monitoring, memory allocation monitoring, memory leak detection and other functions.
Objective-C++
2,290
star
48

tsf

coroutine and Swoole based php server framework in tencent
PHP
2,180
star
49

tmt-workflow

A web developer workflow used by WeChat team based on Gulp, with cross-platform supported and solutions prepared.
CSS
2,175
star
50

Hardcoder

Hardcoder is a solution which allows Android APP and Android System to communicate with each other directly, solving the problem that Android APP could only use system standard API rather than the hardware resource of system.
C++
2,136
star
51

LKImageKit

A high-performance image framework, including a series of capabilities such as image views, image downloader, memory caches, disk caches, image decoders and image processors.
Objective-C
2,081
star
52

TubeMQ

TubeMQ has been donated to the Apache Software Foundation and renamed to InLong, please visit the new Apache repository: https://github.com/apache/incubator-inlong
2,030
star
53

UnLua

A feature-rich, easy-learning and highly optimized Lua scripting plugin for UE.
C++
2,010
star
54

ObjectDetection-OneStageDet

单阶段通用目标检测器
Python
1,956
star
55

cloudbase-framework

腾讯云开发云原生一体化部署工具 🚀 CloudBase Framework:一键部署,不限框架语言,云端一体化开发,基于Serverless 架构。A front-end and back-end integrated deployment tool. One-click deploy to serverless architecture. https://docs.cloudbase.net/framework/index
JavaScript
1,931
star
56

InjectFix

InjectFix is a hot-fix solution library for Unity
C#
1,921
star
57

soter

A secure and quick biometric authentication standard and platform in Android held by Tencent.
Java
1,896
star
58

phxrpc

A simple C++ based RPC framework.
C++
1,894
star
59

phxqueue

A high-availability, high-throughput and highly reliable distributed queue based on the Paxos algorithm.
C++
1,891
star
60

plato

腾讯高性能分布式图计算框架Plato
C++
1,885
star
61

TscanCode

A static code analyzer for C++, C#, Lua
C++
1,868
star
62

TSW

Tencent Server Web
TypeScript
1,798
star
63

MedicalNet

Many studies have shown that the performance on deep learning is significantly affected by volume of training data. The MedicalNet project provides a series of 3D-ResNet pre-trained models and relative code.
Python
1,793
star
64

NeuralNLP-NeuralClassifier

An Open-source Neural Hierarchical Multi-label Text Classification Toolkit
Python
1,741
star
65

GameAISDK

基于图像的游戏AI自动化框架
C++
1,738
star
66

QMUI_Web

An efficient front-end framework for developers building UI on the web.
JavaScript
1,720
star
67

Biny

Biny is a tiny, high-performance PHP framework for web applications
PHP
1,689
star
68

sluaunreal

lua dev plugin for unreal engine 4 or 5
C++
1,664
star
69

paxosstore

PaxosStore has been deployed in WeChat production for more than two years, providing storage services for the core businesses of WeChat backend. Now PaxosStore is running on thousands of machines, and is able to afford billions of peak TPS.
C++
1,647
star
70

Metis

Metis is a learnware platform in the field of AIOps.
Python
1,609
star
71

CodeAnalysis

Static Code Analysis - 静态代码分析
Python
1,563
star
72

TurboTransformers

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.
C++
1,426
star
73

TencentOS-kernel

腾讯针对云的场景研发的服务器操作系统
1,401
star
74

nohost

基于 Whistle 实现的多账号多环境远程配置及抓包调试平台
JavaScript
1,377
star
75

TBase

TBase is an enterprise-level distributed HTAP database. Through a single database cluster to provide users with highly consistent distributed database services and high-performance data warehouse services, a set of integrated enterprise-level solutions is formed.
C
1,364
star
76

WeDemo

WeDemo为微信团队开源项目,用于帮助微信开发者完成微信登录、微信分享等功能的接入和开发。开发者可参考源代码完成开发,也可以直接将代码应用到自己的App开发中,安全、便捷地在App中实现微信分享、微信登录功能。
Objective-C
1,354
star
77

feflow

🚀 A command line tool aims to improve front-end engineer workflow and standard, powered by TypeScript.
TypeScript
1,348
star
78

GAutomator

Automation for mobile games
Objective-C
1,297
star
79

tdesign-vue-next

A Vue3.x UI components lib for TDesign.
TypeScript
1,282
star
80

flare

Flare是广泛投产于腾讯广告后台的现代化C++开发框架,包含了基础库、RPC、各种客户端等。主要特点为易用性强、长尾延迟低。
C++
1,241
star
81

FeatherCNN

FeatherCNN is a high performance inference engine for convolutional neural networks.
C++
1,206
star
82

TFace

A trusty face analysis research platform developed by Tencent Youtu Lab
Python
1,203
star
83

LuaPanda

lua debug and code tools for VS Code
Lua
1,149
star
84

tdesign-miniprogram

A Wechat MiniProgram UI components lib for TDesign.
HTML
1,035
star
85

tgfx

A lightweight 2D graphics library for rendering texts, geometries, and images with high-performance APIs that work across various platforms.
C++
992
star
86

RapidView

RapidView is an android ui and lightapp development framework
Java
978
star
87

TencentPretrain

Tencent Pre-training framework in PyTorch & Pre-trained Model Zoo
Python
953
star
88

FAutoTest

A UI automated testing framework for H5 and applets
Python
927
star
89

TencentKona-8

Tencent Kona is a no-cost, production-ready distribution of the Open Java Development Kit (OpenJDK), Long-term support(LTS) with quarterly updates. Tencent Kona serves as the default JDK internally at Tencent Cloud for cloud computing and other Java applications.
Java
904
star
90

tdesign-vue

A Vue.js UI components lib for TDesign.
TypeScript
862
star
91

hel

A module federation SDK which is unrelated to tool chain for module consumer. 工具链无关的运行时模块联邦sdk.
JavaScript
861
star
92

Pebble

Pebble分布式开发框架
C++
860
star
93

tquic

A high-performance, lightweight, and cross-platform QUIC library
Rust
839
star
94

mxflutter

使用 TypeScript/JavaScript 来开发 Flutter 应用的框架。
Dart
818
star
95

Face2FaceTranslator

面对面翻译小程序是微信团队针对面对面沟通的场景开发的流式语音翻译小程序,通过微信同声传译插件提供了语音识别,文本翻译等功能。
JavaScript
804
star
96

tdesign-react

A React UI components lib for TDesign.
TypeScript
772
star
97

DCache

A distributed in-memory NOSQL system based on TARS framework, support LRU algorithm and data persists on back-end database. Users can easily deploy, publish, and scale services on the web interface.
C++
747
star
98

Real-SR

Real-World Super-Resolution via Kernel Estimation and Noise Injection
Python
740
star
99

PatrickStar

PatrickStar enables Larger, Faster, Greener Pretrained Models for NLP and democratizes AI for everyone.
Python
734
star
100

HaboMalHunter

HaboMalHunter is a sub-project of Habo Malware Analysis System (https://habo.qq.com), which can be used for automated malware analysis and security assessment on the Linux system.
Python
721
star