• Stars
    star
    267
  • Rank 153,621 (Top 4 %)
  • Language
    HTML
  • License
    Apache License 2.0
  • Created about 8 years ago
  • Updated almost 2 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

java 版本的logstash

注释:

  jlogstash前期的有部分代码引用了hangout项目里的代码,这里也感谢hangout的作者。

说明:

用java版本重写logstash目的是提升性能,跟ruby logstash 对比 请看 https://github.com/DTStack/jlogstash-performance-testing

不同的业务,性能会有所不同,dtstack 后台日志解析 java 版本是ruby版本的5倍,在单台4g 4cores 虚拟机上。

jlogstash 的参数配置和使用看wiki介绍,现在的插件跟ruby版本相比还太少,希望更多的人参与开发。

Inputs详情:

https://github.com/DTStack/jlogstash/tree/master/pipeline/inputs/README.md

Filters详情:

https://github.com/DTStack/jlogstash/tree/master/pipeline/filters/README.md

Outputs详情:

https://github.com/DTStack/jlogstash/tree/master/pipeline/outputs/README.md

性能采集:

目前支持将jlogstash运行时的性能数据采集到promethues.

详情:https://github.com/DTStack/jlogstash/tree/master/core/metrics.md

Jar放置目录(编译的jar必须要有版本号 ):

jlogstash 核心代码放在jlogstash/lib/下

插件的代码分别的放到jlogstash/plugin 下的filter,input,output目录下

启动命令:

sh jlogstash.sh -f /home/admin/ysq.yaml vv

启动参数:

-name:任务名称

-f:配置文件 yaml格式路径(必填)

-l:日志文件路径

-i:input queue size coefficient 默认 200f/1024

-w:filter work number 默认是根据的机器cpu核数+2

-o:output work number 默认是根据的机器cpu核数

-c:output queue size coefficient 默认 500f/1024

-dev: 开发模式,直接在pom.xml引用包即可。

v: error级别

vv: warn级别

vvv:info级别

vvvv:debug级别

vvvvv:trace级别

插件开发:

 1.现在各自的plugin 的包 都会有各自的classloder去加载,parent classloder是AppClassLoder,所以各自的plugin的代码即使引用了相同的第三的jar版   本不一样也不会导致版本冲突  

2.各个插件的代码不能相互引用,如果有公共代码需要打入到各自的jar包中

3.所需依赖到maven中心库 搜索 jlogstash(http://search.maven.org/https://oss.sonatype.org)

4.插件开发样列 https://github.com/DTStack/jlogstash/tree/master/src/test/java/com/dtstack/jlogstash

5.每一个plugin打的包名的前缀要跟插件的类名一致,不区分大小写,不然会报类找不到,列如:input.kafka-1.0.0.jar 或    kafka-1.0.0.jar

招聘:

1.大数据平台开发工程师,想了解岗位详细信息可以添加本人微信号ysqwhiletrue,注明招聘,如有意者发送简历至[email protected]

More Repositories

1

chunjun

A data integration framework
Java
3,970
star
2

flinkStreamSQL

基于开源的flink,对其实时sql进行扩展;主要实现了流与维表的join,支持原生flink SQL所有的语法
Java
2,017
star
3

Taier

Taier is a big data development platform for submission, scheduling, operation and maintenance, and indicator information display
Java
1,329
star
4

molecule

🚀 A lightweight Web IDE UI framework.
TypeScript
879
star
5

dt-sql-parser

SQL Parsers for BigData, built with antlr4.
TypeScript
293
star
6

monaco-sql-languages

SQL languages for monaco-editor
TypeScript
217
star
7

chengying

一款支持标准化schema定义、自动化部署产品包的软件,旨在对产品包下每个服务进行部署、升级、卸载、配置等操作,解放人工运维成本。
Go
197
star
8

dt-react-component

React UI component library based on antd package
TypeScript
84
star
9

jfilebeat

类filebeat的轻量级日志采集工具
Java
68
star
10

DatasourceX

Java
65
star
11

catcher

java性能采集工具
Java
51
star
12

flinkx

48
star
13

code-review-practices

Code Review Practices
48
star
14

yice-performance

易测性能检测平台
TypeScript
46
star
15

doraemon

A management tool to help you organize your daily development
TypeScript
32
star
16

chengying-agent

EasyAgent is an infrastructure component, applied to manage the life-cycle of services on the remote host.
Go
32
star
17

dt-utils

前端常用工具函数
TypeScript
30
star
18

dt-python-parser

Python Parsers for BigData, built with antlr4.
JavaScript
25
star
19

ko

Project toolkit for React Applications
JavaScript
24
star
20

dt-react-monaco-editor

Monaco editor for React.
TypeScript
22
star
21

jlogstash-input-plugin

java 版本 logstash input 插件
Java
21
star
22

jlogstash-performance-testing

jlogstash 与 logstash 性能对比
20
star
23

molecule-examples

The collection of Molecule examples
TypeScript
20
star
24

UED

袋鼠云数栈 UED 团队 - http://ued.dtstack.cn/
TypeScript
20
star
25

ant-design-dtinsight-theme

This is a document of DTInsight-theme based on Ant Design.
Less
20
star
26

dt-form-renderer

Render Interaction Form Via JSON
TypeScript
18
star
27

ant-design-testing

TypeScript
16
star
28

babel-plugin-treasure

Base on babel-plugin-import , committed to achieving the AST optimization requirements of a unified library
HTML
11
star
29

chengying-server

Go
10
star
30

Code-Style-Guide

10
star
31

chunjun-web

ChunJun Offical Website https://dtstack.github.io/chunjun-web/
JavaScript
8
star
32

jlogstash-output-plugin

java 版本 logstash output 插件
Java
7
star
33

jlogstash-filter-plugin

java 版本 logstash filter 插件
Java
7
star
34

easyvc-power-meter

an open source component code for demonstrating @easyv/cli.
JavaScript
6
star
35

chengying-schema

Shell
6
star
36

chengying-front

TypeScript
5
star
37

chengying-web

JavaScript
4
star
38

typescript-migration-helper

Help ES6 + React + Redux project migrates to the Typescript project.
Perl
2
star
39

easyv-cli

EasyV官方组件开发工具
2
star
40

dtstack-log-java-sdk

玳数日志java版本sdk
Java
2
star
41

create-molecule

Create Molecule Application with create-react-app.
TypeScript
2
star
42

dt-react-codemirror-editor

Codemirror editor for React.
TypeScript
2
star
43

maven-repository

1
star
44

elasticsearch-sql-old

Java
1
star
45

dt-monaco-editor-nls-webpack-plugin

Simplified Chinese Support For Monaco Editor
JavaScript
1
star