alibaba/yugong

Stars
2,504
Rank 18,332 (Top 0.4 %)
Language
Java
License
GNU General Publi...
Created almost 9 years ago
Updated almost 2 years ago

alibaba/yugong

alibaba

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

阿里巴巴去Oracle数据迁移同步工具(全量+增量,目标支持MySQL/DRDS)

背景

2008年，阿里巴巴开始尝试使用 MySQL 支撑其业务，开发了围绕 MySQL 相关的中间件和工具，Cobar/TDDL(目前为阿里云DRDS产品)，解决了单机 Oracle 无法满足的扩展性问题，当时也掀起一股去IOE项目的浪潮，愚公这项目因此而诞生，其要解决的目标就是帮助用户完成从 Oracle 数据迁移到 MySQL 上，完成去 IOE 的重要一步工作。

项目介绍

名称: yugong

译意: 愚公移山

语言: 纯java开发

定位: 数据库迁移 (目前主要支持oracle / mysql / DRDS)

项目介绍

整个数据迁移过程，分为两部分：

全量迁移
增量迁移

过程描述：

增量数据收集 (创建oracle表的增量物化视图)
进行全量复制
进行增量复制 (可并行进行数据校验)
原库停写，切到新库

架构

{width="584" height="206"}

说明:

一个Jvm Container对应多个instance，每个instance对应于一张表的迁移任务
instance分为三部分 a. extractor (从源数据库上提取数据，可分为全量/增量实现) b. translator (将源库上的数据按照目标库的需求进行自定义转化) c. applier (将数据更新到目标库，可分为全量/增量/对比的实现)

方案设计

快速开始

运维管理

性能报告

相关资料

yugong简单介绍ppt: ppt
分布式关系型数据库服务DRDS (前身为阿里巴巴公司的Cobar/TDDL的演进版本, 基本原理为MySQL分库分表)

沟通与交流

详见 wiki home 页

arthas

Alibaba Java Diagnostic Tool Arthas/Alibaba Java诊断利器Arthas

easyexcel

快速、简洁、解决大文件内存溢出的java处理Excel工具

p3c

Alibaba Java Coding Guidelines pmd implements and IDE plugin

nacos

an easy-to-use dynamic service discovery, configuration and service management platform for building cloud native applications.

canal

阿里巴巴 MySQL binlog 增量订阅&消费组件

druid

阿里云计算平台DataWorks(https://help.aliyun.com/document_detail/137663.html) 团队出品，为监控而生的数据库连接池

spring-cloud-alibaba

Spring Cloud Alibaba provides a one-stop solution for application development for the distributed solutions of Alibaba middleware.

fastjson

FASTJSON 2.0.x has been released, faster and more secure, recommend you upgrade.

flutter-go

flutter 开发者帮助 APP，包含 flutter 常用 140+ 组件的demo 演示与中文文档

Sentinel

A powerful flow control component enabling reliability, resilience and monitoring for microservices. (面向云原生微服务的高可用流控防护组件)

weex

A framework for building Mobile cross-platform UI

ice

🚀 ice.js: The Progressive App Framework Based On React（基于 React 的渐进式应用框架）

DataX

DataX是阿里云DataWorks数据集成的开源版本。

lowcode-engine

An enterprise-class low-code technology stack with scale-out design / 一套面向扩展设计的企业级低代码技术体系

ARouter

💪 A framework for assisting in the renovation of Android componentization (帮助 Android App 进行组件化改造的路由框架)

hooks

A high-quality & reliable React Hooks library. https://ahooks.pages.dev/

tengine

A distribution of Nginx with some advanced features

formily

📱🚀 🧩 Cross Device & High Performance Normal Form/Dynamic(JSON Schema) Form/Form Builder -- Support React/React Native/Vue 2/Vue 3

vlayout

Project vlayout is a powerfull LayoutManager extension for RecyclerView, it provides a group of layouts for RecyclerView. Make it able to handle a complicate situation when grid, list and other layouts in the same recyclerview.

COLA

🥤 COLA: Clean Object-oriented & Layered Architecture

MNN

MNN is a blazing fast, lightweight deep learning framework, battle-tested by business-critical use cases in Alibaba

ali-dbhub

已迁移新仓库，此版本将不再维护

atlas

A powerful Android Dynamic Component Framework.

otter

阿里巴巴分布式数据库同步系统(解决中美异地机房)

rax

🐰 Rax is a progressive framework for building universal application. https://rax.js.org

anyproxy

A fully configurable http/https proxy in NodeJS

fish-redux

An assembled flutter application framework.

x-render

🚴‍♀️ 阿里 - 很易用的中后台「表单 / 表格 / 图表」解决方案

flutter_boost

FlutterBoost is a Flutter plugin which enables hybrid integration of Flutter for your existing native apps with minimum efforts

AndFix

AndFix is a library that offer hot-fix for Android App.

transmittable-thread-local

📌 TransmittableThreadLocal (TTL), the missing Java™ std lib(simple & 0-dependency) for framework/middleware, provide an enhanced InheritableThreadLocal that transmits values between threads even using thread pooling components.

jvm-sandbox

Real - time non-invasive AOP framework container based on JVM

BizCharts

Powerful data visualization library based on G2 and React.

freeline

A super fast build tool for Android, an alternative to Instant Run

UltraViewPager

UltraViewPager is an extension for ViewPager to provide multiple features in a single ViewPager.

jetcache

JetCache is a Java cache framework.

AliSQL

AliSQL is a MySQL branch originated from Alibaba Group. Fetch document from Release Notes at bottom.

AliOS-Things

面向IoT领域的、高可伸缩的物联网操作系统，可去官网了解更多信息https://www.aliyun.com/product/aliosthings

dexposed

dexposed enable 'god' mode for single android application.

butterfly

🦋Butterfly，A JavaScript/React/Vue2 Diagramming library which concentrate on flow layout field. (基于JavaScript/React/Vue2的流程图组件)

QLExpress

QLExpress is a powerful, lightweight, dynamic language for the Java platform aimed at improving developers’ productivity in different business scenes.

BeeHive

🐝 BeeHive is a solution for iOS Application module programs, it absorbed the Spring Framework API service concept to avoid coupling between modules.

HandyJSON

A handy swift json-object serialization/deserialization library

x-deeplearning

An industrial deep learning framework for high-dimension sparse data

Tangram-Android

Tangram is a modular UI solution for building native page dynamically including Tangram for Android, Tangram for iOS and even backend CMS. This project provides the sdk on Android.

coobjc

coobjc provides coroutine support for Objective-C and Swift. We added await method、generator and actor model like C#、Javascript and Kotlin. For convenience, we added coroutine categories for some Foundation and UIKit API in cokit framework like NSFileManager, JSON, NSData, UIImage etc. We also add tuple support in coobjc.

jstorm

Enterprise Stream Process Engine

dragonwell8

Alibaba Dragonwell8 JDK

LuaViewSDK

A cross-platform framework to build native, dynamic and swift user interface - 强大轻巧灵活的客户端动态化解决方案

fastjson2

🚄 FASTJSON2 is a Java JSON library with excellent performance.

Alink

Alink is the Machine Learning algorithm platform based on Flink, developed by the PAI team of Alibaba computing platform.

f2etest

F2etest是一个面向前端、测试、产品等岗位的多浏览器兼容性测试整体解决方案。

GGEditor

A visual graph editor based on G6 and React

GraphScope

🔨 🍇 💻 🚀 GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | 一站式图计算系统

designable

🧩 Make everything designable 🧩

cobar

a proxy for sharding databases and tables

macaca

Automation solution for multi-platform. 多端自动化解决方案

lightproxy

💎 Cross platform Web debugging proxy

pont

🌉数据服务层解决方案

higress

🤖 AI Gateway | AI Native API Gateway

euler

A distributed graph deep learning framework.

sentinel-golang

Sentinel Go enables reliability and resiliency for Go microservices

beidou

🌌 Isomorphic framework for server-rendered React apps

ChatUI

The UI design language and React library for Conversational UI

pipcook

Machine learning platform for Web developers

kiwi

🐤 Kiwi-国际化翻译全流程解决方案

jvm-sandbox-repeater

A Java server-side recording and playback solution based on JVM-Sandbox

tsar

Taobao System Activity Reporter

tidevice

tidevice can be used to communicate with iPhone device

TProfiler

TProfiler是一个可以在生产环境长期使用的性能分析工具

tair

A distributed key-value storage system developed by Alibaba Group

dubbo-spring-boot-starter

Dubbo Spring Boot Starter

RedisShake

redis-shake is a tool for synchronizing data between two redis databases. Redis-shake 是一个用于在两个 redis之间同步数据的工具，满足用户非常灵活的同步、迁移需求。

uirecorder

UI Recorder is a multi-platform UI test recorder.

EasyNLP

EasyNLP: A Comprehensive and Easy-to-use NLP Toolkit

AliceMind

ALIbaba's Collection of Encoder-decoders from MinD (Machine IntelligeNce of Damo) Lab

LVS

A distribution of Linux Virtual Server with some advanced features. It introduces a new packet forwarding method - FULLNAT other than NAT/Tunneling/DirectRouting, and defense mechanism against synflooding attack - SYNPROXY.

GCanvas

A lightweight cross-platform graphics rendering engine. (超轻量的跨平台图形引擎) https://alibaba.github.io/GCanvas

alpha

Alpha是一个基于PERT图构建的Android异步启动框架，它简单，高效，功能完善。在应用启动的时候，我们通常会有很多工作需要做，为了提高启动速度，我们会尽可能让这些工作并发进行。但这些工作之间可能存在前后依赖的关系，所以我们又需要想办法保证他们执行顺序的正确性。Alpha就是为此而设计的，使用者只需定义好自己的task，并描述它依赖的task，将它添加到Project中。框架会自动并发有序地执行这些task，并将执行的结果抛出来。

Tangram-iOS

Tangram is a modular UI solution for building native page dynamically, including Tangram for Android, Tangram for iOS and even backend CMS. This project provides the sdk on iOS platform.

testable-mock

换种思路写Mock，让单元测试更简单

compileflow

🎨 core business process engine of Alibaba Halo platform, best process engine for trade scenes. | 一个高性能流程编排引擎

SREWorks

Cloud Native DataOps & AIOps Platform | 云原生数智运维平台

EasyCV

An all-in-one toolkit for computer vision

LazyScrollView

An iOS ScrollView to resolve the problem of reusability in views.

EasyRec

A framework for large scale recommendation algorithms.

ilogtail

Fast and Lightweight Observability Data Collector

MongoShake

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具，可以满足迁移和同步的需求，进一步实现灾备和多活功能。

xquic

XQUIC Library released by Alibaba is a cross-platform implementation of QUIC and HTTP/3 protocol.

lowcode-demo

An enterprise-class low-code technology stack with scale-out design / 一套面向扩展设计的企业级低代码技术体系

async_simple

Simple, light-weight and easy-to-use asynchronous components

havenask

clusterdata

cluster data collected from production clusters in Alibaba for cluster management research

Jupyter Notebook

mdrill

for千亿数据即席分析

kt-connect

A toolkit for Integrating with your kubernetes dev environment more efficiently

Virtualview-Android

A light way to build UI in custom XML.

yalantinglibs

A collection of modern C++ libraries, include coro_rpc, struct_pack, struct_json, struct_xml, struct_pb, easylog, async_simple

tb_tddl

react-intl-universal

Internationalize React apps. Not only for Component but also for Vanilla JS.

data-juicer

A one-stop data processing system to make data higher-quality, juicier, and more digestible for LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大语言模型提供更高质量、更丰富、更易”消化“的数据！