E-MapReduce DataSources
Requirements
- Spark 1.3+
Introduction
- This project supports interaction with Aliyun's base service, e.g. OSS, ODPS, LogService and ONS, in Spark runtime environment.
Build and Install
git clone https://github.com/aliyun/aliyun-emapreduce-datasources.git
cd aliyun-emapreduce-datasources
mvn clean package -DskipTests
Build emr-maxcompute with Spark3.2.0
git clone https://github.com/aliyun/aliyun-emapreduce-datasources.git
cd aliyun-emapreduce-datasources/emr-maxcompute/
mvn clean package -Pspark3 -DskipTests
Use SDK in Eclipse project directly
- copy sdk jar to your project
- right click Eclipse project -> Properties -> Java Build Path -> Add JARs
- choose and import the sdk
- you can use the sdk in your Eclipse project
Maven
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-maxcompute_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-logservice_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-tablestore</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-ons_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-mns_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-redis_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-hbase_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-jdbc_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-dts_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-kudu_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-datahub_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-druid_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-sql_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-oss</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-common</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.aliyun.emr</groupId>
<artifactId>emr-kafka-client-metrics</artifactId>
<version>2.2.0</version>
</dependency>
Run tests
JindoFS/OSS support
- Hadoop on JindoFS/OSS (Hive, Spark, Presto, Impala, Hbase and Flink are also supported)
MaxCompute support
ONS support
LogService support
TableStore support
License
Licensed under the Apache License 2.0