Apache Hadoop Connectors

Libraries and tools for interoperability between Apache Hadoop related open-source software and Google Cloud Platform.

Google Cloud Storage connector for Apache Hadoop (HCFS)

The Google Cloud Storage connector for Hadoop enables running MapReduce jobs directly on data in Google Cloud Storage by implementing the Hadoop FileSystem interface. For details, see the README.

Building the Cloud Storage connector

Note that build requires Java 11+ and fails with older Java versions.

To build the connector for specific Hadoop version, run the following commands from the main directory:

./mvnw clean package

In order to verify test coverage for specific Hadoop version, run the following commands from the main directory:

./mvnw -P coverage clean verify

The Cloud Storage connector JAR can be found in gcs/target/ directory.

Adding the Cloud Storage connector to your build

Maven group ID is com.google.cloud.bigdataoss and artifact ID for Cloud Storage connector is gcs-connector.

To add a dependency on Cloud Storage connector using Maven, use the following:

<dependency>
  <groupId>com.google.cloud.bigdataoss</groupId>
  <artifactId>gcs-connector</artifactId>
  <version>hadoop3-2.2.10</version>
</dependency>

Resources

On Stack Overflow, use the tag google-cloud-dataproc for questions about the connectors in this repository. This tag receives responses from the Stack Overflow community and Google engineers, who monitor the tag and offer unofficial support.