• Stars
    star
    1,107
  • Rank 41,944 (Top 0.9 %)
  • Language
    Java
  • License
    Apache License 2.0
  • Created over 9 years ago
  • Updated about 2 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

A microservices-based Streaming and Batch data processing in Cloud Foundry and Kubernetes

Spring Data Flow Dashboard

Latest Release Version Latest Snapshot Version
Build Status

Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes.

Data processing pipelines consist of Spring Boot apps, built using the Spring Cloud Stream or Spring Cloud Task microservice frameworks.

This makes Spring Cloud Data Flow ideal for a range of data processing use cases, from import/export to event streaming and predictive analytics.


Components

Architecture: The Spring Cloud Data Flow Server is a Spring Boot application that provides RESTful API and REST clients (Shell, Dashboard, Java DSL). A single Spring Cloud Data Flow installation can support orchestrating the deployment of streams and tasks to Local, Cloud Foundry, and Kubernetes.

Familiarize yourself with the Spring Cloud Data Flow architecture and feature capabilities.

Deployer SPI: A Service Provider Interface (SPI) is defined in the Spring Cloud Deployer project. The Deployer SPI provides an abstraction layer for deploying the apps for a given streaming or batch data pipeline and managing the application lifecycle.

Spring Cloud Deployer Implementations:

Domain Model: The Spring Cloud Data Flow domain module includes the concept of a stream that is a composition of Spring Cloud Stream applications in a linear data pipeline from a source to a sink, optionally including processor application(s) in between. The domain also includes the concept of a task, which may be any process that does not run indefinitely, including Spring Batch jobs.

Application Registry: The App Registry maintains the metadata of the catalog of reusable applications. For example, if relying on Maven coordinates, an application URI would be of the format: maven://<groupId>:<artifactId>:<version>.

Shell/CLI: The Shell connects to the Spring Cloud Data Flow Server's REST API and supports a DSL that simplifies the process of defining a stream or task and managing its lifecycle.


Building

Clone the repo and type

$ ./mvnw clean install 

Looking for more information? Follow this link.

Building on Windows

When using Git on Windows to check out the project, it is important to handle line-endings correctly during checkouts. By default Git will change the line-endings during checkout to CRLF. This is, however, not desired for Spring Cloud Data Flow as this may lead to test failures under Windows.

Therefore, please ensure that you set Git property core.autocrlf to false, e.g. using: $ git config core.autocrlf false. For more information please refer to the Git documentation, Formatting and Whitespace.


Contributing

We welcome contributions! See the CONTRIBUTING guide for details.


Code formatting guidelines

  • The directory ./src/eclipse has two files for use with code formatting, eclipse-code-formatter.xml for the majority of the code formatting rules and eclipse.importorder to order the import statements.

  • In eclipse you import these files by navigating Windows -> Preferences and then the menu items Preferences > Java > Code Style > Formatter and Preferences > Java > Code Style > Organize Imports respectfully.

  • In IntelliJ, install the plugin Eclipse Code Formatter. You can find it by searching the "Browse Repositories" under the plugin option within IntelliJ (Once installed you will need to reboot Intellij for it to take effect). Then navigate to Intellij IDEA > Preferences and select the Eclipse Code Formatter. Select the eclipse-code-formatter.xml file for the field Eclipse Java Formatter config file and the file eclipse.importorder for the field Import order. Enable the Eclipse code formatter by clicking Use the Eclipse code formatter then click the OK button. ** NOTE: If you configure the Eclipse Code Formatter from File > Other Settings > Default Settings it will set this policy across all of your Intellij projects.

License

Spring Cloud Data Flow is Open Source software released under the Apache 2.0 license.

More Repositories

1

spring-cloud-netflix

Integration with Netflix OSS components
Java
4,866
star
2

spring-cloud-gateway

An API Gateway built on Spring Framework and Spring Boot providing routing and more.
Java
4,509
star
3

spring-cloud-kubernetes

Kubernetes integration with Spring Cloud Discovery Client, Configuration, etc...
Java
3,457
star
4

spring-cloud-config

External configuration (server and client) for Spring Cloud
Java
1,953
star
5

spring-cloud-sleuth

Distributed tracing for spring cloud
Java
1,770
star
6

spring-cloud-openfeign

Support for using OpenFeign in Spring Cloud apps
Java
1,202
star
7

spring-cloud-function

Java
1,037
star
8

spring-cloud-stream

Framework for building Event-Driven Microservices
Java
993
star
9

spring-cloud-stream-samples

Samples for Spring Cloud Stream
Java
954
star
10

spring-cloud-release

Spring Cloud Release Train - dependency management across a wide range of Spring Cloud projects.
874
star
11

spring-cloud-consul

Spring Cloud Consul
Java
813
star
12

spring-cloud-contract

Support for Consumer Driven Contracts in Spring
Java
717
star
13

spring-cloud-commons

Common classes used in different Spring Cloud implementations
Java
706
star
14

spring-cloud-zookeeper

Spring Cloud Zookeeper
Java
555
star
15

spring-cloud-task

Short lived microservices with Spring Batch
Java
426
star
16

spring-cloud-bus

Spring Cloud event bus
Java
407
star
17

spring-cloud-circuitbreaker

Spring Cloud Circuit Breaker API and Implementations
Java
335
star
18

spring-cloud-stream-binder-kafka

Spring Cloud Stream binders for Apache Kafka and Kafka Streams
Java
330
star
19

spring-cloud-vault

Configuration Integration with HashiCorp Vault
Java
274
star
20

stream-applications

Functions and Spring Cloud Stream Applications for data driven microservices
Java
251
star
21

spring-cloud-dataflow-samples

Sample starter applications and code for use with the Spring Cloud Data Flow project
Java
223
star
22

spring-cloud-dataflow-ui

This repo provides the Dashboard application of Spring Cloud Data Flow
TypeScript
210
star
23

spring-cloud-build

Common build concerns, shared plugin configuration, etc. for Spring Cloud modules
Shell
191
star
24

spring-cloud-connectors

Library to let cloud applications connect to services
Java
185
star
25

spring-cloud-open-service-broker

Spring Cloud project for creating service brokers that conform to the Open Server Broker API specification
Java
167
star
26

spring-cloud-deployer

The Spring Cloud Deployer project defines an SPI for deploying long lived applications and short lived tasks
Java
161
star
27

spring-cloud-deployer-kubernetes

The Spring Cloud Deployer implementation for Kubernetes
Java
157
star
28

spring-cloud-cli

Spring Cloud CLI features
Java
155
star
29

spring-cloud-stream-binder-rabbit

Java
155
star
30

spring-cloud-core-tests

Integration tests for Spring Cloud (small projects testing classpath combinations)
Java
148
star
31

spring-cloud-skipper

A package manager that installs, upgrades, and rolls back Spring Boot applications on multiple Cloud Platforms.
Java
111
star
32

spring-cloud-stream-binder-aws-kinesis

Spring Cloud Stream binder for AWS Kinesis
Java
99
star
33

spring-cloud-stream-starters

Shell
98
star
34

spring-cloud-cloudfoundry

Integration between Cloudfoundry and the Spring Cloud APIs
Java
80
star
35

spring-cloud-schema-registry

A schema registry implementation for Spring Cloud Stream
Java
47
star
36

spring-cloud-bindings

A library that exposes a rich Java language-binding and auto-configuration for CNB Bindings
Java
41
star
37

spring-cloud-jenkins-jobs

Spring Cloud Jenkins Job DSL
Groovy
41
star
38

spring-cloud-deployer-local

The Spring Cloud Deployer implementation for a "local" machine
Java
39
star
39

spring-cloud-release-tools

Tools used for the Spring Cloud release process
Java
28
star
40

spring-cloud-deployer-cloudfoundry

The Spring Cloud Deployer implementation for Cloud Foundry
Java
27
star
41

spring-cloud-app-broker

Spring Cloud project for implementing service brokers that conform to the Open Server Broker API specification
Java
26
star
42

spring-cloud-static

Static resources for the Spring Cloud website
26
star
43

spring-cloud-common-security-config

A common security infrastructure used by Spring Cloud Data Flow and the projects in its ecosystem
Java
19
star
44

spring-functions-catalog

Reusable Spring Functions for data driven microservices
Java
17
star
45

spring-cloud-dataflow-acceptance-tests

Java
13
star
46

baseimage

Shell
12
star
47

spring-cloud.github.io

GH pages for spring-cloud
HTML
7
star
48

spring-cloud-dataflow-apps-plugin

Java
7
star
49

stream-applications-acceptance-tests

Shell
7
star
50

spring-cloud-stream-jenkins-jobs

Groovy
4
star
51

spring-cloud-dataflow-common

Java
2
star
52

.github

2
star
53

spring-cloud-dataflow-build

1
star