• Stars
    star
    181
  • Rank 212,110 (Top 5 %)
  • Language
    Go
  • License
    Apache License 2.0
  • Created over 3 years ago
  • Updated 5 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Meteor is an easy-to-use, plugin-driven metadata collection framework to extract data from different sources and sink to any data catalog.

Meteor

test workflow build workflow Go Report Card Coverage Status Version License

Meteor is a plugin driven agent for collecting metadata. Meteor has plugins to source metadata from a variety of data stores, services and message queues. It also has sink plugins to send metadata to variety of third party APIs and catalog services.

Key Features

  • No Dependency: Written in Go. It compiles into a single binary with no external dependency.
  • Extensible: Plugin system allows new sources and sinks to be easily added.
  • Ecosystem: Extract metadata for many popular services with a wide number of service plugins.
  • Customizable: Add your own processors and sinks to suit your many use cases.
  • Runtime: Meteor can run inside VMs or containers with minimal memory footprint.

Documentation

Explore the following resources to get started with Meteor:

  • Usage Guides will help you get started on Meteor.
  • Concepts describes all important Meteor concepts.
  • Contribute contains resources for anyone who wants to contribute to Meteor.

Installation

Install Meteor on macOS, Windows, Linux, OpenBSD, FreeBSD, and on any machine.

Binary (Cross-platform)

Download the appropriate version for your platform from releases page. Once downloaded, the binary can be run from anywhere. You donโ€™t need to install it into a global location. This works well for shared hosts and other systems where you donโ€™t have a privileged account. Ideally, you should install it somewhere in your PATH for easy use. /usr/local/bin is the most probable location.

Homebrew

# Install meteor (requires homebrew installed)
$ brew install raystack/tap/meteor

# Upgrade meteor (requires homebrew installed)
$ brew upgrade meteor

# Check for installed meteor version
$ meteor version

Usage

Meteorโ€™s CLI is fully featured but simple to use, even for those who have very limited experience working from the command line. Run meteor --help to see list of all available commands and instructions to use.

# List of commands
$ meteor --help

# Print command reference
$ meteor reference

Running locally

# Clone the repo
$ git clone https://github.com/raystack/meteor.git

# Install all the golang dependencies
$ go mod tidy

# Build meteor binary file
$ make build

# Run meteor on a recipe file
$ ./meteor run sample-recipe.yaml

# Run meteor on multiple recipes in a directory
$ ./meteor run directory-path

Running tests

# Running all unit tests, excluding extractors
$ make test

# Run integration test for any extractor
$ cd plugins/extractors/<name-of-extractor>
$ go test -tags=integration

Contribute

Development of Meteor happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving Meteor.

Read our contributing guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Meteor.

To help you get your feet wet and get you familiar with our contribution process, we have a list of good first issues that contain bugs which have a relatively limited scope. This is a great place to get started.

This project exists thanks to all the contributors.

License

Meteor is Apache 2.0 licensed.

More Repositories

1

optimus

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Go
742
star
2

firehose

Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.
Java
319
star
3

dagger

Dagger is an easy-to-use, configuration over code, cloud-native framework built on top of Apache Flink for stateful processing of real-time streaming data.
Java
261
star
4

frontier

Frontier is an all-in-one user management platform that provides identity, access and billing management to help organizations secure their systems and data. (Open source alternative to Clerk)
Go
252
star
5

stencil

Stencil is a schema registry that provides schema management and validation dynamically, efficiently, and reliably to ensure data compatibility across applications.
Go
221
star
6

raccoon

Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols.
Go
192
star
7

guardian

Guardian is universal data access management tool with automated access workflows and security controls across data stores, analytical systems, and cloud products.
Go
137
star
8

siren

Siren provides an easy-to-use universal alert, notification, channels management framework for the entire observability infrastructure.
Go
77
star
9

compass

Compass is an enterprise data catalog that makes it easy to find, understand, and govern data.
Go
63
star
10

apsara

Apsara is an open-source re-usable UI components built using Radix UI and CSS modules to power Raystack projects.
TypeScript
56
star
11

proton

This repository is home to the original protobuf interface definitions which are used throughout the Raystack ecosystem.
54
star
12

cosmos

Cosmos is an operational analytics server to build custom apps with embedded analytics that deliver data experiences as unique as your business.
TypeScript
46
star
13

charts

This repository is home to the original helm charts for products throughout the open data platform ecosystem.
Smarty
41
star
14

transformers

This repository is home to the Optimus data transformation plugins for various data processing needs.
Python
35
star
15

homebrew-tap

This repository is home to the original homebrew taps for products throughout the Raystack ecosystem.
Ruby
31
star
16

platform

ODPF is the next-gen collaborative and distributed data platform to power data-driven workflows.
30
star
17

entropy

Entropy is a framework to safely and predictably create, change, and improve modern cloud applications and infrastructure using familiar languages, tools, and engineering practices.
Go
19
star
18

handbook

Handbook is the central repository for how we build products within ODPF community.
CSS
14
star
19

salt

Salt is a collection of libraries and tools used in the Raystack ecosystem to improve the experience of developing projects with Go.
Go
13
star
20

depot

Depot contains various common sink implementations and publishes them as a library. This library will be used in firehose, daggers or any other application which wants to send data to destinations.
Java
9
star
21

predator

Go
3
star
22

dex

Data Experience
Go
3
star
23

frontier-go

Go
2
star
24

frontier-python

Python
2
star
25

.github

This repository contains the community health files for the @raystack organization
1
star
26

chronicle

TypeScript
1
star
27

scoop-bucket

This repository is home to the original scoop buckets for products throughout the Open DataOps platform ecosystem.
1
star