opensearch-project/data-prepper

Stars
259
Rank 157,669 (Top 4 %)
Language
Java
License
Apache License 2.0
Created over 3 years ago
Updated about 1 month ago

opensearch-project/data-prepper

opensearch-project

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.

Data Prepper

We envision Data Prepper as an open source data collector for observability data (trace, logs, metrics) that can filter, enrich, transform, normalize, and aggregate data for downstream analysis and visualization. It will support stateful processing across multiple instances of data pipelines for observability use cases such as distributed tracing and multi-line log events (e.g. stack traces, aggregations, and log-to-metric transformations). Currently Data Prepper supports processing of distributed trace data and log ingestion with plans to support metric data in the future.

Please read the Overview to understand what Data Prepper is and how it works.

Getting Started

Our Getting Started guide is the best starting point for anybody who wants to run Data Prepper.

Please read the Trace Analytics guide or Log Analytics to get started with using Data Prepper for trace or log analytics use cases.

Project Resources

Downloads
Documentation
Configuration Reference
Need help? Try the Data Prepper category in the OpenSearch forums
Project Principles

Contribute

We invite developers from the larger OpenSearch community to contribute and help improve test coverage and give us feedback on where improvements can be made in design, code and documentation. You can look at contribution guide for more information on how to contribute.

If you are looking to contribute code, or just to build from source, please see our Developer Guide.

Code of Conduct

This project has adopted an Open Source Code of Conduct.

Security Issue Notifications

If you discover a potential security issue in this project, please refer to the security policy.

License

This library is licensed under the Apache 2.0 License

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.

OpenSearch

🔎 Open source distributed and RESTful search engine.

OpenSearch-Dashboards

📊 Open source visualization dashboards for OpenSearch.

opensearch-k8s-operator

OpenSearch Kubernetes Operator

opensearch-py

Python Client for OpenSearch

security

🔐 Secure your cluster with TLS, numerous authentication backends, data masking, audit logging as well as role-based access control on indices, documents, and fields

opensearch-go

Go Client for OpenSearch

opensearch-js

Node.js Client for OpenSearch

helm-charts

☸️ A community repository for Helm Charts of OpenSearch Project.

k-NN

🆕 Find the k-nearest neighbors (k-NN) for your vector data

opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems

sql

Query your data using familiar SQL or intuitive Piped Processing Language (PPL)

spring-data-opensearch

opensearch-java

Java Client for OpenSearch

opensearch-benchmark

OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch

logstash-output-opensearch

A Logstash plugin that sends event data to a OpenSearch clusters and stores as an index.

opensearch-net

OpenSearch .NET Client

opensearch-php

Official PHP Client for OpenSearch

ml-commons

ml-commons provides a set of common machine learning algorithms, e.g. k-means, or linear regression, to help developers build ML related features within OpenSearch.

opensearch-ruby

Ruby Client for OpenSearch

ansible-playbook

🤖 A community repository for Ansible Playbook of OpenSearch Project.

terraform-provider-opensearch

security-analytics

Security Analytics enables users for detecting security threats on their security event log data. It will also allow them to modify/tailor the pre-packaged solution.

security-dashboards-plugin

🔐 Manage your internal users, roles, access control, and audit logs from OpenSearch Dashboards

documentation-website

The documentation for OpenSearch, OpenSearch Dashboards, and their associated plugins.

anomaly-detection

Identify atypical data and receive automatic notifications

neural-search

Plugin that adds dense neural retrieval into the OpenSearch ecosytem

alerting

📟 Get notified when your data meets certain conditions by setting up monitors, alerts, and notifications

opensearch-rs

OpenSearch Rust Client

index-management

🗃 Automate periodic data operations, such as deleting indices at a certain age or performing a rollover at a certain size

observability

Visualize and explore your logs, traces and metrics data in OpenSearch Dashboards

opensearch-plugins

For all things OpenSearch plugins. You want to install, or develop a plugin? You've come to the right place.

cross-cluster-replication

Synchronize your data across multiple clusters for lower latencies and higher availability

opensearch-devops

😄 We welcome all the community members to help develop third party tools/automations/workflows for OpenSearch/OpenSearch-Dashboards.

opensearch-cli

A full-featured command line interface (CLI) for OpenSearch.

project-website

The source to opensearch.org

opensearch-dsl-py

High level Python client for OpenSearch (deprecated, moved to opensearch-py)

opensearch-migrations

Migrate, upgrade, compare, and replicate OpenSearch clusters with ease.

performance-analyzer

📈 Get detailed performance metrics from your cluster independently of the Java Virtual Machine (JVM)

oui

OpenSearch UI Framework

geospatial

Future home of Geospatial features for OpenSearch

flow-framework

OpenSearch plugin that enables builders to innovate AI apps on OpenSearch

opensearch-py-ml

opensearch-api-specification

API specification for OpenSearch

.github

Provides templates and resources for other OpenSearch project repositories.

opensearch-hadoop

anomaly-detection-dashboards-plugin

Manage your detectors and identify atypical data in OpenSearch Dashboards

opensearch-sdk-java

OpenSearch SDK to build and run extensions

performance-analyzer-rca

The Performance Analyzer RCA is a framework that builds on the Performance Analyzer engine to support root cause analysis (RCA) of performance and reliability problems for OpenSearch instances.

asynchronous-search

▶️ Run queries in the background and retrieve partial results along the way

reporting

Export and automate PNG, PDF, and CSV reports in OpenSearch Dashboards

alerting-dashboards-plugin

📟 Manage your monitors, alerts and notifications in OpenSearch Dashboards

notifications

Notifications plugin for OpenSearch enables other plugins to send notifications via Email, Slack, Amazon Chime, Custom web-hook etc channels

opensearch-plugin-template-java

Template repo for creating OpenSearch plugins

search-processor

Search Request Processor: pipeline for transformation of queries and results inline with a search request.

job-scheduler

Run periodic jobs on the cluster from your plugin

common-utils

Offers a library of utilities for building Java-based OpenSearch plugins

opensearch-testcontainers

opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.

opensearch-benchmark-workloads

Official workloads used by OpenSearch Benchmark (OSB)

dashboards-search-relevance

Tools to help search relevance engineers and business users tune search results for their OpenSearch applications.

opensearch-catalog

The OpenSearch Catalog is designed to make it easier for developers and community to contribute, search and install artifacts like plugins, visualization dashboards, ingestion to visualization content packs (data pipeline configurations, normalization, ingestion, dashboards).

dashboards-assistant

Dashboard assistant is a way for users on OpenSearch Dashboards to interact with an assistant through chat or the different OSD pages

piped-processing-language

Query your data using an intuitive Piped Processing Language (PPL)

dashboards-maps

🌐 User interface for the backend geospatial plugin which can be integrated with OpenSearch Dashboards and a home for all other map related features.

user-behavior-insights

User Behavior Insights plugin for OpenSearch

index-management-dashboards-plugin

🗃 Manage policies and jobs and automate periodic data operations in OpenSearch Dashboards

opensearch-cluster-cdk

dashboards-observability

Visualize and explore your logs, traces and metrics data in OpenSearch Dashboards

perftop

📈 PerfTop: A client for the OpenSearch Performance Analyzer

opensearch-ci

Enables continuous integration across OpenSearch, OpenSearch Dashboards, and plugins.

sql-jdbc

This is the driver for JDBC connectivity to a cluster running with OpenSearch SQL support.

dashboards-anywhere

Cloud native infrastructure for the OpenSearch Dashboards service.

dashboards-visualizations

Visualize the start, end, and duration of unique events in a sequence in OpenSearch Dashboards

opensearch-dashboards-functional-test

Maintains functional tests for OpenSearch Dashboards and Dashboards plugins

security-analytics-dashboards-plugin

Front end (UI) plugin to support security-analytics

opensearch-sdk-py

OpenSearch Extensions SDK for Python.

dashboards-notifications

The Notifications plugin provides a central location for all of your notifications from OpenSearch plugins. You can use either OpenSearch Dashboards or the REST API to configure notifications. Dashboards offers a more organized way of selecting a channel type and selecting which OpenSearch plugin sources you want to use.

dashboards-reporting

logstash-input-opensearch

dashboards-flow-framework

A UI designer for constructing AI applications with OpenSearch

opensearch-clients

For all things about OpenSearch clients.

dashboards-notebooks

Combine code snippets, live visualizations, and narrative text into rich reports in OpenSearch Dashboards

docker-images

sql-odbc

OpenSearchODBC is a read-only ODBC driver for Windows and Mac for connecting to OpenSearch SQL support.

opensearch-oci-object-storage

OpenSearch Repository Plugin For Oci Object Storage

ml-commons-dashboards

User interface for the ml-commons plugin which can be integrated with OpenSearch Dashboards and a health dashboard for machine learning models.

query-insights

Query Insights plugin offers frameworks and APIs for analyzing and optimizing query performance in OpenSearch.

dashboards-query-workbench

The OpenSearch Dashboards Query Workbench enables you to query your OpenSearch data using either SQL or PPL

maps

Maps resources for OpenSearch

simple-schema

The purpose of this Plugin is to enhance open-search engine with user-domain schema by using GraphQL.

custom-codecs

OpenSearch custom lucene codecs for providing different on-disk index encoding (e.g., compression).

opensearch-build-libraries

skills

dashboards-desktop

ux

Public repository for OpenSearch UX. Tracks issues, epics and workstreams related to improving the user experience, usability and accessibility of OpenSearch core, dashboards and related plugins.

reporting-cli

project-meta

Tools to make it easy to manage processes across the opensearch-project org.

sql-cli

The SQL CLI component in OpenSearch is a stand-alone Python application for query

project-tools

OpenSearch project/org tools.

performance-analyzer-commons