Compressed Log Processor (CLP) is a tool capable of losslessly compressing text logs and searching the compressed logs without decompression. To learn more about it, you can read our paper.
Getting Started
You can download a release from the releases page or you can build the latest by using the packager.
For some logs you can use to test CLP, check out our open-source datasets.
Providing Feedback
You can use GitHub issues to report a bug or request a feature.
Join us on Zulip to chat with developers and other community members.
Project Structure
CLP is currently split across a few different components in the components directory:
- clp-package-utils contains Python utilities for operating the CLP package.
- clp-py-utils contains Python utilities common to several of the other components.
- compression-job-handler contains code to submit compression jobs to a cluster.
- core contains code to compress uncompressed logs, decompress compressed logs, and search compressed logs.
- job-orchestration contains code to schedule compression jobs on the cluster.
- package-template contains the base directory structure and files of the CLP package.
GitHub Packages
The artifacts published to GitHub packages in this repo are a set of Docker container images useful for building and running CLP:
| Image name | Image contents | Link |
|---|---|---|
ghcr.io/y-scope/clp/clp-core-dependencies-x86-centos7.4:main |
The dependencies necessary to build CLP core in a Centos 7.4 x86 environment. | ↗ |
ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-focal:main |
The dependencies necessary to build CLP core in an Ubuntu Focal x86 environment. | ↗ |
ghcr.io/y-scope/clp/clp-core-dependencies-x86-ubuntu-jammy:main |
The dependencies necessary to build CLP core in an Ubuntu Jammy x86 environment. | ↗ |
ghcr.io/y-scope/clp/clp-core-x86-ubuntu-focal:main |
The CLP core binaries (clp, clg, clo, etc.) built in an Ubuntu Focal x86 environment. |
↗ |
ghcr.io/y-scope/clp/clp-execution-x86-ubuntu-focal:main |
The dependencies necessary to run the CLP package in an x86 environment. | ↗ |
Next Steps
This is our open-source release which we will be constantly updating with bug fixes, features, etc. If you would like a feature or want to report a bug, please file an issue and we'll be happy to engage. We also welcome any contributions!