SURF
Free Text Search across your infrastructure platforms via CLI.
It's like ... | grep
but for your entire infrastructure!
S.U.R.F is an acronym for: Search-Unified-Recursive-Fast
Supported Platforms
- AWS Route53 and Targets
- AWS ACM
- AWS S3
- AWS DynamoDB
- Hashicorp Vault
- Hashicorp Consul KV
- ElasticSearch / AWS OpenSearch
- Logz.io
- Kubernetes - TODO
Table of Contents
Overview
SURF is built for Infrastructure Engineers as a CLI tool that enables searching any pattern across different platforms. Usually, the results are returned with a direct web URL.
The search process depends on the context, for example: if you're searching in Vault it'll pattern match against keys. Instead, if you're searching in Route53 AWS a DNS address it'll return links to the targets behind it (e.g Load balancer).
Usage Examples
AWS Route53 Usage
Based on AWS Route53: Search what's behind domain api.my-corp.com
:
surf r53 -q api.my-corp.com
AWS ACM Usage
Search inside ACM Certificates in AWS.
Example search: containing a domain:
surf acm -q my-domain.com
Example search: certificate attached to a loab balancer:
surf acm -q 's:elasticloadbalancing:us-west-2:123:loadbalancer/app/alb' --filter-used-by
AWS DynamoDB Usage
Search free text data in DynamoDB
Supported Formats: protobuf
, base64
, json
, binary
, bytes
.
Example: list existing tables
surf ddb --list-tables
Example: use -p
for aws profile, -r
for region
surf ddb -q val -t table -p my-aws-profile -r us-east-1
Example: search all tables with production
in their name, where the data containing the pattern val
surf ddb -q val --all-tables -t production
Example: search all tables data containing the word val
, output as JSON
surf ddb -q val --all-tables -o json
Example: stop on first match
, search all tables data containing the word val
surf ddb -q val -t my-prefix-table --stop-first-match
AWS S3 Usage
Search inside S3 Buckets and Keys in AWS.
Example: Find all keys containing logs
in all buckets containing the name prod-bucket
:
surf s3 -q 'logs' -b 'prod-bucket'
Example: find all keys containng house
with prefix key my
surf s3 --prefix my -q house -b my-bucket
Example: find all keys ending with .json
and use non-default AWS profile:
surf s3 -q '\.json$' -b bucket-prefix -p my-aws-profile
Optional: Configure a default bucket name (same as --bucket
flag) to start search from (any regex pattern):
export SURF_S3_DEFAULT_MOUNT=<my-bucket-pattern>
Hashicorp Vault Usage
Search the query aws
in Vault:
surf vault -q aws
Configure a default mount to start search from in Vault:
export SURF_VAULT_DEFAULT_MOUNT=<my-default-mount>
Store LDAP auth on your OS keychain:
surf config
Hashicorp Consul Usage
Search all keys containing the substring server
surf consul -q "server"
Search under the scripts
path for keys ending with .sh
surf consul --prefix scripts --query "\.sh$"
ElasticSearch and OpenSearch Usage
Search free text and/or KQL.
Supports Standard Elasticsearch and AWS Opensearch.
-
Example: Search docs containing the term
api
return limit 40 results.surf es -q 'api' -l 40
-
Example: Search in indexes
prod-*
andapi-*
surf es -q 'api' -i 'prod-*d' -i 'api-*'
-
Example: Search docs containing the term
api
with client field andxyz*
pattern and NOT containing the termstaging
surf es -q 'api AND client:xyz*' --nq staging
-
Example: List Indexes
surf es --list-indexes
-
Example: Configure Elasticsearch default settings (use
surf es --help
for more):SURF_ELASTICSEARCH_URL SURF_ELASTICSEARCH_USERNAME='token-or-username-and-password' SURF_ELASTICSEARCH_PASSWORD='token-or-username-and-password' SURF_ELASTICSEARCH_TOKEN='token-or-username-and-password' SURF_ELASTICSEARCH_INDEXES='prod-*,api-*'
-
Example: Store Username/Password or Token auth on your OS keychain:
surf config
Logz.io Usage
Free text and KQL Search in logz.io, use the api website to get the address.
-
Example: Search docs containing the word 'exception' with limit size 200
surf logz -q 'exception' -l 200
-
Example: Search docs containing the word 'something' across ALL sub-accounts matching production/automation
surf logz -q 'something' --acc production --acc automation
-
Example: Search docs containing the word 'production', field errorCode with value Access* and are not containing 'dummy'
surf logz -q 'production AND errorCode:Access*' --nq 'dummy'
-
Example: Search docs across 10 day window with 2 days offset (e.g all matches between 12 days ago until 2 days ago)
surf logz -o 2 -w 10d -q 'some pattern'
-
Example: Configure Logz.io default settings (use
surf logz --help
for more):SURF_LOGZ_IO_TOKEN SURF_LOGZ_IO_URL SURF_LOGZ_IO_ACCOUNT_IDS
-
Example: Store Username/Password or Token auth on your OS keychain:
surf config
Install
Brew
MacOS (and ubuntu supported) installation via Brew:
brew tap isan-rivkin/toolbox
brew install surf
Download Binary
-
Move the binary to global dir and change name to
surf
:
cd <downloaded zip dir>
mv surf /usr/local/bin
Install from Source
git clone [email protected]:Isan-Rivkin/surf.git
cd surf
go run main.go
Authentication
Please open a PR and request additional methods if you need.
Supported Authentication Methods
- Vault - LDAP (run
$surf config
) - AWS - via profile on
~/.aws/credentials file
- Consul - None
- Elasticsearch / Opensearch - User/Pass or Token (run
$surf config
orsurf es --help
) - Logz.io - Token (run
$surf config
orsurf logz --help
)
Version check
The CLI will query github.com to check if there is a newer version and print out a message to the terminal.
If you wish to opt out set the environment variable SURF_VERSION_CHECK=false
.
No Data is collected it is purely github.com query.
How it Works
Contributors
- @AliRamberg
- @Isan-Rivkin