A native Go library to streamline and supercharge your interactions with the AWS CloudWatch Logs Insights service using minimalist, native Go, paradigms.
Incite makes it easier to write code to query your logs using AWS CloudWatch Logs Insights, and makes it possible to use Insights to query massive, arbitrary, amounts of log data reliably.
- Streaming. AWS CloudWatch Logs Insights makes you poll your queries, requiring boilerplate code that is hard to write efficiently. Incite does the polling for you and lets you simply read your query results from a stream.
- Auto-Chunking. Every CloudWatch Logs Insights query is limited to 10,000
results. If your query exceeds 10K results, AWS advises you to break it into
smaller time ranges. Incite does this chunking automatically and merges the
results of all chunks into one convenient stream. Use the
Chunk
field inQuerySpec
to enable chunking. - Dynamic Splitting. Since v1.2.0, Incite can dynamically detect when a query
chunk exceeds the 10K result limit, split that chunk into sub-chunks, and
re-query the chunks, all automatically and without intervention. Use the
SplitUntil
field inQuerySpec
to enable dynamic splitting. - Multiplexing. Incite efficiently runs multiple queries at the same time and is smart enough to do this without getting throttled or going over your CloudWatch Logs service quota limits.
- Unmarshalling. The CloudWatch Logs Insights API can only give you
unstructured key/value string pairs, requiring you to write boilerplate code
to put your results into a useful structure for analysis. Incite lets you
unmarshal your results into maps or structs using a single function call.
Incite supports tag-based field mapping just like
encoding/json
. (And it supportsjson:"..."
tags as well as its nativeincite:"..."
tags, right out of the box!) - Go Native. Incite gives you a more Go-friendly coding experience than the
AWS SDK for Go, including getting rid of unnecessary pointers and using
standard types like
time.Time
.
$ go get github.com/gogama/incite
- For quick prototyping and scripting type work, simplify your life using the
global
Query
function. - When you need finer control over what your app is doing, create a new
QueryManager
usingNewQueryManager
and query it using itsQuery
method. - To read all the results from a stream, use the global
ReadAll
function. - To unmarshal the results into a structure of your choice, use the global
Unmarshal
function.
package main
import (
"fmt"
"time"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
"github.com/gogama/incite"
)
func main() {
// Use the AWS SDK for Go to get the CloudWatch API actions Incite needs.
// For simplicity, we assume that the correct AWS region and credentials are
// already set in the environment.
a := cloudwatchlogs.New(session.Must(session.NewSession()))
// Create a QueryManager. An alternative to using a QueryManager is just
// using the global scope Query function.
m := incite.NewQueryManager(incite.Config{Actions: a})
defer func() {
_ = m.Close()
}()
// Look at the last 15 minutes.
end := time.Now().Truncate(time.Millisecond)
start := end.Add(-15*time.Minute)
// Query the results.
s, err := m.Query(incite.QuerySpec{
Text: "fields @timestamp, @message | filter @message =~ /foo/ | sort @timestamp desc",
Start: start,
End: end,
Groups: []string{"/my/log/group"},
Limit: 100,
})
if err != nil {
return
}
data, err := incite.ReadAll(s)
if err != nil {
return
}
// Unpack the results into a structured format.
var v []struct{
Timestamp time.Time `incite:"@timestamp"`
Message string `incite:"@message"`
}
err = incite.Unmarshal(data, &v)
if err != nil {
return
}
// Print the results!
fmt.Println(v)
}
Works with all Go versions 1.14 and up, and AWS SDK for Go V1 versions 1.21.6 and up.
Official AWS documentation: Analyzing log data with CloudWatch Logs Insights.
Find Insights' query syntax documentation here
and the API reference here (look
for StartQuery
, GetQueryResults
, and StopQuery
).
This project is licensed under the terms of the MIT License.
Developer happiness on this project was embiggened by JetBrains, which generously donated an open source license for their lovely GoLand IDE. Thanks JetBrains!