ShadowReader
ShadowReader has the ability to replay production traffic to a destination of your choice by collecting traffic patterns from access logs. It is built on AWS Lambda, S3 and Elastic Load Balancers.
ShadowReader mimics real user traffic by replaying URLs from production at the same rate as the live website. Being serverless, it is more efficient cost and performance wise than traditional distributed load tests and in practice has scaled beyond 50,000 requests / minute.
Support for replaying logs from these load balancers:
- Application Load Balancer
- Classic Load Balancer
- Any logs stored locally, just provide a RegEx for the log format
- (Support for other types of load balancers planned)
Read about it
How we fixed a Node.js memory leak by using ShadowReader to replay production traffic into QA
Documentation
To get started, see:
- Batteries included demo for trying out live replay
- More in-depth guide for setting up past replay
- How to send load test metrics to CloudWatch
- Parse locally stored logs
- Filter requests from replay
Trying Live Replay
This guide will deploy all the necessary AWS resources to try out ShadowReader's live replay feature (replay requests as they come in).
This demo uses us-east-1 AWS region. Substitute all references to us-east-1 with a region of your choice.
Deploy AWS resources for demo
The provided CloudFormation stack will deploy these AWS resources for the demo: A VPC, subnets, 2 ALBs and a S3 bucket.
One ALB comes preconfigured to send logs to S3. This demo will walk you through in setting up ShadowReader to parse those logs then replay it to the 2nd ALB.
Deploy the stack:
curl https://raw.githubusercontent.com/edmunds/shadowreader/master/docs/demo-cf.yml --output demo-cf.yml
aws cloudformation deploy --stack-name sr-demo-resources --template-file demo-cf.yml --region us-east-1
If you'd rather not deploy a new VPC and subnets, this CF will allow you to choose already existing VPC/Subnets.
curl https://raw.githubusercontent.com/edmunds/shadowreader/master/docs/demo-cf-choose.yml --output demo-cf-choose.yml
aws cloudformation deploy --stack-name sr-demo-resources --template-file demo-cf-choose.yml --region us-east-1 --parameter-overrides subnet1=$SUBNET_ID1 subnet2=$SUBNET_ID2 vpcid=$VPCID
Set up shadowreader.yml
Copy shadowreader.example.yml to shadowreader.yml
cp shadowreader.example.yml shadowreader.yml
Set access_logs_bucket in shadowreader.yml like below.
Replace AWS_ACCOUNT_ID with your account id (should be an integer like 123450493079).
Replace AWS_REGION with the region you deployed the CloudFormation to.
environment:
access_logs_bucket: sr-access-logs-$AWS_REGION-$AWS_ACCOUNT_ID/AWSLogs/$AWS_ACCOUNT_ID/elasticloadbalancing/$AWS_REGION/
Set replay_mode to replay_live
plugins:
loader_middleware: loader_middleware
replay_mode: replay_live
Set up serverless.yml
Copy serverless.example.yml to serverless.yml.
cp serverless.example.yml serverless.yml
(Both serverless.yml and shadowreader.yml must be configured before deployment via the Serverless framework.)
Update my_project_name in serverless.yml. This is your project name.
It is to ensure that the S3 bucket used by ShadowReader has unique naming (S3 bucket names must be globally unique).
custom:
my_project_name: my-unique-project-name
Find out the DNS name for the ALB we will be load testing.
$ aws elbv2 describe-load-balancers --names SR-Demo-ALB-receiving --region us-east-1 | grep DNSName
>> "DNSName": "SR-Demo-ALB-receiving-123459376.us-east-1.elb.amazonaws.com",
Find orchestrator-past in serverless.yml and edit base_url to be http://{DNSName}.
orchestrator-past:
handler: functions/orchestrator_past.lambda_handler
events:
- schedule: rate(1 minute)
environment:
test_params: '{
"base_url": "http://SR-Demo-ALB-receiving-123459376.us-east-1.elb.amazonaws.com",
"rate": 100,
"replay_start_time": "2018-08-06T01:00",
"replay_end_time": "2018-08-06T02:00",
"identifier": "oss"
}'
timezone: US/Pacific
3. Install the Serverless framework
npm install -g serverless
serverless plugin install -n serverless-python-requirementsA more detailed guide here: https://serverless.com/framework/docs/getting-started/
3.5 Set up virtual env (optional)
python3 -m venv ~/.virtualenvs/sr-env
source ~/.virtualenvs/sr-env/bin/activate4. Deploy to AWS
# Deploy ShadowReader to your AWS account
serverless deploy --stage dev --region us-east-1
If you installed Python using Brew, you may run into this error:
File "/usr/local/Cellar/python/3.6.4_4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/command/install.py", line 248, in finalize_options
"must supply either home or prefix/exec-prefix -- not both")
distutils.errors.DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both
Run this to fix it. More details at StackOverflow
(while in same directory as serverless.yml)
echo '[install]\nprefix=' > setup.cfg
Start replaying traffic
Now we will start querying one ALB, which will generate access logs for ShadowReader to replay in real-time.
Find the DNSName for the ALB we will hit.
$ aws elbv2 describe-load-balancers --names SR-Demo-ALB-log-generator --region us-east-1 | grep DNSName
>> "DNSName": "SR-Demo-ALB-log-generator-1234567.us-east-1.elb.amazonaws.com"
Use watch to start sending requests to our ALB, this will generate access logs for ShadowReader to replay.
(brew install watch on mac)
watch -n 1 curl http://SR-Demo-ALB-log-generator-1234567.us-east-1.elb.amazonaws.com
You should now start seeing ALB logs being deposited to the sr-access-logs S3 bucket.
In about 8 minutes, requests sent to SR-Demo-ALB-log-generator will be replayed to SR-Demo-ALB-receiving
See the results
Check the CloudWatch HTTP fixed response count metrics for SR-Demo-ALB-receiving. It should be similar to the one for SR-Demo-ALB-log-generator.
Next steps
To start replaying actual application traffic, try attaching an ECS service or EC2 server to log-generator ALB.
Design
This diagram details the AWS components Shadow Reader uses and how they interact. More details in the above case study.
Contributing and Deveploment
Please see the contributing guide for more specifics.
Contact/Support
While ShadowReader has been productionalized at Edmunds, the open source version is still under active development and we greatly appreciate any feedback, suggestions, or comments!
Email me: [email protected] or contact me on Twitter: @yukisww
or use the Issues page
License
Distributed under the Apache License 2.0. See LICENSE for more information.