• Stars
    star
    241
  • Rank 167,643 (Top 4 %)
  • Language
    CSS
  • License
    MIT License
  • Created over 7 years ago
  • Updated over 1 year ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Track website visitors with Serverless Analytics using Kinesis, Lambda, and TypeScript.

Serverless Analytics ⚡️

MIT License Read Tutorial

Example project and proof of concept for a personal serverless Google Analytics clone to track website visitors. You can read more about Serverless Analytics with Amazon Kinesis and AWS Lambda on sbstjn.com

Components

After deploying the service you will have an HTTP endpoint using Amazon API Gateway that accepts requests and puts them into a Kinesis Stream. A Lambda function processes the stream and writes basic metrics about how many visitors you have per absolute URL to DynamoDB.

To access the tracked data, a basic dashboard with a JSON API is included as well. This should be a perfect starting point for you to create your own analytics service.

Tracking Service

  • Amazon Kinesis to stream visitor events
  • Amazon API Gateway for Kinesis HTTP proxy
  • Amazon DynamoDB for data storage
  • AWS Lambda to process visitor events

Dashboard

Example

The use of two API Gateways (data tracking and reading) is intended. You might have different settings for tracking and data access when you build something meaningful out of this example.

Configuration

All settings can be customized in the serverless.yml configuration file. You can easily change the DynamoDB Table, Kinesis Stream and API Gateway tracking resource name:

service: sls-analytics

custom:
  names:
    bucket: 
      website: ${self:service}-website-example
      dashboard: ${self:service}-website-dashboard
    resource: track
    dynamodb: ${self:service}-data
    kinesis: ${self:service}-stream

The S3 Bucket configuration is only needed for the included example website and dashboard. If you don't need the examples, have a look at the scripts/deploy.sh file and disable the deployment and remove the CloudFormation resources from the serverless.yml file.

Amazon requires unique names for S3 buckets and other resources. Please rename at least the service before you try to deploy the example!

Deployment

s3sync requires you to set AWS access keys as environmental variables.

  AWS_ACCESS_KEY=abc123
  AWS_SECRET_KEY=def456

Deploy script uses jq to read variables from generated output during deployment process. If you don't have jq installed, download it here.

Running yarn deploy will trigger a serverless deployment. After the output of your CloudFormation Stack is available, the included static websites will be generated (using the hostname from the stack output) and uploaded to the configured S3 buckets. As the last step, the deploy process will display the URLs of the example website and dashboard:

# Install dependencies
$ > yarn install
# Deploy 
$ > yarn deploy

[…]

Dashboard:  http://sls-analytics-dashboard.s3-website-us-east-1.amazonaws.com/
Website:    http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/

The website includes a simple HTML file, some stylings, and a few JavaScript lines that send a request to the tracking API on every page load. Open the URL in a web browser, hit a few times the refresh button and take a look at the DynamoDB table or the dashboard URL.

Tracking

Basically, tracking is nothing more than sending a HTTP request to the API with a set of payload information (currently url, date, name, and a websiteId). Normally you would have an additional non-JS fallback, like an image e.g., but a simple fetch call does the job for now:

fetch(
  'https://lqwyep8qee.execute-api.us-east-1.amazonaws.com/v1/track',
  {
    method: "POST",
    body: JSON.stringify(
      {
        date: new Date().getTime(),
        name: document.title,
        url: location.href,
        website: 'yfFbTv1GslRcIkUsWpa7'
      }
    ),
    headers: new Headers({ "Content-Type": "application/json" })
  }
)

Data Access

An example dashboard to access tracked data is included and deployed to S3. The URL will be displayed after the deploy task. You can access the metrics using basic curl requests as well. Just provide the website and date parameters:

Top Content

The ranking resource scans the DynamoDB for pages with the most hits on a specific date value:

$ > curl https://lqwyep8qee.execute-api.us-east-1.amazonaws.com/dev/ranking?website=yfFbTv1GslRcIkUsWpa7&date=MONTH:2017-08

[
  {
    "name": "Example Website - Serverless Analytics",
    "url": "http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/baz",
    "value": 19
  },
  {
    "name": "Example Website - Serverless Analytics",
    "url": "http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/",
    "value": 10
  },
  {
    "name": "Example Website - Serverless Analytics",
    "url": "http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/bar",
    "value": 4
  }
]

Requests per URL

The series resource scans the DynamoDB for data about a specific url in a given date period.

$ > curl https://lqwyep8qee.execute-api.us-east-1.amazonaws.com/dev/series?website=yfFbTv1GslRcIkUsWpa7&date=HOUR:2017-08-25T13&url=http://sls-analytics-website.s3-website-us-east-1.amazonaws.com/baz

[
  {
    "date": "MINUTE:2017-08-25T13:33",
    "value": 1
  },
  {
    "date": "MINUTE:2017-08-25T13:37",
    "value": 1
  },
  {
    "date": "MINUTE:2017-08-25T13:46",
    "value": 14
  },
  {
    "date": "MINUTE:2017-08-25T13:52",
    "value": 1
  }
]

Date Parameter

The DynamoDB stores the absolute hits number for the dimensions YEAR, MONTH, DATE, HOUR, and MINUTE per default. This may cause lots of write capacities when processing events, but with the serverless-dynamodb-autoscaling plugin DynamoDB will scale the capacities when needed.

All dates are UTC values!

Infrastructure

Infrastructure

License

Feel free to use the code, it's released using the MIT license.

Contribution

You are welcome to contribute to this project! 😘

To make sure you have a pleasant experience, please read the code of conduct. It outlines core values and beliefs and will make working together a happier experience.

More Repositories

1

timesheet.js

JavaScript library for HTML5 & CSS3 time sheets
JavaScript
6,946
star
2

noduino

JavaScript and Node.js Framework for controlling Arduino with HTML and WebSockets
CSS
1,229
star
3

sqs-worker-serverless

Example for SQS Worker in AWS Lambda using Serverless
JavaScript
178
star
4

cra-serverless

Serverless pre-rendering (SSR) for React SPA using AWS Lambda, S3, and CloudFront.
TypeScript
158
star
5

hanu

Golang Framework for writing Slack bots
Go
148
star
6

serverless-dynamodb-autoscaling

Serverless Plugin for Amazon DynamoDB Auto Scaling configuration.
TypeScript
148
star
7

serverless-stack-output

Serverless plugin to process AWS CloudFormation Stack Output
TypeScript
92
star
8

node-blog

Markdown Blog written in Node.js with twitter bootstrap UI and ace editor for composing…
JavaScript
83
star
9

68Keys.io

Build your own 68% Custom Mechanical Keyboard.
Sass
61
star
10

appsync-resolvers-example

Example project for AppSync, GraphQL, and AWS Lambda resolvers using Go.
Go
61
star
11

allot

Parse placeholder and wildcard text commands
Go
56
star
12

go-lambda-example

Example AWS Lambda function in Go and SAM
Go
49
star
13

serverless-geoip

Use MaxMind GeoLite2 database with AWS Lambda
JavaScript
43
star
14

appsync-resolvers

AWS AppSync Resolvers for GraphQL using AWS Lambda functions in Go.
Go
41
star
15

cra-pipeline

Use AWS CDK with CodePipeline to deploy a TypeScript Create-React-App.
TypeScript
40
star
16

serverless-sqs-alarms-plugin

Serverless wrapper to setup CloudWatch Alarms for SQS
JavaScript
36
star
17

lawos

SQS Worker for AWS Lambda.
JavaScript
29
star
18

go-appsync-graphql-cloudformation

AWS AppSync GraphQL API Proxy with Lambda, CloudFormation, and SAM
Makefile
29
star
19

serverless-s3bucket-sync

Serverless Plugin to sync local folders with an S3 bucket
TypeScript
25
star
20

mongoclikker

MongoDB viewer and editor written in Node
JavaScript
22
star
21

latenz

JavaScript HTTP latency analyzer
JavaScript
20
star
22

cdk-lambda-fleet

Deploy AWS Lambda functions using docker images to ECR with AWS CDK
TypeScript
19
star
23

github-releases-slack

Notify Slack about GitHub Releases.
Go
17
star
24

kirbycms-oauth-consumer

Kirby CMS Plugin for using Twitter and GitHub OAuth login
PHP
14
star
25

typescript-npm-boilerplate

Publish to NPM with TypeScript, CircleCI and GitHub Releases.
TypeScript
13
star
26

node-arduino-socket

Simple node.js application for piping a serial connection over websockets to your browser
JavaScript
12
star
27

DropFile.php

HTML, JavaScript and PHP file uploader
JavaScript
11
star
28

liveblog.js

Node.js App for Liveblogs and Event Tickers with Socket.IO and Express.js
JavaScript
10
star
29

aws-lambda-docker-node

Continuous Integration and Continuous Deployment setup for AWS Lambda functions with Docker images using GitHub Actions.
Makefile
10
star
30

node-kickstart

Simple Node.js example app with support for jade, less, ssl certificates…
JavaScript
8
star
31

requirejs-express

Example app using Express and RequireJS in Node.js
JavaScript
8
star
32

lawos-serverless

Serverless example for an SQS Worker in AWS Lambda using Lawos
JavaScript
7
star
33

appsync-example-dynamodb

GraphQL API using AWS AppSync, SAM, and DynamoDB
Makefile
7
star
34

serverless-alexa-skill

Serverless example for a custom Amazon Alexa Skill
JavaScript
7
star
35

faas.club

Curated list for serverless resources and everything for Functions as a Service at https://faas.club
JavaScript
7
star
36

taco.js

Rating awesomeness with Express.js
CSS
5
star
37

node-markdownblog

Engine for managing markdown files…
JavaScript
4
star
38

cdk-template

My GitHub Template for AWS CDK projects and GitHub actions.
TypeScript
4
star
39

node-kickstart-example

Example application for node-kickstart
JavaScript
4
star
40

node-simple-cdn

Simple CDN host ballancing for node
JavaScript
4
star
41

faas-sam-lambda-go

AWS Serverless Application Model w/ Go
Go
4
star
42

XMLBuilder.js

Create XML data with JavaScript
JavaScript
4
star
43

alexa-example-skill

Example code of a custom Skill for Amazon Echo & Alexa.
JavaScript
4
star
44

archive

Amazon S3 Replication with KMS, CDK, and CloudFormation StackSets.
TypeScript
4
star
45

SimplePHP

A bunch of static PHP classes making everyday code easier and comfortable.
PHP
4
star
46

hanu-example

Example usage of the hanu Go framework
Go
3
star
47

rdm

Scaffold common GitHub project files like README, CoC, or LICENSE.md
Go
3
star
48

php-jade

PHP example app using jade template engine
PHP
3
star
49

faas-sam-lambda-typescript

AWS Serverless Application Model w/ TypeScript
TypeScript
3
star
50

SocialStream.php

PHP Framework to fetch your recent Social Media events
PHP
3
star
51

super-arduino-bro

Basic pixel Jump'n'Run for 8x8 RGB LED matrix and Arduino — v. 0.1
C++
3
star
52

gatsby-disable-404

Disable 404 pages for Gatsby websites
TypeScript
2
star
53

lambda-error

Node.js error object for streamlined errors using AWS Lambda and API Gateway
JavaScript
2
star
54

Snug.php

Static Page Generator using PHP, HAML and LESS
PHP
1
star
55

actions

GitHub Actions
Dockerfile
1
star
56

reng

Release Name Generator in PHP 5.3 - aka. handy-hyena
PHP
1
star
57

sqs-object

Convert JavaScript objects to SQS Meta Attributes
JavaScript
1
star
58

cdk-encrypted-bucket

Example for polyglot CDK construct for NPM, PyPi, and NuGet
TypeScript
1
star
59

gauges-referrer

Tiny Node.JS Helper for the Gaug.es API
JavaScript
1
star
60

pinged.php

Lightweight PHP logging service with HTTP interface
PHP
1
star
61

aws-codepipeline-example

Go
1
star
62

SimpleObjects

A bunch of PHP objects making everyday code easier and comfortable
PHP
1
star
63

node-proclog

Basic event logger with Node and MongoDB and Websocket Live Monitoring
1
star
64

Boilerplate.swift

Swift3 code with UnitTests that run on macOS, tvOS, iOS, watchOS and Linux.
Swift
1
star
65

discogs-collection

Export your Discogs.com collection value to CloudWatch Metrics
TypeScript
1
star
66

circleci-orbs

CircleCI Orbs for common tasks.
1
star
67

stopper

Minimal stopwatch object to count and measure time in JavaScript.
JavaScript
1
star
68

falke

Alexa skill to check when a Tatort is broadcasted
JavaScript
1
star
69

feedback

Command line feedback for Go
Go
1
star
70

node-cloudstream

Query latest items stored at http://getCloudApp.com
JavaScript
1
star
71

docker-node

Toolbox for Node.
Dockerfile
1
star
72

SliPHP

Atomic view and layout engine for PHP 5.4 and above.
PHP
1
star
73

aws-edukit-example

Example baseline for prototyping with the Core2 AWS IoT EduKit.
C
1
star
74

layers

Manage, build, and deploy AWS Lambda Layers
Makefile
1
star
75

faas-sam-lambda-python

AWS Serverless Application Model w/ Python
Python
1
star
76

function-path

Deeplink into nested JavaScript module functions
TypeScript
1
star
77

faas-makefile

Reusable Makefile to bootstrap AWS::Serverless CloudFormation projects
Makefile
1
star
78

docker-go

Docker image for CircleCI projects using Golang.
Dockerfile
1
star