• This repository has been archived on 11/Dec/2023
  • Stars
    star
    142
  • Rank 256,980 (Top 6 %)
  • Language
    TypeScript
  • License
    Apache License 2.0
  • Created about 4 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

TUNA on AWS

Open TUNA project!

This is the infrastructure project of Open TUNA on AWS, orchestrated by AWS CDK.

See below for the overall architecture, overall-arch

It consists of the following independent stacks,

  • Network stack(optional)
    • Create a dedicated VPC with public and private subnets across three AZs with NAT gateways
    • Create S3 Endpoint
  • Storage stack(optional)
    • EFS file system
  • Common stack
    • SNS notification topic
    • Slack webhook subscription(optional)
  • Open TUNA stack
    • S3 asset bucket
    • application load balancer
    • CloudFront distribution
    • log analysis for CloudFront distribution
    • ECS cluster for content server and web portal
    • issue SSL certificate from ACM(only for using Route53 as DNS resolver)
    • create DNS record in R53 for ALB and CloudFront(only for using Route53 as DNS resolver)
    • invalidate CloudFront cache when content updates
    • S3 buckets for S3-backed repos e.g. rubygems
    • Tunasync Manager stack
      • auto scaling group for tunasync manager
      • intranet application load balancer for manager's API
      • elasticache redis cluster for database storage
    • Tunasync Worker stack
      • auto scaling group for tunasync worker
      • install necessary third party tools for mirroring tasks
      • use systemctl as daemon to start tunasync worker
      • send custom CloudWatch metrics of tunasync process info
      • publish large configuration files to s3 asset bucket
    • Content Server stack
      • build custom nginx container
      • use Fargate service to serve mirror contents
      • internet facing application load balancer
      • collect network metrics by CloudWatch agent
      • setup auto scaling rules based on metrics
    • Web Portal stack
      • use tuna/mirror-web
      • route tunasync.json to tunasync manager
      • create and run lambda function periodically to generate iso download links
    • Monitor stack
      • create several CodeBuild projects to verify integrity of index files
      • run projects periodically and report upon failure
      • trigger re-sync when failure is detected
    • Certificate stack
      • create CodeBuild project to manual renew Let's Encrypt certificate
  • Pipeline stack

Prerequisites

  • VPC with both public and private subnets crossing two AZs at least and NAT gateway. You can deploy the network stack if you don't have a VPC which satisfies the requirements.
  • EFS file system associated with above VPC. You can deploy stack with provisioning a EFS file system without specifying the existing filesystem id of EFS.

How to deploy it

Prerequisites

  • An AWS account
  • Configure credential of aws cli
  • Install node.js LTS version, such as 12.x
  • Install npm 7.x
  • Install Docker Engine

Checkout submodules

git submodule init
git submodule update

Install project dependencies

npm run init

Deploy network stack(optional)

npm run deploy-network

Deploy storage stack(optional)

npx cdk deploy OpenTunaStorageStack -c vpcId=<existing vpc Id>

Deploy open tuna stack

# deploy storage and common stack as well
npx cdk deploy OpenTunaStack -c vpcId=<existing vpc Id>

# subscribe SNS topic alaram to slack channel
# Go to https://slack.com/apps/A0F7XDUAZ-incoming-webhooks to apply for webhook
npx cdk deploy OpenTunaStack -c vpcId=<existing vpc Id> -c slackHookUrl=<webhook url>

# or deploy with existing EFS filesystem
npx cdk deploy OpenTunaStack -c vpcId=<existing vpc Id> -c fileSystemId=<existing filesystem id> -c fileSystemSGId=<existing sg id of the given file system>

# deploy with domain name and use Route53 as DNS resolver
npx cdk deploy OpenTunaStack -c vpcId=<existing vpc Id> -c domainName=<domain name of site> -c domainZone=<public hosted zone of your domain in Route53>

# deploy with SSL cert of CloudFront for China regions
# upload SSL cert to IAM, for China region only
aws iam upload-server-certificate --server-certificate-name my-domain --certificate-body file://cert.pem --private-key file://privkey.pem --certificate-chain file://chain.pem --path '/cloudfront/'
# get cert id from above output
npx cdk deploy OpenTunaStack -c vpcId=<existing vpc Id> -c domainName=<domain name of site> -c domainZone=<public hosted zone of your domain in Route53> -c iamCertId=<cert id>

The CertificateStack will be created when using IAM cert for the CloudFront SSL. The stack will create a CodeBuild project to issue new SSL certificates from Let's encrypt. And the build event can be pushed to the pipeline account when deploying the stack via the below context option,

-c certTopicArn=<arn of sns topic created by pipeline stack>

Docker image for content server is automatically built and published. You can build and publish to ecr manually:

$ sudo docker build -t content-server:1.18-alpine .
$ sudo docker tag content-server:1.18-alpine ${uid}.dkr.ecr.${region}.amazonaws.com/content-server:1.18-alpine
$ sudo docker push ${uid}.dkr.ecr.${region}.amazonaws.com/content-server:1.18-alpine

How to test

npm run test

Post deployment

  • Add email addresses or other subscribers to the notification topic created in the common stack. The alarm notifications related to Open TUNA will be sent to those subscribers.

More Repositories

1

thuthesis

LaTeX Thesis Template for Tsinghua University
TeX
4,498
star
2

tunasync

Mirror job management tool.
Go
1,406
star
3

blogroll

世界一流兼容并包TUNA协会收集的周围同学们的Blog
Python
827
star
4

mirror-web

Source code of the web interface of https://mirrors.tuna.tsinghua.edu.cn/
HTML
690
star
5

oh-my-tuna

Keep calm and use TUNA mirrors.
Python
480
star
6

ipv6.tsinghua.edu.cn

CSS
399
star
7

tunasync-scripts

Custom scripts for mirror jobs
Python
390
star
8

issues

Code Unrelated Issues
336
star
9

THU-Beamer-Theme

A LaTeX beamer theme template for Tsinghua students.
TeX
305
star
10

fishroom

Message forwarding for multiple IM protocols
Python
283
star
11

collection

idea collections from TUNA
254
star
12

danmaQ

danmaku implemented in Qt5
JavaScript
185
star
13

freedns-go

Optimized DNS Server for Chinese users.
Go
180
star
14

gdanmaku-server

Web-based danmaku server
Python
123
star
15

tuna.moe

HTML
75
star
16

gdanmaku

play danmaku on any screen
Python
36
star
17

artwork

tuna artwork
26
star
18

QSerial

An advanced cross-platform serial port utility
C++
23
star
19

podcast

Voice from TUNA
CSS
19
star
20

scripts

Python
15
star
21

pub-mirror

Dart application to fully mirror the content on https://pub.dartlang.org/
Dart
15
star
22

mirrorhub

Redirects the user to a nearby mirror site.
Go
11
star
23

Books

Tracking Books of TUNA Library
9
star
24

registr

JavaScript
6
star
25

dataset-issues

Issues about academic dataset mirrors
5
star
26

tunaccount

Go
5
star
27

topdf

Automactically convert Office files to PDF via a web interface
PHP
4
star
28

live

CSS
3
star
29

pdns-ldap

Python
3
star
30

fakeTunet

a fake net.tsinghua.edu.cn
JavaScript
2
star
31

feed_tuna

Telegram shop/payment bot 金枪鱼喂食计划
Python
2
star
32

tunasay

tunasay is a configurable talking tuna
JavaScript
2
star
33

playbooks

Python
1
star
34

Cash

1
star
35

loglyzer

log analyzer for tuna mirrors
Python
1
star
36

docker-images

Repo for various docker images
Dockerfile
1
star
37

tunasync-monitor

Rust
1
star