• Stars
    star
    1,084
  • Rank 40,959 (Top 0.9 %)
  • Language
    Python
  • License
    Apache License 2.0
  • Created almost 7 years ago
  • Updated 9 months ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

AWS Least Privilege for Distributed, High-Velocity Deployment

Repokid

NetflixOSS Lifecycle Build Status PyPI version Coverage Status Discord chat

Repokid Logo

Repokid uses Access Advisor provided by Aardvark to remove permissions granting access to unused services from the inline policies of IAM roles in an AWS account.

Getting Started

Install

mkvirtualenv repokid
git clone [email protected]:Netflix/repokid.git
cd repokid
pip install -e .
repokid config config.json

DynamoDB

You will need a DynamoDB table called repokid_roles (specify account and endpoint in dynamo_db in config file).

The table should have the following properties:

  • RoleId (string) as a primary partition key, no primary sort key
  • A global secondary index named Account with a primary partition key of Account and RoleId and Account as projected attributes
  • A global secondary index named RoleName with a primary partition key of RoleName and RoleId and RoleName as projected attributes

For development, you can run dynamo locally.

To run locally:

docker-compose up

The endpoint for DynamoDB will be http://localhost:8000. A DynamoDB admin panel can be found at http://localhost:8001.

If you run the development version the table and index will be created for you automatically.

IAM Permissions

Repokid needs an IAM Role in each account that will be queried. Additionally, Repokid needs to be launched with a role or user which can sts:AssumeRole into the different account roles.

RepokidInstanceProfile:

  • Only create one.
  • Needs the ability to call sts:AssumeRole into all of the RepokidRoles.
  • DynamoDB permissions for the repokid_roles table and all indexes (specified in assume_role subsection of dynamo_db in config) and the ability to run dynamodb:ListTables

RepokidRole:

  • Must exist in every account to be managed by repokid.
  • Must have a trust policy allowing RepokidInstanceProfile.
  • Name must be specified in connection_iam in config file.
  • Has these permissions:
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Action": [
       "iam:DeleteInstanceProfile",
       "iam:DeleteRole",
       "iam:DeleteRolePolicy",
       "iam:GetAccountAuthorizationDetails",
       "iam:GetInstanceProfile",
       "iam:GetRole",
       "iam:GetRolePolicy",
       "iam:ListInstanceProfiles",
       "iam:ListInstanceProfilesForRole",
       "iam:ListRolePolicies",
       "iam:PutRolePolicy",
       "iam:UpdateRoleDescription"
     ],
     "Effect": "Allow",
     "Resource": "*"
   }
 ]
}

So if you are monitoring n accounts, you will always need n+1 roles. (n RepokidRoles and 1 RepokidInstanceProfile).

Editing config.json

Running repokid config config.json creates a file that you will need to edit. Find and update these fields:

  • dynamodb: If using dynamo locally, set the endpoint to http://localhost:8010. If using AWS hosted dynamo, set the region, assume_role, and account_number.
  • aardvark_api_location: The location to your Aardvark REST API. Something like https://aardvark.yourcompany.net/api/1/advisors
  • connection_iam: Set assume_role to RepokidRole, or whatever you have called it.

Optional Config

Repokid uses filters to decide which roles are candidates to be repoed. Filters may be configured to suit your environment as described below.

Blocklist Filter

Roles may be excluded by adding them to the Blocklist filter. One common reason to exclude a role is if the corresponding workload performs occasional actions that may not have been observed but are known to be required. There are two ways to exclude a role:

  • Exclude role name for all accounts: add it to a list in the config filter_config.BlocklistFilter.all
  • Exclude role name for specific account: add it to a list in the config filter_config.BlocklistFilter.<ACCOUNT_NUMBER>

Blocklists can also be maintained in an S3 blocklist file. They should be in the following form:

{
  "arns": ["arn1", "arn2"],
  "names": {"role_name_1": ["all", "account_number_1"], "role_name_2": ["account_number_2", "account_number_3"]}
}

Exclusive Filter

If you prefer to repo only certain roles you can use the Exclusive Filter. Maybe you want to consider only roles used in production or by certain teams. To select roles for repo-ing you may list their names in the configuration files. Shell style glob patterns are also supported. Role selection can be specified per individual account or globally. To activate this filter put "repokid.filters.exclusive:ExclusiveFilter"in the section active_filters of the config file. To configure it you can start with the autogenerated config file, which has an example config in the "filter_config" section:

"ExclusiveFilter": {
                   "all": [
                     "<GLOB_PATTERN>"
                     ],
                   "<ACCOUNT_NUMBER>": [
                     "<GLOB_PATTERN>"
                    ]
                   }

Age Filter

By default the age filter excludes roles that are younger than 90 days. To change this edit the config setting: filter_config.AgeFilter.minimum_age.

Active Filters

New filters can be created to support internal logic. At Netflix we have several that are specific to our use cases. To make them active make sure they are in the Python path and add them in the config to the list in the section active_filters.

Extending Repokid

Hooks

Repokid is extensible via hooks that are called before, during, and after various operations as listed below.

Hook name Context
AFTER_REPO role, errors
AFTER_REPO_ROLES roles, errors
BEFORE_REPO_ROLES account_number, roles
AFTER_SCHEDULE_REPO roles
DURING_REPOABLE_CALCULATION role_id, arn, account_number, role_name, potentially_repoable_permissions, minimum_age
DURING_REPOABLE_CALCULATION_BATCH role_batch, potentially_repoable_permissions, minimum_age

Hooks must adhere to the following interface:

from repokid.hooks import implements_hook
from repokid.types import RepokidHookInput, RepokidHookOutput

@implements_hook("TARGET_HOOK_NAME", 1)
def custom_hook(input_dict: RepokidHookInput) -> RepokidHookOutput:
    """Hook functions are called with a dict containing the keys listed above based on the target hook.
    Any mutations made to the input and returned in the output will be passed on to subsequent hook funtions.
    """
    ...

Examples of hook implementations can be found in repokid.hooks.loggers.

Filters

Custom filters can be written to exclude roles from being repoed. Filters must adhere to the following interface:

from repokid.filters import Filter
from repokid.types import RepokidFilterConfig
from repokid.role import RoleList


class CustomFilterName(Filter):
    def __init__(self, config: RepokidFilterConfig = None) -> None:
        """Filters are initialized with a dict containing the contents of `filter_config.FilterName`
        from the config file. This example would be initialized with `filter_config.CustomFilterName`.
        The configuration can be accessed via `self.config`

        If you don't need any custom initialization logic, you can leave this function out of your
        filter class.
        """
        super().__init__(config=config)
        # custom initialization logic goes here
        ...

    def apply(self, input_list: RoleList) -> RoleList:
        """Determine roles to be excluded and return them as a RoleList"""
        ...

A simple filter implementation can be found in repokid.filters.age. A more complex example is in repokid.blocklist.age.

How to Use

Once Repokid is configured, use it as follows:

Standard flow

  • Update role cache: repokid update_role_cache <ACCOUNT_NUMBER>
  • Display role cache: repokid display_role_cache <ACCOUNT_NUMBER>
  • Display information about a specific role: repokid display_role <ACCOUNT_NUMBER> <ROLE_NAME>
  • Repo a specific role: repokid repo_role <ACCOUNT_NUMBER> <ROLE_NAME>
  • Repo all roles in an account: repokid repo_all_roles <ACCOUNT_NUMBER> -c

Scheduling

Rather than running a repo right now you can schedule one (schedule_repo command). The duration between scheduling and eligibility is configurable, but by default roles can be repoed 7 days after scheduling. You can then run a command repo_scheduled_roles to only repo roles which have already been scheduled.

Targeting a specific permission

Say that you find a given permission especially dangerous in your environment. Here I'll use s3:PutObjectACL as an example. You can use Repokid to find all roles that have this permission (even those hidden in a wildcard), and then remove just that single permission.

Find & Remove:

  • Ensure the role cache is updated before beginning.
  • Find roles with a given permission: repokid find_roles_with_permissions <permission>... [--output=ROLE_FILE]
  • Remove permission from roles: repokid remove_permissions_from_roles --role-file=ROLE_FILE <permission>... [-c]

Example:

$ repokid find_roles_with_permissions "s3:putobjectacl" "sts:assumerole" --output=myroles.json
...
$ repokid remove_permissions_from_roles --role-file=myroles.json "s3:putobjectacl" "sts:assumerole" -c

Rolling back

Repokid stores a copy of each version of inline policies it knows about. These are added when a different version of a policy is found during update_role_cache and any time a repo action occurs. To restore a previous version run:

See all versions of roles: repokid rollback_role <ACCOUNT_NUMBER> <ROLE_NAME> Restore a specific version: repokid rollback_role <ACCOUNT_NUMBER> <ROLE_NAME> --selection=<NUMBER> -c

Stats

Repokid keeps counts of the total permissions for each role. Stats are added any time an update_role_cache or repo_role action occur. To output all stats to a CSV file run: repokid repo_stats <OUTPUT_FILENAME>. An optional account number can be specified to output stats for a specific account only.

Library

New in v0.14.2

Repokid can be called as a library using the repokid.lib module:

from repokid.lib import display_role, repo_role, update_role_cache

account_number = "123456789012"

display_role(account_number, "superCoolRoleName")
update_role_cache(account_number)
repo_role(account_number, "superCoolRoleName", commit=True)

Dispatcher

Repokid Dispatcher is designed to listen for messages on a queue and perform actions. So far the actions are:

  • List repoable services from a role
  • Set or remove an opt-out
  • List and perform rollbacks for a role

Repokid will respond on a configurable SNS topic with information about any success or failures. The Dispatcher component exists to help with operationalization of the repo lifecycle across your organization. You may choose to expose the queue directly to developers, but more likely this should be guarded because rolling back can be a destructive action if not done carefully.

Development

Releasing

Versioning is handled by setupmeta. To create a new release:

python setup.py version --bump patch --push

# Inspect output and make sure it's what you expect
# If all is well, commit and push the new tag:
python setup.py version --bump patch --push --commit

More Repositories

1

Hystrix

Hystrix is a latency and fault tolerance library designed to isolate points of access to remote systems, services and 3rd party libraries, stop cascading failure and enable resilience in complex distributed systems where failure is inevitable.
Java
23,594
star
2

chaosmonkey

Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures.
Go
14,410
star
3

zuul

Zuul is a gateway service that provides dynamic routing, monitoring, resiliency, security, and more.
Java
12,993
star
4

conductor

Conductor is a microservices orchestration engine.
Java
12,920
star
5

eureka

AWS Service registry for resilient mid-tier load balancing and failover.
Java
11,991
star
6

falcor

A JavaScript library for efficient data fetching
JavaScript
10,338
star
7

pollyjs

Record, Replay, and Stub HTTP Interactions.
JavaScript
10,184
star
8

SimianArmy

Tools for keeping your cloud operating in top form. Chaos Monkey is a resiliency tool that helps applications tolerate random instance failures.
Java
7,955
star
9

metaflow

๐Ÿš€ Build and manage real-life ML, AI, and data science projects with ease!
Python
7,498
star
10

fast_jsonapi

No Longer Maintained - A lightning fast JSON:API serializer for Ruby Objects.
Ruby
5,078
star
11

dispatch

All of the ad-hoc things you're doing to manage incidents today, done for you, and much more!
Python
4,548
star
12

ribbon

Ribbon is a Inter Process Communication (remote procedure calls) library with built in software load balancers. The primary usage model involves REST calls with various serialization scheme support.
Java
4,468
star
13

security_monkey

Security Monkey monitors AWS, GCP, OpenStack, and GitHub orgs for assets and their changes over time.
Python
4,349
star
14

vmaf

Perceptual video quality assessment based on multi-method fusion.
Python
4,159
star
15

dynomite

A generic dynamo implementation for different k-v storage engines
C
4,104
star
16

vizceral

WebGL visualization for displaying animated traffic graphs
JavaScript
4,047
star
17

vector

Vector is an on-host performance monitoring framework which exposes hand picked high resolution metrics to every engineerโ€™s browser.
JavaScript
3,588
star
18

atlas

In-memory dimensional time series database.
Scala
3,331
star
19

concurrency-limits

Java
3,117
star
20

consoleme

A Central Control Plane for AWS Permissions and Access
Python
3,065
star
21

flamescope

FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.
Python
2,979
star
22

dgs-framework

GraphQL for Java with Spring Boot made easy.
Kotlin
2,963
star
23

bless

Repository for BLESS, an SSH Certificate Authority that runs as a AWS Lambda function
Python
2,722
star
24

archaius

Library for configuration management API
Java
2,435
star
25

asgard

[Asgard is deprecated at Netflix. We use Spinnaker ( www.spinnaker.io ).] Web interface for application deployments and cloud management in Amazon Web Services (AWS). Binary download: http://github.com/Netflix/asgard/releases
Groovy
2,235
star
26

curator

ZooKeeper client wrapper and rich ZooKeeper framework
Java
2,138
star
27

titus

1,996
star
28

EVCache

A distributed in-memory data store for the cloud
Java
1,900
star
29

lemur

Repository for the Lemur Certificate Manager
Python
1,651
star
30

bpftop

bpftop provides a dynamic real-time view of running eBPF programs. It displays the average runtime, events per second, and estimated total CPU % for each program.
Rust
1,647
star
31

genie

Distributed Big Data Orchestration Service
Java
1,635
star
32

metacat

Java
1,555
star
33

netflix.github.com

HTML
1,419
star
34

servo

Netflix Application Monitoring Library
Java
1,408
star
35

mantis

A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications
Java
1,385
star
36

vectorflow

D
1,287
star
37

hubcommander

A Slack bot for GitHub organization management -- and other things too
Python
1,262
star
38

rend

A memcached proxy that manages data chunking and L1 / L2 caches
Go
1,174
star
39

hollow

Hollow is a java library and toolset for disseminating in-memory datasets from a single producer to many consumers for high performance read-only access.
Java
1,148
star
40

astyanax

Cassandra Java Client
Java
1,034
star
41

Priam

Co-Process for backup/recovery, Token Management, and Centralized Configuration management for Cassandra.
Java
1,024
star
42

aminator

A tool for creating EBS AMIs. This tool currently works for CentOS/RedHat Linux images and is intended to run on an EC2 instance.
Python
938
star
43

Turbine

SSE Stream Aggregator
Java
831
star
44

governator

Governator is a library of extensions and utilities that enhance Google Guice to provide: classpath scanning and automatic binding, lifecycle management, configuration to field mapping, field validation and parallelized object warmup.
Java
821
star
45

Fido

C#
816
star
46

suro

Netflix's distributed Data Pipeline
Java
783
star
47

security-bulletins

Security Bulletins that relate to Netflix Open Source
734
star
48

spectator

Client library for collecting metrics.
Java
720
star
49

Fenzo

Extensible Scheduler for Mesos Frameworks
Java
703
star
50

msl

Message Security Layer
C++
687
star
51

unleash

Professionally publish your JavaScript modules in one keystroke
JavaScript
588
star
52

denominator

Portably control DNS clouds using java or bash
Java
573
star
53

blitz4j

Logging framework for fast asynchronous logging
Java
559
star
54

edda

AWS API Read Cache
Scala
554
star
55

PigPen

Map-Reduce for Clojure
Clojure
551
star
56

netflix-graph

Compact in-memory representation of directed graph data
Java
548
star
57

go-env

a golang library to manage environment variables
Go
542
star
58

karyon

The nucleus or the base container for Applications and Services built using the NetflixOSS ecosystem
Java
495
star
59

Prana

A sidecar for your NetflixOSS based services.
Java
492
star
60

iceberg

Iceberg is a table format for large, slow-moving tabular data
Java
465
star
61

Lipstick

Pig Visualization framework
JavaScript
464
star
62

Surus

Java
453
star
63

aws-autoscaling

Tools and Documentation about using Auto Scaling
Shell
429
star
64

go-expect

an expect-like golang library to automate control of terminal or console based programs.
Go
422
star
65

nf-data-explorer

The Data Explorer gives you fast, safe access to data stored in Cassandra, Dynomite, and Redis.
TypeScript
420
star
66

Workflowable

Ruby
370
star
67

osstracker

Github organization OSS metrics collector and metrics dashboard
Scala
365
star
68

vizceral-example

Example Vizceral app
JavaScript
363
star
69

ndbench

Netflix Data Store Benchmark
HTML
360
star
70

Raigad

Co-Process for backup/recovery, Auto Deployments and Centralized Configuration management for ElasticSearch
Java
346
star
71

recipes-rss

RSS Reader Recipes that uses several of the Netflix OSS components
Java
339
star
72

aegisthus

A Bulk Data Pipeline out of Cassandra
Java
323
star
73

titus-control-plane

Titus is the Netflix Container Management Platform that manages containers and provides integrations to the infrastructure ecosystem.
Java
316
star
74

weep

The ConsoleMe CLI utility
Go
311
star
75

metaflow-ui

๐ŸŽจ UI for monitoring your Metaflow executions!
TypeScript
300
star
76

dyno-queues

Dyno Queues is a recipe that provides task queues utilizing Dynomite.
Java
264
star
77

image_compression_comparison

Image Compression Comparison Framework
Python
258
star
78

falcor-express-demo

Demonstration Falcor end point for a Netflix-style Application using express
HTML
246
star
79

gradle-template

Java
244
star
80

ember-nf-graph

Composable graphing component library for EmberJS.
JavaScript
241
star
81

falcor-router-demo

A demonstration of how to build a Router for a Netflix-like application
JavaScript
236
star
82

titus-executor

Titus Executor is the container runtime/executor implementation for Titus
Go
233
star
83

photon

Photon is a Java implementation of the Interoperable Master Format (IMF) standard. IMF is a SMPTE standard whose core constraints are defined in the specification st2067-2:2013
Java
233
star
84

dial-reference

C
228
star
85

s3mper

s3mper - Consistent Listing for S3
Java
218
star
86

ReactiveLab

Experiments and prototypes with reactive application design.
Java
209
star
87

inviso

JavaScript
205
star
88

NfWebCrypto

Web Cryptography API Polyfill
C++
205
star
89

staash

A language-agnostic as well as storage-agnostic web interface for storing data into persistent storage systems, the metadata layer abstracts a lot of storage details and the pattern automation APIs take care of automating common data access patterns.
Java
204
star
90

zeno

Netflix's In-Memory Data Propagation Framework
Java
200
star
91

brutal

A multi-network asynchronous chat bot framework using twisted
Python
200
star
92

vizceral-react

JavaScript
199
star
93

dispatch-docker

Shell
193
star
94

pytheas

Web Resources and UI Framework
JavaScript
187
star
95

dyno

Java client for Dynomite
Java
184
star
96

hal-9001

Hal-9001 is a Go library that offers a number of facilities for creating a bot and its plugins.
Go
178
star
97

metaflow-service

๐Ÿš€ Metadata tracking and UI service for Metaflow!
Python
173
star
98

Nicobar

Java
171
star
99

lemur-docker

Docker files for the Lemur certificate orchestration tool
Python
170
star
100

yetch

Yet-another-fetch polyfill library. Supports AbortController/AbortSignal
JavaScript
168
star