misc-scripts

This is a collection of miscellaneous scripts that I've written or modified for my use. Hopefully they may be of some use to others. When I get a chance, I'll try and update this readme with descriptions of all of the scripts.

Unless otherwise noted, these are distributed under the terms of the LICENSE file.

• add_team_to_github_org_repos.py - Python script to add a given GitHub Team to all of the specified Organization's repositories.
• apache_log_verify_site_move.py - Python script that parses Apache HTTPD access logs, finds all unique URLs, and compares the current HTTP response code to that of another server. Useful when moving a site.
• artifactory_support_bundle.py - Python script using requests to generate, list, and download JFrog Artifactory support bundles via the ReST API, from one or more instances/nodes.
• asg_instances.py - Script to list instances in an ASG and their IP addresses, given an ASG name.
• avery5160_code128_numeric.py - Script to generate PDFs of numeric-series Code128 barcodes (with optional prefixes) to fit Avery 5160-size labels.
• avery5160_code128_numeric-example.pdf - Example output of the above script.
• aws_api_gateway_lint.py - Script using boto3 to attempt to identify unused or idle API Gateways.
• aws-count-tag-names.py - Using boto3, scan all AWS resources in the current account, and produce a report detailing all of the distinct tag names and the number of resources having each one.
• aws_creds_report_csv_filter.py - Filter the AWS IAM Credentials Report (CSV) by credential age and/or last used time.
• aws_cw_log_group_daily_stats.py - Python script using boto3 to query CloudWatch Metrics for the IncomingBytes and IncomingLogEvents metrics for one or more LogGroups, across all regions. Retrieves metrics at the 24-hour resolution for N (default 7) days.
• aws_delete_user.py - Python/boto3 script to delete an IAM user including all of its associated resources, with optional dry-run.
• aws_find_duplicate_sgs.py - Using boto3, find duplicated EC2 Security Groups (optionally limited to one VPC).
• aws_limit_increases_for_service.py - Script using boto3 to show all Limit Increase support tickets for a specified service.
• aws_region_stats.py - Python script to print a table with some statistics from each AWS region. Stats include number of RDS instances, EC2 instances, volumes, snapshots, VPCs, and AMIs, ECS clusters, ELBs and ASGs.
• aws_sg_summary.py - Print a summary of all SGs in the current account/region, their rules, and what's in them.
• aws_subnet_available_ips.py - Print information on used and available IPs in an AWS subnet.
• aws_subnet_ip_usage.py - Given an AWS Subnet ID or CIDR, report on the usable number of IPs, used IPs, and how many additional IPs would be taken up if all ELBs and ASGs scaled out.
• bigipcookie.pl - Perl script to de/encode F5 BigIp persistence cookies.
• bgw210-700_to_graphite.py - Pull stats from AT&T / Arris BWM210-700 router web UI and push to graphite.
• centos7_rpmbuild.Vagrantfile - Moved to https://github.com/jantman/rpmbuild-vagrant-boxes
• check_url_list.py - Script to check a list of URLs (passed on stdin) for response code, and for response code of the final path in a series of redirects.
• cm600_to_graphite.py - Pull stats from Netgear CM600 modem web UI and push to graphite.
• cmd-wrapper.c - C wrapper for a setuid/gid command, to ensure that ONLY a certain command and args can be run.
• collectRsyslogInfo.php - Script to collect information on a crashed/hung rsyslogd process, and log it all somewhere.
• cookies_from_pdml.py - Script to parse http Cookie header field from WireShark PDML XML.
• dashsnap.py - script to snapshot a graphite dashboard at specified intervals in the past (i.e. the last 2,4,6 hours) or a single specified time range. Snapshots both PNG images and raw JSON data, builds directory with HTML files.
• disqus_backup.py - script to backup all Disqus comments for a site.
• dot_find_cycles.py - uses Pydot and NetworkX to find cycles in a dot file directed graph (i.e. the graph output of Puppet)
• dump_firefox_session.py - Script to dump all URLs from a Firefox3 sessionstore.js or modern Firefox sessionstore.jsonlz4.
• dumpMysqlGrants.sh - Script to dump all grants from a MySQL server, for input into another.
• dump_skype_logs.py - Script to dump all Skype logs from a main.db file to HTML
• dump_sphinx_objects_inventory.py - Process URL for intersphinx targets and emit html or text.
• dynamodb_to_csv.py - Python/boto3 script to dump all data in a DynamoDB table to CSV.
• ec2-list-all-tags.py - Using boto3, list all distinct tag names on all EC2 instances in all regions.
• find_outdated_puppets.py - Script to look at a Puppet Dashboard unhidden-nodes.csv and extract the latest report time for each node, optionally, list nodes with runtime BEFORE a string.
• find_test_order_problems.py - Script to run tests multiple times and analyze JUnit results XML, to find tests with order-dependent failures.
• firefox_recovery_to_html.py - Script to convert Firefox profile sessionstore-backups/recovery.js to HTML links
• get_addons.py - Unmaintained script to download WoW addons from CurseForge.
• gist.py - Simple python script to upload a file as a private Gist on GitHub. Prompts interactively for Auth token, so usable from shared servers.
• git_repo_diff.py - uses GitPython to compare 2 git repo clones and report on branches that only exist in one, or have different head commits in the two repos
• github_clone_setup.py - script using github3.py to add upstream remote for any clone of a github fork, and add refs to check out pull requests from origin and upstream.
• github_find_member_with_key.py - Script using PyGithub to list an organization's members, and then find who has a specified public key.
• github_irc_hooks.py - script to setup GitHub repo IRC service hooks
• github_issue_watch_pushover.py - Poll github for updates to issues, notify via Pushover if there are any.
• github_label_setup.py - script to setup a given set of labels on all of your (or an org's) GitHub repos.
• har_urls.py - Script to dump all URLs and their status codes from a JSON HTTP Archive (HAR) file, such as those generated by the Firebug NetExport extension
• hipchat_date_history.py - Python script to retrieve HipChat room history for a specific date.
• htmldata.py - Perl library to manipulate HTML or XHTML documents, required by mw2html-auth
• increment_zone_serial - This script updates/increments the bind zone file serial number in a file specified as the first argument
• ismerged - shell script that takes a git branch name, and tells if it is merged into master or not
• jenkins_list_plugins.py - Python script to query Jenkins for all installed plugins, and list them. Optionally output as a block of Puppet code for the puppet-jenkins module.
• jenkins_node_labels.py - Python script to list all Jenkins slaves/executors and their labels.
• jenkins_plugins_to_puppet.py - DEPRECATED in favor of jenkins_list_plugins.py. Python script to query Jenkins for all installed plugins, and generate a block of Puppet code for the puppet-jenkins module.
• jira2trello.py - Python script to update a Trello board with some details from Jira.
• js2phpdoc.php - script to take comments and function prototypes from JS files and make them PHP-ish to be parsed by phpdoc
• kickRsyslog.php - script to restart rsyslog if it appears to have stopped logging
• lastpass2vault.py - Interactive script using HVAC and lastpass-python to copy your LastPass saved passwords to a HashiCorp Vault server.
• libvirt_csv.py - Use libvirt python bindings to print CSV lists of dom0_host,domU_name,ID,state,UUID for all qemu+kvm VMs running on libvirt hosts passed in as arguments
• LICENSE - License for these files - GPLv3 with additional provisions
• linode_ddns_update.sh - simple script to use Linode's HTTP API to update Linode DNS for a dynamic IP
• linodeDnsToCsv.php - Script to pull DNS information for all of your Linode hosted zones, output as CSV
• linode_list_records.py - Simple script to list all records in Linode DNS via API, along with their type, DomainID and ResourceID
• list_all_aws_resources_skew.py - Script using skew to list all AWS resources in your account
• list_github_org_repos.py - List information about an org's repositories using PyGithub (GitHub API library)
• make_puppet_param_markdown.py - # Python script to generate MarkDown docblock fragment for all parameters of a Puppet parameterized class or define. Simple, naive regex matching. Assumes you style your manifests properly.
• mw2html-auth - Produce an HTML version (standalone backup/export) of a MediaWiki site that's behind authentication
• nagios_log_problem_interval.pl - Chart intervals between problem and recovery from Nagios/Icinga logs
• nethogs2statsd.py - Python script to push nethogs data to statsd.
• nethogs2statsd.service - Example systemd unit for the above script.
• nmap-xml-to-table.php - Script to transform multiple nmap XML output files (presumably of the same host/port range with different scan options) into a HTML table
• pacman_compare.py - Compare packages in two files containing pacman -Q output. Ignores versions.
• pagerduty_list_incidents.py - Python script to list and filter PagerDuty incidents.
• print-cmd.sh - Simple script to log environment variables and original command for forced ssh commands
• print-cmd-wrapper.c - C wrapper like cmd-wrapper.c, but just echoes back the command that was called
• pushover - script to wrap execution of a command and send Pushover and notify-send notifications about its duration and exit code.
• quick_cloudtrail.py - Python script to parse AWS CloudTrail log JSON files and search for a user, IP, request ID, etc.
• README.VCS - Note on my CVS/SVN to github migration
• rebuild_srpm.sh - Script to rebuild a SRPM 1:1, useful when you want to build a RHEL/CentOS 6 SRPM on a RHEL/CentOS 5 system that doesn't support newer compression (cpio: MD5 sum mismatch)
• reconcile_git_repos.py - Script to reconcile the state of GitHub repos with local repos (from a private or other git server).
• reviewboard_reminder_mail.py - ReviewBoard - Script to send reminder emails for any open reviews, targeted at a specific group, not updated in more than X days.
• route53_ddns_update.sh - Bash script to update Route53 dynamic DNS
• rss_to_mail_config.py - sample configuration file for rss_to_mail.py*
• rss_to_mail.py - Dead simple python script to find new entries in an RSS feed, and email listing of new entries matching a regex to you. Intended to be run via cron.
• rsync-wrapper.c - setuid/gid wrapper around rsync. Useful to allow members of a specified group to do rsync backups as root over SSH.
• rsyslogIsHung.php - script to investigate rsyslog hangs, write output, and send mail
• rsyslogPstats.php - script to parse rsyslog impstats output and generate a simple report
• ruby_simplecov_diff.rb - Given two Ruby simplecov output directories, show the differences
• s3sync_inotify.py - Python script using boto and pyinotify to watch a directory and sync all files in it to an S3 bucket, and update an index.html file for the bucket.
• savescreen.py - Python script to save screen windows and titles, and write a screenrc to recreate them.
• scrape_domain.py - Python script using requests and BeautifulSoup4 to request all URLs/links/images/CSS/feeds/etc. found on a domain.
• show_cf_template_params.py - Show all parameters and their defaults for a CloudFormation template
• show_dhcp_fixed_ACKs.pl - script to show the most recent DHCP ACKs per IP address for ISC DHCPd, from a log file. Originally written for Vyatta routers that just show the dynamic leases
• simpleLCDproc.py - Simple LCDproc replacement in Python. Uses LCDd server.
• skeleton.py - Skeleton of a one-off Python CLI script, including optparse and logging.
• smart_check.py - Check SMART status of all attached and SMART-enabled disks via pySMART. Report on status. Cache status on disk, and exit non-zero if status of any disks changes.
• sms_backup_dump.py - Dump the XML SMS logs from the SMS Backup and Restore Android app to HTML
• sync_git_clones.sh - WIP / Alpha - Script to sync all local git clones in a list of paths with origin (and upstream, if configured). If present, uses github_clone_setup.py to setup upstream branches for any GitHub forks, and set refs to check out pull requests from origin and upstream.
• sync_git_clones.conf - config file for sync_git_clones.sh
• syslogAgeChecker.php - script to check timestamp of last syslog line in some files, and send mail if >= X seconds
• syslogDatesGraph.php - script to help visualize time distribution of syslog messages. This is the graph host part.
• syslogDatesToArray.php - script to help visualize time distribution of syslog messages. This is the log host part.
• syslogDatesToArray-sample.ser - example serialized data for syslogDatesGraph.php
• test_libvirt.py - some tests using the libvirt python bindings, for qemu+kvm hosts accessed over SSH
• timeout - shell script to execute a command with a timeout
• tomtom_tsp.py - Script to take a list of TomTom WoW addon coordinates and output them in the optimal order.
• toxit.py - Script to parse a tox.ini file, and run the test commands inside an existing virtualenv, against the code already installed there.
• transmission-alphabetical.py - Python script (using the transmission-rpc package) to adjust files in the Transmission bittorrent client so they download in alphabetical order.
• trello_copy_checklist.py - Python script to copy a checklist from one Trello card to another.
• trello_ensure_card.py - Script to ensure that a card with the given title (and optionally other attributes) exists in the specified column of the specified board.
• trello_push_due_dates.py - Script to push all due dates on a Trello board (optionally in one list) back by N days.
• twitter_find_followed_not_in_list.py - Simple script to list anyone whom you're following but isn't in one of your lists.
• ubiquiti-mac-acl/ - PHP script and MySQL schema to manage the MAC ACL on Ubiquiti AirOS2 devices.
• unifi_switch_to_statsd.py - Script to pull per-port stats from a UniFi switch and push them to statsd.
• unifi_websocket_log.py - Script to connect to UniFi controller websocket server and print all events, for testing.
• VipToInternalHosts.pl - script to take F5 BigIp VIP address and display the members of the pool it is served by
• watch_all_my_github_repos.py - Python script to ensure you're watching all of your own GitHub repos.
• watch_circleci.py - Python script to watch a CircleCI build, optionally notify via Pushover, and optionally retry failed builds.
• watch_cloudformation.py - Python script to watch a CloudFormation stack's events, and exit when the CF stack update or create finishes. Optional notification via PushOver.
• watch_elasticsearch.py - Python script to watch an ElasticSearch cluster's status and exit/notify when the status changes. Optional notifivation via PushOver.
• watch_jenkins.py - Python script using python-jenkins (https://pypi.python.org/pypi/python-jenkins) to watch a job (specified by URL), and exit 0 on success or 1 on failure. Optional notification via PushOver.
• watts_up_pro_logger.py - Logs data from a Watts Up Pro USB data collector to a file, and optionally to a Graphite instance.
• whendoiwork.py - Script to find all git repositories in a list of local filesystem paths, iterate over all commits in them (in the last N days), and build a histogram of the day of week and hour of day of your commits (using information from your git configuration).
• where_is_my_pi_zero.py - Python script to find in-stock Raspberry Pi Zero
• wiki-to-deckjs.py - simple, awful script to change markdown-like (very restricted markup set) markup to deck.js-ready html
• wordpress_daily_post.php - Script to publish the oldest post with a given status, if no other post has been published in 24 hours. Intended to be run via cron on weekdays
• xb3_to_graphite.py - Pull stats from Comcast XB3 modem web UI and push to graphite.