• Stars
    star
    447
  • Rank 97,700 (Top 2 %)
  • Language
    Python
  • Created about 10 years ago
  • Updated over 8 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Like Facebook's OSQuery, but for Postgres

TL;DR Example

--------------------------------------------------------
-- get the name, pid and attached port of all processes
-- which are listening on localhost interfaces
--------------------------------------------------------
SELECT DISTINCT
    process.name,
    listening.port,
    process.pid
FROM processes AS process
JOIN listening_ports AS listening
ON process.pid = listening.pid
WHERE listening.address = '127.0.0.1';
   name   | port | pid
----------+------+------
 postgres | 5432 | 6932

(1 row)

About

So I saw Facebook's OSQuery, and thought "That looks awesome, but complicated to build on top of SQLite. Postgres' Foreign Data Wrappers seem like a much better foundation. How long would it take to write the same app on top of Postgres?". Turns out it takes about 15 minutes, for someone who's never written an FDW before :-)

This approach does have the downside that it runs as the postgres user rather than as root, so it can't see the full details of other people's processes, but I'm sure that could be worked around if you really want to.

Currently this is just a proof-of-concept to see how useful Postgres' foreign data wrappers are, and how easy they are to create with the Multicorn python library. Seems the answers are "very useful" and "very easy". If people want to make this more useful by adding more virtual tables, pull requests are welcome~

Installation

TL;DR

git clone https://github.com/shish/pgosquery.git
cd pgosquery
sudo apt-get install pgxnclient
sudo pgxn install multicorn
sudo pip install psutil
sudo python setup.py develop
psql -h localhost -U postgres -w < queries.sql

Let your system python install know about this module:

$ sudo python setup.py develop

"setup.py develop" will link the current directory so you can modify it; "setup.py install" will copy a snapshot of current code to the OS folder.

Note that either way, you need to restart the postgres server to pick up python code changes.

Create a database with multicorn loaded (See http://multicorn.org/#installation for multicorn installation)

CREATE DATABASE pgosquery;
\c pgosquery;

CREATE EXTENSION multicorn;

Create a FDW table for PgOSQuery:

CREATE SERVER pgosquery_srv foreign data wrapper multicorn options (
    wrapper 'pgosquery.PgOSQuery'
);

CREATE FOREIGN TABLE processes (
    pid integer,
    name character varying,
	username character varying
) server pgosquery_srv options (
    tabletype 'processes'
);

CREATE FOREIGN TABLE listening_ports (
    pid integer,
    address character varying,
	port integer
) server pgosquery_srv options (
    tabletype 'listening_ports'
);

Select data:

--------------------------------------------------------
-- get the name, pid and attached port of all processes
-- which are listening on all interfaces
--------------------------------------------------------
SELECT DISTINCT
    process.name,
    listening.port,
    process.pid
FROM processes AS process
JOIN listening_ports AS listening
ON process.pid = listening.pid
WHERE listening.address = '127.0.0.1';
   name   | port | pid
----------+------+------
 postgres | 5432 | 6932

(1 row)

Table Types

processes: Columns are based on psutil's Process attributes, see http://pythonhosted.org/psutil/#psutil.Process

listening_ports: pid, address, port

net_connections: pid, address, port, type, status

More Repositories

1

rosettaboy

A gameboy emulator in several different languages
C++
444
star
2

shimmie2

Shish's official Shimmie code repository
PHP
353
star
3

context2

Golang flame chart viewer
Go
69
star
4

sikulpy

A reimplementation of the Sikuli API for CPython
Python
30
star
5

mic2midi

Whistle / Hum / Sing into a microphone, generate MIDI signals to drive a sequencer
Python
21
star
6

devtools-py

A Python client for Chrome's DevTools protocol / a headless chrome control library
Python
15
star
7

pyge

A game archive extraction tool written in Python, designed to make it easy to support new formats
Python
14
star
8

eve-mlp

Mobile Launch Platform for EVE Online
Python
12
star
9

cview3

AJAX comic viewer
JavaScript
12
star
10

clearskies-gui

A simple GUI for clearskies
Python
10
star
11

firehose

GPG-based chat network, resistant to traffic analysis
Python
10
star
12

esphome-projects

8
star
13

apache2rrd

Makes pretty graphs out of apache (or similar) log files
Python
7
star
14

Git-Flow-Eclipse

TODO: an eclipse plugin for easy git-flow based development ("TODO" as in "doesn't do anything useful yet")
Java
5
star
15

shimmie2-utils

Various utility scripts for Shimmie2
PHP
4
star
16

sqliteshelf

Allows an SQLite database to be used as a python dictionary with persistant storage
Python
4
star
17

ugh

A tinder client for people who think that tinder is awful
Python
4
star
18

csb

A curses-based browser for SQL databases
Python
4
star
19

context-apis

A collection of libraries for creating .ctxt files
Python
3
star
20

travmap

Travian Mapping Tool
PHP
3
star
21

rav

random avatar host
Python
3
star
22

lemonade

A somewhat fruity Ascii Demo Engine. Compilation of demos requires perl and 7-zip; playback requires perl, GNU tail and zcat (or compatible).
Perl
3
star
23

wp_deusex

Deus Ex Theme for Wordpress
PHP
2
star
24

guts

A proof-of-concept sparkleshare back-end
C#
2
star
25

eventtracer-py

Event Tracing for Python
Python
2
star
26

shm-cached

A shimmie cache daemon
Rust
2
star
27

link

See what you've got in common
Python
2
star
28

git-extras

A few somewhat hacky third-party git commands
Python
2
star
29

kuri2d

A simple game loosely based on Kurushi
C
2
star
30

spawn-fcgi.php

A shim to host FastCGI webapps on a PHP webhost
PHP
2
star
31

cinema

For watching with friends
TypeScript
2
star
32

sharedraw

A shared doodlepad, demonstrating python UDP networking
Python
2
star
33

shimmie1

Shish's Image Board, ancient edition
PHP
2
star
34

patch-archive

Various patches to third-party software
1
star
35

seagame

A game, involving the sea
Erlang
1
star
36

filemon

See what apps are making use of a given file
Python
1
star
37

microhtml

A tiny PHP HTML generating library
PHP
1
star
38

sdog

systemd-style process monitor
Python
1
star
39

shimpy

A direct translation of shimmie to python
Python
1
star
40

divetools2

a website with some scuba notes
TypeScript
1
star
41

tidydns

Tiny Dynamic DNS
Python
1
star
42

orpen

Drop-dead simple VPN
Python
1
star
43

votabo

Python
1
star
44

context

Python .ctxt viewer
Python
1
star
45

chunker

Yet another P2P file sync app, mostly abandoned
Python
1
star
46

slek3

Shish's Linux Enhancement Kit (aka shish's dotfiles)
Python
1
star
47

python-clearskies

A python library for communicating with the ClearSkies daemon
Python
1
star
48

java4k

A java game in 4KB (or less)
Java
1
star
49

packetstats

Packet sniffing and statsing
Rust
1
star
50

browserslist-logs

Turn webserver access logs into a browserslist file
TypeScript
1
star
51

std-solver

An experiment in solving Spot-the-Difference games
Python
1
star
52

travis-py35-opencv3

A docker recipe to build a Travis-CI environment with Python 3.5 and OpenCV 3.2 included
Shell
1
star
53

hyperapp-navigation

A polished version of the abandoned @hyperapp/navigation
HTML
1
star
54

commentonthis

Comment on This!
Python
1
star
55

microcrud

A tiny library for managing create / read / update / delete operations
PHP
1
star
56

shimmie2fe

Mobile-friendly Shimmie Frontend
TypeScript
1
star
57

zelda

Apache log file compressor
Python
1
star