• This repository has been archived on 06/Dec/2017
  • Stars
    star
    176
  • Rank 216,987 (Top 5 %)
  • Language
    Python
  • License
    Other
  • Created almost 13 years ago
  • Updated almost 13 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

Realtime log reader in Flask

Logmon

Logmon is a realtime log reader written with Flask and Juggernaut.

Realtime log reader in Flask

Installation

Start by cloning this repository:

$ git clone git://github.com/maxcountryman/logmon.git

This application does assume you have Node.js installed so that you can make use of npm and run Juggernaut. You will need npm to install Juggernaut if not already installed:

$ curl http://npmjs.org/install.sh | sh

Now you should be able to install Juggernaut:

$ npm install -g juggernaut 

Once installed you will now need to run Juggernaut. Juggernaut serves as an interface between the frontend of the application and the backend.

Finally before you can use the included runner you will need to install gevent. Gevent is a coroutine-based module for concurrency. Here it serves the simple task of reading off a file without blocking execution:

$ pip install -U gevent

At this point you should be able to execute the runner script and point your browser to http://127.0.0.1:5000.

Usage

Once set up, ensure the Juggernaut server is already running and then simply execute the runner script:

$ python runner.py

A Note On Rotating Logs

Depending upon how log rotating is setup, the follow function in the runner script will break when the log is rotated. In order to get around this you may need to do some hacking related to your specific set up.

Assuming you're using logrotated and nginx you can edit the logrotated config to use a customized sh command when rotating. For instance, let's assume your config is located /etc/logrotate.d/nginx, then you might edit it to look like this:

    /var/log/nginx/*log {
            create 640 http log
            compress
            postrotate
                    /bin/kill -USR1 `cat /var/run/nginx.pid 2>/dev/null` 2> /dev/null || true
                    /bin/kill -HUP `cat /tmp/Logmon.pid`
            endscript
    }

Notice we are sending a SIGHUP to the logmon script. In order for this to work we need to modify the runner to save its PID as well as respond to SIGHUP in a favorable way.

Because the runner is making use of gevent and due to some pecularities of the specifics of the way libev is wrapped, the gevent.signals wrapper must be used in place of Python's built-in module.

The rewrite should look something like this, although depending on your situation you may find this needs to be modified:

if __name__ == '__main__':
    from logmon import app
    
    from flask import escape
    from juggernaut import Juggernaut
    
    import gevent
    from gevent import monkey
    from gevent.wsgi import WSGIServer
    from werkzeug.contrib.fixers import ProxyFix
    
    import os
    import time
    import signal
    
    monkey.patch_all()
    jug = Juggernaut()
    f = None
    
    LOG_FILE = app.config['LOG_FILE']
    
    
    def write_pid(filepath='/tmp/{}.pid'):
        with open(filepath.format(app.config['SITE_NAME']), 'w') as f:
            pid = str(os.getpid())
            f.write(pid)
    
    
    def follow(filepath):
        global f
        f = open(filepath)
        f.seek(0, 2)
        
        def rotate_handler():
            global f
            f.close()
            f = open(filepath)
        
        # bind to SIGHUP
        gevent.signal(signal.SIGHUP, rotate_handler)
        
        while True:
            line = f.readline()
            if not line:
                time.sleep(0.1)
                continue
            line = escape(line)
            jug.publish('logger', line)
    
    
    # write the PID to a file
    write_pid()
    
    # fixes the X-Real-IP header
    app.wsgi_app = ProxyFix(app.wsgi_app)
    http_server = WSGIServer(('127.0.0.1', 5051), app)
    jobs = [gevent.spawn(follow, LOG_FILE),
            gevent.spawn(http_server.serve_forever)]
    gevent.joinall(jobs)

More Repositories

1

flask-login

Flask user session management.
Python
3,569
star
2

axum-login

🪪 User identification, authentication, and authorization for Axum.
Rust
550
star
3

flask-bcrypt

Flask-Bcrypt is a Flask extension that provides bcrypt hashing utilities for your application.
Python
324
star
4

tower-sessions

🥠 Sessions as a `tower` and `axum` middleware.
Rust
214
star
5

flask-uploads

File uploads for Flask.
Python
208
star
6

flask-seasurf

SeaSurf is a Flask extension for preventing cross-site request forgery (CSRF).
Python
190
star
7

flake

Decentralized, k-ordered unique IDs in Clojure
Clojure
142
star
8

atomos

Atomic primitives for Python.
Python
119
star
9

warc-parquet

🗄️ A simple CLI for converting WARC to Parquet.
Rust
103
star
10

axum-sessions

🥠 Cookie-based sessions for Axum via async-session.
Rust
74
star
11

aquamarine

A demo of zero-downtime deploys with Docker Compose and Traefik
Shell
53
star
12

irctk

A simple framework for writing IRC applications
Python
44
star
13

quanta

Distributed CRDT of sparse integer vectors.
Clojure
33
star
14

forma

🐚 An opinionated SQL formatter.
Rust
27
star
15

axum-messages

🛎️ One-time notification messages for Axum.
Rust
26
star
16

tower-sessions-stores

🚃 Previously bundled session stores for `tower-sessions`.
Rust
23
star
17

hyperlight

A performance-focused HTTP reverse proxy
Clojure
19
star
18

flask-themes

Flask Themes
Python
19
star
19

cryptotrade

A simple Python API wrapper for Bitcoin trading platforms such as MtGox and TradeHill
Python
14
star
20

flog

A blog written with Flask
Python
9
star
21

flask-wepay

A Flask wrapper for WePay's Python API
Python
8
star
22

blizzard

HTTP unique ID generation service
Clojure
8
star
23

st

Fast and simple statistics on the command line.
Rust
6
star
24

markov-domains

Finds available domains using Markov chains.
Clojure
6
star
25

nautilus

User authentication and management service
Clojure
5
star
26

yelp-api

A wrapper for Yelp's public API
PHP
4
star
27

affinis

An IRC library for Clojure.
Clojure
4
star
28

wtforms

Python
4
star
29

rauth

A Python library for OAuth 1.0/a, 2.0, and Ofly
Python
4
star
30

simpleirc

An IRC connection layer written in Python.
Python
4
star
31

headers-accept

🤝 The missing `Accept` implementation for `headers::Header`.
Rust
4
star
32

pyxine-branch

Branch of the Python extension for xine
Python
3
star
33

fluyt

ClojureScript HTTP requests
Clojure
3
star
34

cozy

A modern Node API template for the weary traveller
JavaScript
3
star
35

dotfiles

Development environment configuration files.
Shell
3
star
36

ewt

EDN Web Tokens
Clojure
3
star
37

flask-simpleoauth

A dead simple OAuth 1.0a provider in Flask
Python
3
star
38

simpleoauth

Simple, correct OAuth 1.0 and 2.0 signing methods.
Python
2
star
39

kaa

Kaa is the resident IRC bot on VoxInfinitus, written with IrcTK
Python
2
star
40

voxinfinitus

Basic Django apps providing CMS and blog functionality for Voxi
Python
2
star
41

mage

A Clojure-like Lisp.
Python
2
star
42

tasker

simple task manager
Python
2
star
43

ChatOnMacWebAPI-Swift

Swift
2
star
44

chatter

Chatter is a quick and dirty realtime chat application written in Flask
Python
2
star
45

bitpit-https-bridge

A simple Flask app to bridge the unsecured service with a secured page
Python
1
star
46

conceptis.org

My personal site and blog
Python
1
star
47

primes

A simple Clojure program for generating a multiplication table of primes
Clojure
1
star
48

konvej

Httpbin in Clojure.
Clojure
1
star
49

atrium

HTTP Authentication Service
1
star
50

clasp

A dead simple routing DSL for Clojure's ring.
Clojure
1
star
51

celeb

Incomplete Flask gallery project, now abandoned
Python
1
star
52

accord

A simple OAuth 1.0/a, 2.0 consumer client for Clojure.
Clojure
1
star
53

locksmithing

Lock-free, concurrent data structure experiments.
Clojure
1
star