draino

Heroku logplex drain framework for node.js

This repo is a work in progress / spike of the following processing architecture:

  +------+                            +-------------------+                +-------+
  | App  |   +------+       HTTP(S)   |                   |     STDIN      |       |
  +------+   | App  |    +----------->|      Draino       |+-------------->|Filter |
   +         +------+    |            |                   |<--------------+|       |
   |STDOUT     +         |            +-------------------+     STDOUT     +-------+
   |           |STDOUT   |                  ++        ^  +
   v           v         |                  ||        |  |
 +---------------+       |                  ||        |  |
 |               |       |                  ||        |  | STDIN  +-------+
 | h drains:add  |+------+                  ||        |  +------->|       |
 |               |                          vv        |           |Filter |
 +---------------+                        STDOUT      +----------+|       |
      ^        ^                                           STDOUT +-------+
      |STDOUT  |
      +        |
     +------+  | STDOUT   +------+
     | App  |  +--------+ | App  |
     +------+             +------+

All you have to do is write the filters, which are any UNIX program that can parse JSON.

Draino spawns everything in the ./filters directory and pipes log data into STDIN as one JSON object per line.

Modes

usage

npm install draino

-f single filter

draino -f path/to/filter

-d directory of filters

draino -f path/to/filter

--heroku normalize logplex metadata to logfmt

draino -f path/to/filter --heroku

--logfmt parse logfmt logs and emit JSON

draino -f path/to/filter --heroku --logfmt