nicer

An Http Multipart/Form-Data Request Body Parser.

Usage no npm install needed!

<script type="module">
  import nicer from 'https://cdn.skypack.dev/nicer';
</script>

README

nicer

npm version

nicer is An Http Multipart/Form-Data Request Body Parser. It can receive form data fields and files headers and stream their data.

yarn add nicer
Benchmark
Library 1 2 3
dicer 137.21mb/s 100.65mb/s 165.95mb/s
multiparty 25.78mb/s 29.48mb/s 29.60mb/s
nicer 106.55mb/s 112.46mb/s 134.24mb/s
nicerc 114.42mb/s 115.04mb/s 116.57mb/s
Stable Benchmark (18 Jul)
Library Max Speed
(1) dicer 92.93mb/s
(2) nicerc 79.95mb/s
nicer 72.23mb/s
(3) multiparty 28.79mb/s
Nicer is comparable to the faster streaming parser, Dicer since the real-world data (uploading 2 fields, 2 text files and 50 photos) is processed at speeds that are close to max 90mb/s. In the benchmark, nicer is the source code of this package, whereas nicerc is the compiled JavaScript optimised Closure Compiler, which probably increases the speed by 5-10%.

Table Of Contents

API

The package is available by importing its default constructor function:

import Nicer from 'nicer'

constructor(
  boundary: string,
): Nicer

Creates a transform that emits objects with a header buffer and the body stream. The body stream is a pass-through so all data must be written as it comes, the request doesn't pause for data to be consumed. The header is a buffer which can be parsed more and/or decrypted, but it does not stream. The assumption is the headers are short therefore a header buffer is accumulated until \r\n is found. Just make sure to run behind NginX then it should be alright.

Nicer: A stream that emits objects with a header buffer and the body PassThrough stream.

Name Type Description
boundary* string The mandatory field separator.
import { Transform } from 'stream'
import Nicer from 'nicer'

const detected = []

await http.startPlain((req, res) => {
  const boundary = getBoundary(req, res)
  console.log('Boundary detected: %s', boundary)
  req.pipe(new Nicer({ boundary })).pipe(new Transform({
    objectMode: true,
    transform(obj, enc, next) {
      const { header: HEADER, stream: STREAM } = obj

      // to print in sync have to wait for all data
      // since STREAM is a pass-through
      let d = []
      detected.push(['%s\n====\n', HEADER, d])

      STREAM.on('data', (data) => {
        d.push(data)
      })
      next()
    },
    final() {
      res.statusCode = 200
      res.end(JSON.stringify(detected))
    },
  }))
})
A new instance of Nicer can be piped into by an http.IncomingMessage stream in the Node.JS server. Then a transform stream must be created to listen for the data emitted by Nicer in object mode.

Part: A part that gets emitted by Nicer.

Name Type Description
stream* !stream.PassThrough The mandatory field separator.
header* !Buffer The header found before data.
Boundary detected: u2KxIV5yF1y+xUspOQCCZopaVgeV6Jxihv35XQJmuTx8X3sh

Content-Disposition: form-data; name="key"
====
 [ 'value' ] 


Content-Disposition: form-data; name="alan"
====
 [ 'watts' ] 


Content-Disposition: form-data; name="file"; filename="test/fixture/test.txt"
Content-Type: application/octet-stream
====
 [ 'a test file\n' ]
The data received by the 'transform' method, contains the { header, stream } properties. The data from the stream must be accumulated.

Errors

The errors are spawned when the buffer remaining the stream after the final event, and processed to extract the rest of the fields, still contains symbols different from -- ([45,45]).

Extra Buffer (Source)
-------example
Content-Disposition: form-data; name="key"

data
-------exampleWAT
The data remaining after the last boundary detected after the final method is called does not have any meaning and is discarded. This is not the case with parts that arrived before the stream was closed, i.e., the file limit is not implemented.
Boundary detected: -----example
[!] Error Unexpected end of request body, wanted to see "--" but saw WA.
    Detected Data:

Content-Disposition: form-data; name="key"
====
 [ 'data' ]
The parser will always check for the closing -- and emit an error in the end, however the headers and data streams emitted by it, would have been all closed, i.e., the data can still be used.

Debug

The software can write debug information, when the DEBUG=nicer env variable is set.

Debug (Source)
import { Writable } from 'stream'
import Nicer from '../src'

const detected = []

await http.startPlain((req, res) => {
  const boundary = getBoundary(req, res)
  console.log('Boundary detected: %s', boundary)
  const nicer = new Nicer({ boundary })
  const bt = new BufferTransform(50)

  req.pipe(bt).pipe(nicer).pipe(new Writable({
    objectMode: true,
    write(obj, enc, next) {
      const { header: HEADER, stream: STREAM } = obj
      next()
    },
    final() {
      res.statusCode = 200
      res.end(JSON.stringify(detected))
    },
  }))
})
The transform method appends data to the left-over buffer (which can usually be small enough to accommodate [--boundary.length-1] symbols) and consumes data. The data is consumed by first trying to find the boundary in the new buffer. If this is possible, then depending on the state of the parser, the data found before the separator is either flushed in an existing data stream, or appended to the existing header, which can then lead to body-flushing.
nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer 🔎  Finished boundary scan, buffer of length 50B left, separators found: 0 +2ms
  nicer one consume safe consumed 0B and left 50B +1ms
  nicer <concat-transform> +1ms
  nicer <concat-transform> 100B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer   ⭐  Found starting boundary at index 2 +0ms
  nicer 🔎  Finished boundary scan, buffer of length 48B left, separators found: 1 +0ms
  nicer one consume safe consumed 52B and left 48B +1ms
  nicer <concat-transform> +0ms
  nicer <concat-transform> 98B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer 🔎  Finished boundary scan, buffer of length 98B left, separators found: 0 +0ms
  nicer one consume safe consumed 46B and left 52B +0ms
  nicer <concat-transform> +0ms
  nicer <concat-transform> 102B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer   🔛  Found boundary, data size 7B +0ms
  nicer   🗒  Found header and data of size <53B> +1ms
  nicer      Content-Disposition: form-data; name="key" +0ms
  nicer      value +0ms
  nicer 🔎  Finished boundary scan, buffer of length 43B left, separators found: 1 +2ms
  nicer one consume safe consumed 59B and left 43B +0ms
  nicer <concat-transform> +0ms
  nicer <concat-transform> 93B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer 🔎  Finished boundary scan, buffer of length 93B left, separators found: 0 +0ms
  nicer one consume safe consumed 41B and left 52B +0ms
  nicer <concat-transform> +1ms
  nicer <concat-transform> 102B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer   🔛  Found boundary, data size 13B +0ms
  nicer   🗒  Found header and data of size <54B> +0ms
  nicer      Content-Disposition: form-data; name="alan" +0ms
  nicer      watts +0ms
  nicer 🔎  Finished boundary scan, buffer of length 37B left, separators found: 1 +0ms
  nicer one consume safe consumed 65B and left 37B +1ms
  nicer <concat-transform> +0ms
  nicer <concat-transform> 87B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer 🔎  Finished boundary scan, buffer of length 87B left, separators found: 0 +0ms
  nicer one consume safe consumed 35B and left 52B +0ms
  nicer <concat-transform> +0ms
  nicer <concat-transform> 102B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +11ms
  nicer 🔎  Finished boundary scan, buffer of length 102B left, separators found: 0 +0ms
  nicer       <concat-header> +0ms
  nicer       <concat-header> 85B +1ms
  nicer one consume safe consumed 50B and left 52B +0ms
  nicer <concat-transform> +0ms
  nicer <concat-transform> 102B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +1ms
  nicer   🔛  Found boundary, data size 50B +0ms
  nicer   🗒  Found header and data of size <135B> +0ms
  nicer      Content-Disposition: form-data; name="file"; filename="test/fixture/test.txt" +0ms
  nicer      Content-Type: ap... +0ms
  nicer 🔎  Finished boundary scan, buffer of length 0B left, separators found: 1 +1ms
  nicer one consume safe consumed 102B and left 0B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +0ms
  nicer 🔎  Finished boundary scan, buffer of length 4B left, separators found: 0 +0ms
  nicer one consume safe consumed 0B and left 4B +0ms
  nicer 🔍  Staring boundary --u2KxIV5yF1y+x... scan +2ms
  nicer 🔎  Finished boundary scan, buffer of length 4B left, separators found: 0 +1ms
  nicer one consume safe consumed 0B and left 4B +0ms
After knowing what's left after the last found boundary, the Nicer parser takes only the safe amount of data to consume more which equals to the length of the boundary (including prior --), otherwise there might be a partial boundary leaking into the data stream. The remainder is saved as the new buffer, to which the following chunk in the transform method will be appended, and so on.

Copyright

(c) Idio 2019