async-iterable-limiter

Iterate over async iterable with concurrency limit.

Usage no npm install needed!

<script type="module">
  import asyncIterableLimiter from 'https://cdn.skypack.dev/async-iterable-limiter';
</script>

README

async-iterable-limiter

Iterate over async iterable with concurrency limit.

Difference from many other similar modules is that it supports iterating over async iterables (like ReadableStream or database cursor).

Note: documentation and tests are work in progress, but it should already work :)

Usage

Simple:

const Limiter = require('async-iterable-limiter');

const cursor = db.collection().find(); // todo: better example

await Limiter.iterate(cursor, 5, async (item) => {
    await doWork(item);
});

Complex (internal implementation of iterate function):

const Limiter = require('async-iterable-limiter');

const limiter = new Limiter({ concurrency: 5 });
for await (const item of iterable) {
    await limiter.ready();
    limiter.add(taskReturningPromise(item));
}
await limiter.finished();

TODO:

Functionality:

  • add stopOnError parameter (true by default in current implementation)
  • maybe add Limiter.map function that would produce an array with sequential results
  • maybe add Limiter.mapIterable function that would produce another async iterable
  • make inProgress and concurrency readonly properties, to prevent mess

Tests:

  • add test for error handling of async iterable
  • add test for error handling of task
  • add test for invalid input of params
  • add test for expected behavior of multiple calls of Limiter.ready/finished class

Documentation:

  • readme
  • api surface documentation
  • typescript definitions