@thi.ng/transducers

Lightweight transducer implementations for ES6 / TypeScript

Usage no npm install needed!

<script type="module">
  import thiNgTransducers from 'https://cdn.skypack.dev/@thi.ng/transducers';
</script>

README

thi.ng/umbrella

Build status Code Climate Become a patron Discord chat Twitter Follow

About

"A collection of functional programming libraries that can be composed together. Unlike a framework, thi.ng is a suite of instruments and you (the user) must be the composer of. Geared towards versatility, not any specific type of music." — @loganpowell via Twitter

Please visit thi.ng for additional information & topic based search of packages relevant to your use cases...

This mono-repository is home to 160+ thi.ng TypeScript/ES6 projects, a wide and jointly developed collection of packages and building blocks for (non-exhaustive list of topics):

  • Functional programming (ES6 iterators/generators, composition, memoization, transducers, multi-methods)
  • Reactive programming, stream / transducer based dataflow graphs / pipelines
  • Data structures & data transformations for wide range of use cases (maps, sets, heaps, queues, graphs etc.)
  • 2D geometry generation, shape primitives, math, manipulation, intersections, conversions & visualizations
  • Canvas abstractions, pixel buffers & SVG serialization/conversion
  • Vector, matrix (dense & sparse), ECS implementations with optional support for strided memory layouts
  • Semi-declarative WebGL 1/2 abstraction layer
  • DSL for shader functions defined in TypeScript and cross-compilation to GLSL, JS, VEX etc.
  • Value-based equivalence
  • DSP building blocks: oscillators, noise generators, filters, 1D FFT/IFFT, muxers, rate converters
  • Immutable data handling, state containers, transacted state updates, Undo-Redo history
  • Data driven UI component toolkits (DOM-based, canvas-based, immediate-mode, multiple approaches...)
  • Multi-format, multi-channel pixel buffers (int & float based), conversions, dithering, Porter-Duff alpha-blending operators
  • Color space/format conversions, matrix based color manipulation, gradient generators, color palettes
  • Date-time abstraction, relative dates, iterators, formatters, math
  • WebWorker workflow abstractions
  • PEG-style functional parser combinators w/ (optional) custom grammar definition language
  • Forth-style pointfree DSL for functional composition and DSL development/extension
  • S-expression parser & runtime (interpreter) infrastructure for custom DSL creation
  • Low-level tooling for binary data, shared memory/WASM/WebGL interop
  • SIMD batch-processing of vector data
  • Various interpolations, math helpers, automatic differentiation (Dual numbers)
  • etc. (see package overview below)

This project is NOT a framework, provides no turn-key, one-size-fits-all approach and instead encourages a mix & match philosophy for various key aspects of application design (in & outside the browser). Most customization points only expect certain interfaces/type signatures rather than concrete implementations.

All packages:

  • are versioned independently
  • distributed as ESM modules (ES2020 syntax) with export maps, TypeScript typings & change logs
  • highly modular with largely only single function / class (only closely related functions) per file to help w/ selective imports and tree shaking
  • provide re-exports of all their publics for full library imports
  • have either none or only @thi.ng internal runtime dependencies
  • declare public interfaces, enums & types in an api.ts and/or constants.ts file (larger packages only)
  • have auto-generated online documentation at docs.thi.ng
  • licensed under Apache Software License 2.0

Getting started

The sheer number and varied nature & purpose of these packages makes it impossible to provide traditional "Getting started" tutorials. To compensate, this repo provides a large number of example projects, detailed readmes (at the very least for all the larger and/or more important packages) and smaller usage examples in the docstrings.

Examples

There's a steadily growing number (100+) of standalone examples (different complexities, often combining functionality from several packages) in the examples directory.

Example screenshots (small selection)

Blog posts

Community, contributing, getting help

Join our little community on our Discord server or get in touch via Twitter or the issue tracker. If you'd like to contribute, please first read this document.

In general, we welcome contributions of all kinds (docs, examples, bug fixes, feature requests, financial contributions etc.). You can find a fairly detailed overview for contributors here: CONTRIBUTING.md.

Note: The default branch for this repo is develop and all PRs should be created based on this branch. This too means, the README files on this branch might refer to yet-unreleased features or packages. Please use the main branch for viewing the most recently released version(s)!.

Also please be sure to check the wiki for other project-wide information, tidbits, useful snippets etc.

Projects

Important changes (10/2021)

All packages are now only published in ESM format (ES2020 syntax) with TypeScript typings. You can find more details & rationale in issue #315.

As a result (and opportunity), many packages have undergone internal restructuring, some have been split up. All are now fully supporting deep-imports (based on declared export maps), leading to drastically smaller userland bundle sizes.

Previously we also created & published CommonJS & UMD formats, but deemed them an unnecessary burden & hindrance, since all important JS tooling/platforms do support ES modules by now.

Latest updates

As of: 2021-11-10

Project Version Changelog Description
@thi.ng/api version changelog Common types, decorators, mixins
@thi.ng/checks version changelog Type & value checks
@thi.ng/dl-asset version changelog Asset download trigger helper
@thi.ng/pixel version changelog Multi-format pixel buffers
@thi.ng/rasterize version changelog Shape drawing, filling & rasterization
@thi.ng/text-canvas version changelog Text-mode canvas, drawing, tables, charts

Fundamentals

Project Version Changelog Description
@thi.ng/args version changelog Declarative & functional CLI arg parsing & coercions
@thi.ng/api version changelog Common types, decorators, mixins
@thi.ng/bench version changelog Basic benchmarking helpers
@thi.ng/checks version changelog Type & value checks
@thi.ng/compare version changelog Comparators
@thi.ng/compose version changelog Functional composition helpers
@thi.ng/date version changelog Date/time iterators, formatters, rounding
@thi.ng/defmulti version changelog Dynamic multiple dispatch
@thi.ng/distance version changelog n-D distance metrics & K-nearest neighborhoods
@thi.ng/equiv version changelog Deep value equivalence checking
@thi.ng/errors version changelog Custom error types
@thi.ng/expose version changelog Conditional global variable exposition
@thi.ng/hex version changelog Hex value formatters for U4-64 words
@thi.ng/logger version changelog Basis infrastructure for arbitrary logging
@thi.ng/memoize version changelog Function memoization w/ customizable caching
@thi.ng/oquery version changelog Pattern based query engine for JS objects
@thi.ng/parse version changelog Parser combinators & AST generator/transformer
@thi.ng/paths version changelog Immutable nested object accessors
@thi.ng/strings version changelog Higher-order string formatting utils
@thi.ng/system version changelog Minimal life cycle container for stateful app components
@thi.ng/testament version changelog Minimal test runner

Maths

Project Version Changelog Description
@thi.ng/dual-algebra version changelog Dual number algebra / automatic differentiation
@thi.ng/dsp version changelog DSP utils, composable signal gens/processors
@thi.ng/fuzzy version changelog Fuzzy logic primitives & rule inference engine
@thi.ng/intervals version changelog Open/closed intervals, queries, set ops
@thi.ng/math version changelog Assorted common math functions & utilities
@thi.ng/matrices version changelog Matrix operations
@thi.ng/sparse version changelog Sparse matrix & vector impls
@thi.ng/vectors version changelog Fixed & arbitrary-length vector ops

Randomness

Project Version Changelog Description
@thi.ng/colored-noise version changelog 1D colored noise generators
@thi.ng/ksuid version changelog K-sortable unique identifiers, binary & base-N encoded
@thi.ng/lowdisc version changelog n-D Low discrepancy sequence generators
@thi.ng/random version changelog Seedable PRNG implementations, distributions & utilities

File format support

Project Version Changelog Description
@thi.ng/bencode version changelog Bencode binary format encoding
@thi.ng/csv version changelog Customizable CSV parser/object mapper
@thi.ng/dot version changelog Graphviz DOM & export
@thi.ng/dsp-io-wav version changelog WAV file format exporter
@thi.ng/geom-io-obj version changelog Wavefront OBJ model parser
@thi.ng/hiccup-css version changelog CSS from nested JS data structures
@thi.ng/hiccup-html version changelog Type-checked HTML5 element wrappers for hiccup
@thi.ng/hiccup-markdown version changelog Hiccup-to-Markdown serialization
@thi.ng/hiccup-svg version changelog hiccup based SVG vocab
@thi.ng/iges version changelog IGES format geometry serialization
@thi.ng/markdown-table version changelog Markdown table generator / formatter
@thi.ng/mime version changelog File extension to MIME type mappings
@thi.ng/pixel-io-netpbm version changelog 1/8/16/24bit NetPBM image format reader/writer
@thi.ng/prefixes version changelog Linked Data, RDF & xmlns prefixes/URLs
@thi.ng/sax version changelog SAX-like XML parser / transducer

Iterator, stream & sequence processing

Project Version Changelog Description
@thi.ng/csp version changelog Channel based async ops
@thi.ng/fsm version changelog FSM / parser primitives
@thi.ng/grid-iterators version changelog 2D grid iterator strategies
@thi.ng/iterators version changelog ES6 generators / iterators
@thi.ng/seq version changelog Lisp/Clojure-style sequence abstraction
@thi.ng/transducers version changelog Composable data transformations
@thi.ng/transducers-binary version changelog Binary data related transducers
@thi.ng/transducers-fsm version changelog Finite state transducer
@thi.ng/transducers-hdom version changelog Transducer based hdom UI updates
@thi.ng/transducers-patch version changelog Patch-based, array & object editing
@thi.ng/transducers-stats version changelog Technical / statistical analysis

Reactive programming

Project Version Changelog Description
@thi.ng/rstream version changelog Push-based, reactive event stream primitves
@thi.ng/rstream-csp version changelog Adapter bridge CSP -> rstream
@thi.ng/rstream-dot version changelog Graphviz visualization of rstream topologies
@thi.ng/rstream-gestures version changelog Mouse & touch event stream abstraction
@thi.ng/rstream-graph version changelog Declarative dataflow graph construction
@thi.ng/rstream-log version changelog Hierarchical structured data logging
@thi.ng/rstream-log-file version changelog Log-file output handler
@thi.ng/rstream-query version changelog Triple store & query engine

Algorithms & data structures

Project Version Changelog Description
@thi.ng/adjacency version changelog Adjacency matrices & graph algorithms
@thi.ng/arrays version changelog Array utilities
@thi.ng/associative version changelog Alt Set & Map implementations
@thi.ng/atom version changelog Immutable value wrappers, views, history
@thi.ng/bitfield version changelog 1D/2D bit field implementations
@thi.ng/cache version changelog In-memory caches / strategies
@thi.ng/dcons version changelog Doubly-linked list
@thi.ng/diff version changelog Array & object diffing
@thi.ng/dgraph version changelog Dependency graph
@thi.ng/ecs version changelog Entity-Component System
@thi.ng/egf version changelog Extensible Graph Format
@thi.ng/gp version changelog Genetic programming helpers / AST gen
@thi.ng/heaps version changelog Binary & d-ary heap impls
@thi.ng/idgen version changelog Versioned ID generation / free-list
@thi.ng/k-means version changelog K-means clustering of n-D data
@thi.ng/ramp version changelog Parametric, interpolated lookup tables
@thi.ng/quad-edge version changelog Quad-edge, dual-graph data structure
@thi.ng/resolve-map version changelog DAG computations & value resolution
@thi.ng/vclock version changelog Vector clock functions / comparators
@thi.ng/zipper version changelog Immutable tree editing / navigation

Frontend / UI

Project Version Changelog Description
@thi.ng/adapt-dpi version changelog HDPI canvas adaptation / styling util
@thi.ng/dl-asset version changelog Asset download trigger helper
@thi.ng/hdiff version changelog String diffing w/ hiccup output (includes CLI)
@thi.ng/hdom version changelog Hiccup based VDOM & diffing
@thi.ng/hdom-canvas version changelog hdom adapter for hiccup-canvas
@thi.ng/hdom-components version changelog hdom based UI components
@thi.ng/hdom-mock version changelog hdom mock implementation (testing / prototyping)
@thi.ng/hiccup version changelog S-expression based HTML/XML serialization
@thi.ng/hiccup-canvas version changelog hiccup interpreter for canvas api
@thi.ng/hiccup-carbon-icons version changelog IBM Carbon icons in hiccup format
@thi.ng/imgui version changelog Immediate mode GUI
@thi.ng/interceptors version changelog Composable event handlers & processor
@thi.ng/rdom version changelog Reactive, diff-less, async UI components
@thi.ng/rdom-canvas version changelog rdom component wrapper for thi.ng/hiccup-canvas
@thi.ng/rdom-components version changelog Unstyled, customizable component collection
@thi.ng/router version changelog Customizable browser & non-browser router
@thi.ng/text-canvas version changelog Text-mode canvas, drawing, tables, charts
@thi.ng/text-format version changelog Color text formatting w/ ANSI & HTML presets

Geometry, image & visualization

Project Version Changelog Description
@thi.ng/color version changelog Color conversions, gradients
@thi.ng/color-palettes version changelog Collection of color palettes
@thi.ng/dgraph-dot version changelog Dependency graph -> Graphviz
@thi.ng/fuzzy-viz version changelog Visualization, instrumentation for @thi.ng/fuzzy
@thi.ng/geom version changelog 2D only geometry types & ops
@thi.ng/geom-accel version changelog Spatial indexing data structures
@thi.ng/geom-api version changelog Shared types & interfaces
@thi.ng/geom-arc version changelog 2D elliptic arc utils
@thi.ng/geom-clip-line version changelog 2D line clipping
@thi.ng/geom-clip-poly version changelog 2D convex polygon clipping
@thi.ng/geom-closest-point version changelog Closest point helpers
@thi.ng/geom-fuzz version changelog Fuzzy 2D shape drawing / filling
@thi.ng/geom-hull version changelog 2D convex hull (Graham scan)
@thi.ng/geom-isec version changelog Point & shape intersection tests
@thi.ng/geom-isoline version changelog 2D contour line extraction
@thi.ng/geom-poly-utils version changelog 2D polygon helpers
@thi.ng/geom-resample version changelog nD polyline / curve resampling
@thi.ng/geom-splines version changelog nD cubic / quadratic spline ops
@thi.ng/geom-subdiv-curve version changelog nD iterative subdivision curves
@thi.ng/geom-tessellate version changelog nD convex polygon tessellators
@thi.ng/geom-voronoi version changelog 2D iterative delaunay/voronoi
@thi.ng/lsys version changelog Extensible L-System architecture
@thi.ng/pixel version changelog Multi-format pixel buffers
@thi.ng/pixel-dither version changelog Image dithering w/ various algorithm presets
@thi.ng/poisson version changelog nD Poisson disk sampling
@thi.ng/porter-duff version changelog Alpha blending / compositing ops
@thi.ng/rasterize version changelog Shape drawing, filling & rasterization
@thi.ng/scenegraph version changelog Extensible 2D/3D scenegraph
@thi.ng/simd version changelog WebAssembly SIMD vector batch processing
@thi.ng/viz version changelog Declarative & functional data visualization toolkit

WebGL / GPGPU

Project Version Changelog Description
@thi.ng/shader-ast version changelog AST DSL for x-platform shader code
@thi.ng/shader-ast-glsl version changelog GLSL code generator
@thi.ng/shader-ast-js version changelog JS code generator
@thi.ng/shader-ast-optimize version changelog AST code optimization strategies
@thi.ng/shader-ast-stdlib version changelog 100+ useful AST shader functions
@thi.ng/webgl version changelog WebGL 1/2 / GPGPU facilities
@thi.ng/webgl-msdf version changelog MSDF font rendering
@thi.ng/webgl-shadertoy version changelog Shadertoy-like WebGL setup

Low-level, binary, memory management

Project Version Changelog Description
@thi.ng/base-n version changelog Arbitrary base-n encoding/decoding with presets
@thi.ng/binary version changelog Assorted binary / bitwise ops, utilities
@thi.ng/bitstream version changelog Bitwise input / output streams
@thi.ng/dlogic version changelog Digital logic ops / constructs
@thi.ng/leb128 version changelog WASM based LEB128 varint encoder / decoder
@thi.ng/malloc version changelog Raw & typed array memory pool & allocator
@thi.ng/morton version changelog Z-order-curve / Morton coding
@thi.ng/range-coder version changelog Binary data Range encoder / decoder
@thi.ng/rle-pack version changelog Run-length encoding data compression
@thi.ng/soa version changelog Memory mapped data structures & serialization
@thi.ng/unionstruct version changelog Wrapper for C-like structs / unions
@thi.ng/vector-pools version changelog Data structures for memory mapped vectors

DSLs

Project Version Changelog Description
@thi.ng/pointfree version changelog Stack-based DSL & functional composition
@thi.ng/pointfree-lang version changelog Forth-like syntax layer for @thi.ng/pointfree
@thi.ng/sexpr version changelog S-Expression parser & runtime infrastructure

Building

git clone https://github.com/thi-ng/umbrella.git
cd umbrella

yarn install
yarn build

Once the entire mono-repo has been fully built at least once before, individual packages can then be (re)built like so:

yarn workspace @thi.ng/transducers run build

# or

(cd packages/transducers && yarn build)

Building example projects

Please see the example build instructions in the wiki for further details.

Testing

(Most, but not all packages have tests)

Due to various build/config issues/complexities, we're now using our own minimal test runner @thi.ng/testament

yarn test

# or individually
yarn workspace @thi.ng/rstream run test

Documentation

Autogenerated documentation (using TypeDoc) will be saved under /packages/*/doc/ and is also available at docs.thi.ng.

yarn doc

Furthermore, an experimental tsdoc-based documentation repo provides an alternative markdown output including cross-references between all packages, but currently isn't frequently updated (only every few months):

https://github.com/thi-ng/umbrella-docs-temp

License

© 2015 - 2021 Karsten Schmidt // Apache Software License 2.0

Contributors ✨

Thanks goes to these wonderful people (emoji key):


Karsten Schmidt

πŸ’» πŸ“– 🚧 πŸ’΅

Alberto

πŸ’» πŸ’‘ πŸ› πŸ€” πŸ’΅

Arthur Carabott

πŸ’» πŸ€” πŸ’‘ πŸ“ πŸ’΅

AndrΓ© Wachter

πŸ’» πŸ€” πŸ›

Gavin Cannizzaro

πŸ’» πŸ› πŸ€”

Logan Powell

πŸ“– πŸ› πŸ€” πŸ’΅

Marcin Ignac

πŸ›

arcticnoah

πŸ’»

allforabit

πŸ› πŸ’» πŸ€” πŸ’΅

Yifan Wu

πŸ› πŸ“–

stwind

πŸ’» πŸ›

evilive

πŸ’»

Bnaya Peretz

πŸ’» πŸ› πŸ€”

oljeger

πŸ›

Nik Shevchenko

πŸ› πŸ’»

Matei Adriel

πŸ’» πŸ› πŸ€”

Pierre Grimaud

πŸ“–

Matt Huebert

πŸ’΅

Raphael Saunier

πŸ’΅

Eric Ihli

πŸ’΅

David Pham

πŸ’΅

TBD

πŸ› πŸ€”

Pedro Henriques dos Santos Teixeira

πŸ’΅

Jamie Owen

πŸ’» πŸ›

Robert Kesteson

πŸ› πŸ’»

Chancy Kennedy

πŸ’΅

Jarred Sumner

πŸ›

Jamie Slome

πŸ› πŸ›‘οΈ

d3v53c

πŸ› πŸ›‘οΈ

Jannis Pohlmann

πŸ›

Shakthi Prasad G S

πŸ› πŸ’»

Robin Gower

πŸ›

Michael Latzoni

πŸ›

Z Yin

πŸ›

Damien Seguin

πŸ› πŸ’»

Rui Gil

πŸ›

Ja|nz

πŸ’» πŸš‡ 🚧

This project follows the all-contributors specification. Contributions of any kind welcome!