wordmap-lexer

A lexer used in wordMAP

Usage no npm install needed!

<script type="module">
  import wordmapLexer from 'https://cdn.skypack.dev/wordmap-lexer';
</script>

README

Build Status codecov TypeScript

wordMAP-lexer

A lexical tool for wordMAP

The lexer provides tools generating tokens which can be fed into WordMAP.

Install

npm i wordmap-lexer

Usage


require Lexer, {Token} from 'wordmap-lexer';

const tokens : Token[] = Lexer.tokenize("hello, world");
// produces tokens "hello" and "world".

const puct_tokens : Token[] = Lexer.tokenize("hello, world", {punctuation: true});
// produces tokens "hello", ",", and "world".