README
tokenizer-next
iterator based tokenizer for writing parsers
🔧 Install · 🧩 Example · 📜 API docs · 🔥 Releases · 💪🏼 Contribute · 🖐️ Help
Install
$ npm i tokenizer-next
What is this?
A tokenizer for writing parsers based on RegExp's named groups as returned, for example, by String.prototype.matchAll()
.
For a more advanced version of this, check out lexer-next which includes methods for writing Recursive descent parsers.
Example
See TokenizerFactory.
API
Table of Contents
createTokenizer
Create a TokenizerFactory for the given RegExps.
To capture, RegExps must use a named group.
const tokenize = createTokenizer(
/(?<ident>[a-z]+)/, // named groups determine token `group`
/(?<number>[0-9]+)/
)
Parameters
TokenizerFactory
Create a TokenizerCallableIterable for given input string.
// using next()
const next = tokenize('hello 123')
console.log(next()) // => {group: 'ident', value: 'hello', index: 0}
console.log(next()) // => {group: 'number', value: '123', index: 6}
console.log(next()) // => undefined
// using for of
for (const token of tokenize('hello 123')) {
console.log(token)
// => {group: 'ident', value: 'hello', index: 0}
// => {group: 'number', value: '123', index: 6}
}
// using spread
const tokens = [...tokenize('hello 123')]
console.log(tokens)
// => [
// {group: 'ident', value: 'hello', index: 0},
// {group: 'number', value: '123', index: 6}
// ]
Type: function (input: string): TokenizerCallableIterable
Parameters
input
The string to tokenize.
TokenizerCallableIterable
Can be called to return next Token or can be used as an Iterable on for-of and spread operations.
Type: any
Contribute
All contributions are welcome!
License
MIT © 2021 stagas