lexer-next

lexer for recursive descent parsers

Usage no npm install needed!

<script type="module">
  import lexerNext from 'https://cdn.skypack.dev/lexer-next';
</script>

README

lexer-next

lexer for recursive descent parsers

🔧 Install · 🧩 Example · 📜 API docs · 🔥 Releases · 💪🏼 Contribute · 🖐️ Help

Install

$ npm i lexer-next

What's this?

It is a lexer generator for writing Recursive descent parsers and is compatible with RegExp's named group matches as returned, for example, by String.prototype.matchAll().

Example

import { createLexer } from 'lexer-next'

const tokenize = (input: string) =>
  input.matchAll(/(?<ident>[a-z]+)|(?<number>[0-9]+)/g)

const lexer = createLexer(tokenize)

const { advance, peek, accept, expect } = lexer('hello 1337 world')

console.log(advance())
// => { group: 'ident', value: 'hello', index: 0 }

console.log(accept('ident'))
// => undefined

console.log(accept('number'))
// => { group: 'number', value: '1337', index: 6 }

console.log(peek())
// => { group: 'ident', value: 'world', index: 11 }

console.log(expect('number'))
// => SyntaxError: Unexpected token: world
//       expected: number
//   but received: ident "world"
//    at position: 11

API

Table of Contents

ErrorHandler

src/index.ts:12-12

Error handler.

Type: function (error: Error): void

Parameters

  • error The error object

FilterFunction

src/index.ts:20-20

Filter function.

Type: function (token: Token): boolean

Parameters

  • token The token to match.

Returns any true if it passes or false if it's rejected

Lexer

src/index.ts:47-89

Lexer interface.

advance

src/index.ts:51-51

Returns token under current position and advances.

Returns Token

peek

src/index.ts:60-60

Returns token under current position. When passed a group and maybe a value it will only return the token if they match, otherwise will return null.

Parameters
  • group The group name to examine
  • value The value to match

Returns Token

accept

src/index.ts:70-70

Advances position only when current token.group matches group, and optionally when token.value matches value, otherwise does nothing.

Parameters
  • group string The group name to examine
  • value string? The value to match

Returns (Token | null)

expect

src/index.ts:78-78

Same as accept() except it throws when token.group does not match group, or (optionally) when token.value does not match value,

Parameters
  • group string The group name to examine
  • value string? The value to match

Returns Token

onerror

src/index.ts:83-83

Sets a function to handle errors. The error handler accepts an Error object.

Parameters

Returns void

filter

src/index.ts:88-88

Sets a filter function. The filter function receives a Token as first parameter.

Parameters

Returns void

LexerFactory

src/index.ts:94-94

Generate a Lexer for given input string.

Type: function (input: string): Lexer

createLexer

src/index.ts:105-177

Create a LexerFactory with the given Tokenizer.

It can be anything that, when called with an input string, returns an interface that conforms to the IterableIterator<RegExpMatchArray> type as is returned, for example, by String.prototype.matchAll(regexp)

Parameters

  • tokenize Tokenizer A tokenizer Iterable factory.

Contribute

Fork or edit and submit a PR.

All contributions are welcome!

License

MIT © 2021 stagas