text-tokenizer

A simple perfomant tokenizer

Usage no npm install needed!

<script type="module">
  import textTokenizer from 'https://cdn.skypack.dev/text-tokenizer';
</script>

README

Text Tokenizer

Overview

A simple and fast TypeScript text tokenizer using tokenizr

Getting Started

npm install -S text-tokenizer