simple-text2word-tokens

simple text to word tokenizer. accepts a custom regex. returns an array of tokens

Usage no npm install needed!

<script type="module">
  import simpleText2wordTokens from 'https://cdn.skypack.dev/simple-text2word-tokens';
</script>

README

Simple tokenize text to word tokens

Install package

npm install simple-text2word-tokens

Usage

import textToWordTokens from 'simple-text2word-tokens'
const text = 'text to be tokenized'

console.log(JSON.stringify(textToWordTokens(text), null, 2))
/**
 [
  {
    "value": "text",
    "index": 0,
    "offset": 4
  },
  {
    "value": "to",
    "index": 5,
    "offset": 2
  },
  {
    "value": "be",
    "index": 8,
    "offset": 2
  },
  {
    "value": "tokenized",
    "index": 11,
    "offset": 9
  }
 ]
*/

Development

npm install # install dependencies

npm start # start webpack-dev-server

npm test # run tests