README

Please note that issues have been observed when piping using node version 15. We are on the case to identify the exact problem

RexReplace

RexReplace is a versatile tool to search and replace text in files from the command line. Its inspired by how developers often need to do quick fixes or one-liners for build scripts.

Key features:

Easy and intuitive notation makes you trust what you are doing
Replacement can be javascript code - giving you Turing complete flexibility
Pinpoint the exact files with glob notation (docs/*.md represents each markdown file in docs/)
No more brute-forcing the right combination of find, cat, sed, tr, and awk to replace a text pattern in the right files

Install

To use RexReplace from your command line

> npm install -g rexreplace

To use RexReplace from an npm build script:

> yarn add rexreplace --dev
# or
> npm install rexreplace --save-dev

RE2

If you want to lazy load RE2 you will need to explicitly add it as a dependency. It has issues and I no longer want it as a direct dependency until it's post build part can go away.

Examples

Let 'foobar' become 'xxxbar' in myfile.md

> rexreplace 'Foo' 'xxx' myfile.md

Hard for your fingers to write on your keyboard? We got you covered with the rr alias for rexreplace

> rr Foo xxx myfile.md

Catch the beginning

Let all markdown files in the docs/ dir get headlines moved one level deeper

> rexreplace '^#' '##' docs/*.md

Using glob notation to pinpoint files

Fix a spell error in all javascript and typescript files in the folders src/ and test/ recursively.

> rexreplace 'foubar' 'foobar' '{src,test}/**/*.{js,ts}'

Dynamically generated content

Let the version number from package.json get into your distribution js files (use the string VERSION in your source files).

> rexreplace 'VERSION' 'require("./package.json").version' -j dist/*.js

Require have been given the alias r and both are expanded to understand relative paths even without ./ prepended. As the file extension is not needed eighter you will get the same result writing:

> rexreplace 'VERSION' 'r("package").version' -j dist/*.js

Reference a matching group

Let 'foobar' become 'barfoo' in myfile.md

> rexreplace '(foo)(.*)' '$2$1' myfile.md

RexReplace defaults to treating € as an alias for $ so the following will do the same as the previous example

> rexreplace '(foo)(.*)' '€2€1' myfile.md

Surviving backslash-escape hell

Let foo[bar] become foo.0.[bar] in myfile.md

> rexreplace '\[' '.0.[' myfile.md

The [ as a literate char must be escaped according to regex. If you run the command as a parameter (this could be from a script in package.json) you need to escape the escape:

"scripts":{
  "fix": "rexreplace '\\[' '.0.[' myfile.md"
}

RexReplace defaults to treating § as an alias for \ so the following give same result:

> rexreplace '§[' '.0.[' myfile.md

"scripts":{
  "fix": "rexreplace '§[' '.0.[' myfile.md"
}

More relevant examples

Per file info

Add creation time, name of the file and human readable file size as the first line in each file in test-run recursively.

> rexreplace '^' 'ctime_ + name_ + size + nl' -j -M 'dist/**/*.*'

Matching pairs

Let both "foo 'is' bar" and ' foo "was" bar' ' become » foo ... bar « independent of using ' or "

> rexreplace '(['"])([^\1]+)\1' '» $2 «' myfile.md

Usage

> rexreplace pattern replacement [fileGlob|option]+

Flag	Effect
`-v`	`--version` Print rexreplace version (can be given as only argument) [boolean]
`-V`	`--verbose` More chatty output [boolean]
`-L`	`--literal` Literal string search (no regex used when searching) [boolean]
`-I`	`--void-ignore-case` Void case insensitive search pattern. [boolean]
`-G`	`--void-global` Void global search (stop looking after the first match). [boolean]
`-s`	`--dot-all` Have `.` also match newline. [boolean]
`-M`	`--void-multiline` Void multiline search pattern. Makes ^ and $ match start/end of whole content rather than each line. [boolean]
`-u`	`--unicode` Treat pattern as a sequence of unicode code points. [boolean]
`-e`	`--encoding` Encoding of files/piped data. [default: "utf8"]
`-E`	`--engine` What regex engine to use: [choices: "V8", "RE2"] [default: "V8"]
`-q`	`--quiet` Only display errors (no other info) [boolean]
`-Q`	`--quiet-total` Never display errors or info [boolean]
`-H`	`--halt` Halt on first error [boolean] [default: false]
`-d`	`--debug` Print debug info [boolean]
`-€`	`--void-euro` Void having `€` as alias for `$` in pattern and replacement parameters [boolean]
`-§`	`--void-section` Void having `§` as alias for `` in pattern and replacement parameters [boolean]
`-o`	`--output` Output the final result instead of saving to file. Will also output content even if no replacement has taken place. [boolean]
`-A`	`--void-async` Handle files in a synchronous flow. Good to limit memory usage when handling large files. [boolean]
`-B`	`--void-backup` Avoid temporary backing up file. Works async (independent of -A flag) and will speed up things but at one point data lives only in memory, and you will lose the content if the process is abrupted. [boolean]
`-b`	`--keep-backup` Keep a backup file of the original content. [boolean]
`-m`	`--output-match` Output each match on a new line. Will not replace any content but you still need to provide a dummy value (like `_`) as replacement parameter. If search pattern does not contain matching groups the full match will be outputted. If search pattern does contain matching groups only matching groups will be outputted (same line with no delimiter). [boolean]
`-T`	`--trim-pipe` Trim piped data before processing. If piped data only consists of chars that can be trimmed (new line, space, tabs...) it will become an empty string. [boolean]
`-R`	`--replacement-pipe` Replacement will be piped in. You still need to provide a dummy value (like `_`) as replacement parameter. [boolean]
`-j`	`--replacement-js` Treat replacement as javascript source code. The statement from the last expression will become the replacement string. Purposefully implemented the most insecure way possible to remove any incentive to consider running code from an untrusted part. The full match will be available as a javascript variable named $0 while each captured group will be available as $1, $2, $3, ... and so on. At some point, the $ char will give you a headache when used from the command line, so use €0, €1, €2, €3... instead. If the javascript source code references to the full match or a captured group the code will run once per match. Otherwise, it will run once per file. The code has access to the following variables: `r` as an alias for `require` with both expanded to understand a relative path even if it is not starting with `./`, `fs` from node, `path` from node, `globs` from npm, `pipe`: the data piped into the command (null if no piped data), `find`: pattern searched for (the needle), `text`: full text being searched i.e. file content or piped data (the haystack), `bytes`: total size of the haystack in bytes, `size`: human-friendly representation of the total size of the haystack, `time`: String representing the local time when the command was invoked, `time_obj`: date object representing `time`, `now`: alias for `time`, `cwd`: current process working dir, `nl`: a new-line char, `_`: a single space char (for easy string concatenation). The following values defaults to `❌` if haystack does not originate from a file: `file`: contains the full path of the active file being searched (including full filename), `file_rel`: contains `file` relative to current process working dir, `dirpath`: contains the full path without filename of the active file being searched, `dirpath_rel`: contains `dirpath` relative to current process working dir, `filename`: is the full filename of the active file being searched without path, `name`: filename of the active file being searched with no extension, `ext`: extension of the filename including leading dot, `mtime`: ISO inspired representation of the last local modification time of the current file, `ctime`: ISO representation of the local creation time of the current file. `mtime_obj`: date object representing `mtime`, `ctime_obj`: date object representing `ctime`. All variables, except from module, date objects, `nl` and `_`, has a corresponding variable name followed by `_` where the content has an extra space at the end (for easy concatenation). [boolean]
`-h`	`--help` Display help. [boolean]

Good to know

Features

Patterns are described as javascript notation regex
Pattern defaults to global multiline case-insensitive search
Supports regex lookaheads in the pattern
Supports backreference to matching groups in the replacement
Data to be treated can be piped in
See the release note for a log of changes. Descriptions are given in latest patch version.

Quirks

Options can only be set after the replacement parameter. "But I like to put my options as the first thing, so I know what I am doing" I agree, but we must sometimes sacrifice habits for consistency.
Per default € is treated as an alias for $ in the CLI input. The main reason is for you not to worry about how command line tools often have a special relationship with the $ char. You can escape your way out of this old love story, but it often pops up in unexpected ways. Use the -€ flag if you need to search or replace the actual euro char.
Per default § is treated as an alias for \ in the CLI input. The main reason is for you not to worry about escaping the escape of an escape. Use the -§ flag if you need to search or replace the actual section char.

Limitations

RexReplace reads each file fully into memory, so working on your 8Gb log files will probably not be ideal.
For versions of Node prior to 6, please use version 2.2.x. For versions of Node prior to 0.12, please use the legacy version of RexReplace called rreplace

Priorities

Flexibility regarding text pattern matching
Easy to filter what files to be treated
Helpful interface
Tests (if you know how to do a test cover report on javascript code ran via the command line, please let me know)

Not a priority

Speed. Well... obviously, speed is important, but to what extent does a 29 millisecond command really satisfy the user compared to a 294 millisecond command? See test->speed for more info.

> time cat README.md | sed 's/a/x/g'  > /dev/null
cat myfile  0,00s user 0,00s system 45% cpu 0,011 total
sed 's/a/x/g' > /dev/null  0,00s user 0,00s system 43% cpu 0,029 total

> time rr a x README.md -o > /dev/null 
rr x y myfile -o > /dev/null  0,21s user 0,04s system 86% cpu 0,294 total

Test

Regression

All CLI end to end tests are defined in test/cli/run.sh and all unit test are described in test/*.js. After git clone'ing the repo and npm install'ing you can invoke them with:

> npm test

Speed

tl;dr: Files over 5 Mb are faster with rr than with sed - but - it does not matter as any file under 25 Mb has less than 0.7 seconds in difference.

The speed test is initiated by npm run test-speed. The test takes files in different sizes and compares the processing time for RexReplace (rr) and the Unix tool sed. The test uses the sources of a website displaying the book 1984 by George Orwell. The task for the tests is to remove all HTML tags by search-and-replace, so only the final text is left. The source is 888Kb, so all files up to 500Kb are generated directly from the source, while larger files are created by combining the first 500Kb several times. Each test runs 10 times to even out any temporary workload fluctuations. Results from the latest speed test can always be seen in the speed test log.

The graph visualises speed as relative to fastest overall run (sed on a 1kb file). This chart also has an interactive version in log scale, so the details in the low end can be studied better. Interestingly files of 1Kb, 5Kb takes longer for rr than 10Kb files.

Now, what is relevant to notice is how sed only takes 4.9 seconds longer for the 100Mb file - even if the difference looks drastic on the graph.

Speed relative to fastest tool for each file size
---------------------------------------------------
Bytes    sed    rr    Time it took longer (seconds)
1          1    39    0,5    <= sed is 39x faster  
5          1    32    0,4     
10         1    27    0,4
100        1    19    0,3
500        1     7    0,3
1000       1     4    0,3
5000       1     1    0,0    <= same speed for 5Mb file
10000      2     1    0,3
25000      2     1    1,1
50000      3     1    3,1
100000     3     1    4,9    <= rr 3.1.0 is 3x faster

So even though the speed evolves very differently, there is only little practical use of the focus on speed for most use cases. Replacing in 10.000 small files? Use RexReplace and get yourself a cup of coffee - or spend half an hour getting sed to work as you want it to and enjoy the thrilling few seconds it takes to do its magic.

Please note that speeds might look very different when files get as large as the memory available.

Rumours

Inspiration

.oO(What should "sed" have looked like by now?)

Future ideas

Add support to require hjson, jsonh, yaml, ini files directly
Test-run with info outputted about what will happen (sets -t and does not change anything)
Let search and replace be within the names of the files (ask for overwriting. -Y = no questions)
Let search and replace be within the path of the files (ask for overwriting. -Y = no questions)
Let pattern and globs be piped
Let Pattern, replacement, and globs come from a file
Let pattern and glob be javascript code returning a string as the result
Auto string search / replace if no regex magic is used (and verify that speed is better)
Error != warning
Flag for String-literal (no regex, no special chars, no escape chars) to avoid backslashes or remembering which characters needs to be escaped
Check if https://github.com/eugeneware/replacestream is good to rely on
Check if regex engine from spider monkey can be wrapped in something that does not need node
Check if sd can be wrapped in a WASM module
Set engine to RE2 https://www.npmjs.com/package/re2
Set engine to run on streams with https://www.npmjs.com/package/replacestream
Implement in go so that all platforms can be supported with no need for node (might be based on)
Let https://github.com/dthree/vorpal deal with the interface? Or maybe https://www.npmjs.com/package/pretty-cli

Related projects

Many projects are seeking to solve the same problem as RexReplace. I my oppinion they all lack the flexibility of the RexReplace CLI interface and diverse the replacement can be, but have other strong features. If our way does not suit you, we suggest you have a look at:

sd - Blazingly fast! 5.9x faster than RexReplace. Almost perfect. Using "Good ol' unix philosophy to the rescue." for pinpointing files so not always easy. Get it via cargo for Rust.
replace-in-file - same speed as RexReplace. Very good support to be used as a module for a node application. A bit cumbersome CLI notation for pinpointing files. Get via npm.
replace-x - 3.8x slower than RexReplace. A bit cumbersome CLI notation for pinpointing files. Get it via npm.
replace - 2.6x slower than RexReplace. Very chatty default output. A bit more cumbersome CLI notation for pinpointing files. Get it via npm.

RexReplace mascot Benny on the RexReplace logBo

Please note that RexReplace is an OPEN open source software project. This means that individuals making significant and valuable contributions are given commit access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.

Icon inspired by Freepik from www.flaticon.com

@znemz/rexreplace

Usage no npm install needed!