README
verify-md5
A standalone program to compare a file of md5 hash signatures in /path/to/md5.json
to the md5 hashes of the contents of the listed files in that same directory
Installation
To install globally:
npm i verify-md5 -g
Prerequisites
nodejs
is installed and accesible by invokingnode
- An
md5.json
file consisting of an object with filenames as keys and base64 md5 as value for each key.
Example md5.json file:
{ "data.csv": "fi5F17hHxxqZ9E6gpUlvfA==", "profit.csv": "NbLa4M1gKG5Mi1gREVtwWg==", "trade.csv": "PTIe1aHM/ww85vzA6UbMAg==", "volume.csv": "z8pesXd3MJ7PjmQq+MhNrw==", "secretcode.js": "ho5ejYGt7uYwtEZSg8M7QQ==", "someotherfile.json": "7Vl0EF+dhruo4YhBFlaW8g==" }
Usage
If the path begins with gs://
it refers to a Google Cloud Storage[tm] bucket, e.g. gs://bucketName/path/to/md5.json
verify-md5 --key /path/to/secret/myserviceaccountkey.json gs://bucketName/path/to/md5.json
If the path is a filesystem path, the local filesystem is used:
verify-md5 /path/to/md5.json
--json
enables json output:
json
output is a 4 element array:
[true|false, goodFiles, badFiles, failErrors, dirname]
Otherwise, "human-readable" output is used.
Examples
Success example, using storage bucket.
verify-md5 --key secret.json gs://somebucket/path/to/md5.json bucket is somebucket path is path/to/md5.json OK gs://somebucket/path/to/md5.json ok: effalloc.csv ok: ohlc.csv ok: profit.csv ok: sim.json ok: trade.csv ok: volume.csv
Failure example, with local file system.
verify-md5 /tmp/test/md5.json FAIL /tmp/test/md5.json ok: effalloc.csv ok: ohlc.csv ok: sim.json ok: profit.csv ok: trade.csv fail: volume.csv md5 File Verification Error on file: /tmp/test/volume.csv : expected: z8pesXd3MJ7PjmQq+MhNrw== actual: 8VMdzVVyB0zpNxwmtJRwUw==
verify-md5 --json /tmp/test/md5.json [ false, [ "effalloc.csv", "ohlc.csv", "sim.json", "profit.csv", "trade.csv" ], [ "volume.csv" ], { "volume.csv": { "name": "MD5FileVerificationError", "message": "md5 File Verification Error on file: /tmp/test/volume.csv : expected: z8pesXd3MJ7PjmQq+MhNrw== actual: 8VMdzVVyB0zpNxwmtJRwUw==", "stack": "Error\n at new MD5FileVerificationError (/usr/local/lib/node_modules/verify-md5/node_modules/verify-fsdir-md5/index.js:17:19)\n at hasha.fromFile.then (/usr/local/lib/node_modules/verify-md5/node_modules/verify-fsdir-md5/index.js:50:19)\n at process._tickCallback (internal/process/next_tick.js:103:7)", "actual": "8VMdzVVyB0zpNxwmtJRwUw==", "expected": "z8pesXd3MJ7PjmQq+MhNrw==", "file": "/tmp/test/volume.csv" } }, "/tmp/test/" ]
Authentication / Authorization
verify-md5
on a filesystem path runs with the permissions of the current user.
verify-md5
on a gs://
path will attempt to authenticate with the Google Storage API.
Unless you run in the Google Cloud, you will need to include a --key keyfilename
to access your buckets.
If you need to generate a key for your project, look in the IAM section of the Google Cloud Platform Console and generate either:
- a service account key for role: Storage Object Viewer
- or, for granularity, a new service account with no permissions, added as a reader to an individual bucket acl.
Important: It is solely your responsibility to keep API keys secret and manage related security and financial risks.
Storage API usage
Only the bucket's md5.json
file is downloaded with a storage.bucket.file.download
call.
Then storage.bucket.file.get
calls are used to access md5 metadata for each file, not .download()
calls.
Creating the necessary md5 metadata in the bucket is normally automatic when objects are stored.
Creating the md5.json
file in the bucket to enable these comparisons is your responsibility.
Help
verify-md5 --help
prints this reminder message
Usage: verify-md5 [options]Options: -h, --help output usage information -V, --version output the version number --key [keyfile] keyfile to access Google Cloud Storage API --json output test results in json
Copyright
Copyright 2017 Paul Brewer, Economic and Financial Technology Consulting LLC drpaulbrewer@eaftc.com
License
The MIT License
Trademarks
Google Cloud Storage[tm] is a trademark of Google, Inc.
This software is not a product of Google, Inc.