@mapbox/s3scan

Streaming operations on S3 objects

Usage no npm install needed!

<script type="module">
  import mapboxS3scan from 'https://cdn.skypack.dev/@mapbox/s3scan';
</script>

README

Build Status

s3scan

Provides native Node.js streams for S3 operations such as

  • listing all the keys in a bucket/prefix
  • retrieving all the objects under a given bucket/prefix
  • deleting all the children of a given bucket/prefix

Also provides CLI tools:

s3keys: prints a line-delimited list of keys to stdout

$ s3keys s3://my-bucket/some-prefix

s3purge: deletes all objects beneath the provided prefix

$ s3purge s3://my-bucket/all-finished-with-these

s3scan: GET and print a prefix-worth of objects to stdout

$ s3scan s3://my-bucket/some-prefix
$ s3scan s3://my-bucket/some-prefix --gunzip
$ s3scan s3://my-bucket/some-prefix --gunzip --concurrency=1

Running tests

You can run tests against your own bucket/prefix by specifying them as environment variables:

TestBucket=my-bucket TestPrefix=my-prefix npm test

Otherwise these values default to buckets owned by Mapbox which require appropriate authentication.