s3-cached

S3 File Access Abstraction providing Memory and Disk Caching Layer

Usage no npm install needed!

<script type="module">
  import s3Cached from 'https://cdn.skypack.dev/s3-cached';
</script>

README

Cached S3 Wrapper

Build Status Test Coverage Dependabot Status Dependencies NPM Downloads Semantic-Release Gardener

S3 File Access Abstraction providing Memory and Disk Caching Layer. Useful e.g. in lambda functions if you want to reduce the amount of s3 access for serving (semi-)static files.

What it does

  • Access abstraction to access JSON and GZipped data on AWS S3
  • Two Layer caching (memory and disk)
  • Allows you to define cache constraints like TTL

Getting Started

Install

$ npm install --save s3-cached

Request S3 Files

const s3 = require('s3-cached')({
  bucket: 'YOUR_BUCKET_NAME',
  s3Options: {
    accessKeyId: 'YOUR_ACCESS_KEY_ID',
    secretAccessKey: 'YOUR_SECRET_ACCESS_KEY'
  }
});

s3.getJsonObjectCached('large.json').then((json) => {
  // do something with the json data
}).catch((err) => {
  // there has been an error
});

Available functions

  • getBinaryObjectCached: retrieve file content, caching additional modifications possible through parameter
  • getTextObjectCached: retrieve file content as string and return as promise
  • getJsonObjectCached: retrieve file content as string, parse as json and return as promise
  • getGzipObjectCached: retrieve file content, gunzip and return as promise
  • getKeysCached: retrieve all file names in bucket with given prefix

Note that you can specify the ttl and/or custom bucket on a per file basis by calling e.g. s3.getJsonObjectCached(FILE_NAME, { ttl, bucket }). For exact method signatures please check the code.

Other Function / Exports

  • resetCache(): Reset everything in cache
  • aws: The underlying aws-sdk-wrap instance

Options

bucket

Type: string
Default: undefined

Specify the Bucket name you want to retrieve data from. It either has to be defined here or on every request (overwrites).

s3Options

Type: object
Default: -

Passed into aws-wrap-sdk for AWS.S3() initialization.

logger

Type: logger
Default: null

Passed into aws-wrap-sdk.

ttl

Type: integer
Default: 600

Define how long a cached file is kept by default. This can be overwritten on a per-file basis by passing a second parameter into the function.

memoryLimit

Type: integer
Default: 100

Define how many cached entities can be hold in memory at the same time. If more entities are present, the earliest are discarded from memory cache.

diskMaxSize

Type: integer
Default: 469762048

Maximum amount of disk space in bytes used by disk cache. Earliest files are discarded from file cache if more space is used.

diskTmpDirectory

Type: string
Default: /tmp

Location to store temporary data for disk cache.