dynamodb-onetable

One Table to Rule Them All

The Easiest Way to Create DynamoDB Single Table Designs.

OneTable is the most evolved API for DynamoDB. It provides a high-level, elegant dry syntax while still enabling full access to DynamoDB API.

OneTable works with AWS V2 and V3 SDKs for JavaScript and TypeScript. For TypeScript, OneTable will create fully typed entities from your data schemas automatically.

Join the active community using OneTable on our GitHub Discussion Hub and learn about the growing set of extentions for migrations, CLI and graphical monitoring tools for your single-table designs.

OneTable Features

• Schema supported one-table access to DynamoDB APIs.
• Efficient storage and access of multiple entities in a single DynamoDB table.
• High level API with type marshaling, validations, and extended query capability for get/delete/update operations.
• Bidirectional conversion of DynamoDB types to Javascript types.
• Generation of Conditional, Filter, Key and Update expressions.
• Schema item definitions for attribute types, default values, enums, unique attributes and validations.
• Option to invoke DynamoDB or simply generate API parameters.
• Powerful field level validations with "required" and "unique" attributes.
• Easy parameterization of filter and conditional queries.
• Detailed metrics by Table, Tenant, Source, Index, Model and Operation.
• Multi-page response aggregation.
• Compound and templated key management.
• Attribute mapping and packing.
• Support for sparse GSIs that project keys and overloaded attributes.
• Encrypted fields.
• CreateTable, DeleteTable table and index admin operations.
• Support for Batch, Transactions, GSI, LSI indexes.
• Intercept hooks to modify DynamoDB requests and responses.
• Controllable logging to see exact parameter, data and responses.
• Simple and easy to read source.
• Integrated statistics.
• Safety options to prevent "rm -fr *".
• No module dependencies.
• Support for the AWS SDK v3.
• TypeScript type inference from schema for full type validation on APIs, parameters, returns, and entities and attributes.
• Migrations support via OneTable Migrate and OneTable CLI.
• Graphical monitoring of single-table performance via SenseDeep.

Installation

npm i dynamodb-onetable

Quick Tour

Import the OneTable library. If you are not using ES modules or TypeScript, use require to import the libraries.

import {Table} from 'dynamodb-onetable'

If you are using the AWS SDK V2, import the AWS DynamoDB class and create a DocumentClient instance.

import DynamoDB from 'aws-sdk/clients/dynamodb'
const client = new DynamoDB.DocumentClient(params)

This version includes prototype support for the AWS SDK V3.

If you are using the AWS SDK V3, import the AWS V3 DynamoDBClient class and the OneTable Dynamo helper. Then create a DynamoDBClient instance and Dynamo wrapper instance. Note: you will need Node v14 or later for this to work.

Note: you can use the Table.setClient API to defer setting the client or replace the client at any time.

import Dynamo from 'dynamodb-onetable/Dynamo'
import {Model, Table} from 'dynamodb-onetable'
import {DynamoDBClient} from '@aws-sdk/client-dynamodb'
const client = new Dynamo({client: new DynamoDBClient(params)})

Initialize your OneTable Table instance and define your models via a schema.

const table = new Table({
    client: client,
    name: 'MyTable',
    schema: MySchema,
})

This will initialize your OneTable Table instance and define your models via a schema.

Schemas

Schemas define how items will be stored in your database and look like this:

const MySchema = {
    format: 'onetable:1.1.0',
    version: '0.0.1',
    indexes: {
        primary: { hash: 'pk', sort: 'sk' },
        gs1:     { hash: 'gs1pk', sort: 'gs1sk', follow: true },
        ls1:     { sort: 'id', type: 'local' },
    },
    models: {
        Account: {
            pk:          { type: String, value: 'account:${name}' },
            sk:          { type: String, value: 'account:' },
            id:          { type: String, generate: 'ulid', validate: /^[0-9A-F]{32}$/i },
            name:        { type: String, required: true },
            status:      { type: String, default: 'active' },
            zip:         { type: String },
        },
        User: {
            pk:          { type: String, value: 'account:${accountName}' },
            sk:          { type: String, value: 'user:${email}', validate: EmailRegExp },
            id:          { type: String, required: true },
            accountName: { type: String, required: true },
            email:       { type: String, required: true },
            firstName:   { type: String, required: true },
            lastName:    { type: String, required: true },
            username:    { type: String, required: true },
            role:        { type: String, enum: ['user', 'admin'], required: true, default: 'user' },
            balance:     { type: Number, default: 0 },

            gs1pk:       { type: String, value: 'user-email:${email}' },
            gs1sk:       { type: String, value: 'user:' },
        }
    },
    params: {
        'isoDates': true,
        'timestamps': true,
    },
}

To create an item:

let account = await Account.create({
    id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name: 'Acme Airplanes',
})

This will write the following to DynamoDB:

{
    pk:         'account:8e7bbe6a-4afc-4117-9218-67081afc935b',
    sk:         'account:98034',
    id:         '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name:       'Acme Airplanes',
    status:     'active',
    zip:        '98034',
    created:    1610347305510,
    updated:    1610347305510,
}

Get an item:

let account = await Account.get({
    id: '8e7bbe6a-4afc-4117-9218-67081afc935b',
})

which will return:

{
    id:       '8e7bbe6a-4afc-4117-9218-67081afc935b',
    name:     'Acme Airplanes',
    status:   'active',
    zip:      '98034',
}

To use a secondary index:

let user = await User.get({email: 'user@example.com'}, {index: 'gs1'})

To find a set of items:

let users = await User.find({accountId: account.id})

let adminUsers = await User.find({accountId: account.id, role: 'admin'})

let users = await User.find({accountId: account.id}, {
    where: '${balance} > {100.00}'
})

//  Get a count of matching users without returning the actual items
let users = await User.find({accountId: account.id, role: 'admin'}, {count: true})
let count = users.count

To update an item:

await User.update({id: userId, balance: 50})
await User.update({id: userId}, {add: {balance: 10.00}})
await User.update({id: userId}, {set: {status: '{active}'}})

To do a transactional update:

let transaction = {}
await Account.update({id: account.id, status: 'active'}, {transaction})
await User.update({id: user.id, role: 'user'}, {transaction})
await table.transact('write', transaction)

TypeScript

OneTable provides TypeScript type declaration files so that OneTable APIs, requests and responses can be fully type checked.

OneTable also creates type declarations for your table entities and attributes. TypeScript will catch any invalid entity or entity attribute references.

Using the magic of TypeScript dynamic typing, OneTable automatically converts your OneTable schema into fully typed generic Model APIs.

For example:

import {Entity, Model, Table} from 'dynamodb-onetable'

const MySchema = {
    ...
    models: {
        Account: {
            pk:    { type: String, value: 'account:${name}' },
            name:  { type: String },
        }
    } as const     // Required for TypeScript
}

//  Fully typed Account object based on the schema (must include "as const" after the models above)
type AccountType = Entity<typeof MySchema.models.Account>

let account: AccountType = {
    name: 'Coyote',        //  OK
    unknown: 42,           //  Error
}

//  Get an Account access model
let Account = table.getModel<AccountType>('Account')

let account = await Account.create({
    name: 'Acme',               //  OK
    unknown: 42,                //  Error
})

account.name = 'Coyote'         //  OK
account.unknown = 42            //  Error

Why OneTable?

DynamoDB is a great NoSQL database that comes with a learning curve. Folks migrating from SQL often have a hard time adjusting to the NoSQL paradigm and especially to DynamoDB which offers exceptional scalability but with a fairly low-level API.

The standard DynamoDB API requires a lot of boiler-plate syntax and expressions. This is tedious to use and can unfortunately can be error prone at times. I doubt that creating complex attribute type, key, filter, condition and update expressions are anyone's idea of a good time.

Net/Net: it is not easy to write terse, clear, robust Dynamo code for one-table patterns.

Our goal with OneTable for DynamoDB was to keep all the good parts of DynamoDB and to remove the tedium and provide a more natural, "JavaScripty / TypeScripty" way to interact with DynamoDB without obscuring any of the power of DynamoDB itself.

Working Samples

To get you going quickly, try out the working samples in the OneTable repository at:

• OneTable Overview Sample — A quick tour through OneTable.
• OneTable CRUD Sample — Basic CRUD.
• OneTable TypeScript Sample
• OneTable Migrate Sample — how to use OneTable Migrate.
• OneTable Packed Attributes Sample — How to use packed attributes.
• OneTable SenseDeep Sample — How to access SenseDeep log data.
• All OneTable Samples

History and Credits

After watching the famous Rick Houlihan DynamoDB ReInvent Video, we changed how we used DynamoDB for our SenseDeep serverless developer studio to use one-table design patterns. However, we found the going tough and thus this library was created to make our one-table patterns less tedious, more natural and a joy with DynamoDB.

A big thank you to Alex DeBrie and his excellent DynamoDB Book. Highly recommended.

Database Migrations

To manage your database migrations, consider the OneTable CLI which provides command line migration control and the OneTable Migrate library for inclusion in your services to manage database migrations.

Dynamo Class

The Dynamo class is used ease the configuration of the AWS SDK v3. The class is only used with AWS SDK V3 to wrap the DynamoDBClient instance and provide helper methods for OneTable. It does not expose any other methods.

Dynamo Constructor

The Dynamo constructor takes a parameter of type object with the following properties:

Property	Type	Description
client	DynamoDB	An AWS SDK v3 DynamoDBClient instance.
marshall	object	Marshall options for converting to DynamoDB attribute types. See: util-dynamodb for details.
unmarshall	object	Unmarshall options for converting from DynamoDB attribute types. See: util-dynamodb for details.

Table Class

The Table class is the top-most OneTable class and it represents a single DynamoDB table. The table class configures access to a DynamoDB table, defines the model (entity) schema, indexes, crypto and defaults. You can create a single Table instance or if you are working with multiple tables, you can create one instance per table.

The Table class provides APIs for transactions and batch API operations. While most access to the database is via the Model methods, the Table class also provides a convenience API to wrap the Model methods so you can specify the required model by a string name. The is helpful for factory design patterns.

Table Examples

import {Table} from 'dynamodb-onetable'

const table = new Table({
    client: DocumentClientInstance,
    name: 'MyTable',
    schema: Schema,
})

//  Fetch an item collection (will return parsed entities in an item collection)
let items = await table.fetch(['User', 'Product'], {pk: 'account:AcmeCorp'})
let users = items.User
let products = items.Product

//  Alternatively, group after a standard query
let items = await table.queryItems({pk: 'account:AcmeCorp'}, {parse: true, hidden: true})
items = table.groupByType(items)
let users = items.User
let products = items.Product

//  Fetch an account by the ID which is used to create the primary key value
let account = await table.get('Account', {id})

//  Update Account and User in a transaction
let transaction = {}
await table.update('Account', {id: account.id, status: 'active'}, {transaction})
await table.update('User', {id: user.id, role: 'user'}, {transaction})
await table.transact('write', transaction)

//  Fetch an Account using the Account model
let account = await table.find('Account', {id})

//  Get the number of accounts without reading the items
let accounts = await table.scan('Account')
let count = accounts.count

Table Constructor

The Table constructor takes a parameter of type object with the following properties:

Property	Type	Description
client	DocumentClient	An AWS DocumentClient instance.
crypto	object	Optional properties defining a crypto configuration to encrypt properties.
generate	function	Define a custom ID generator function that is used to create model IDs if required.
logger	boolean|object	Set to true to log to the console or set to a logging function(type, message, properties). Type is info
metrics	object	Configure metrics. Default null.
name	string	The name of your DynamoDB table.
schema	string	Definition of your DynamoDB indexes and models.
senselogs	object	Set to a SenseLogs logger instance instead logger. Default null.
transform	function	Callback function to be invoked to format and parse the data before reading and writing.
validate	function	Function to validate properties before issuing an API.
value	function	Function to evaluate value templates. Default null.

In previous versions the following properties could be specified via the table constructor options. They should now be supplied via the schema.params. The deprecated options and their defaults are:

| createdField | string | Name of the "created" timestamp attribute. Defaults to "created". | | hidden | boolean | Hide templated (value) attributes in Javascript properties. Default true. | | isoDates | boolean | Set to true to store dates as Javascript ISO strings vs epoch numerics. Default false. | | nulls | boolean | Store nulls in database attributes vs remove attributes set to null. Default false. | | timestamps | boolean | Make "created" and "updated" timestamps in items. Default false. | | typeField | string | Name of the "type" attribute. Default "_type". | | updatedField | string | Name of the "updated" timestamp attribute. Default "updated". |

The client property must be an initialized AWS DocumentClient. The DocumentClient API is currently supported by the AWS v2 API. The recently released AWS v3 API does not yet support the DocumentClient API (stay tuned - See Issue).

By default, OneTable will not write null values to the database rather, it will remove the corresponding attribute from the item. If you set the nulls property to true, null values will be written via create or update. You can also define nulls on a model attribute basis via the schema.

The metrics property may be set to a map that configures detailed CloudWatch EMF metrics. See Metrics below.

Transforming Data

The optional Table transform function will be invoked on read and write requests to transform data before reading or writing to the table. The transform function can be used for custom storage formats or to assist with data migrations. The transform function can modify the item as it sees fit and return the modified item. The invocation signature is:

item = transform(model, operation, item, properties, params, raw)

Where operation is set to 'read' or 'write'. The params and properties are the original params and properties provided to the API call. When writing, the item will contain the already transformed properties by the internal transformers. You can overwrite the value in item using your own custom transformation logic using property values from properties.

When reading, the item will contain the already transformed read data and the raw parameter will contain the raw data as read from the table before conversion into Javascript properties in item via the internal transformers.

You can also use a params.transform with many Model APIs. See Model API Params for details.

Table Validations

The optional Table validate function will be invoked on requests to enable property validation before writing to the table. The invocation signature is:

details = validate(model, properties, params)

The validation function must return a map of validation messages for properties that fail validation checks. The map is indexed by the property field name.

Value Template Function

Value templates are defined in the schema for model fields. These are typically literal strings with property variable references. In some use cases, more complex logic for a value template requires a function to calculate the property value at runtime. The Table params.value function provides a centralized place to evaluate value templates. It will be invoked for fields that define their value template to be true.

The value template function is called with the signature:

str = value(model, fieldName, properties, params)

The value template should return a string to be used for the given fieldName. The properties and params are corresponding arguments given to the API.

Crypto

The crypto property defines the configuration used to encrypt and decrypt attributes that specify encrypt: true in their schema. This is useful as an additional layer of security for passwords, keys and other especially sensitive information. The crypto property should be set to a hash that contains the cipher to use and an encryption secret/password.

{
    "cipher": "aes-256-gcm",
    "password": "16719023-772f-133d-1111-aaaa7722188f"
}

Logger

OneTable can log complete request parameters and responses to assist you in debugging and understanding how your API requests are being translated to DynamoDB.

You can set logger parameter to true for simple logging to the console. Alternatively, the logger may be set to logging callback that will be invoked as required to log data. The logger function has the signature:

const table = new Table({
    ...
    logger: (level, message, context) => {
        if (level == 'trace' || level == 'data') return
        console.log(`${new Date().toLocaleString()}: ${level}: ${message}`)
        console.log(JSON.stringify(context, null, 4) + '\n')
    }
})

Where level is set to info, error, warn, exception, trace or data. The trace level is for verbose debugging messages. The data level logs user data retrieved find and get API calls.

The message is a simple String containing a descriptive message. The context is a hash of contextual properties regarding the request, response or error.

If you use {log: true} in the various OneTable Model API options, the more verbose trace and data levels will be changed to info for that call before passing to the logging callback. In this way you can emit trace and data output on a per API basis.

SenseLogs

OneTable also integrates with SenseLogs which is a simple, dynamic logger designed for serverless.

import SenseLogs from 'senselogs'
const senselogs = new SenseLogs()
const table = new Table({senselogs})

This will log request details in JSON. Use SenseLogs({destination: 'console'}) for plain text logging to the console.

Metrics

OneTable can emit detailed CloudWatch custom metrics to track DynamoDB performance and usage on a per app/function, index, entity model and operation basis.

The metrics are emitted using the CloudWatch EMF format with dimensions for: Table, Source, Index, Model and Operation.

The following metrics are emitted for each dimension combination:

• read — Read capacity units consumed
• write — Write capacity units consumed
• latency — Aggregated request latency in milliseconds
• count — Count of items returned
• scanned — Number of items scanned
• requests — Number of API requests issued

SenseDeep and other tools can present and analyze these metrics to gain insights and graph into how your single-table designs are performing.

The properties of Table constructor params.metrics property are:

Property	Type	Description
chan	string	Log channel to use to emit metrics. Defaults to 'metrics'.
dimensions	array	Ordered array of dimensions to emit. Defaults to [Table, Tenant, Source, Index, Model, Operation].
enable	boolean	Set to true to enable metrics. Defaults to true.
env	boolean	Set to true to enable dynamic control via the LOG_FILTER environment variable. Defaults to true.
max	number	Number of DynamoDB API calls for which to buffer metrics before flushing. Defaults to 100.
namespace	string	CloudWatch metrics namespace for the metrics. Defaults to SingleTable/metrics.
period	number	Number of seconds to buffer metrics before flushing. Defaults to 30 seconds.
properties	map|function	Set to a map of additional properties to be included in EMF log record. These are not metrics. Set to a function that will be invoked as properties(operation, params, result) and should return a map of properties. Defaults to null.
queries	boolean	Set to true to enable per-query profile metrics. Defaults to true.
source	string	Name of application or function name that is calling DynamoDB. Default to the Lambda function name.
tenant	string	Set to an identifying string for the customer or tenant. Defaults to null.

Metrics can be dynamically controlled by the LOG_FILTER environment variable. If this environment variable contains the string dbmetrics and the env params is set to true, then Metrics will be enabled. If the env parameter is unset, LOG_FILTER will be ignored.

const table = new Table({
    metrics: {source: 'acme:launcher', env: true}
})

You can also generate metrics for specially profiled queries and scans via the params.profile tag. The profile param takes a unique string tag and metrics will be created for the dimensions [Profile, profile-tag-name]. These metrics exist outside the standard dimensions specified via the Metrics dimensions parameter.

await User.find({}, {profile: 'find-all-users'})

Read more about how to use and configure metrics at Understanding Your DynamoDB Performance.

The metrics can be viewed in CloudWatch or best via the free SenseDeep Developer plan which has detailed graphs for your single-table monitoring for DynamoDB.

.

Metrics Under the Hood

The metric are emitted using CloudWatch EMF via the metrics method. This permits zero-latency creation of metrics without impacting the performance of your Lambdas.

Metrics will only be emitted for dimension combinations that are active. If you have many application entities and indexes, you may end up with a large number of metrics. If your site uses all these dimensions actively, your CloudWatch Metric costs may be high. You will be charged by AWS CloudWatch for the total number of metrics that are active each hour at the rate of $0.30 cents per hour.

If your CloudWatch costs are too high, you can minimize your charges by reducing the number of dimensions via the dimensions property. You could consider disabling the source or operation dimensions. Alternatively, you should consider SenseLogs which integrates with OneTable and can dynamically control your metrics to enable and disable metrics dynamically.

DynamoDB Metrics are buffered and aggregated to minimize the load on your system. If a Lambda function is reclaimed by AWS Lambda, there may be a few metric requests that are not emitted before the function is reclaimed. This should be a very small percentage and should not significantly impact the quality of the metrics. You can control this buffering via the max and period parameters.

Schema

The schema property describes the indexes and models (entities) on your DynamoDB table. Models may be defined via the schema or alternatively may be constructed using the Model constructor and the Table.addModel method.

The valid properties of the schema object are:

Property	Type	Description
format	string	Reserved. Must be set to 'onetable:1.1.0'
indexes	object	Hash of indexes used by the table.
models	object	Hash of model entities describing the model keys, indexes and attributes.
params	object	Hash of properties controlling how data is stored in the table.
version	string	A SemVer compatible version string.

The format property must be set to onetable:1.1.0.

The schema must contain a version property set to a SemVer compatible version string. This may be used by consumers of the schema to understand compatibility constraints for the schema and stored data.

Params

The schema.params is a hash map of properties that control how data is stored. It may contain the following properties:

| createdField | string | Name of the "created" timestamp attribute. Defaults to "created". | | hidden | boolean | Hide templated (value) attributes in Javascript properties. Default true. | | isoDates | boolean | Set to true to store dates as Javascript ISO strings vs epoch numerics. Default false. | | nulls | boolean | Store nulls in database attributes vs remove attributes set to null. Default false. | | timestamps | boolean | Make "created" and "updated" timestamps in items. Default false. | | typeField | string | Name of the "type" attribute. Default "_type". | | updatedField | string | Name of the "updated" timestamp attribute. Default "updated". |

Indexes

The schema.indexes property can contain one or more indexes and must contain the primary key. Additional indexes will be treated as Global Secondary Indexes (GSIs) unless they are defined with a type: "local" property in which case they will be designated as Local Secondary Indexes (LSIs). An LSI index should not specify a hash attribute. If one is specified, it must equal that of the primary index.

{
    primary: {
        hash: 'pk',         //  Schema property name of the hash key
        sort: 'sk',         //  Schema property name of the sort key
    },
    //  Zero or more global secondary or local secondary indexes
    gs1: {
        hash: 'gs1pk',      //  Omit the hash for an LSI or set to the primary index hash name
        sort: 'gs1sk',
        project: 'all',
        follow: true,
    },
    ls1: {
        type: 'local'
        sort: 'id',
    }
    ...
}

Note the hash and sort names are schema property names which may differ from table attribute names if you are using mapping.

The project property can be set to 'all' to project all attributes to the secondary index, set to 'keys' to project only keys and may be set to an array of attributes (not properties) to specify an explicit list of attributes to project. The project property is used by the Table.createTable and updateTable APIs only.

The follow property is used to support GSI indexes that project KEYS_ONLY or only a subset of an items properties. When follow is true, any fetch of an item via the GSI will be transparently followed by a fetch of the full item using the primary index and the GSI projected keys. This incurs an additional request for each item, but for large data sets, it is useful to minimize the size of a GSI and yet retain access to full items.

Models

The schema.models property contains one or more models with attribute field descriptions. The models collections define the attribute names, types, mappings, validations and other properties. For example:

{
    album: {
        pk:     { type: String, value: '${_type}:${name}' },
        sk:     { type: String, value: '${_type}:' },
        name:   { type: String, required: true },
        songs:  { type: Number },
    },
    artist: {
        pk:     { type: String, value: '${_type}:${name}' },
        sk:     { type: String, value: '${_type}:' },
        name:   { type: String, required: true },
        address: {
            type: Object, schema: {
                street: { type: String },
                city: { type: String },
                zip: { type: String },
            },
        },
    }
}

The name of the entity model is model map name (album and artist above).

The valid types are: Array, Binary, Boolean, Buffer, Date, Number, Object, Set and String.

OneTable will ensure that values are of the correct type before writing to the database. Where possible, values will be cast to their correct types. For example: 'false' will be cast to false for Boolean types and 1000 will be cast to '1000' for String types.

These JavaScript types map onto the equivalent DynamoDB types. For Binary types, you can supply data values with the types: ArrayBuffer and Buffer.

For Sets, you should set the schema type to Set and supply values as instances of the JavaScript Set type. DynamoDB supports sets with elements that are strings, numbers or binary data.

OneTable will automatically add a _type attribute to each model that is set to the name of the model. However, you can explicitly define your type attribute in your model schema if you wish.

The type field can be used in PK/SK value templates by using ${_type}. You can change the name of the type field from _type by setting the params.typeField in the Table constructor.

Schema Attribute Properties

The following attribute properties are supported:

Property	Type	Description
crypt	boolean	Set to true to encrypt the data before writing.
default	string	Default value to use when creating model items or when reading items without a value.
enum	array	List of valid string values for the attribute.
filter	boolean	Enable a field to be used in a filter expression. Default true.
generate	string|boolean	Set to 'ulid' or 'uuid' to automatically create a new ID value for the attribute when creating new items. Set to true to use a custom ID generator defined via the Table params.generate option. Default to null.
hidden	boolean	Set to true to omit the attribute in the returned Javascript results. Attributes with a "value" template defined will by hidden by default. Default to the schema params value.
isoDates	boolean	Set to true to store dates as Javascript ISO strings vs epoch numerics. If unset, the field will use the table default value for isoDates. Default to the schema params value.
map	string	Map the field value to a different attribute name when storing in the database. Can be a simple attribute name or a compound "obj.name" where multiple fields can be stored in a single attribute containing an object with all the fields.
nulls	boolean	Set to true to store null values or false to remove attributes set to null. Default false.
required	boolean	Set to true if the attribute is required. Default to the schema params value.
reference	string	Describes a reference to another entity item. Format is: model:index:attribute=src-attribute,...
schema	object	Nested schema.
ttl	boolean	Set to true to store the date value as a Unix epoch in seconds suitable for use as a DynamoDB TTL attribute.
type	Type or string	Type to use for the attribute.
unique	boolean	Set to true to enforce uniqueness for this attribute. Default false.
validate	RegExp	Regular expression to use to validate data before writing.
value	string	Template to derive the value of the attribute. These attributes are "hidden" by default.

If the default property defines the default value for an attribute. If no value is provided for the attribute when creating a new item, the default value will be used.

If the hidden property is set to true, the attribute will be defined in the DynamoDB database table, but will be omitted in the returned Javascript results.

If the isoDates property is defined and not-null, it will override the table isoDates value. Set to true to store the field date value as an ISO date string. Set to false to store the date as a Unix epoch date number.

The map property can be used to set an alternate or shorter attribute name when storing in the database. The map value may be a simple string that will be used as the actual attribute name.

Alternatively, the map value can be a pair of the form: 'obj.name', where the attribute value will be stored in an object attribute named "obj" with the given name name. Such two-level mappings may be used to map multiple properties to a single table attribute. This is helpful for the design pattern where GSIs project keys plus a single 'data' field and have multiple models map relevant attributes into the projected 'data' attribute. OneTable will automatically pack and unpack attribute values into the mapped attribute. Note: APIs that write to a mapped attribute must provide all the properties that map to that attribute on the API call. Otherwise an incomplete attribute would be written to the table.

The reference attribute documents a reference to another entity by using this attribute in combination with other attributes. The format is:

model:index:attribute=source-attribute,...

The "model" selects that target entity model of the reference using the nominated "index" where the target "attribute" is determined by the associated source-attribute. Multiple attributes can be specified. Tools can use this reference to navigate from one entity item to another.

The schema property permits nested field definitions. The parent property must be an Object as Arrays are not yet supported. Note: TypeScript typings are not created for nested schemas.

The ttl property supports DynamoDB TTL expiry attributes. Set to true to store a supplied date value as a Unix epoch in seconds suitable for use as a DynamoDB TTL attribute.

The type properties defines the attribute data type. Valid types include: String, Number, Boolean, Date, Object, Null, Array, Buffer (or Binary) and Set. The object type is mapped to a map, the array type is mapped to a list. Dates are stored as Unix numeric epoch date stamps unless the isoDates parameter is true, in which case the dates are store as ISO date strings. Binary data is supplied via buffer types and is stored as base64 strings in DynamoDB.

The validate property defines a regular expression that is used to validate data before writing to the database. Highly recommended.

The value property defines a literal string template that is used to compute the attribute value. This is useful for computing key values from other properties, creating compound (composite) sort keys or for packing fields into a single DynamoDB attribute when using GSIs.

String templates are similar to JavaScript string templates. The template string may contain ${name} references to other fields defined in the entity model. If any of the variable references are undefined when an API is called, the computed field value will be undefined and the attribute will be omitted from the operation. The variable name may be of the form: ${name:size:pad} where the name will be padded to the specified size using the given pad character (which default to '0'). This is useful for zero padding numbers so that they sort numerically.

If you call find or any query API and do not provide all the properties needed to resolve the complete value template. i.e. some of the ${var} references are unresolved, OneTable will take the resolved leading portion and create a begins with key condition for that portion of the value template.

Table Contexts

Each Table has a context of properties that are blended with Model properties before executing APIs. The context is used to provide keys and attributes that apply to more than just one API invocation. A typical use case is for a central authorization module to add an accountId or userId to the context which is then used in keys for items belonging to that account or user. This is useful for multi-tenant applications.

When creating items, context properties are written to the database. When updating, context properties are not, only explicit attributes provided in the API properties parameter are written.

Context properties take precedence over supplied properties. This is to prevent accidental updating of context keys. To force an update of context attributes, provide the context properties either by updating the context via Table.addContext, replacing the context via Table.setContext or supplying an explicit context via params.context to the individual API.

Use the Table.setContext method to initialize the context and Table.clear to reset.

Table Methods

The Table API provides a utility methods and low-level data API to manage DynamoDB. The low-level methods are: deleteItem, getItem, putItem, updateItem. Use these methods to do raw I/O on your table. In general, you should prefer the Model APIs that are based on their schema definition and provide a higher level of operation. The model methods are: create, get, find, remove and update.

addContext(context = {})

Add the table context properties. The context properties are merged with (overwrite) the existing context.

addModel(name, fields)

Add a new model to a table. This invokes the Model constructor and then adds the model to the table. The previously defined Table indexes are used for the model.

async batchGet(operation, params = {})

Invoke a prepared batch operation and return the results. Batches are prepared by creating a bare batch object {} and passing that via params.batch to the various OneTable APIs to build up a batched operation. Invoking batch will execute the accumulated API calls in a batch.

The batch parameter should initially be set to {} and then be passed to API calls via params.batch.

For example:

let batch = {}
await Account.get({id: accountId}, {batch})
await User.get({id: userId}, {batch})
let results = await table.batchGet(batch)

Set batch params.consistent for a consistent read.

If using params.fields to return a field set, you must provide actual attribute names in the field list and not mapped property names like when using normal Model params.fields.

async batchWrite(batch, params = {})

Same as batchGet but for write operations.

clearContext()

Clear the table context properties. The Table has a context of properties that are blended with Model properties before writing items to the database.

async create(modelName, properties, params = {})

Create a new item in the database of the given model modelName as defined in the table schema. Wraps the Model.create API. See Model.create for details.

async createTable(params)

Create a DynamoDB table based upon the needs of the specified OneTable schema. The table configuration can be augmented by supplying additional createTable configuration via the params.provisioned. See DynamoDB CreateTable for details.

async deleteItem(properties, params = {})

Delete an item in the database. This wraps the DynamoDB deleteItem method.

The properties parameter is a Javascript hash containing the required keys or fields that are used to create the primary key.

Additional fields supplied in properties may be used to construct a filter expression. In this case, a find query is first executed to identify the item to remove. Superfluous property fields will be ignored.

The optional params are fully described in Model API Params. Some relevant params include:

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression. The properties must include the key attributes if you wish to use params.prev for reverse pagination.

If params.execute is set to false, the command will not be executed and the prepared DynamoDB API parameters will be returned.

If params.many is set to true, the API may be used to delete more than one item. Otherwise, for safety, it is assume the API will only remove one item.

The params.where clause may be used to define a filter expression. This will define a FilterExpression and the ExpressionAttributeNames and ExpressionAttributeValues. See Where Clause for more details.

async deleteTable(confirmation)

Delete a DynamoDB table. Because this is a destructive operation, a confirmation string of 'DeleteTableForever' must be provided.

async exists()

Test if the table name exists in the database.

async fetch(models, properties, params = {})

Fetch an item collection of items that share the same primary key. Models should be a list of model type names to return. The properties should provide the primary key shared by those model types. The return result is a map with items organized by their model type.

For example:

let items = await table.fetch(['User', 'Product'], {pk: 'account:AcmeCorp'})
let users = items.User
let products = items.Product
users.forEach(user => /* operate on user */)
products.forEach(product => /* operate on product */)

async find(modelName, properties, params = {})

Find an item in the database of the given model modelName as defined in the table schema. Wraps the Model.find API. See Model.find for details.

async get(modelName, properties, params = {})

Get an item in the database of the given model modelName as defined in the table schema. Wraps the Model.get API. See Model.get for details.

getContext()

Return the current context properties.

getCurrentSchema(): OneSchema

Return the schema currently used by the table.

getLog()

Return the current logger object.

async getKeys()

Return the current primary table and global secondary index keys. Returns a map indexed by index name or 'primary'. The partition key property is named 'hash' and the sort key 'sort'.

async getItem(properties, params = {})

Get an item from the database. This API wraps the DynamoDB getItem method.

The properties parameter is a Javascript hash containing the required keys or fields that are used to create the primary key.

Additional fields supplied in properties may be used to construct a filter expression. In this case, a find query is first executed to identify the item to retrieve. Superfluous property fields will be ignored.

The get method returns Javascript properties for the item after applying any schema mappings. Hidden attributes will not be returned.

The optional params are fully described in Model API Params. Some relevant params include:

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression.

If params.execute is set to false, the command will not be executed and the prepared DynamoDB API parameters will be returned.

If params.parse is set to true, the results will be parsed and mapped into a set of Javascript properties. By default, the unmodified DynamoDB results are returned.

The params.where clause may be used to define a filter expression. This will define a FilterExpression and the ExpressionAttributeNames and ExpressionAttributeValues. See Where Clause for more details.

getModel(name)

Return a model for the given model name.

groupByType(items)

Return the items grouped by the configured table typeField property. Returns a map indexed by type name.

listModels()

Return a list of models defined on the Table.

async listTables()

Return a list of tables in the database.

async putItem(properties, params = {})

Create an item in the database. This API wraps the DynamoDB putItem method.

The properties parameter is a Javascript hash containing all the required attributes for the item and must contain the required keys or fields that are used to create the primary key.

OneTable will only write fields in properties that correspond to the schema attributes for the model. Superfluous property fields will be ignored.

The property names are those described by the schema. NOTE: these are not the same as the attribute names stored in the Database. If a schema uses map to define a mapped attribute name, the Javascript field name and the DynamoDB attribute name may be different.

The method returns the unmodified DynamoDB put response. If params.parse is set to true, it will return the Javascript properties created for the item with hidden attributes will not be returned.

Before creating the item, all the properties will be validated according to any defined schema validations and all required properties will be checked. Similarly, properties that use a schema enum definition will be checked that their value is a valid enum value. Encrypted fields will be encrypted transparently before writing.

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression.

If params.execute is set to false, the command will not be executed and the prepared DynamoDB API parameters will be returned.

If params.parse is set to true, the results will be parsed and mapped into a set of Javascript properties. Otherwise, the unmodified DynamoDB response will be returned.

async queryItems(properties, params)

This API invokes the DynamoDB query API and return the results.

The properties should include the relevant key properties.

The sort key may be defined as a key condition by setting the property to an object that defines the condition. The condition operator is specified as the key, and the operand as the value.

These operators may only be used with the sort key property. If the sort key uses a value template, you cannot use the operator on the sort key value directly and not on the properties that are referenced in the value template.

For example:

let user = await table.queryItems({pk, sk: {begins: 'user:john'}})
let tickets = await table.queryItems({pk, sk: {between: [1000, 2000]}})
let invoices = await table.queryItems({pk, sk: {'<=': 1000}})

The operators include:

< <= = <> >= >
begins or begins_with
between

Filter Expressions

Non-key fields are used to construct a filter expression which is applied by DynamoDB after reading the data but before returning it to the caller. OneTable will utilize fields in properties that correspond to the schema attributes for the model. Superfluous property fields will be ignored in the filter expression.

More complex filter expressions may be created via a params.where property. For example:

let invoices = await table.queryItems({pk}, {where: '${sk} <= {1000}'})

See Where Clause for more details.

If queryItems is called without a sort key, queryItems will utilize the model type as a sort key prefix and return all matching model items. This can be used to fetch all items that match the primary hash key and are of the specified model type.

The queryItems method returns an array of items after applying any schema mappings. Hidden attributes in items will not be returned.

Some useful params for queryItems include:

The params.index may be set to the desired index name.

The params.where clause may be used to define a filter expression. This will define a FilterExpression and the ExpressionAttributeNames and ExpressionAttributeValues. See Where Clause for more details.

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression.

If params.execute is set to false, the command will not be executed and the prepared DynamoDB API parameters will be returned.

If params.parse is set to true, the results will be parsed and mapped into a set of Javascript properties. Otherwise, the unmodified DynamoDB response will be returned.

async readSchema(): OneSchema

Read the Current schema from the table if it has been stored there via saveSchema.

async readSchemas(): OneSchema[]

Read all stored schemas from the table.

async remove(modelName, properties, params = {})

Delete an item in the database of the given model modelName as defined in the table schema. Wraps the Model.remove API. See Model.remove for details.

removeModel(name)

Remove a model from the current schema in use by the table. This does not impact the persisted schemas.

removeSchema(schema)

Remove a schema from the persisted Table schema items. The schema should include a name property that describes the schema.

async saveSchema(schema?: OneSchema): OneSchema

Save the current schema to the table using the _Schema:_Schema hash/sort key pair.

If the schema parameter is null or not provided, the currently configured schema will be saved. If a schema is provided and the schema.params is unset, the saved schema will include the current Table parms.

async scanItems(params)

Invokes the DynamoDB scan API and return the results.

Some relevant params include:

The params.where clause may be used to define a filter expression. This will define a FilterExpression and the ExpressionAttributeNames and ExpressionAttributeValues. See Where Clause for more details.

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression.

If params.execute is set to false, the command will not be executed and the prepared DynamoDB API parameters will be returned.

If params.parse is set to true, the results will be parsed and mapped into a set of Javascript properties. Otherwise, the unmodified DynamoDB response will be returned.

The scan method supports parallel scan where you invoke scan simultaneously from multiple workers. Using the async/await pattern, you can start the workers and then use a Promise.all to wait for their completion. To perform parallel scans, you should set the params.segments to the number of parallel segements and the params.segment to the numeric segment to be scaned for that worker.

const segments = 4
let promises = []
for (let segment = 0; segment < segments; segment++) {
    promises.push(table.scanItems({}, {segment, segments}))
}
let results = await Promise.all(promises)

setClient(client)

Assign an AWS SDK V2 DocumentClient or AWS SDK V3 Dynamo helper client to be used for communiction with DynamoDB. Note the V3 DocumentClient instance is a native AWS SDK DocumentClient instance. For AWS SDK V3, the client is an instance of the OneTable Dynamo helper.

setContext(context = {}, merge = false)

Set the table context properties. If merge is true, the properties are blended with the existing context.

async setSchema(schema?: OneSchema)

Set the current schema for the table instance. This will reset the current schema. If the schema parameter contains a schema.params, these will be applied and overwrite the current Table params.

If the schema property is null, the current schema will be removed.

async transact(operation, transaction, params = {})

Invoke a prepared transaction and return the results. Transactions are prepared by creating a bare transaction object {} and passing that via params.transaction to the various OneTable APIs to build up a transactional operation. Finally invoking transact will execute the accumulated API calls within a DynamoDB transaction.

The operation parameter should be set to write or get.

The transaction parameter should initially be set to {} and then be passed to API calls via params.transaction.

A get operation will return an array containing the items retrieved.

The Table.groupBy can be used to organize the returned items by model. E.g.

let transaction = {}
await table.get('Account', {id: accountId}, {transaction})
await table.get('User', {id: userId}, {transaction})
let items = await table.transact('get', transaction, {parse: true, hidden: true})
items = table.groupByType(items)
let accounts = items.Account
let users = items.User

async update(modelName, properties, params = {})

Update an item in the database of the given model modelName as defined in the table schema. Wraps the Model.update API. See Model.update for details.

async updateItem(properties, params)

Update an item in the database. This method wraps the DynamoDB updateItem API.

The properties parameter is a Javascript hash containing properties to update including the required keys or fields that are used to create the primary key.

OneTable will only update fields in properties that correspond to the schema attributes for the model. Superfluous property fields will be ignored.

The property names are those described by the schema. NOTE: these are not the same as the attribute names stored in the Database. If a schema uses map to define a mapped attribute name, the Javascript field name and the DynamoDB attribute name may be different.

The method returns the unmodified DynamoDB response. If params.parse is true, the call returns the Javascript properties for the item with hidden attributes removed.

The optional params are described in Model API Params.

The params.fields may be set to a list of properties to return. This defines the ProjectionExpression.

If params.execute is set to false, the command will not be executed and the prepared DynamoDB API parameters will be returned.

If params.parse is set to true, the results will be parsed and mapped into a set of Javascript properties. Otherwise, the unmodified DynamoDB response will be returned.

async updateTable(params)

Update a table and create or remove a Global Secondary Index.

Set params.create to an index to create. Set create to a map with properties for the hash and sort attributes. E.g.

await table.updateTable({create: {
    hash: 'gs1pk',
    hash: 'gs2pk',
    name: 'gs1',
}})

Set params.remove to remove an index. Set remove to a map with a name property of the table to remove. E.g.

await table.updateTable({remove: {
    name: 'gs1'
}})

uuid()

Internal routine to generate a simple, fast non-cryptographic UUID string. This routine is provided for use in the browser where Node crypto is not availble. The uuid function will generate IDs that have the same format as a UUID v4 string. However they are not crypto-grade in terms of uniqueness nor are they fully compliant in the composition of the UUID sub-components. In general, use ulid in preference to uuid.

This routine

ulid()

Generate a ULID. Useful when you need a time-based sortable, cryptographic, unique sequential number. This is preferable to using uuid.

Model Class

The Model class represents an entity (item) in the database that implements the specified model schema. With one-table design patterns, different model items are store in a single DynamoDB table and are distinguished via their unique primary keys.

Models define attributes in the database which may overlap with the attributes of other models. There is no problem with this.

A model instance is typically created via a model constructor or via the Table factory.

Errors will thow an instance of the OneTableError error class. See Error Handling for more details.

Model Examples

import {Table} from 'dynamodb-onetable'

const table = new Table({})

let Account = table.getModel('Account')
let User = table.getModel('User')

//  Get an item where the name is sufficient to construct the primary key
let account = await Account.get({name: 'Acme Airplanes'})
let user = await User.get({email: 'user@example.com'}, {index: 'gs1'})

//  find (query) items
let users = await User.find({accountName: 'Acme Airplanes'})

//  Update an item
await User.update({email: 'user@example.com', balance: 0})

Model Constructor

Models are typically created via the Table schema definition and factory and retrieved using the Table.getModel(name) method.

let User = table.getModel('User')

or in TypeScript to return a fully typed model:

type UserType = Entity<typeof Schema.models.User>
let User = table.getModel<UserType>('User')

Thereafter, the references to User instances will be fully type checked. Note: you must add "as const" to the end of your models after the closing brace.

Where table is a configured instance of Table. Name is the name of the model and options are an optional hash.

The Model options are:

Property	Type	Description
fields	object	Field attribute definitions. Same format as in the Table schema
indexes	object	Index definition. Same format as in the Table schema
timestamps	boolean	Make "created" and "updated" timestamps in items

Model High-Level API

async create(properties, params = {})

Create an item in the database. This API wraps the DynamoDB putItem method.

The properties parameter is a Javascript hash containing all the required attributes for the item and must contain the required keys or fields that are used to create the primary key.

OneTable will only write fields in properties that correspond to the schema attributes for the model. Superfluous property fields will be ignored.

The property names are those described by the schema. NOTE: these are not the same as the attribute names stored in the Database. If a schema uses map to define a mapped attribute name, the Javascript field name and the DynamoDB attribute name may be different.

The method returns the Javascript properties created for the item. Hidden attributes will not be returned.

Before creating the item, all the properties will be validated according to any defined schema validations and all required properties will be checked. Similarly, properties that use a schema enum definition will be checked that their value is a valid enum value. Encrypted fields will be encrypted transparently before writing.

For create, the params.exists will default to a false value to ensure an item of the same key does not already exist. If set to null, a create will be allowed to overwrite an existing item.

Unique Fields

If the schema specifies that an attribute must be unique, OneTable will create a special item in the database to enforce the uniqueness. This item will be an instance of the Unique model with the primary key set to _unique:Model:Attribute:Value. The created item and the unique item will be created in a transparent transaction so that the item will be created only if all the unique fields are truly unique. The remove API will similarly remove the special unique item.

When a unique field for an item is updated, the prior item value must be read first so that the unique item can be deleted. If you do an update and do not specify params.return == 'NONE', the update API must return the full updated item. This incurs an additional get request to fetch the updated values as DynamoDB transactions do not return item values. There is thus additional I/O and overhead for unique items, so take care when designating attributes as unique in high update volume items.

The optional params are described in Model API Params.

async find(properties, params = {})

Find items in the database. This API wraps the DynamoDB query method.

The properties parameter is a Javascript hash containing the required keys or fields that are used to create the primary key.

The sort key may be defined as a simple value or as a key condition by setting the property to an object that defines the condition. The condition operator is specified as the key, and the operand as the value. For example:

let user = await User.find({pk, sk: {begins: 'user:john'}})
let tickets = await Ticket.find({pk, sk: {between: [1000, 2000]}})
let invoices = await Invoice.find({pk, sk: {'<=': 1000}})
let invoices = await Invoice.find({pk}, {where: '${sk} <= {1000}'})

let items = await Invoice.find({pk}, {where: '${sk} <= {1000}'}, {count: true})
let count = items.count

The operators include:

< <= = >= >
begins or begins_with
between

Additional fields supplied in properties are used to construct a filter expression which is applied by DynamoDB after reading the data but before returning it to the caller. OneTable will utilize fields in properties that correspond to the schema attributes for the model. Superfluous property fields will be ignored in the filter expression.

More complex filter expressions may be created via a params.where property. For example:

let adminUsers = await User.find({}, {
    where: '(${role} = {admin}) and (${status} = {current})'
})

Use params.count set to true to return the number of