dms-kinesis-gen

Generate Kinesis Base64 Records, based on JSON Files

Usage no npm install needed!

<script type="module">
  import dmsKinesisGen from 'https://cdn.skypack.dev/dms-kinesis-gen';
</script>

README

kinesis-dms-record-generator

Summary

This is simple CLI, to help developers simulate the records that Amazon DMS sends to a kinesis stream, when running Migration Tasks.

The problem

When developers are working with DMS + Kinesis, it's very difficult to simulate the stream locally, therefore, developer's productivity is impaired. This tool aims in improving the developer experience and productivity

The solution

By using tools such as Data Grip, the developer can right-click the table, and export the table content to a JSON File. This tool will feed from this JSON file, and it will automatically generate the Kinesis Record, and use the aws-cli to put the records in the stream

Dependencies

Before using this tool, you must have the following tools configured in your machine

Running as bin

npx dms-kinesis-gen

or install it globally with

npm i -g dms-kinesis-gen

And then use the command

dms-kinesis-gen

Pattern on filenames

All the files inside the folder should follow the following pattern.

loadOrder.schema.table.json

For example

1.OT.CUSTOMER.json
2.OT.PRODUCTS.json

The bigger the load-order, first it will be loaded, following the same patter from Amazon DMS Docs.

For the example files, the order will be:

2.OT.PRODUCTS.json
1.OT.CUSTOMER.json

The JSON can have a single record, or an array of records, for example:

[
  {
    "ID": 1,
    "NAME": "Joselito Silva"
  }
]

This JSON file will generate the following Kinesis Record

{
  "data": {
    "ID": 1,
    "NAME": "Joselito Silva"
  },
  "metadata": {
    "timestamp": "2021-02-21T23:02:36.5680Z",
    "record-type": "data",
    "operation": "load",
    "partition-key-type": "primary-key",
    "schema-name": "OT",
    "table-name": "CUSTOMER"
  }
}

Prompts

√ What's the folder where the JSON files are located ? ... C:\Users\my-user\Documents\cdc-files
√ What's the name of the kinesis stream? ... my-localstack-stream
√ What's the partition key? ... 1
√ What's the localstack endpoint? ... http://localhost:4566
√ What's the name operation you want to simulate? » LOAD

The cli will load all files inside the files folder, and load them to the kinesis stream

Running Tests

yarn test

Contributing

Check the contributing.md file for more information