POLITICO Service for moving data from APIs to S3.
CORS is the devil. This tool is used to create a "shipping lane". Shipping lanes take data from an API endpoint and put it on S3... and then it keeps doing that over and over again to keep the S3 file up to date. Because it's a node process, you don't have to deal with CORS errors when fetching from the API.
This is meant as a tool for development and should not be used in a production environment. If you need up-to-date data on S3 for a production build, you should set up a proper publishing pipeline.
$ npm install -g @politico/postal-service
To start a new shipping lane run...
$ postal ship <URL_FOR_DATA> <S3_BUCKET_KEY>
By default, ship will go on forever until the process is cancelled. You can set an explicit number of intervals by using
-i and setting it equal to the number of intervals you want to run. You can also change the delay between each interval (defaults to
5 seconds) by using
-d and setting it equal to a number of milliseconds.
By default, ship will publish to staging (since this is a development tool). If you want to publish to the production bucket set the
--production flag or use the
-b flag and set it equal to a bucket.