README

overlord

Overlord compliments system monorepos.

It provide shared scripts that works with infrastructure and development tasks.

It helps create system repo and seed it with structure with sample docker related files.

Overload's scripts

The shared scripts are group into the follow categories:

buildkite

This contains scripts that generates new build steps in a buildkite build. The generation logic depends on convention on monorepos.

ci-scripts

This contains scripts that is to execute on hosts. It is usually referenced in step definition uploaded by buildkite steps generated by buildkite scripts mentioned above.

These scripts assumes very little on what is available on hosts, it performs most of the work through docker containers vis docker compose provided/configured by system's monorepo.

Its contract with components/libs/frontends are all via npm run.

playpen

This contains docker-compose wrappers that deals with docker compose project name, standard env variables and support of multiple compose yaml files for components (not infrastructure aka root playpen).

project

This contains scripts that are meant to run in machines that are setup to develop system's monorepo. e.g. your laptop or docker compose containers.

How to create a brand new system

Run npx @siteminder/dx github create-system-repos --help to see the menu

How to release overlord

See arch-packages' README

How to see what playpen provides

Run the following command to see high level actions.

cd <system>
$(npm bin)/playpen --help

Subcommand also has help too.

$(npm bin)/playpen start --help

App artefact, tests, linting etc

Zip files & jar files will be produced if the following step is added to your monorepo's pipeline.yml

- label: 'Define package steps'
  command: './node_modules/\@siteminder/overlord/shared/buildkite/package.sh'

- label: 'Define test steps'
  command: './node_modules/\@siteminder/overlord/shared/buildkite/test.sh'

- label: ':nodejs: Define lint steps'
  command: './node_modules/\@siteminder/overlord/shared/buildkite/lint.sh'

package.sh will search the following paths for docker-compose files named docker-compose.yaml, where xxxx is any directory e.g. core-api etc:

components/xxx/
frontends/xxx/

test.sh will search the following paths for docker-compose files named docker-compose.yaml, where xxxx is any directory e.g. core-api etc:

components/xxx/
frontends/xxx/
libs/xxx/
packages/xxx/

A test step will be added for each docker-compose.yaml found.

test-component.sh will search the following paths for docker-compose files named docker-compose.yaml, where xxxx is any directory e.g. core-api etc:

components/xxx/

The test-component ci step uses docker-compose.yaml & docker-compose.test-component.yaml of the component.

docker-compose.test-component.yaml is where you configure the component's compose service to run via the component's docker image. Take care to set any env or working dir setting so that the image is started under the same setting that it would be under kubernetes. You also need to define a test runner service in docker-compose.test-component.yaml, if the test runner service name is not passed to test-component.sh then by default a <component>-test-runner service will be executed.

components/xxx/

It will check if npm run test-component is defined in package.json (using jq), if so then a test-component step will be added for each docker-compose.yaml found.

cypress.sh will search the following paths for docker-compose files named docker-compose.yaml, where xxxx is any directory e.g. admin etc:

frontends/xxx/

It will check if npm run cypress is defined in package.json (using jq), if so then a cypress step will be added for each docker-compose.yaml found.

The cypress ci step uses docker-compose.yaml & docker-compose.cypress.yaml of the frontend's beef.

docker-compose.cypress.yaml is where you start up beef via beef's harness (which handles data generation, stubs & server startup) and other dependencies that needs to be running for the beef.

lint.sh will search the following paths for docker-compose files named docker-compose.yaml, where xxxx is any directory e.g. core-api etc:

components/xxx/
frontends/xxx/
libs/xxx/
packages/xxx/

It will check if npm run lint is defined in package.json (using jq), if so then a lint step will be added for each docker-compose.yaml found.

audit.sh will search the following paths for docker-compose files named docker-compose.yaml, where xxxx is any directory e.g. core-api etc:

components/xxx/
frontends/xxx/
libs/xxx/
packages/xxx/

It will check determine the appropriate audit command. The logic is as follows:

if package.json exists and npm run audit exists then npm run audit is it
if package.json exists but npm run audit does not exists then npm audit is it
otherwise it will assume npm run audit is callable once inside the container

App Images & Components

App images will be built if the following step is added to your monorepo's pipeline.yml

- label: ':docker: Define build steps'
  command: './node_modules/\@siteminder/overlord/shared/buildkite/build.sh'

build.sh will search within components/xxxx/ directory for Dockerfiles named Dockerfile or Dockerfile.app, where xxxx is any directory e.g. core-api etc:

For each Dockerfile found matching this convention a build step will be added.

Migration Images & Components

Building Migrations

Migration images will be built if the following step is added to your monorepo's pipeline.yml

  - label: 'define migration steps'
    command: './node_modules/\@siteminder/overlord/shared/buildkite/migrations.sh'

migrations.sh will search the following paths for Dockerfile(s) matching the following naming convention Dockerfile.*-migrations, where xxxx is any directory e.g. domain-model:

libs/xxxx/
migrations/xxxx/

For each Dockerfile found matching this convention a build step will be added.

Example:

Given the following directory structure:

  libs
    - domain-model
      - Dockerfile.hotel-db-migrations
      - Dockerfile.inventory-db-migrations

these build steps will be generated:

  - label: 'domain-model: build hotel-db-migrations'
    command: "./node_modules/@siteminder/overlord/shared/ci-scripts/build-migrations.sh libs/domain-model/Dockerfile.hotel-db-migrations"

  - label: 'domain-model: build inventory-db-migrations'
    command: "./node_modules/@siteminder/overlord/shared/ci-scripts/build-migrations.sh libs/domain-model/Dockerfile.inventory-db-migrations"

System deployment

All docker images (component / migration), frontend zip files, lambda zip files will be deployed if the following step is added to your monorepo's pipeline.yml.

  - label: 'define prod deployment steps'
    command: './node_modules/\@siteminder/overlord/shared/buildkite/deployments.sh -r prod -e pciprod'

Deploying apps

deployments.sh will search the following paths for Dockerfile(s) named Dockerfile or Dockerfile.app, where xxxx is any directory e.g. core-api:

components/xxxx/

Supported arguments for components

--use-tf-helm: Set type of deployment to terraform rather than helm
--docker-version: Tune behaviour of version arg passed to subsequent deployment steps for docker components.
- --docker-version build: The default option, which is the same as not supplying the option.
- --docker-version none: Do not output version option. Useful when running with versions locked in config repo & image.pullPolicy: Always in dev
- --docker-version branch: The version option will be set to branch name in Buildkite pipeline. Useful when running with versions locked in config repo to branch name & image.pullPolicy: Always in dev. Providing the branch name as version give faster feedback when deployment fails due to config repo having different value.
--config-branch: Set preferred config repo branch. (Must be master in prod & pciprod, so useful in dev only)
--app-infras-branch: Set preferred infrastructure repo branch. (Must be master in prod & pciprod, so useful in dev only)
-R: Region arg.
-e: environment arg.
-r: realm arg.
--include: explicitly specify what are in scope to be deployed
--exclude: explicitly skip some components/frontends from been deployed

Deploying migrations

deployments.sh will search the following paths for Dockerfile(s) matching the following naming convention Dockerfile.*-migrations, where xxxx is any directory e.g. domain-model:

libs/xxxx/
migrations/xxxx/

For each Dockerfile found matching this convention a deploy step will be added for each Dockerfile found.

NOTE: The name of the migration components in your monorepo's corresponding infrastructure repo, must match the suffix of the migration Dockerfile.

Example

Given the following directory structure:

  libs
    - domain-model
      - Dockerfile.hotel-db-migrations
      - Dockerfile.inventory-db-migrations

and a version of 1.0.0, these deploy steps will be generated:

  - label: 'deploy - domain-model hotel-db-migrations 1.0.0 to pciprod.prod'
    command: './node_modules/\@siteminder/overlord/shared/buildkite/deploy.sh -c hotel-db-migrations -t helm -V "1.0.0"'

  - label: 'deploy - domain-model inventory-db-migrations 1.0.0 to pciprod.prod'
    command: './node_modules/\@siteminder/overlord/shared/buildkite/deploy.sh -c inventory-db-migrations -t helm -V "1.0.0"'

how to install lib's dependency in a component

Code that is shared amongst multiple components within the same system are stored in the libs dir. It is compiled into the component's node_modules folder which is why you can access the shared code via import <folder-name-of-the-shared-lib>.

Because shared code is not really a npm package, you will have to install npm packages that your lib uses.

To simply the process, in your component dir run

../../node_modules/@siteminder/overlord/shared/project/install-lib-dependencies.sh <shared-lib-folder-name>

It will install the dependencies (no dev dependencies) in your component.

@vdtn359/overlord

Usage no npm install needed!