README

Laoban

Laoban or 老板 is chinese for 'boss'. It is a tool for controlling multiple projects. While it is language agnostic it probably offers the most value to javascript/typescript projects

NPM usage

NPM does not handle multiple projects well. Each project is a separate project.json that is managed separately. There is no ability to 'inherit' or share configuration, so in a project with many moving parts each of which is implemented with a small bit of javascript/typescript, it can be difficult to keep all the dependancies in line.

Laoban makes the following easy:

Managing config files
- There are a number of template files (might be just one)
  - These holds files that are copied to the project whenever 'laoban update' is called
  - The package.json in it is 'modified' during the copying based on a file called 'project.details.json' in the project
  - In my projects these files are things like:
    - jest.config.json
    - babel.config.json
    - tsconfig.json
    - the jest adapter for the version of jest
Executing things in parallel across all projects
- tsc: to compile all the typescript
- npm test: to run all the tests
- npm install: to make sure everything is loaded
- npm
- Any command at all...
It keeps track of the status of important things: such as last test execution, last compile, last install

Other package managers

Laoban is not opinionated. Replaceing npm with yarn in the config will let you use all the features with yarn. If you want to use it with maven or sbt or... it works fine (although those tools already have much of the capabliities that laoban brings to the javascript world)

What are the 'golden moments'

Running all the tests in parallel across multiple projects
- Without this I have to either use a pipeline after a commit, or make a script to call them one at a time
Seeing the status of the important commands
- When working with ten or more projects I found it very hard to get a simple of view of how well the code was behaving in each project
Updating all the 'react' project settings in one go
- You can update the template settings, call laoban update followed by laoban install and laoban status
- Now you know how all the projects have responded to the upgrade: they are all using it, and they have been compiled and tested
Updating a global version number
- If the projects are tightly coupled, I like them to share a version number.
When the commands take a long time you can see the tail of the logs of the commands easily
- Press ? while the commands are running for a menu

Typical usage

When loading a project with many subprojects from git

git clone the project
laoban install will setup and test the subprojects in parallel

When publishing

Change the version in the template directory
laoban update will update all the projects
laoban publish will publish all the projects

Important ideas

laoban.json

This is a file that configures laoban. The variable values can be seen by laoban config

The existance of the file marks that this is the root of a 'big project' which is composed of one or more sub projects

It holds:

"templateDir": the directory that the templates can be found in
"log": the name of the log file found in the project directories holding the log of executing the commands
"status": the name of the file (in each project directory) that holds the status of 'important commands' executed
"scriptDir": A place for bash scripts that can be accessed by the laoban commands. You can put your own scripts here
"packageManager": defaults to npm

Templates

under templateDir. Each template is a directory holding files that are used by the update command

variables

The syntax ${variableName} allows access to variables. These can be used

In other variables
In commands

Legal variables are the variables in laoban.json and the variables in the project.details.json. Examples are

laoban run 'echo ${projectDetail.template}' -a. Note the use of '' which is used to prevent bash from attemption to dereference the variables
laoban run 'echo ${scriptDir}'

When debugging executing laoban <scriptname> -adsv can provide a lot of help. This will show the value of the command being executed in each direcoty, and if the variable is not 'correct' you can often see what are legal values, and why it's not working

project.details.json

{
  "template"      : "noreact",
  "name"          : "@phil-rice/lens",
  "description"   : "A simple implementation of lens using type script",
  "projectDetails": {
    "generation"  : 0,
    "publish"     : true,
    "links"       : [],
    "extraDeps"   : {},
    "extraDevDeps": {}
  }
}

If this is present in a directory it tells laoban that the directory is a project directory.

template is the name of the subdirectory that holds the configuration files that laoban will place in the project
name is the name of the project. This is injected into package.json by update
description is the name of the project. This is injected into package.json by update
extraDeps are the names of dependancies that this project needs and are to be added to the template
extraDevDeps are the names of developer dependancies that this project needs and are to be added to the template
links are used within the 'master project' that laoban is looking after. * It allows laoban to set up symbolic links so that changes in one project are immediately reflected * These are added as dependencies to the project, with the 'current version number'
publish should this project be affected by commands with the guard condition ${projectDetails.details.publish} * Typically these are projects to be published to npmjs * typicall commands are laoban pack, laoban publish, laoban ls-publish
generation the projects are sorted in generation order so that all generation 0 projects are processed before generation 1 * See the 'TODO' section at the end: generations are only respected in display order at the moment
throttle sets the maximum number of parallel activites that will be executed. The default is 0 which doesn't limit things

Scripts

Scripts are lists of commands (sometimes just one) that are executed when you type laoban scriptName.

Scripts can be run in more than one directory at once. Most commands run in 'all projects' unless the current directory is a project directory, in which case they run in the current * -a means 'run in all project directories' * -1 means 'just run in this directory'
Scripts have access to variables

OsGuard

Scripts can be marked so that they only run in a particularly OS. Examples can be seen in the laoban.json. If called from the wrong os then an error is given. guardReason can be set to give an error message (and document why)

    "pack"       : {
       ...
      "osGuard":  "Linux",
      "guardReason": "uses the linux 'cp' command",
      "commands"   : [
          ....
      ]
    },

pmGuard

If a script requires a particular package manager (example npm test and yarn test are both OK but yarn install is not allowed), then a pmGuard can be set. If the command is executed then it will give an error message. As with osGuard, guardReason can be set to document why

guard

A command can be set to only execute if a guard is defined. Unlike the osGuard and pmGuard this does not cause an error message: only scripts that match are executed. The example of ls-ports here:

    "ls-ports"  : {
      "description": "lists the projects that have a port defined in project.details.json",
      "guard"      : "${projectDetails.details.port}",
      "commands"   : ["js:process.cwd()"]
    },

Another good example is

    "start"     : {
      "description": "${packageManager} start for all projects that have a port defined in project.details.json",
      "guard"      : "${projectDetails.details.port}",
      "commands"   : ["${packageManager} start"],
      "env"        : {"PORT": "${projectDetails.details.port}"}
    },

So by setting 'ports' to a numeric value in the project.details.json we have 'marked' the directory in such a way that executing laoban start will start up the project. This lets us spin up multiple react projects at once. It's a good idea if all the projects have different ports...

environment variables

cwd is added as an environment variable to represent the current directory other environment variables can be added to scripts such as

    "ls-pact":        {
      "osGuard":     "Windows_NT",
      "description": "lists the projects with pact files in them",
      "guard":       "${projectDetails.details.packport}",
      "commands":    ["echo %PORT%  %cwd%"],
      "env":         {"PORT": "${projectDetails.details.packport}"}
    },

inLinksOrder

Some scripts don't parallelise that well. For example if we are compiling projects, and some projects depend on other projects then we want to compile them in 'the right order'.

Setting 'inLinksOrder' means that the links in the projectDetails are used to determine the order in which things are executed

This can be seen using '-g | --generationPlan' as an option. This behavior can also be forced on any command by selecting -l, --links

env

If a command needs access to environment variables (for example a port) these can be added. It is not uncommon to have a guard condition on the command.

Commands

These are added to laoban by means of the laoban.json file. An inspection of it should give you a good idea of how to add your own command. Each command can have multiple sections or steps. Each step is executed in it's own shell so for example changing directory or setting environment variables in one step will not impact others

Simple commands

    "log"       : {"description": "displays the log file", "commands": ["cat ${log}"]},

After adding this command laoban --help will now display the command and the description. The command simply executes cat ${log}.

Commands with step names and (optional) status

"test"      : {
"description": "runs ${packageManager} test",
"commands"   : [{"name": "test", "command": "${packageManager} test", "status": true}]
},

Here we can see that the command has one step with name test. Because status is true the step results will be visible in the status

Javascript

If the text of a command starts with js: then the command will be executed in javascript.

Examples

js:process.cwd()
js:"Hello World"

This is primarily for js:process.cwd() so that we can run scripts on both windows and linux that want to show the current directory

eachLink

Some commands need to be accessed once for each link defined in the projectDetails file.

    "remoteLink"   : {
...
      "commands"       : [
        {"name": "remoteLink", "command": "${packageManager} link ${link}", "eachLink": true, "status": true},

Here we can see that the command will be executed once for each link. The variable ${link} holds the value of the link

osGuard and pmGuard

Scripts can also be set with these. If set at the script it is used to say 'this script will throw an error if you try and use it'. When set at the command level the command is ignored if the guard is not valid. This can be used to give multiple implementations for different operating systems / package managers

    "install": {
      "description"    : "(not working yet ) doing the initial updateConfigFilesFromTemplates/install/link/tsc/test... etc in each project",
      "commands"       : [
...
        {"name": "copyPackageJsonLinux", "command": "cp package.json dist", "osGuard": "Linux"},
        {"name": "copyPackageJsonWindows", "command": "copy package.json dist", "osGuard": "Windows"},
      ], "inLinksOrder": true

Monitoring

While laoban is running you can press ? to get an interactive menu.

You can press (capital) S to see the status of all your scripts as they are executed in different directories
- The status includes 'which commands have finished', 'how long they ran for'
You can press (capital) L for the names of the log files if you want to do something like tail -f
You can press a number (or letter if there are more than 10 directory) to see the tail of the log for that log

options

option `-a`

The -a means 'in all projects'. Without this laoban looks at the current directory

If it contains a project.details.json, the command is executed in this directory only
If it doesn't contain a project.details.json the command is executed as though -a had been specified

options `-p <project>`

You can give a regex for the project name and the command will be executed in those projects

Example

You are in a project X that depends on another project Y
you type laoban tsc -p X
- Now the tsc is executed in project X

option `-s`

This allows a bit of debugging of scripts. If you are having problems adding -s gives a little more information about what is happening

This is a great option when you want 'information from many places'. Like

laoban run 'ls *.config' -as

option `-d`

This is a dryrun. Instead of executing the command, it is just printed. This includes dereferencing the variables. A combination of -ds gives quite nice information about what is executing

option `-v`

Only used when debugging to help work out what are legal variables

option `-g`

Rather like '-a' in that it does not display any commands. Instead it outputs the 'generation plan': the directories that will be processed in parallel.

option `-t xxx`

Sets the maximum number of items executed at once, so that the computer doesn't get over loaded. This overrides the setting in laoban.json

status

The idea of status is to give a way to visualise what is happening across all the sub projects

Some commands are marked as 'status: true'
When these are executed they are recorded in the status file for that project
laoban status will display the latest status
laoban compactStatus will crunch the status down

TODO

Make easily available on npmjs with name laoban

Have reserved name
Need to make it so that we can just mention it and it gets installed.

Monitor improvements

Had a situation where everything was finished but one thing that was hanging
It would be nice to be able to 'kill' the one thing that is hanging

Move to the ts project so we can easily dogfood

Consider if this is a good idea

Make the `laoban.json` composible

Because it can get very big...
and we want to be able to be able to reuse bits across projects

Need a plugin story so that we don't have to shell out for common things

For example updating package json... only important for npm/yarn. Currently done by a script
We want to be able to use other people's scripts easily

Bugs?

check laoban pack and loaban pack -a. Not sure why isn't executing

Usage no npm install needed!