README
Stencila is an open source platform for authoring, collaborating on, and sharing executable documents. This is the repository for the Stencila Hub.
✨ Features
A brief overview of features implemented by the Stencila Hub. These are current features, more are planned as we are under active development (specific features are labelled with the Future Feature Unicorn 🦄).
We also welcome your suggestions or code contributions for feature you'd like to see.
Authentication
- Store user credentials securely inside the Hub database, or
- Authenticate with third party services such as Github, Google, Twitter and Orcid
Projects
- Create projects
- Upload, manage and edit files and data in the browser
- Link remote sources (Github, Google Drive 🦄 and Dropbox 🦄)
- Seamlessly integrate all code and data no matter the source
- Allow Projects to be publicly accessible (or added to the Project Gallery 🦄)
User Management
- Build Teams of Users
- Assign access to Projects at a User or Team level
- Fine-grained permissions to limit reading, writing, creation, etc per User or Team
Sessions
- Start execution Sessions on Stencila Cloud
- Apply resource limits to execution contexts on a per Project basis
- Allow public or controlled access to execution sessions with a Token or Key
- Limit the number of concurrent Sessions or the total number of Sessions that have been spawned
⚙️ Services
Stencila Hub consists of several services, each with it's own sub-folder. The README.md
file for each service (🦄 not all of these are written yet) provides further details on the service, including its purpose, the current and potential alternative technical approaches, and tips for development:
router
: A Nginx server that routes requests to other services.manager
: A Django project containing most of the application logic.broker
: A RabbitMQ instance that acts as a message queue broker.worker
: A Celery process that runs jobs from the broker's queue.scheduler
: A Celery process that places periodic, scheduled jobs on the broker's queue.overseer
: A Celery process that monitors events associated with workers and job queues.database
: A PostgreSQL database used by the manager.monitor
: A Prometheus instance that monitors the health of the other services.
🛠️ Develop
Prerequisites
The prerequisites for development vary somewhat by service (see the service README.md
s for more details). However, most of the services will require you to have at least one of the following installed:
To run the service integration tests described below you will need:
and/or,
Getting started
The top level Makefile
contains recipes for common development tasks e.g make lint
. To run all those recipes, culminating in standing up containers for each of the services, simply do:
make
The top level make
recipes mostly just invoke the corresponding recipe in the Makefile
for each service. You can run them individually by either cd
ing into the service's folder or using the make -C
option e.g. to just run the manager
service,
make -C manager run
💁 Tip: The individual
run
recipes are useful for quickly iterating during development and, in the case of themanager
, will hot-reload when source files are edited. Where possible, therun
recipes will use a local Python virtual environment. In other cases, they will use the Docker image for the service. In both cases therun
recipes define the necessary environment variables, set at their defaults.
💁 Tip: If you need to run a couple of the services together you can
make run
them in separate terminals. This can be handy if you want to do iterative development of one service while checking that it is talking correctly to one or more of the other services.
🛈 Info: We use
Makefile
s throughout this repo becausemake
is a ubiquitous and language agnostic development tool. However, they are mostly there to guide and document the development workflow. You may prefer to bypassmake
in favour of using your favorite tools directly e.g.python
,npx
, PyCharm,pytest
, VSCode or whatever.
Linting and formatting
Some of services define code linting and formatting recipes. Often they get run together e.g.
make format lint
Unit testing
Some of services define unit tests. Run them using,
make test
Or, with coverage,
make cover
🛈 Info: Test coverage reports are generated on CI for each push and are available on Codecov here.
Integration testing
docker-compose
With To test the integration between services, use the docker-compose.yaml
file. To bring up the whole stack of services,
docker-compose up --build
Or, to just bring up one or two of the services and their dependents,
docker-compose up manager worker
To create a development database for integration testing, stand up the database
service and then use the manager
's create-devdb-pg
recipe to populate it:
docker-compose start database
make -C manager create-devdb-pg
💁 Tip: pgAdmin is useful for inspecting the development PostgreSQL database
minikube
and kompose
With To test deployment within a Kubernetes cluster you can use Minikube and Kompose,
minikube start
make run-in-minikube
🛈 Info: the
run-in-minikube
recipe sets up Minikube to be able to do local builds of images (rather than pulling them down from Docker Hub), then builds the images in Minikube and runskompose up
.
💁 Tip: The
minikube dashboard
is really useful for debugging. And don't forget tominikube stop
when you're done!
Committing
Commit messages should follow the conventional commits specification. This is important (but not required) because commit messages are used to determine the semantic version of releases (and thus deployments) and to generate the project's CHANGELOG.md. If appropriate, use the sentence case service name as the scope (to help make both git log
and the change log more readable). Some examples,
fix(Monitor): Fix endpoint for scraping metrics
feat(Director): Add pulling from Google Drive
docs(README): Add notes on commit messages
Dependency updates
We use Renovate to keep track of updates to packages this repo depends upon. For more details, see
the list of currently open and scheduled Renovate PRs and the renovate
configuration in package.json
.
🚀 Continuous integration and deployment
We use Azure Pipelines as a continuous integration (CI) service. On each push, the CI does linting and run tests. On each tag, the Docker image for each service is built and pushed to Docker Hub:
Continuous deployment (CD) to https://hub.stenci.la is done using FluxCD from another repository: