Basics#
Stimela is workflow management framework. The two basic elements of Stimela are cabs and recipes.
Cabs: a cab is an atomic task that can be invoked in a workflow. Cabs come in a few different flavours:
an executable command
a Python function
a snippet of Python code
a CASA task
A cab definition (see Cab definition reference) is a YaML document that tells Stimela how to invoke the task, and what its inputs and outputs are (this is collectively known as a schema). You can write your own cab definitions, and mix-and-match them with standard cabs shipped in stimela’s companion
cult-cargo
package.Recipes: a recipe is a YaML document describing a workflow, in terms of a sequence of steps. Each step invokes a cab, or another recipe (a.k.a. nested recipe). Steps have parameters which are matched (validated) against the schema of the underlying cab. Stimela provides a number of powerful mechanisms to pass parameters between steps.
Recipes also have inputs and outputs, described by the recipe’s schema (which is what allows them to be nested.)
Cabs and recipes cab be executed natively (i.e. directly on the host OS), inside a Python virtual environment, inside a container, on a Kubernetes cluster, and/or as a Slurm job. See backends for more detail.
Stimela and cult-cargo#
Stimela by itself does not predefine any cabs. Instead, the idea is that separate packages such as cult-cargo
(just use pip install) provide cab collections that the user can employ. cult-cargo
is a curated set of cabs for radio interferometry software, maintained by the Stimela developers. Users are also free to roll their own cab definitions, and provide their own cab collections as installable packages.
Anatomy of a simple recipe#
Here is a (rather notional and idealized) recipe:
#!/usr/bin/env -S stimela run
### (the above is just a handy trick that lets us execute the recipe file directly)
# include some of my cab definitions
_include:
- mycabs.yml
# set some stimela options
opts:
log:
dir: .logs # changes the logfiles directory
# this recipe is named thus
calibration-recipe:
info: "a notional recipe for calibration & imaging"
# this recipe has some input parameters
inputs:
ms:
dtype: MS
required: true
info: "measurement set to use"
image-name:
dtype: str
required: true
info: "base name for output images"
image-size:
dtype: int
default: 4096
info: "image size, in pixels"
# this recipe consists of three steps, "image", "predict" and "calibrate"
steps:
image:
info: "make initial image and model from DATA column"
# this is the unerlying tool that the step invokes (defined in mycabs.yml, presumably)
cab: imager-tool
# and these are the parameters of the step...
params:
ms: =recipe.ms
mode: image
size: =recipe.image-size * 2
column: DATA
output.image: '{recipe.image-name}.image-{info.suffix}-{current.size:05d}.fits'
output.model: '{recipe.image-name}.model-{info.suffix}.fits'
predict:
info: "predict model into MODEL_DATA"
cab: imager-tool
params:
ms: =recipe.ms
mode: predict
model: =previous.output.model
column: MODEL_DATA
calibrate:
info: "calibrate model against data"
cab: calibration-tool
params:
ms: =recipe.ms
model.column: =steps.predict.column
output.column: CORRECTED_DATA
The following sections will explain what’s going on in more detail.