API Reference

This reference covers the main authoring surface — the functions and classes you use directly when writing workflows. For task-by-task explanation and worked examples, see Tasks and Flows and Core Concepts.

Top-Level Package

Ginkgo — a dynamic, reproducible workflow orchestrator for scientific and data analyses.

class ginkgo.AssetKey

Stable logical identifier for one asset.

Parameters:

  • namespace (str) – Asset namespace (the asset’s kind).

  • name (str) – Human-readable logical asset name within the namespace.

namespace: str
name: str
to_dict()

Return a JSON/YAML-safe mapping.

Return type:

dict[str, str]

classmethod from_dict(data)

Build an asset key from serialized metadata.

Parameters:

data (dict[str, Any]) – Serialized key payload.

Return type:

AssetKey

classmethod parse(text, *, strict=False)

Parse a namespace:name (or bare name) string.

Parameters:

  • text (str) – Key text. namespace:name yields an explicit key; a bare name (no separator) defaults to the file namespace.

  • strict (bool) – When True, malformed input (empty text, or a : with an empty namespace or name) raises ValueError. When False, such input falls back to file:<text>.

Return type:

AssetKey

__init__(*, namespace, name)
Parameters:

  • namespace (str)

  • name (str)

Return type:

None

class ginkgo.AssetRef

Resolved reference passed to downstream tasks.

Parameters:

  • key (AssetKey) – Logical asset identity.

  • version_id (str) – Immutable version identifier.

  • kind (str) – Physical asset kind.

  • artifact_id (str) – Backing artifact identifier.

  • content_hash (str) – Content hash of the stored bytes.

  • artifact_path (str) – Absolute filesystem path to the immutable stored artifact.

  • metadata (dict[str, Any]) – Asset metadata copied from the registered version.

key: AssetKey
version_id: str
kind: str
artifact_id: str
content_hash: str
artifact_path: str
metadata: dict[str, Any]
property namespace: str

Return the asset namespace.

property name: str

Return the asset name.

load()

Return the stored artifact path.

Returns:

Absolute path to the immutable artifact content.

Return type:

str

as_file()

Return the artifact path as a ginkgo.file marker.

Return type:

file

to_dict()

Return a JSON/YAML-safe mapping.

Return type:

dict[str, Any]

classmethod from_dict(data)

Build a reference from serialized metadata.

Parameters:

data (dict[str, Any]) – Serialized asset reference payload.

Return type:

AssetRef

__init__(*, key, version_id, kind, artifact_id, content_hash, artifact_path, metadata=<factory>)
Parameters:

  • key (AssetKey)

  • version_id (str)

  • kind (str)

  • artifact_id (str)

  • content_hash (str)

  • artifact_path (str)

  • metadata (dict[str, Any])

Return type:

None

class ginkgo.AssetResult

Task-return sentinel produced by asset() and its shorthands.

An AssetResult tags a task output for registration as an immutable asset. The evaluator consumes the sentinel, serialises its payload, and replaces it with a resolved AssetRef.

Parameters:

  • payload (Any) – The user-provided value. For file-kind assets this is a path (str / Path). For semantic kinds this is the live object (pandas.DataFrame / numpy.ndarray / figure / text body / trained model) or, for the path-backed sub-kinds (csv/tsv/png/svg/html and Path for text), a filesystem path.

  • kind (str) – Asset kind. One of file/table/array/fig/text /model. file is the fallback kind for bytes whose semantic shape Ginkgo does not track; the other kinds are semantically typed and drive kind-specific serialization, preview rendering, and rehydration behaviour.

  • sub_kind (str | None) – Backend sub-kind detected at construction time (e.g. "pandas" / "matplotlib" / "sklearn"). None for file assets.

  • name (str | None) – Optional explicit local asset name. When omitted, the evaluator assigns a name based on the producing task function.

  • group (str | None) – Optional report grouping label. This affects presentation only and does not participate in asset identity.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output. This affects presentation only and does not participate in asset identity.

  • metadata (dict[str, Any]) – Optional user-supplied metadata stored on the asset version.

  • checks (tuple[Callable[[Any], bool], ...]) – Ordered asset checks run during registration. Each check receives the wrapped payload and must return True or False. Checks must be importable module-level functions when the task runs in a worker.

  • kind_fields (dict[str, Any]) – Internal bag carrying kind-specific construction-time fields (e.g. "format" for text / "framework" and "metrics" for model). Populated by the factory from its typed keyword arguments; readers should use the well-known keys defined in the corresponding kind spec.

payload: Any
kind: Literal['file', 'table', 'array', 'fig', 'text', 'model'] = 'file'
sub_kind: str | None = None
name: str | None = None
group: str | None = None
caption: str | None = None
metadata: dict[str, Any]
checks: tuple[Callable[[Any], bool], ...] = ()
kind_fields: dict[str, Any]
property path: Path

Return the wrapped path as a Path.

Only valid for file assets and the path-backed sub-kinds (csv/tsv/png/svg/html and Path text). Raises TypeError for in-memory payloads.

__init__(*, payload, kind='file', sub_kind=None, name=None, group=None, caption=None, metadata=<factory>, checks=(), kind_fields=<factory>)
Parameters:

  • payload (Any)

  • kind (Literal['file', 'table', 'array', 'fig', 'text', 'model'])

  • sub_kind (str | None)

  • name (str | None)

  • group (str | None)

  • caption (str | None)

  • metadata (dict[str, Any])

  • checks (tuple[Callable[[Any], bool], ...])

  • kind_fields (dict[str, Any])

Return type:

None

class ginkgo.AssetVersion

Immutable metadata for one asset materialization.

Parameters:

  • key (AssetKey) – Logical identity of the asset.

  • version_id (str) – Immutable version identifier.

  • kind (str) – Physical asset kind.

  • artifact_id (str) – Backing content-addressed artifact identifier.

  • content_hash (str) – Content hash of the stored asset bytes.

  • run_id (str) – Run identifier that produced this version.

  • producer_task (str) – Fully-qualified producing task name.

  • created_at (str) – ISO-8601 creation timestamp.

  • metadata (dict[str, Any]) – User-supplied asset metadata.

key: AssetKey
version_id: str
kind: str
artifact_id: str
content_hash: str
run_id: str
producer_task: str
created_at: str
metadata: dict[str, Any]
to_dict()

Return a YAML-safe mapping.

Return type:

dict[str, Any]

classmethod from_dict(data)

Build an asset version from serialized metadata.

Parameters:

data (dict[str, Any]) – Serialized version payload.

Return type:

AssetVersion

__init__(*, key, version_id, kind, artifact_id, content_hash, run_id, producer_task, created_at, metadata=<factory>)
Parameters:

  • key (AssetKey)

  • version_id (str)

  • kind (str)

  • artifact_id (str)

  • content_hash (str)

  • run_id (str)

  • producer_task (str)

  • created_at (str)

  • metadata (dict[str, Any])

Return type:

None

class ginkgo.ExecutionDirective

A value a task body returns to request further execution.

Each concrete directive type carries the parameters the evaluator needs to dispatch the appropriate runner. The four built-in directive types (ShellDirective, NotebookDirective, ScriptDirective, SubWorkflowDirective) subclass this.

class ginkgo.Expr

An opaque node representing a deferred computation.

Parameters:

  • task_def (TaskDef) – The task definition that produced this expression.

  • args (dict[str, object]) – Mapping of parameter names to argument values. Values may be concrete Python objects or nested Expr / ExprList instances that must be resolved before this task can execute.

task_def: TaskDef
args: dict[str, object]
mapped: bool = False
display_label_parts: tuple[str, ...]
concurrency_group: str | None = None
concurrency_group_limit: int | None = None
property output: _OutputProxy

Return a proxy for indexing into this expression’s tuple result.

__init__(task_def, args=<factory>, mapped=False, display_label_parts=<factory>, concurrency_group=None, concurrency_group_limit=None)
Parameters:

  • task_def (TaskDef)

  • args (dict[str, object])

  • mapped (bool)

  • display_label_parts (tuple[str, ...])

  • concurrency_group (str | None)

  • concurrency_group_limit (int | None)

Return type:

None

class ginkgo.ExprList

A typed list of independent Expr[T] nodes produced by fan-out.

The scheduler evaluates all elements in parallel. When passed as an argument to a downstream task expecting list[T], the evaluator resolves all constituent expressions before executing the consumer.

Parameters:

  • exprs (list[Expr[T]]) – The individual expression nodes.

  • task_def (TaskDef | None) – Optional originating task definition for empty or chained fan-out.

exprs: list[Expr[T]]
task_def: TaskDef | None = None
property output: _OutputProxy

Return a proxy for indexing into each element’s tuple result.

map(*, max_concurrent=None, **varying)

Extend each existing branch by zipping new varying arguments.

Parameters:

  • max_concurrent (int | None) – When set, the scheduler will run at most this many of the generated branches concurrently. Independent of the global --jobs and --cores budgets.

  • **varying – Per-branch keyword arguments.

Return type:

ExprList[T]

product_map(*, max_concurrent=None, **varying)

Extend each existing branch across Cartesian varying arguments.

Parameters:

  • max_concurrent (int | None) – When set, the scheduler will run at most this many of the generated branches concurrently.

  • **varying – Per-branch keyword arguments.

Return type:

ExprList[T]

__init__(exprs=<factory>, task_def=None)
Parameters:
Return type:

None

class ginkgo.FlowDef

A wrapper around a flow function.

Calling a FlowDef executes the flow body (building the expression tree) and returns whatever the flow function returns.

Parameters:

fn (Callable) – The original flow function.

fn: Callable[[...], Any]
property name: str

Fully qualified name of the wrapped function.

__init__(fn)
Parameters:

fn (Callable[[...], Any])

Return type:

None

final class ginkgo.NotebookDirective

Execution directive representing a notebook execution request.

Parameters:

  • path (Path) – Resolved source notebook path (.ipynb or .py for marimo).

  • output (str | AssetResult | list[those] | None) – Declared output path or paths. When provided, every path is validated for existence after execution. When omitted, the managed HTML artifact path is returned as the task result.

  • log (str | None) – Optional path to capture stdout/stderr.

  • source_hash (str) – BLAKE3 hash of the notebook source file, used for cache invalidation.

path: Path
output: str | AssetResult | list[str | AssetResult] | None
log: str | None
source_hash: str
__init__(path, output, log, source_hash)
Parameters:
Return type:

None

class ginkgo.PartialCall

A partially applied task call, awaiting .map() for remaining args.

Parameters:

  • task_def (TaskDef) – The task definition.

  • fixed_args (dict[str, object]) – Arguments already supplied.

task_def: TaskDef
fixed_args: dict[str, object]
map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per element by zipping varying columns.

All varying argument columns must be the same length.

Parameters:

  • max_concurrent (int | None) – When set, the scheduler will run at most this many generated branches concurrently, independently of --jobs and --cores limits. Use this to throttle classes of work that should not run in parallel (e.g. model training).

  • **varying – Keyword arguments where each value is an iterable (list, Series, or ExprList) of per-element values.

Returns:

One Expr per element in the varying columns.

Return type:

ExprList

Raises:

  • ValueError – If varying columns have different lengths or no varying args given.

  • TypeError – If a varying argument name is not a valid parameter.

product_map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per Cartesian combination.

Parameters:

  • max_concurrent (int | None)

  • varying (Any)

Return type:

ExprList

__init__(task_def, fixed_args=<factory>)
Parameters:

  • task_def (TaskDef)

  • fixed_args (dict[str, object])

Return type:

None

final class ginkgo.ScriptDirective

Execution directive representing a script execution request.

Parameters:

  • path (Path) – Resolved source script path.

  • output (str | AssetResult | list[those] | None) – Declared output path or paths. When provided, every path is validated for existence after execution.

  • log (str | None) – Optional path to capture stdout/stderr.

  • interpreter (str) – Interpreter command (e.g. "python" or "rscript").

  • source_hash (str) – BLAKE3 hash of the script source file, used for cache invalidation.

path: Path
output: str | AssetResult | list[str | AssetResult] | None
log: str | None
interpreter: str
source_hash: str
__init__(path, output, log, interpreter, source_hash)
Parameters:

  • path (Path)

  • output (str | AssetResult | list[str | AssetResult] | None)

  • log (str | None)

  • interpreter (str)

  • source_hash (str)

Return type:

None

class ginkgo.SecretRef

Reference to a runtime-resolved secret value.

Parameters:

  • name (str) – Logical secret name or path.

  • backend (str) – Resolver backend identifier.

name: str
backend: str = 'env'
__init__(*, name, backend='env')
Parameters:

  • name (str)

  • backend (str)

Return type:

None

final class ginkgo.ShellDirective

Execution directive representing a shell command to execute.

Parameters:

  • cmd (str) – The shell command (already interpolated with resolved values).

  • output (str | list[str] | tuple[str, ...]) – Expected output path or paths. Used for cache checking and post- execution validation.

  • log (str | None) – Optional path to capture stdout/stderr.

cmd: str
output: str | AssetResult | list[str | AssetResult] | tuple[str | AssetResult, ...]
log: str | None = None
__init__(cmd, output, log=None)
Parameters:
Return type:

None

final class ginkgo.SubWorkflowDirective

Execution directive representing a nested Ginkgo workflow invocation.

Parameters:

  • path (str) – Path to the child workflow file, resolved by the caller.

  • params (dict) – Parameter overrides to pass to the child run. Written to a temporary YAML config file at dispatch time and passed via --config.

  • config (tuple of str) – Additional --config paths to forward to the child run.

path: str
params: dict
config: tuple[str, ...] = ()
__init__(path, params=<factory>, config=())
Parameters:

  • path (str)

  • params (dict)

  • config (tuple[str, ...])

Return type:

None

class ginkgo.SubWorkflowResult

Outcome of a completed sub-workflow invocation.

Parameters:

  • run_id (str) – The child run’s identifier.

  • status (str) – "success" on a clean exit. Failures raise SubWorkflowError before a result is returned, so this is always "success" when a result object is observed.

  • manifest_path (str) – Path to the child run’s manifest.yaml.

run_id: str
status: str
manifest_path: str
__init__(run_id, status, manifest_path)
Parameters:

  • run_id (str)

  • status (str)

  • manifest_path (str)

Return type:

None

class ginkgo.TaskDef

Wraps a user function so that calls produce expression nodes.

Parameters:

  • fn (Callable) – The original user function.

  • env (str | None) – Foreign execution environment for shell tasks.

  • version (int) – Cache-busting version tag.

  • retries (int) – Additional retry attempts after the initial execution.

  • retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – When set, only retry failures matching these exception classes. None (default) retries every failure up to retries.

  • retry_backoff (float) – Base delay in seconds between retry attempts. 0.0 (default) reruns immediately.

  • retry_backoff_multiplier (float) – Exponential factor applied to the base delay. Delay for attempt k (1-indexed) is retry_backoff * retry_backoff_multiplier ** (k - 1), capped at retry_backoff_max.

  • retry_backoff_max (float) – Upper bound on the computed retry delay, in seconds.

  • retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. When set, only retry failures whose exit code is in this tuple. Ignored for non-shell tasks.

  • priority (int) – Relative scheduling priority. When several tasks are ready at the same time, higher-priority tasks are dispatched first. Range is [-1000, 1000]; default 0.

  • kind (str) – Execution contract for the task body.

  • threads (int) – Static CPU footprint for the scheduler. Used as the task’s core budget against --cores and made available to the task body when the function signature declares a threads parameter. Shell tasks also receive GINKGO_THREADS=<n> in the subprocess environment.

  • memory (str | None) – Static memory footprint for the scheduler, in Kubernetes resource notation (e.g. "4Gi", "512Mi"). When set, the scheduler reserves this amount against --memory and remote executors map it to pod resource requests.

  • gpu (int) – Number of GPUs to request for remote execution. Has no effect on local runs. Remote executors map this to the appropriate accelerator resource (e.g. nvidia.com/gpu on Kubernetes).

  • remote (bool) – When True, dispatch this task to the remote executor (if configured via --executor). Tasks with gpu > 0 are implicitly remote. Local-only tasks ignore this flag.

  • export_thread_env (bool) – When True, shell tasks additionally receive OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, and NUMEXPR_NUM_THREADS set to the declared thread count. Default is False so existing tool configuration is not silently overridden.

fn: Callable[[...], Any]
env: str | None = None
version: int = 1
retries: int = 0
retry_on: type[BaseException] | tuple[type[BaseException], ...] | None = None
retry_backoff: float = 0.0
retry_backoff_multiplier: float = 2.0
retry_backoff_max: float = 60.0
retry_on_exit_codes: tuple[int, ...] | None = None
priority: int = 0
kind: str = 'python'
threads: int = 1
memory: str | None = None
gpu: int = 0
remote: bool = False
export_thread_env: bool = False
remote_input_access: str | None = None
streaming_compatible: bool = True
fuse_prefetch: tuple[tuple[str, str], ...] = ()
property name: str

Fully qualified name of the wrapped function.

property required_params: frozenset[str]

Parameter names that have no default value.

property execution_mode: str

Return whether the task body runs on the driver or a worker.

property all_params: dict[str, Parameter]

All parameters from the function signature.

property signature: Signature

The inspected function signature.

property type_hints: dict[str, Any]

Resolved runtime type hints for the wrapped function.

property source_hash: str

BLAKE3 digest of task source and local imported modules.

property memory_gb: int

Parsed memory footprint in whole GiB (0 when unset).

property cache_source_hash: str

Digest used for cache invalidation.

For notebook and script tasks, the source file hash is incorporated at execution time via the NotebookDirective/ScriptDirective.

should_retry_exception(*, exc)

Return whether exc matches the configured retry policy.

Parameters:

exc (BaseException)

Return type:

bool

retry_delay_seconds(*, attempt)

Return the retry delay to wait before attempt (1-indexed).

Parameters:

attempt (int)

Return type:

float

__init__(fn, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', threads=1, memory=None, gpu=0, remote=False, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=())
Parameters:

  • fn (Callable[[...], Any])

  • env (str | None)

  • version (int)

  • retries (int)

  • retry_on (type[BaseException] | tuple[type[BaseException], ...] | None)

  • retry_backoff (float)

  • retry_backoff_multiplier (float)

  • retry_backoff_max (float)

  • retry_on_exit_codes (tuple[int, ...] | None)

  • priority (int)

  • kind (str)

  • threads (int)

  • memory (str | None)

  • gpu (int)

  • remote (bool)

  • export_thread_env (bool)

  • remote_input_access (str | None)

  • streaming_compatible (bool)

  • fuse_prefetch (tuple[tuple[str, str], ...])

Return type:

None

ginkgo.array(payload, *, name=None, group=None, caption=None, metadata=None, checks=None)

Wrap an n-dimensional array value as an asset return.

Parameters:

  • payload (Any) – The array value. Supports numpy ndarray, xarray DataArray/Dataset, zarr array/group, and dask array.

  • name (str | None) – Optional explicit local asset name.

  • group (str | None) – Optional report grouping label.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

  • metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

  • checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

ginkgo.asset(payload, *, kind='file', name=None, group=None, caption=None, metadata=None, checks=None, **kind_fields)

Wrap a task output for registration as an asset.

Canonical constructor for every asset kind. asset(df, kind="table") and table() produce identical AssetResult values; the semantic factories are shorthand around this function.

Parameters:

  • payload (Any) – The value to register. For file assets this must be a path-like value. For the semantic kinds, this is the live object (DataFrame / ndarray / figure / text body / model) or a path to a sub-kind-specific file format.

  • kind (str) – Asset kind. Defaults to "file". Must be one of the registered kinds (file/table/array/fig/text /model).

  • name (str | None) – Optional explicit local asset name.

  • group (str | None) – Optional report grouping label. Grouping is persisted with the asset version but does not affect the stable asset key.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output. Captions are persisted with the asset version but do not affect the stable asset key.

  • metadata (dict[str, Any] | None) – Optional user-supplied metadata persisted on the asset version.

  • checks (Iterable[Callable[[Any], bool]] | None) – Optional ordered checks run against the payload during asset registration. Every check must return True or False.

  • **kind_fields (Any) – Kind-specific construction-time fields. For text this accepts format; for model this accepts framework and metrics. Other kinds accept no extra fields.

Returns:

Sentinel consumed by the evaluator after task execution.

Return type:

AssetResult

ginkgo.config(path)

Load a project configuration file via the top-level package API.

Parameters:

path (str | Path)

Return type:

dict[str, Any]

ginkgo.evaluate(expr, *, jobs=None, cores=None, memory=None, backend=None, provenance=None, secret_resolver=None, event_bus=None)

Resolve an expression tree to concrete values.

Parameters:

  • expr (Any) – The root expression or nested container to resolve.

  • jobs (int | None) – Maximum number of concurrently running tasks.

  • cores (int | None) – Maximum total thread budget across running tasks.

  • memory (int | None) – Maximum total declared memory budget across running tasks in GiB.

  • backend (ExecutionEnvironment | None) – Execution environment for environment-isolated tasks.

  • event_bus (EventBus | None) – Optional event bus to receive lifecycle events. Useful for tests and ad-hoc programmatic callers that want to observe task progress.

  • provenance (RunProvenanceRecorder | None)

  • secret_resolver (SecretResolver | None)

Returns:

The concrete result of evaluating the input.

Return type:

Any

ginkgo.expand(template, **wildcards)

Expand a string template across wildcard combinations.

Parameters:

  • template (str) – Template containing named str.format placeholders.

  • **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic Cartesian-product order.

Return type:

list[str]

ginkgo.fig(payload, *, name=None, group=None, caption=None, metadata=None, checks=None)

Wrap a figure or plot value as an asset return.

Parameters:

  • payload (Any) – The figure value. Supports matplotlib Figure, plotly Figure, bokeh Figure, or a path to an existing PNG/SVG/HTML file.

  • name (str | None) – Optional explicit local asset name.

  • group (str | None) – Optional report grouping label.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

  • metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

  • checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

class ginkgo.file

A path to a single file.

Validated to exist on disk before task execution. Return values declared as file are validated to exist after execution. Cache key contribution is the BLAKE3 digest of file contents.

ginkgo.flatten(items)

Flatten nested lists and tuples into a single list.

Parameters:

items (list[Any] | tuple[Any, ...]) – Nested list or tuple structure.

Returns:

Flat list preserving left-to-right order.

Return type:

list[Any]

ginkgo.flow(fn)

Decorator that marks a function as a flow (pipeline entry point).

Unlike @task(), @flow is used without parentheses.

Parameters:

fn (Callable) – The flow function.

Return type:

FlowDef

class ginkgo.folder

A path to a directory.

Validated to exist and be a directory before execution. Cache key contribution is the BLAKE3 digest of sorted recursive contents.

ginkgo.model(payload, *, name=None, group=None, caption=None, framework=None, metrics=None, metadata=None, checks=None)

Wrap a trained model as an asset return.

Parameters:

  • payload (Any) – The trained model object. Supports scikit-learn estimators, XGBoost and LightGBM sklearn-wrapped models, PyTorch nn.Module instances, and Keras/TensorFlow models.

  • name (str | None) – Optional explicit local asset name.

  • group (str | None) – Optional report grouping label.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

  • framework (str | None) – Optional explicit framework override, bypassing module-based detection. Must be one of "sklearn", "xgboost", "lightgbm", "pytorch", "keras".

  • metrics (dict[str, float] | None) – Optional scalar metrics captured at training time. Stored as a first-class field on the asset version for ginkgo models and UI rendering.

  • metadata (dict[str, Any] | None) – Optional free-form metadata stored on the asset version.

  • checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

ginkgo.notebook(path, *, output=None, log=None)

Create a notebook execution expression.

Called from inside a @task("notebook") body with fully resolved argument values. Relative paths resolve from the current working directory at the time of the call.

Parameters:

  • path (str | Path) – Source notebook file (.ipynb for Jupyter/Papermill or .py for marimo).

  • output (str | AssetResult | list[those] | None) – Declared output path or paths, validated for existence after execution. When omitted, the managed rendered HTML artifact path is returned instead.

  • log (str | None) – Optional path to capture stdout/stderr during execution.

Return type:

NotebookDirective

ginkgo.remote_file(uri, *, version_id=None, access=None)

Construct a remote file reference from a URI.

Parameters:

  • uri (str) – Remote URI (e.g. s3://bucket/key or oci://namespace/bucket/key).

  • version_id (str | None) – Optional version ID for immutable pinning.

  • access (str | None) – Preferred access mode ("stage" or "fuse"). None defers the choice to the configured default / policy resolver.

Return type:

RemoteFileRef

Raises:

ValueError – If the URI scheme is unsupported or the URI is malformed, or if access is not one of "stage" / "fuse".

ginkgo.remote_folder(uri, *, version_id=None, access=None)

Construct a remote folder reference from a URI.

Parameters:

  • uri (str) – Remote URI pointing to a prefix (e.g. s3://bucket/prefix/).

  • version_id (str | None) – Optional version ID for immutable pinning.

  • access (str | None) – Preferred access mode ("stage" or "fuse"). None defers the choice to the configured default / policy resolver.

Return type:

RemoteFolderRef

Raises:

ValueError – If the URI scheme is unsupported or the URI is malformed, or if access is not one of "stage" / "fuse".

ginkgo.script(path, *, output=None, log=None, interpreter=None)

Create a script execution expression.

Called from inside a @task("script") body with fully resolved argument values. Resolved task inputs are forwarded to the script as --param-name value CLI arguments.

Parameters:

  • path (str | Path) – Source script file. Relative paths resolve from the current working directory at the time of the call.

  • output (str | AssetResult | list[those] | None) – Declared output path or paths, validated for existence after execution.

  • log (str | None) – Optional path to capture stdout/stderr during execution.

  • interpreter (str | None) – Interpreter command override. When None, the interpreter is inferred from the file extension: .py → python, .R/.r → rscript.

Return type:

ScriptDirective

Raises:

  • FileNotFoundError – If the script file does not exist.

  • ValueError – If the interpreter cannot be inferred from the extension and no explicit interpreter is given.

ginkgo.secret(name, *, backend='env')

Return a runtime secret reference.

Parameters:

  • name (str) – Logical secret name or path.

  • backend (str, default="env") – Resolver backend identifier.

Returns:

Deferred secret reference to resolve at execution time.

Return type:

SecretRef

ginkgo.shell(*, cmd, output, log=None)

Create a shell command expression.

Called from inside a @task(kind="shell") body with fully resolved argument values. The cmd is a standard Python f-string — all variables are concrete at the point this is called.

Parameters:

  • cmd (str) – The shell command to run.

  • output (str | list[str] | tuple[str, ...]) – The expected output path or paths.

  • log (str | None) – Optional path to capture stdout/stderr.

Return type:

ShellDirective

ginkgo.slug(value)

Return a deterministic file-safe slug.

Parameters:

value (str) – Input text to normalize.

Returns:

Lowercased slug with non-alphanumeric runs collapsed to underscores.

Return type:

str

ginkgo.subworkflow(path, *, params=None, config=None)

Create a sub-workflow invocation expression.

Called from inside a @task(kind="subworkflow") body with fully resolved argument values. The child workflow runs as a self-contained ginkgo run subprocess; its run_id and manifest path are returned to the parent task.

Parameters:

  • path (str or Path) – Path to the child workflow file.

  • params (dict, optional) – Parameter overrides for the child run. Serialised as YAML and passed via a temporary --config file.

  • config (str, Path, or sequence of either, optional) – Additional --config paths to forward to the child.

Return type:

SubWorkflowDirective

ginkgo.table(payload, *, name=None, group=None, caption=None, metadata=None, checks=None)

Wrap a tabular value as an asset return.

Parameters:

  • payload (Any) – The tabular value. Supports pandas DataFrame, polars DataFrame/LazyFrame, pyarrow Table/Dataset, DuckDB relation, or a path to a CSV/TSV file.

  • name (str | None) – Optional explicit local asset name.

  • group (str | None) – Optional report grouping label.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

  • metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

  • checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

ginkgo.task(_kind=None, /, *, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', threads=1, memory=None, gpu=0, remote=False, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=None)

Decorator that turns a function into a lazy task definition.

The task kind may be given as the first positional argument or via the kind keyword. @task("shell"), @task("notebook"), and @task("script") are the preferred short forms.

Parameters:

  • _kind (str | None) – Task kind as a positional argument. When provided, takes precedence over the kind keyword.

  • env (str | None) – Foreign execution environment for shell tasks. If None, the task runs in the current environment.

  • version (int) – Cache-busting version tag. Bump when task logic changes.

  • retries (int) – Additional retry attempts after the initial execution.

  • retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – Narrow retries to specific exception classes. None retries any failure.

  • retry_backoff (float) – Base delay (seconds) before each retry. 0.0 disables the delay.

  • retry_backoff_multiplier (float) – Exponential factor applied between attempts.

  • retry_backoff_max (float) – Upper bound on the computed delay, in seconds.

  • retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. Narrow retries to specific exit codes.

  • priority (int) – Relative scheduling priority. Higher runs first among ready tasks. Range [-1000, 1000]; default 0.

  • kind (str) – Execution contract for the task body. Ignored when _kind is given.

  • threads (int) – Static CPU footprint for the scheduler. The task body receives the same value when its function signature declares a threads parameter; shell tasks also see GINKGO_THREADS=<n> in the subprocess environment.

  • memory (str | None) – Static memory footprint for the scheduler in Kubernetes resource notation (e.g. "4Gi", "512Mi").

  • gpu (int) – Number of GPUs to request for remote execution. Has no effect on local runs. Remote executors map this to the appropriate accelerator resource (e.g. nvidia.com/gpu on Kubernetes).

  • remote (bool) – When True, dispatch this task to the remote executor. Tasks with gpu > 0 are implicitly remote.

  • export_thread_env (bool) – Export common BLAS/OpenMP thread environment variables (OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS) to shell-task subprocesses. Default False.

  • remote_input_access (str | None)

  • streaming_compatible (bool)

  • fuse_prefetch (dict[str, str] | None)

Returns:

A decorator that wraps the function in a TaskDef.

Return type:

Callable

Raises:

ValueError – If both a positional kind and a non-default kind keyword are supplied and they differ.

ginkgo.text(payload, *, name=None, group=None, caption=None, format=None, metadata=None, checks=None)

Wrap a text or structured document value as an asset return.

Parameters:

  • payload (Any) – The document value. Supports strings, dicts, or paths. Dicts are serialised as JSON; strings are stored as the requested format.

  • name (str | None) – Optional explicit local asset name.

  • group (str | None) – Optional report grouping label.

  • caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

  • format ({"plain", "markdown", "json"} | None) – Document format. Auto-detected from the payload when omitted.

  • metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

  • checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

class ginkgo.tmp_dir

A ginkgo-managed scratch directory, unique per task execution.

Created automatically before task execution and deleted on success. Kept on failure for debugging. Does not participate in the cache key.

ginkgo.zip_expand(template, **wildcards)

Expand a string template by zipping wildcard values positionally.

Parameters:

  • template (str) – Template containing named str.format placeholders.

  • **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic positional order.

Return type:

list[str]

Flow API

The @flow decorator for marking pipeline entry points.

A @flow-decorated function is the entry point for a workflow. When called, it executes its body to build the expression tree, then returns the resulting Expr or ExprList. No task execution happens during this phase.

class ginkgo.core.flow.FlowDef

A wrapper around a flow function.

Calling a FlowDef executes the flow body (building the expression tree) and returns whatever the flow function returns.

Parameters:

fn (Callable) – The original flow function.

property name: str

Fully qualified name of the wrapped function.

__init__(fn)
Parameters:

fn (Callable[[...], Any])

Return type:

None

ginkgo.core.flow.flow(fn)

Decorator that marks a function as a flow (pipeline entry point).

Unlike @task(), @flow is used without parentheses.

Parameters:

fn (Callable) – The flow function.

Return type:

FlowDef

ginkgo.core.flow.discover_flow(module)

Return the unique @flow-decorated function defined in module.

Raises RuntimeError when module contains zero or more than one FlowDef; identification is by Python object identity, so a single FlowDef re-exported under multiple names still counts as one.

Parameters:

module (ModuleType)

Return type:

FlowDef

Task API

The @task decorator and supporting classes.

A @task()-decorated function does not execute when called. Instead it returns an Expr[T] (full call) or a PartialCall (subset of required arguments), enabling lazy expression tree construction.

class ginkgo.core.task.TaskDef

Wraps a user function so that calls produce expression nodes.

Parameters:

  • fn (Callable) – The original user function.

  • env (str | None) – Foreign execution environment for shell tasks.

  • version (int) – Cache-busting version tag.

  • retries (int) – Additional retry attempts after the initial execution.

  • retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – When set, only retry failures matching these exception classes. None (default) retries every failure up to retries.

  • retry_backoff (float) – Base delay in seconds between retry attempts. 0.0 (default) reruns immediately.

  • retry_backoff_multiplier (float) – Exponential factor applied to the base delay. Delay for attempt k (1-indexed) is retry_backoff * retry_backoff_multiplier ** (k - 1), capped at retry_backoff_max.

  • retry_backoff_max (float) – Upper bound on the computed retry delay, in seconds.

  • retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. When set, only retry failures whose exit code is in this tuple. Ignored for non-shell tasks.

  • priority (int) – Relative scheduling priority. When several tasks are ready at the same time, higher-priority tasks are dispatched first. Range is [-1000, 1000]; default 0.

  • kind (str) – Execution contract for the task body.

  • threads (int) – Static CPU footprint for the scheduler. Used as the task’s core budget against --cores and made available to the task body when the function signature declares a threads parameter. Shell tasks also receive GINKGO_THREADS=<n> in the subprocess environment.

  • memory (str | None) – Static memory footprint for the scheduler, in Kubernetes resource notation (e.g. "4Gi", "512Mi"). When set, the scheduler reserves this amount against --memory and remote executors map it to pod resource requests.

  • gpu (int) – Number of GPUs to request for remote execution. Has no effect on local runs. Remote executors map this to the appropriate accelerator resource (e.g. nvidia.com/gpu on Kubernetes).

  • remote (bool) – When True, dispatch this task to the remote executor (if configured via --executor). Tasks with gpu > 0 are implicitly remote. Local-only tasks ignore this flag.

  • export_thread_env (bool) – When True, shell tasks additionally receive OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, and NUMEXPR_NUM_THREADS set to the declared thread count. Default is False so existing tool configuration is not silently overridden.

property name: str

Fully qualified name of the wrapped function.

property required_params: frozenset[str]

Parameter names that have no default value.

property execution_mode: str

Return whether the task body runs on the driver or a worker.

property all_params: dict[str, Parameter]

All parameters from the function signature.

property signature: Signature

The inspected function signature.

property type_hints: dict[str, Any]

Resolved runtime type hints for the wrapped function.

property source_hash: str

BLAKE3 digest of task source and local imported modules.

property memory_gb: int

Parsed memory footprint in whole GiB (0 when unset).

property cache_source_hash: str

Digest used for cache invalidation.

For notebook and script tasks, the source file hash is incorporated at execution time via the NotebookDirective/ScriptDirective.

should_retry_exception(*, exc)

Return whether exc matches the configured retry policy.

Parameters:

exc (BaseException)

Return type:

bool

retry_delay_seconds(*, attempt)

Return the retry delay to wait before attempt (1-indexed).

Parameters:

attempt (int)

Return type:

float

__init__(fn, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', threads=1, memory=None, gpu=0, remote=False, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=())
Parameters:

  • fn (Callable[[...], Any])

  • env (str | None)

  • version (int)

  • retries (int)

  • retry_on (type[BaseException] | tuple[type[BaseException], ...] | None)

  • retry_backoff (float)

  • retry_backoff_multiplier (float)

  • retry_backoff_max (float)

  • retry_on_exit_codes (tuple[int, ...] | None)

  • priority (int)

  • kind (str)

  • threads (int)

  • memory (str | None)

  • gpu (int)

  • remote (bool)

  • export_thread_env (bool)

  • remote_input_access (str | None)

  • streaming_compatible (bool)

  • fuse_prefetch (tuple[tuple[str, str], ...])

Return type:

None

class ginkgo.core.task.PartialCall

A partially applied task call, awaiting .map() for remaining args.

Parameters:

  • task_def (TaskDef) – The task definition.

  • fixed_args (dict[str, object]) – Arguments already supplied.

map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per element by zipping varying columns.

All varying argument columns must be the same length.

Parameters:

  • max_concurrent (int | None) – When set, the scheduler will run at most this many generated branches concurrently, independently of --jobs and --cores limits. Use this to throttle classes of work that should not run in parallel (e.g. model training).

  • **varying – Keyword arguments where each value is an iterable (list, Series, or ExprList) of per-element values.

Returns:

One Expr per element in the varying columns.

Return type:

ExprList

Raises:

  • ValueError – If varying columns have different lengths or no varying args given.

  • TypeError – If a varying argument name is not a valid parameter.

product_map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per Cartesian combination.

Parameters:

  • max_concurrent (int | None)

  • varying (Any)

Return type:

ExprList

__init__(task_def, fixed_args=<factory>)
Parameters:

  • task_def (TaskDef)

  • fixed_args (dict[str, object])

Return type:

None

ginkgo.core.task.task(_kind=None, /, *, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', threads=1, memory=None, gpu=0, remote=False, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=None)

Decorator that turns a function into a lazy task definition.

The task kind may be given as the first positional argument or via the kind keyword. @task("shell"), @task("notebook"), and @task("script") are the preferred short forms.

Parameters:

  • _kind (str | None) – Task kind as a positional argument. When provided, takes precedence over the kind keyword.

  • env (str | None) – Foreign execution environment for shell tasks. If None, the task runs in the current environment.

  • version (int) – Cache-busting version tag. Bump when task logic changes.

  • retries (int) – Additional retry attempts after the initial execution.

  • retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – Narrow retries to specific exception classes. None retries any failure.

  • retry_backoff (float) – Base delay (seconds) before each retry. 0.0 disables the delay.

  • retry_backoff_multiplier (float) – Exponential factor applied between attempts.

  • retry_backoff_max (float) – Upper bound on the computed delay, in seconds.

  • retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. Narrow retries to specific exit codes.

  • priority (int) – Relative scheduling priority. Higher runs first among ready tasks. Range [-1000, 1000]; default 0.

  • kind (str) – Execution contract for the task body. Ignored when _kind is given.

  • threads (int) – Static CPU footprint for the scheduler. The task body receives the same value when its function signature declares a threads parameter; shell tasks also see GINKGO_THREADS=<n> in the subprocess environment.

  • memory (str | None) – Static memory footprint for the scheduler in Kubernetes resource notation (e.g. "4Gi", "512Mi").

  • gpu (int) – Number of GPUs to request for remote execution. Has no effect on local runs. Remote executors map this to the appropriate accelerator resource (e.g. nvidia.com/gpu on Kubernetes).

  • remote (bool) – When True, dispatch this task to the remote executor. Tasks with gpu > 0 are implicitly remote.

  • export_thread_env (bool) – Export common BLAS/OpenMP thread environment variables (OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS) to shell-task subprocesses. Default False.

  • remote_input_access (str | None)

  • streaming_compatible (bool)

  • fuse_prefetch (dict[str, str] | None)

Returns:

A decorator that wraps the function in a TaskDef.

Return type:

Callable

Raises:

ValueError – If both a positional kind and a non-default kind keyword are supplied and they differ.

Shell API

Shell task execution primitive.

shell() is called from inside a @task(kind="shell") body and returns a ShellDirective. The evaluator detects this and dispatches the command to the configured shell runner.

final class ginkgo.core.shell.ShellDirective

Execution directive representing a shell command to execute.

Parameters:

  • cmd (str) – The shell command (already interpolated with resolved values).

  • output (str | list[str] | tuple[str, ...]) – Expected output path or paths. Used for cache checking and post- execution validation.

  • log (str | None) – Optional path to capture stdout/stderr.

__init__(cmd, output, log=None)
Parameters:
Return type:

None

ginkgo.core.shell.shell(*, cmd, output, log=None)

Create a shell command expression.

Called from inside a @task(kind="shell") body with fully resolved argument values. The cmd is a standard Python f-string — all variables are concrete at the point this is called.

Parameters:

  • cmd (str) – The shell command to run.

  • output (str | list[str] | tuple[str, ...]) – The expected output path or paths.

  • log (str | None) – Optional path to capture stdout/stderr.

Return type:

ShellDirective

Authoring Helpers

Template expansion and wildcard utilities for workflow authoring.

ginkgo.wildcards.expand(template, **wildcards)

Expand a string template across wildcard combinations.

Parameters:

  • template (str) – Template containing named str.format placeholders.

  • **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic Cartesian-product order.

Return type:

list[str]

ginkgo.wildcards.zip_expand(template, **wildcards)

Expand a string template by zipping wildcard values positionally.

Parameters:

  • template (str) – Template containing named str.format placeholders.

  • **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic positional order.

Return type:

list[str]

ginkgo.wildcards.slug(value)

Return a deterministic file-safe slug.

Parameters:

value (str) – Input text to normalize.

Returns:

Lowercased slug with non-alphanumeric runs collapsed to underscores.

Return type:

str

ginkgo.wildcards.flatten(items)

Flatten nested lists and tuples into a single list.

Parameters:

items (list[Any] | tuple[Any, ...]) – Nested list or tuple structure.

Returns:

Flat list preserving left-to-right order.

Return type:

list[Any]