API Reference

This reference covers the main authoring surface — the functions and classes you use directly when writing workflows. For task-by-task explanation and worked examples, see Tasks and Flows and Core Concepts.

Top-Level Package

Ginkgo — a dynamic, reproducible workflow orchestrator for scientific and data analyses.

class ginkgo.AssetKey

Stable logical identifier for one asset.

Parameters:

• namespace (str) – Asset namespace (the asset’s kind).

• name (str) – Human-readable logical asset name within the namespace.

namespace: str

name: str

to_dict()

Return a JSON/YAML-safe mapping.

Return type:

dict[str, str]

classmethod from_dict(data)

Build an asset key from serialized metadata.

Parameters:

data (dict[str, Any]) – Serialized key payload.

Return type:

AssetKey

classmethod parse(text)

Parse a canonical <kind>:<name> key string.

A bare name is not a key: the kind is half of the identity, so inferring one would silently address a different asset than the caller wrote. Callers holding a bare name resolve it against the catalog instead (see ginkgo.cli.commands.asset).

Parameters:

text (str) – Key text as rendered by __str__().

Return type:

AssetKey

Raises:

ValueError – If text is not <kind>:<name> with both parts non-empty.

__init__(*, namespace, name)

Parameters:

• namespace (str)

• name (str)

Return type:

None

class ginkgo.AssetRef

Resolved reference passed to downstream tasks.

Parameters:

• key (AssetKey) – Logical asset identity.

• version_id (str) – Immutable version identifier.

• kind (str) – Physical asset kind.

• artifact_id (str) – Backing artifact identifier.

• content_hash (str) – Content hash of the stored bytes.

• artifact_path (str) – Absolute filesystem path to the immutable stored artifact.

• metadata (dict[str, Any]) – Asset metadata copied from the registered version.

key: AssetKey

version_id: str

kind: str

artifact_id: str

content_hash: str

artifact_path: str

metadata: dict[str, Any]

property namespace: str

Return the asset namespace.

property name: str

Return the asset name.

as_file(*, execution_mode=None)

Return the artifact path as a ginkgo.file marker.

Available for the kinds whose artifact holds the payload’s own bytes (file, fig, text). A kind stored in Ginkgo’s own encoding (table, array, model) has no readable file path, so it raises rather than wrapping a serialized blob in a file marker.

Parameters:

execution_mode (str | None) – TaskDef.execution_mode of the consuming task, when known, so the error offers remedies that work for that kind of task.

Returns:

Absolute path to the immutable artifact content.

Return type:

file

Raises:

TypeError – When the artifact holds an encoded payload rather than the bytes its path implies.

to_dict()

Return a JSON/YAML-safe mapping.

Return type:

dict[str, Any]

classmethod from_dict(data)

Build a reference from serialized metadata.

Parameters:

data (dict[str, Any]) – Serialized asset reference payload.

Return type:

AssetRef

__init__(*, key, version_id, kind, artifact_id, content_hash, artifact_path, metadata=<factory>)

Parameters:

• key (AssetKey)

• version_id (str)

• kind (str)

• artifact_id (str)

• content_hash (str)

• artifact_path (str)

• metadata (dict[str, Any])

Return type:

None

class ginkgo.AssetResult

Task-return sentinel produced by asset() and its shorthands.

An AssetResult tags a task output for registration as an immutable asset. The evaluator consumes the sentinel, serialises its payload, and replaces it with a resolved AssetRef.

Parameters:

• payload (Any) – The user-provided value. For file-kind assets this is a path (str / Path). For semantic kinds this is the live object (pandas.DataFrame / numpy.ndarray / figure / text body / trained model) or, for the path-backed sub-kinds (csv/tsv/png/svg/html and Path for text), a filesystem path.

• kind (str) – Asset kind. One of file/table/array/fig/text /model. file is the fallback kind for bytes whose semantic shape Ginkgo does not track; the other kinds are semantically typed and drive kind-specific serialization, preview rendering, and rehydration behaviour.

• sub_kind (str | None) – Backend sub-kind detected at construction time (e.g. "pandas" / "matplotlib" / "sklearn"). None for file assets.

• name (str | None) – Optional explicit local asset name. When omitted, the evaluator assigns a name based on the producing task function.

• group (str | None) – Optional report grouping label. This affects presentation only and does not participate in asset identity.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output. This affects presentation only and does not participate in asset identity.

• metadata (dict[str, Any]) – Optional user-supplied metadata stored on the asset version.

• checks (tuple[Callable[[Any], bool], ...]) – Ordered asset checks run during registration. Each check receives the wrapped payload and must return True or False. Checks must be importable module-level functions when the task runs in a worker.

• kind_fields (dict[str, Any]) – Internal bag carrying kind-specific construction-time fields (e.g. "format" for text / "framework" and "metrics" for model). Populated by the factory from its typed keyword arguments; readers should use the well-known keys defined in the corresponding kind spec.

payload: Any

kind: Literal['file', 'table', 'array', 'fig', 'text', 'model'] = 'file'

sub_kind: str | None = None

name: str | None = None

group: str | None = None

caption: str | None = None

metadata: dict[str, Any]

checks: tuple[Callable[[Any], bool], ...] = ()

kind_fields: dict[str, Any]

property path: Path

Return the wrapped path as a Path.

Only valid for file assets and the path-backed sub-kinds (csv/tsv/png/svg/html and Path text). Raises TypeError for in-memory payloads.

__init__(*, payload, kind='file', sub_kind=None, name=None, group=None, caption=None, metadata=<factory>, checks=(), kind_fields=<factory>)

Parameters:

• payload (Any)

• kind (Literal['file', 'table', 'array', 'fig', 'text', 'model'])

• sub_kind (str | None)

• name (str | None)

• group (str | None)

• caption (str | None)

• metadata (dict[str, Any])

• checks (tuple[Callable[[Any], bool], ...])

• kind_fields (dict[str, Any])

Return type:

None

class ginkgo.AssetVersion

Immutable metadata for one asset materialization.

Parameters:

• key (AssetKey) – Logical identity of the asset.

• version_id (str) – Immutable version identifier.

• kind (str) – Physical asset kind.

• artifact_id (str) – Backing content-addressed artifact identifier.

• content_hash (str) – Content hash of the stored asset bytes.

• run_id (str) – Run identifier that produced this version.

• producer_task (str) – Fully-qualified producing task name.

• created_at (str) – ISO-8601 creation timestamp.

• metadata (dict[str, Any]) – User-supplied asset metadata.

key: AssetKey

version_id: str

kind: str

artifact_id: str

content_hash: str

run_id: str

producer_task: str

created_at: str

metadata: dict[str, Any]

to_dict()

Return a YAML-safe mapping.

Return type:

dict[str, Any]

classmethod from_dict(data)

Build an asset version from serialized metadata.

Parameters:

data (dict[str, Any]) – Serialized version payload.

Return type:

AssetVersion

__init__(*, key, version_id, kind, artifact_id, content_hash, run_id, producer_task, created_at, metadata=<factory>)

Parameters:

• key (AssetKey)

• version_id (str)

• kind (str)

• artifact_id (str)

• content_hash (str)

• run_id (str)

• producer_task (str)

• created_at (str)

• metadata (dict[str, Any])

Return type:

None

class ginkgo.ExecutionDirective

A value a task body returns to request further execution.

Each concrete directive type carries the parameters the evaluator needs to dispatch the appropriate runner. The four built-in directive types (ShellDirective, NotebookDirective, ScriptDirective, SubWorkflowDirective) subclass this.

class ginkgo.Expr

An opaque node representing a deferred computation.

A call is not the tuple its task’s return annotation describes, so unpacking, indexing, and len() all refuse, and point at output instead.

Parameters:

• task_def (TaskDef) – The task definition that produced this expression.

• args (dict[str, object]) – Mapping of parameter names to argument values. Values may be concrete Python objects or nested Expr / ExprList instances that must be resolved before this task can execute.

task_def: TaskDef

args: dict[str, object]

mapped: bool = False

display_label_parts: tuple[str, ...]

concurrency_group: str | None = None

concurrency_group_limit: int | None = None

property output: _OutputProxy

Return a proxy for indexing into this expression’s tuple result.

property display_label: str

Return the label under which this call is reported to the user.

Built from display_label_parts, which fan-out fixes at graph-build time, so the label is the same before dispatch as after it.

__init__(task_def, args=<factory>, mapped=False, display_label_parts=<factory>, concurrency_group=None, concurrency_group_limit=None)

Parameters:

• task_def (TaskDef)

• args (dict[str, object])

• mapped (bool)

• display_label_parts (tuple[str, ...])

• concurrency_group (str | None)

• concurrency_group_limit (int | None)

Return type:

None

class ginkgo.ExprList

A typed list of independent Expr[T] nodes produced by fan-out.

The scheduler evaluates all elements in parallel. When passed as an argument to a downstream task expecting list[T], the evaluator resolves all constituent expressions before executing the consumer.

Parameters:

• exprs (list[Expr[T]]) – The individual expression nodes.

• task_def (TaskDef | None) – Optional originating task definition for empty or chained fan-out.

exprs: list[Expr[T]]

task_def: TaskDef | None = None

property output: _OutputProxy

Return a proxy for indexing into each element’s tuple result.

map(*, max_concurrent=None, **varying)

Extend each existing branch by zipping new varying arguments.

Parameters:

• max_concurrent (int | None) – When set, the scheduler will run at most this many of the generated branches concurrently. Independent of the global --jobs and --cores budgets.

• **varying – Per-branch keyword arguments.

Return type:

ExprList[T]

product_map(*, max_concurrent=None, **varying)

Extend each existing branch across Cartesian varying arguments.

Parameters:

• max_concurrent (int | None) – When set, the scheduler will run at most this many of the generated branches concurrently.

• **varying – Per-branch keyword arguments.

Return type:

ExprList[T]

__init__(exprs=<factory>, task_def=None)

Parameters:

• exprs (list[Expr[T]])

• task_def (TaskDef | None)

Return type:

None

class ginkgo.FlowDef

A wrapper around a flow function.

Calling a FlowDef executes the flow body (building the expression tree) and returns whatever the flow function returns.

Parameters:

fn (Callable) – The original flow function.

fn: Callable[[...], Any]

property name: str

Fully qualified name of the wrapped function.

__init__(fn)

Parameters:

fn (Callable[[...], Any])

Return type:

None

final class ginkgo.NotebookDirective

Execution directive representing a notebook execution request.

Parameters:

• path (Path) – Resolved source notebook path (.ipynb or .py for marimo).

• output (str | AssetResult | list[those] | None) – Declared output path or paths. When provided, every path is validated for existence after execution. When omitted, the managed HTML artifact path is returned as the task result.

• log (str | None) – Optional path to capture stdout/stderr.

• source_hash (str) – BLAKE3 hash of the notebook source file, used for cache invalidation.

path: Path

output: str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | None

log: str | None

source_hash: str

__init__(path, output, log, source_hash)

Parameters:

• path (Path)

• output (str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | None)

• log (str | None)

• source_hash (str)

Return type:

None

final class ginkgo.OptionalOutput

A declared output path that may legitimately be absent.

Parameters:

payload (str | AssetResult) – The wrapped output declaration, exactly as it would have been written without the wrapper.

payload: str | AssetResult

__init__(payload)

Parameters:

payload (str | AssetResult)

Return type:

None

class ginkgo.PartialCall

A partially applied task call, awaiting .map() for remaining args.

Parameters:

• task_def (TaskDef) – The task definition.

• fixed_args (dict[str, object]) – Arguments already supplied.

task_def: TaskDef

fixed_args: dict[str, object]

map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per element by zipping varying columns.

All varying argument columns must be the same length.

Parameters:

• max_concurrent (int | None) – When set, the scheduler will run at most this many generated branches concurrently, independently of --jobs and --cores limits. Use this to throttle classes of work that should not run in parallel (e.g. model training).

• **varying – Keyword arguments where each value is an iterable (list, Series, or ExprList) of per-element values, or a per_branch() template rendered from each row’s own values.

Returns:

One Expr per element in the varying columns.

Return type:

ExprList

Raises:

• ValueError – If varying columns have different lengths or no varying args given.

• TypeError – If a varying argument name is not a valid parameter.

product_map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per Cartesian combination.

Every varying list is an axis of the grid. Arguments that are a function of the grid cell — output paths above all — must be passed as per_branch("...{arg}...") templates, which are rendered from each cell’s own values. expand() output is rejected here: it is already one value per cell, so crossing it with the axes it came from would mislabel every branch.

Parameters:

• max_concurrent (int | None)

• varying (Any)

Return type:

ExprList

__init__(task_def, fixed_args=<factory>)

Parameters:

• task_def (TaskDef)

• fixed_args (dict[str, object])

Return type:

None

class ginkgo.Resources

Resource requirements declared by a task.

Parameters:

• threads (int) – CPU footprint for the scheduler. Reserved against the --cores budget wherever the task runs.

• memory (str | None) – Memory footprint in Kubernetes resource notation (e.g. "4Gi", "512Mi"). Reserved against --memory locally and mapped to resource requests by remote executors.

• gpu (int) – Number of GPUs the task requires. Reserved against the --gpus budget locally, or mapped to accelerator requests by remote executors.

• gpu_type (str | None) – Accelerator type for remote execution (e.g. "nvidia-tesla-t4"). Overrides any executor-level default. Only meaningful together with gpu > 0.

• memory_retry_multiplier (float) – Factor applied to memory on each retry, for tasks whose first attempt may run out of memory. Attempt k (0-indexed) is scheduled with memory_gb * memory_retry_multiplier ** k, capped at the run’s --memory budget. 1.0 (default) disables escalation. Requires memory to be set.

• custom (dict[str, int]) – User-defined resource demands (e.g. {"api_calls": 2}), scheduled against run-level budgets from [resources.budgets] config or repeated --resource name=value flags. A dimension no budget names is unconstrained. Unlike the built-in dimensions, custom demands also count for remote-placed tasks — budgets such as API quotas or database connections apply wherever the task runs.

threads: int = 1

memory: str | None = None

gpu: int = 0

gpu_type: str | None = None

memory_retry_multiplier: float = 1.0

custom: dict[str, int]

property memory_gb: int

Parsed memory footprint in whole GiB (0 when unset).

memory_gb_for_attempt(attempt)

Memory footprint in GiB for a 0-indexed retry attempt.

Applies memory_retry_multiplier exponentially so OOM-prone tasks can retry with more memory (attempt 0 is the first execution).

Parameters:

attempt (int)

Return type:

int

__init__(threads=1, memory=None, gpu=0, gpu_type=None, memory_retry_multiplier=1.0, custom=<factory>)

Parameters:

• threads (int)

• memory (str | None)

• gpu (int)

• gpu_type (str | None)

• memory_retry_multiplier (float)

• custom (dict[str, int])

Return type:

None

final class ginkgo.ScriptDirective

Execution directive representing a script execution request.

Parameters:

• path (Path) – Resolved source script path.

• output (str | AssetResult | list[those] | None) – Declared output path or paths. When provided, every path is validated for existence after execution.

• log (str | None) – Optional path to capture stdout/stderr.

• interpreter (str) – Interpreter command (e.g. "python" or "rscript").

• source_hash (str) – BLAKE3 hash of the script source file, used for cache invalidation.

path: Path

output: str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | None

log: str | None

interpreter: str

source_hash: str

__init__(path, output, log, interpreter, source_hash)

Parameters:

• path (Path)

• output (str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | None)

• log (str | None)

• interpreter (str)

• source_hash (str)

Return type:

None

class ginkgo.SecretRef

Reference to a runtime-resolved secret value.

Parameters:

• name (str) – Logical secret name or path.

• backend (str) – Resolver backend identifier.

name: str

backend: str = 'env'

__init__(*, name, backend='env')

Parameters:

• name (str)

• backend (str)

Return type:

None

final class ginkgo.ShellDirective

Execution directive representing a shell command to execute.

Parameters:

• cmd (str) – The shell command (already interpolated with resolved values).

• output (str | list[str] | tuple[str, ...]) – Expected output path or paths. Used for cache checking and post- execution validation.

• log (str | None) – Optional path to capture stdout/stderr.

cmd: str

output: str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | tuple[str | AssetResult | OptionalOutput, ...]

log: str | None = None

__init__(cmd, output, log=None)

Parameters:

• cmd (str)

• output (str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | tuple[str | AssetResult | OptionalOutput, ...])

• log (str | None)

Return type:

None

final class ginkgo.SubWorkflowDirective

Execution directive representing a nested Ginkgo workflow invocation.

Parameters:

• path (str) – Path to the child workflow file, resolved by the caller.

• params (dict) – Parameter overrides to pass to the child run. Written to a temporary YAML config file at dispatch time and passed via --config.

• config (tuple of str) – Additional --config paths to forward to the child run.

path: str

params: dict

config: tuple[str, ...] = ()

__init__(path, params=<factory>, config=())

Parameters:

• path (str)

• params (dict)

• config (tuple[str, ...])

Return type:

None

class ginkgo.SubWorkflowResult

Outcome of a completed sub-workflow invocation.

Parameters:

• run_id (str) – The child run’s identifier.

• status (str) – "success" on a clean exit. Failures raise SubWorkflowError before a result is returned, so this is always "success" when a result object is observed.

run_id: str

status: str

__init__(run_id, status)

Parameters:

• run_id (str)

• status (str)

Return type:

None

class ginkgo.TaskDef

Wraps a user function so that calls produce expression nodes.

Parameters:

• fn (Callable) – The original user function.

• env (str | None) – Foreign execution environment for shell tasks.

• version (int) – Cache-busting version tag.

• retries (int) – Additional retry attempts after the initial execution.

• retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – When set, only retry failures matching these exception classes. None (default) retries every failure up to retries.

• retry_backoff (float) – Base delay in seconds between retry attempts. 0.0 (default) reruns immediately.

• retry_backoff_multiplier (float) – Exponential factor applied to the base delay. Delay for attempt k (1-indexed) is retry_backoff * retry_backoff_multiplier ** (k - 1), capped at retry_backoff_max.

• retry_backoff_max (float) – Upper bound on the computed retry delay, in seconds.

• retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. When set, only retry failures whose exit code is in this tuple. Ignored for non-shell tasks.

• priority (int) – Relative scheduling priority. When several tasks are ready at the same time, higher-priority tasks are dispatched first. Range is [-1000, 1000]; default 0.

• kind (str) – Execution contract for the task body.

• resources (Resources) – Declarative resource requirements (threads, memory, gpu, gpu_type). States what the task needs; placement is decided separately by the evaluator. threads is made available to the task body when the function signature declares a threads parameter, and shell tasks receive GINKGO_THREADS=<n> in the subprocess environment.

• remote (bool) – When True, dispatch this task to the run’s default executor — the one named by --executor. Requires that flag to be set.

• executor (str | None) – Name of a configured executor ([remote.executors.<name>]) this task always routes to, regardless of the run’s default. Implies remote dispatch.

• export_thread_env (bool) – When True, shell tasks additionally receive OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, and NUMEXPR_NUM_THREADS set to the declared thread count. Default is False so existing tool configuration is not silently overridden.

fn: Callable[[...], Any]

env: str | None = None

version: int = 1

retries: int = 0

retry_on: type[BaseException] | tuple[type[BaseException], ...] | None = None

retry_backoff: float = 0.0

retry_backoff_multiplier: float = 2.0

retry_backoff_max: float = 60.0

retry_on_exit_codes: tuple[int, ...] | None = None

priority: int = 0

kind: str = 'python'

resources: Resources

remote: bool = False

executor: str | None = None

export_thread_env: bool = False

remote_input_access: str | None = None

streaming_compatible: bool = True

fuse_prefetch: tuple[tuple[str, str], ...] = ()

property name: str

Fully qualified name of the wrapped function.

property required_params: frozenset[str]

Parameter names that have no default value.

property execution_mode: str

Return whether the task body runs on the driver or a worker.

property all_params: dict[str, Parameter]

All parameters from the function signature.

property signature: Signature

The inspected function signature.

property type_hints: dict[str, Any]

Resolved runtime type hints for the wrapped function.

property source_hash: str

BLAKE3 digest of task source and local imported modules.

property threads: int

Declared CPU footprint (see Resources).

property memory: str | None

Declared memory footprint string (see Resources).

property memory_gb: int

Parsed memory footprint in whole GiB (0 when unset).

property gpu: int

Declared GPU count (see Resources).

property gpu_type: str | None

Declared accelerator type (see Resources).

property cache_source_hash: str

Digest used for cache invalidation.

For notebook and script tasks, the source file hash is incorporated at execution time via the NotebookDirective/ScriptDirective.

should_retry_exception(*, exc)

Return whether exc matches the configured retry policy.

Parameters:

exc (BaseException)

Return type:

bool

retry_delay_seconds(*, attempt)

Return the retry delay to wait before attempt (1-indexed).

Parameters:

attempt (int)

Return type:

float

__init__(fn, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', resources=<factory>, remote=False, executor=None, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=())

Parameters:

• fn (Callable[[...], Any])

• env (str | None)

• version (int)

• retries (int)

• retry_on (type[BaseException] | tuple[type[BaseException], ...] | None)

• retry_backoff (float)

• retry_backoff_multiplier (float)

• retry_backoff_max (float)

• retry_on_exit_codes (tuple[int, ...] | None)

• priority (int)

• kind (str)

• resources (Resources)

• remote (bool)

• executor (str | None)

• export_thread_env (bool)

• remote_input_access (str | None)

• streaming_compatible (bool)

• fuse_prefetch (tuple[tuple[str, str], ...])

Return type:

None

ginkgo.array(payload, *, name=None, group=None, caption=None, metadata=None, checks=None)

Wrap an n-dimensional array value as an asset return.

Parameters:

• payload (Any) – The array value. Supports numpy ndarray, xarray DataArray/Dataset, zarr array/group, and dask array.

• name (str | None) – Optional explicit local asset name.

• group (str | None) – Optional report grouping label.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

• metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

• checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

ginkgo.asset(payload, *, kind='file', name=None, group=None, caption=None, metadata=None, checks=None, **kind_fields)

Wrap a task output for registration as an asset.

Canonical constructor for every asset kind. asset(df, kind="table") and table() produce identical AssetResult values; the semantic factories are shorthand around this function.

Parameters:

• payload (Any) – The value to register. For file assets this must be a path-like value. For the semantic kinds, this is the live object (DataFrame / ndarray / figure / text body / model) or a path to a sub-kind-specific file format.

• kind (str) – Asset kind. Defaults to "file". Must be one of the registered kinds (file/table/array/fig/text /model).

• name (str | None) – Optional explicit local asset name.

• group (str | None) – Optional report grouping label. Grouping is persisted with the asset version but does not affect the stable asset key.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output. Captions are persisted with the asset version but do not affect the stable asset key.

• metadata (dict[str, Any] | None) – Optional user-supplied metadata persisted on the asset version.

• checks (Iterable[Callable[[Any], bool]] | None) – Optional ordered checks run against the payload during asset registration. Every check must return True or False.

• **kind_fields (Any) – Kind-specific construction-time fields. For text this accepts format; for model this accepts framework and metrics. Other kinds accept no extra fields.

Returns:

Sentinel consumed by the evaluator after task execution.

Return type:

AssetResult

ginkgo.config(path)

Load a project configuration file via the top-level package API.

Parameters:

path (str | Path)

Return type:

dict[str, Any]

ginkgo.evaluate(expr, *, jobs=None, cores=None, memory=None, gpus=None, resource_overrides=None, resource_budgets=None, backend=None, run_dir=None, secret_resolver=None, event_bus=None)

Resolve an expression tree to concrete values.

Parameters:

• expr (Any) – The root expression or nested container to resolve.

• jobs (int | None) – Maximum number of concurrently running tasks.

• cores (int | None) – Maximum total thread budget across running tasks.

• memory (int | None) – Maximum total declared memory budget across running tasks in GiB.

• gpus (int | None) – Local GPU budget across running tasks. Defaults to 0 (no local GPUs); tasks declaring gpu > 0 then require a remote executor.

• resource_overrides (ResourceOverrides | None) – Site-level resource overrides merged over each task’s declaration.

• resource_budgets (dict[str, int] | None) – Run-level budgets for user-defined resource dimensions (e.g. {"api_calls": 10}). Dimensions tasks request but this mapping omits are unconstrained.

• backend (ExecutionEnvironment | None) – Execution environment for environment-isolated tasks.

• run_dir (RunDir | None) – The run’s directory, for per-task log paths and lockfile copies. None outside a live run.

• event_bus (EventBus | None) – Optional event bus to receive lifecycle events. Useful for tests and ad-hoc programmatic callers that want to observe task progress.

• secret_resolver (SecretResolver | None)

Returns:

The concrete result of evaluating the input.

Return type:

Any

ginkgo.expand(template, **wildcards)

Expand a string template across wildcard combinations.

The result is one string per combination, aligned row-for-row with the wildcard values, so it pairs with .map(). It is not an axis: passing it to .product_map() is rejected, since that would cross it with the very axes it was derived from. For a grid, use per_branch() instead.

Parameters:

• template (str) – Template containing named str.format placeholders.

• **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic Cartesian-product order.

Return type:

ExpandedTemplate

ginkgo.fig(payload, *, name=None, group=None, caption=None, metadata=None, checks=None)

Wrap a figure or plot value as an asset return.

Parameters:

• payload (Any) – The figure value. Supports matplotlib Figure, plotly Figure, bokeh Figure, or a path to an existing PNG/SVG/HTML file.

• name (str | None) – Optional explicit local asset name.

• group (str | None) – Optional report grouping label.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

• metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

• checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

class ginkgo.file

A path to a single file.

Validated to exist on disk before task execution. Return values declared as file are validated to exist after execution. Cache key contribution is the BLAKE3 digest of file contents.

ginkgo.flatten(items)

Flatten nested lists and tuples into a single list.

Parameters:

items (list[Any] | tuple[Any, ...]) – Nested list or tuple structure.

Returns:

Flat list preserving left-to-right order.

Return type:

list[Any]

ginkgo.flow(fn)

Decorator that marks a function as a flow (pipeline entry point).

Unlike @task(), @flow is used without parentheses.

Parameters:

fn (Callable) – The flow function.

Return type:

FlowDef

class ginkgo.folder

A path to a directory.

Validated to exist and be a directory before execution. Cache key contribution is the BLAKE3 digest of sorted recursive contents.

ginkgo.model(payload, *, name=None, group=None, caption=None, framework=None, metrics=None, metadata=None, checks=None)

Wrap a trained model as an asset return.

Parameters:

• payload (Any) – The trained model object. Supports scikit-learn estimators, XGBoost and LightGBM sklearn-wrapped models, PyTorch nn.Module instances, and Keras/TensorFlow models.

• name (str | None) – Optional explicit local asset name.

• group (str | None) – Optional report grouping label.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

• framework (str | None) – Optional explicit framework override, bypassing module-based detection. Must be one of "sklearn", "xgboost", "lightgbm", "pytorch", "keras".

• metrics (dict[str, float] | None) – Optional scalar metrics captured at training time. Stored as a first-class field on the asset version for ginkgo models and UI rendering.

• metadata (dict[str, Any] | None) – Optional free-form metadata stored on the asset version.

• checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

ginkgo.notebook(path, *, output=None, log=None)

Create a notebook execution expression.

Called from inside a @task("notebook") body with fully resolved argument values. Relative paths resolve from the current working directory at the time of the call.

Parameters:

• path (str | Path) – Source notebook file (.ipynb for Jupyter/Papermill or .py for marimo).

• output (str | AssetResult | list[those] | None) – Declared output path or paths, validated for existence after execution. When omitted, the managed rendered HTML artifact path is returned instead.

• log (str | None) – Optional path to capture stdout/stderr during execution.

Return type:

NotebookDirective

ginkgo.optional(payload)

Declare an output path that may be absent after the task runs.

A present path is hashed, stored, restored, and validated like any other file output. An absent one resolves to None in the task result, so the declaring task must annotate that element file | None and consumers must handle absence explicitly.

Parameters:

payload (str | AssetResult) – The output path, or an asset wrapping one.

Returns:

A marker consumed by shell(), script(), and notebook().

Return type:

OptionalOutput

Raises:

TypeError – If the payload is already optional, or is not a path or asset.

ginkgo.param(name, *, type=<class 'str'>, default=REQUIRED, help=None, choices=None, multiple=False)

Declare a workflow parameter and return its resolved value.

Parameters:

• name (str) – Parameter name. Must start with a letter and contain only letters, digits, and underscores. The command-line flag is the dashed form, so n_replicates is supplied as --n-replicates.

• type (Callable[[str], Any], optional) – Callable applied to string values, following argparse’s type convention. Defaults to str. bool is handled specially: the flag may be given bare (--flag) or with a literal (--flag false).

• default (Any, optional) – Value used when the parameter is supplied neither on the command line nor in config. Omit to make the parameter required.

• help (str | None, optional) – One-line description shown by ginkgo run <workflow> --help.

• choices (Sequence[Any] | None, optional) – Permitted values. Checked after type conversion.

• multiple (bool, optional) – When true the flag may be repeated and the resolved value is a tuple.

Returns:

The resolved value: from the command line if supplied there, otherwise from the config [params] table, otherwise the default.

Return type:

Any

Raises:

ParamError – If the name is invalid, the declaration conflicts with an earlier declaration of the same name, a required parameter was not supplied, or a supplied value fails type conversion or is outside choices.

ginkgo.per_branch(template)

Derive one value per fan-out branch from that branch’s own arguments.

Use this for arguments that are a function of the branch — output paths above all — rather than an axis to sweep. Placeholders name other arguments of the same .map() / .product_map() call (or arguments fixed on the task call), and are rendered per branch, so the value can never drift out of step with the values it describes.

Parameters:

template (str) – Template containing named str.format placeholders, each naming an argument of the fan-out call.

Returns:

Marker consumed by .map() and .product_map().

Return type:

PerBranch

Examples

>>> per_branch("results/{temperature}_{defect_density}.json").template
'results/{temperature}_{defect_density}.json'

ginkgo.project_root()

Return the root directory of the project containing the current directory.

Walks upward from the current working directory to the nearest directory holding a ginkgo configuration file, so this resolves to the same directory whether a workflow is run from the project root or from a subdirectory such as workflow/.

The starting point is deliberately the working directory rather than the calling module’s location: a result that depended on which file asked would be harder to predict than one that depends on where the command was run.

Returns:

An absolute path to the project root. A project configuration file is optional, so when no marker is found this falls back to the current working directory — the same directory the rest of ginkgo already treats as the project root.

Return type:

Path

ginkgo.remote_file(uri, *, version_id=None, access=None)

Construct a remote file reference from a URI.

Parameters:

• uri (str) – Remote URI (e.g. s3://bucket/key or oci://namespace/bucket/key).

• version_id (str | None) – Optional version ID for immutable pinning.

• access (str | None) – Preferred access mode ("stage" or "fuse"). None defers the choice to the configured default / policy resolver.

Return type:

RemoteFileRef

Raises:

ValueError – If the URI scheme is unsupported or the URI is malformed, or if access is not one of "stage" / "fuse".

ginkgo.remote_folder(uri, *, version_id=None, access=None)

Construct a remote folder reference from a URI.

Parameters:

• uri (str) – Remote URI pointing to a prefix (e.g. s3://bucket/prefix/).

• version_id (str | None) – Optional version ID for immutable pinning.

• access (str | None) – Preferred access mode ("stage" or "fuse"). None defers the choice to the configured default / policy resolver.

Return type:

RemoteFolderRef

Raises:

ValueError – If the URI scheme is unsupported or the URI is malformed, or if access is not one of "stage" / "fuse".

ginkgo.script(path, *, output=None, log=None, interpreter=None)

Create a script execution expression.

Called from inside a @task("script") body with fully resolved argument values. Resolved task inputs are forwarded to the script as --param-name value CLI arguments.

Parameters:

• path (str | Path) – Source script file. Relative paths resolve from the current working directory at the time of the call.

• output (str | AssetResult | list[those] | None) – Declared output path or paths, validated for existence after execution.

• log (str | None) – Optional path to capture stdout/stderr during execution.

• interpreter (str | None) – Interpreter command override. When None, the interpreter is inferred from the file extension: .py → python, .R/.r → rscript.

Return type:

ScriptDirective

Raises:

• FileNotFoundError – If the script file does not exist.

• ValueError – If the interpreter cannot be inferred from the extension and no explicit interpreter is given.

ginkgo.secret(name, *, backend='env')

Return a runtime secret reference.

Parameters:

• name (str) – Logical secret name or path.

• backend (str, default="env") – Resolver backend identifier.

Returns:

Deferred secret reference to resolve at execution time.

Return type:

SecretRef

ginkgo.shell(*, cmd, output, log=None)

Create a shell command expression.

Called from inside a @task(kind="shell") body with fully resolved argument values. The cmd is a standard Python f-string — all variables are concrete at the point this is called.

Parameters:

• cmd (str) – The shell command to run.

• output (str | list[str] | tuple[str, ...]) – The expected output path or paths.

• log (str | None) – Optional path to capture stdout/stderr.

Return type:

ShellDirective

ginkgo.slug(value)

Return a deterministic file-safe slug.

Parameters:

value (str) – Input text to normalize.

Returns:

Lowercased slug with non-alphanumeric runs collapsed to underscores.

Return type:

str

ginkgo.subworkflow(path, *, params=None, config=None)

Create a sub-workflow invocation expression.

Called from inside a @task(kind="subworkflow") body with fully resolved argument values. The child workflow runs as a self-contained ginkgo run subprocess; its run_id is returned to the parent task, and ginkgo runs show <run_id> reads what it did.

Parameters:

• path (str or Path) – Path to the child workflow file.

• params (dict, optional) – Values for parameters the child declares with ginkgo.param. Passed as a [params] table in a temporary --config file, layering over the child’s own table, so a parameter not named here keeps the value the child’s config gives it.

• config (str, Path, or sequence of either, optional) – Additional --config paths to forward to the child.

Return type:

SubWorkflowDirective

ginkgo.table(payload, *, name=None, group=None, caption=None, metadata=None, checks=None)

Wrap a tabular value as an asset return.

Parameters:

• payload (Any) – The tabular value. Supports pandas DataFrame, polars DataFrame/LazyFrame, pyarrow Table/Dataset, DuckDB relation, or a path to a CSV/TSV file.

• name (str | None) – Optional explicit local asset name.

• group (str | None) – Optional report grouping label.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

• metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

• checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

ginkgo.task(_kind=None, /, *, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', threads=1, memory=None, gpu=0, gpu_type=None, memory_retry_multiplier=1.0, resources=None, remote=False, executor=None, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=None)

Decorator that turns a function into a lazy task definition.

The task kind may be given as the first positional argument or via the kind keyword. @task("shell"), @task("notebook"), and @task("script") are the preferred short forms.

Parameters:

• _kind (str | None) – Task kind as a positional argument. When provided, takes precedence over the kind keyword.

• env (str | None) – Foreign execution environment for shell tasks. If None, the task runs in the current environment.

• version (int) – Cache-busting version tag. Bump when task logic changes.

• retries (int) – Additional retry attempts after the initial execution.

• retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – Narrow retries to specific exception classes. None retries any failure.

• retry_backoff (float) – Base delay (seconds) before each retry. 0.0 disables the delay.

• retry_backoff_multiplier (float) – Exponential factor applied between attempts.

• retry_backoff_max (float) – Upper bound on the computed delay, in seconds.

• retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. Narrow retries to specific exit codes.

• priority (int) – Relative scheduling priority. Higher runs first among ready tasks. Range [-1000, 1000]; default 0.

• kind (str) – Execution contract for the task body. Ignored when _kind is given.

• threads (int) – Static CPU footprint for the scheduler. The task body receives the same value when its function signature declares a threads parameter; shell tasks also see GINKGO_THREADS=<n> in the subprocess environment.

• memory (str | None) – Static memory footprint for the scheduler in Kubernetes resource notation (e.g. "4Gi", "512Mi").

• gpu (int) – Number of GPUs the task requires. Locally this reserves against the --gpus budget; a requirement the local budget cannot satisfy is dispatched to the remote executor when one is configured, and is a build error otherwise. Remote executors map the count to the appropriate accelerator resource (e.g. nvidia.com/gpu on Kubernetes).

• gpu_type (str | None) – Accelerator type for remote execution (e.g. "nvidia-tesla-t4"). Overrides the executor-level default. Requires gpu > 0.

• memory_retry_multiplier (float) – Factor applied to memory on each retry attempt, capped at the run’s --memory budget. Use with retries for OOM-prone tasks (e.g. memory="16Gi", retries=2, memory_retry_multiplier=2 runs attempts at 16, 32, and 64 GiB). Requires memory.

• resources (dict[str, int] | None) – User-defined resource demands (e.g. {"api_calls": 2}), scheduled against run-level budgets from [resources.budgets] config or repeated --resource name=value flags. Dimensions without a configured budget are unconstrained. Counted wherever the task runs, including remote executors.

• remote (bool) – When True, dispatch this task to the run’s default executor — whichever one --executor names. Keeps the workflow portable across sites; use executor= to pin a specific one.

• executor (str | None) – Name of a configured executor ([remote.executors.<name>]) to route this task to, e.g. executor="gpu-k8s". Implies remote dispatch and overrides the run default, so a mixed workflow can send training to a GPU cluster and everything else elsewhere. Unknown names fail at build time.

• export_thread_env (bool) – Export common BLAS/OpenMP thread environment variables (OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS) to shell-task subprocesses. Default False.

• remote_input_access (str | None)

• streaming_compatible (bool)

• fuse_prefetch (dict[str, str] | None)

Returns:

A decorator that wraps the function in a TaskDef.

Return type:

Callable

Raises:

ValueError – If both a positional kind and a non-default kind keyword are supplied and they differ.

ginkgo.text(payload, *, name=None, group=None, caption=None, format=None, metadata=None, checks=None)

Wrap a text or structured document value as an asset return.

Parameters:

• payload (Any) – The document value. Supports strings, dicts, or paths. Dicts are serialised as JSON; strings are stored as the requested format.

• name (str | None) – Optional explicit local asset name.

• group (str | None) – Optional report grouping label.

• caption (str | None) – Optional human-readable annotation shown in reports and asset inspection output.

• format ({"plain", "markdown", "json"} | None) – Document format. Auto-detected from the payload when omitted.

• metadata (dict[str, Any] | None) – Optional user-defined metadata stored with the asset version.

• checks (Iterable[Callable[[Any], bool]] | None) – Optional checks run against the payload during asset registration.

Return type:

AssetResult

class ginkgo.tmp_dir

A ginkgo-managed scratch directory, unique per task execution.

Created automatically before task execution and deleted on success. Kept on failure for debugging. Does not participate in the cache key.

ginkgo.zip_expand(template, **wildcards)

Expand a string template by zipping wildcard values positionally.

Like expand(), the result is a row-aligned column for .map(), not an axis for .product_map().

Parameters:

• template (str) – Template containing named str.format placeholders.

• **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic positional order.

Return type:

ExpandedTemplate

Flow API

The @flow decorator for marking pipeline entry points.

A @flow-decorated function is the entry point for a workflow. When called, it executes its body to build the expression tree, then returns the resulting Expr or ExprList. No task execution happens during this phase.

class ginkgo.core.flow.FlowDef

A wrapper around a flow function.

Calling a FlowDef executes the flow body (building the expression tree) and returns whatever the flow function returns.

Parameters:

fn (Callable) – The original flow function.

property name: str

Fully qualified name of the wrapped function.

__init__(fn)

Parameters:

fn (Callable[[...], Any])

Return type:

None

ginkgo.core.flow.flow(fn)

Decorator that marks a function as a flow (pipeline entry point).

Unlike @task(), @flow is used without parentheses.

Parameters:

fn (Callable) – The flow function.

Return type:

FlowDef

ginkgo.core.flow.discover_flow(module)

Return the unique @flow-decorated function defined in module.

Raises RuntimeError when module contains zero or more than one FlowDef; identification is by Python object identity, so a single FlowDef re-exported under multiple names still counts as one.

Parameters:

module (ModuleType)

Return type:

FlowDef

Task API

The @task decorator and supporting classes.

A @task()-decorated function does not execute when called. Instead it returns an Expr[T] (full call) or a PartialCall (subset of required arguments), enabling lazy expression tree construction.

class ginkgo.core.task.TaskDef

Wraps a user function so that calls produce expression nodes.

Parameters:

• fn (Callable) – The original user function.

• env (str | None) – Foreign execution environment for shell tasks.

• version (int) – Cache-busting version tag.

• retries (int) – Additional retry attempts after the initial execution.

• retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – When set, only retry failures matching these exception classes. None (default) retries every failure up to retries.

• retry_backoff (float) – Base delay in seconds between retry attempts. 0.0 (default) reruns immediately.

• retry_backoff_multiplier (float) – Exponential factor applied to the base delay. Delay for attempt k (1-indexed) is retry_backoff * retry_backoff_multiplier ** (k - 1), capped at retry_backoff_max.

• retry_backoff_max (float) – Upper bound on the computed retry delay, in seconds.

• retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. When set, only retry failures whose exit code is in this tuple. Ignored for non-shell tasks.

• priority (int) – Relative scheduling priority. When several tasks are ready at the same time, higher-priority tasks are dispatched first. Range is [-1000, 1000]; default 0.

• kind (str) – Execution contract for the task body.

• resources (Resources) – Declarative resource requirements (threads, memory, gpu, gpu_type). States what the task needs; placement is decided separately by the evaluator. threads is made available to the task body when the function signature declares a threads parameter, and shell tasks receive GINKGO_THREADS=<n> in the subprocess environment.

• remote (bool) – When True, dispatch this task to the run’s default executor — the one named by --executor. Requires that flag to be set.

• executor (str | None) – Name of a configured executor ([remote.executors.<name>]) this task always routes to, regardless of the run’s default. Implies remote dispatch.

• export_thread_env (bool) – When True, shell tasks additionally receive OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, and NUMEXPR_NUM_THREADS set to the declared thread count. Default is False so existing tool configuration is not silently overridden.

property name: str

Fully qualified name of the wrapped function.

property required_params: frozenset[str]

Parameter names that have no default value.

property execution_mode: str

Return whether the task body runs on the driver or a worker.

property all_params: dict[str, Parameter]

All parameters from the function signature.

property signature: Signature

The inspected function signature.

property type_hints: dict[str, Any]

Resolved runtime type hints for the wrapped function.

property source_hash: str

BLAKE3 digest of task source and local imported modules.

property threads: int

Declared CPU footprint (see Resources).

property memory: str | None

Declared memory footprint string (see Resources).

property memory_gb: int

Parsed memory footprint in whole GiB (0 when unset).

property gpu: int

Declared GPU count (see Resources).

property gpu_type: str | None

Declared accelerator type (see Resources).

property cache_source_hash: str

Digest used for cache invalidation.

For notebook and script tasks, the source file hash is incorporated at execution time via the NotebookDirective/ScriptDirective.

should_retry_exception(*, exc)

Return whether exc matches the configured retry policy.

Parameters:

exc (BaseException)

Return type:

bool

retry_delay_seconds(*, attempt)

Return the retry delay to wait before attempt (1-indexed).

Parameters:

attempt (int)

Return type:

float

__init__(fn, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', resources=<factory>, remote=False, executor=None, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=())

Parameters:

• fn (Callable[[...], Any])

• env (str | None)

• version (int)

• retries (int)

• retry_on (type[BaseException] | tuple[type[BaseException], ...] | None)

• retry_backoff (float)

• retry_backoff_multiplier (float)

• retry_backoff_max (float)

• retry_on_exit_codes (tuple[int, ...] | None)

• priority (int)

• kind (str)

• resources (Resources)

• remote (bool)

• executor (str | None)

• export_thread_env (bool)

• remote_input_access (str | None)

• streaming_compatible (bool)

• fuse_prefetch (tuple[tuple[str, str], ...])

Return type:

None

class ginkgo.core.task.PartialCall

A partially applied task call, awaiting .map() for remaining args.

Parameters:

• task_def (TaskDef) – The task definition.

• fixed_args (dict[str, object]) – Arguments already supplied.

map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per element by zipping varying columns.

All varying argument columns must be the same length.

Parameters:

• max_concurrent (int | None) – When set, the scheduler will run at most this many generated branches concurrently, independently of --jobs and --cores limits. Use this to throttle classes of work that should not run in parallel (e.g. model training).

• **varying – Keyword arguments where each value is an iterable (list, Series, or ExprList) of per-element values, or a per_branch() template rendered from each row’s own values.

Returns:

One Expr per element in the varying columns.

Return type:

ExprList

Raises:

• ValueError – If varying columns have different lengths or no varying args given.

• TypeError – If a varying argument name is not a valid parameter.

product_map(*, max_concurrent=None, **varying)

Fan-out: produce one Expr per Cartesian combination.

Every varying list is an axis of the grid. Arguments that are a function of the grid cell — output paths above all — must be passed as per_branch("...{arg}...") templates, which are rendered from each cell’s own values. expand() output is rejected here: it is already one value per cell, so crossing it with the axes it came from would mislabel every branch.

Parameters:

• max_concurrent (int | None)

• varying (Any)

Return type:

ExprList

__init__(task_def, fixed_args=<factory>)

Parameters:

• task_def (TaskDef)

• fixed_args (dict[str, object])

Return type:

None

ginkgo.core.task.task(_kind=None, /, *, env=None, version=1, retries=0, retry_on=None, retry_backoff=0.0, retry_backoff_multiplier=2.0, retry_backoff_max=60.0, retry_on_exit_codes=None, priority=0, kind='python', threads=1, memory=None, gpu=0, gpu_type=None, memory_retry_multiplier=1.0, resources=None, remote=False, executor=None, export_thread_env=False, remote_input_access=None, streaming_compatible=True, fuse_prefetch=None)

Decorator that turns a function into a lazy task definition.

The task kind may be given as the first positional argument or via the kind keyword. @task("shell"), @task("notebook"), and @task("script") are the preferred short forms.

Parameters:

• _kind (str | None) – Task kind as a positional argument. When provided, takes precedence over the kind keyword.

• env (str | None) – Foreign execution environment for shell tasks. If None, the task runs in the current environment.

• version (int) – Cache-busting version tag. Bump when task logic changes.

• retries (int) – Additional retry attempts after the initial execution.

• retry_on (type[BaseException] | tuple[type[BaseException], ...] | None) – Narrow retries to specific exception classes. None retries any failure.

• retry_backoff (float) – Base delay (seconds) before each retry. 0.0 disables the delay.

• retry_backoff_multiplier (float) – Exponential factor applied between attempts.

• retry_backoff_max (float) – Upper bound on the computed delay, in seconds.

• retry_on_exit_codes (tuple[int, ...] | None) – Shell-task only. Narrow retries to specific exit codes.

• priority (int) – Relative scheduling priority. Higher runs first among ready tasks. Range [-1000, 1000]; default 0.

• kind (str) – Execution contract for the task body. Ignored when _kind is given.

• threads (int) – Static CPU footprint for the scheduler. The task body receives the same value when its function signature declares a threads parameter; shell tasks also see GINKGO_THREADS=<n> in the subprocess environment.

• memory (str | None) – Static memory footprint for the scheduler in Kubernetes resource notation (e.g. "4Gi", "512Mi").

• gpu (int) – Number of GPUs the task requires. Locally this reserves against the --gpus budget; a requirement the local budget cannot satisfy is dispatched to the remote executor when one is configured, and is a build error otherwise. Remote executors map the count to the appropriate accelerator resource (e.g. nvidia.com/gpu on Kubernetes).

• gpu_type (str | None) – Accelerator type for remote execution (e.g. "nvidia-tesla-t4"). Overrides the executor-level default. Requires gpu > 0.

• memory_retry_multiplier (float) – Factor applied to memory on each retry attempt, capped at the run’s --memory budget. Use with retries for OOM-prone tasks (e.g. memory="16Gi", retries=2, memory_retry_multiplier=2 runs attempts at 16, 32, and 64 GiB). Requires memory.

• resources (dict[str, int] | None) – User-defined resource demands (e.g. {"api_calls": 2}), scheduled against run-level budgets from [resources.budgets] config or repeated --resource name=value flags. Dimensions without a configured budget are unconstrained. Counted wherever the task runs, including remote executors.

• remote (bool) – When True, dispatch this task to the run’s default executor — whichever one --executor names. Keeps the workflow portable across sites; use executor= to pin a specific one.

• executor (str | None) – Name of a configured executor ([remote.executors.<name>]) to route this task to, e.g. executor="gpu-k8s". Implies remote dispatch and overrides the run default, so a mixed workflow can send training to a GPU cluster and everything else elsewhere. Unknown names fail at build time.

• export_thread_env (bool) – Export common BLAS/OpenMP thread environment variables (OMP_NUM_THREADS, MKL_NUM_THREADS, OPENBLAS_NUM_THREADS, NUMEXPR_NUM_THREADS) to shell-task subprocesses. Default False.

• remote_input_access (str | None)

• streaming_compatible (bool)

• fuse_prefetch (dict[str, str] | None)

Returns:

A decorator that wraps the function in a TaskDef.

Return type:

Callable

Raises:

ValueError – If both a positional kind and a non-default kind keyword are supplied and they differ.

Shell API

Shell task execution primitive.

shell() is called from inside a @task(kind="shell") body and returns a ShellDirective. The evaluator detects this and dispatches the command to the configured shell runner.

final class ginkgo.core.shell.ShellDirective

Execution directive representing a shell command to execute.

Parameters:

• cmd (str) – The shell command (already interpolated with resolved values).

• output (str | list[str] | tuple[str, ...]) – Expected output path or paths. Used for cache checking and post- execution validation.

• log (str | None) – Optional path to capture stdout/stderr.

__init__(cmd, output, log=None)

Parameters:

• cmd (str)

• output (str | AssetResult | OptionalOutput | list[str | AssetResult | OptionalOutput] | tuple[str | AssetResult | OptionalOutput, ...])

• log (str | None)

Return type:

None

ginkgo.core.shell.shell(*, cmd, output, log=None)

Create a shell command expression.

Called from inside a @task(kind="shell") body with fully resolved argument values. The cmd is a standard Python f-string — all variables are concrete at the point this is called.

Parameters:

• cmd (str) – The shell command to run.

• output (str | list[str] | tuple[str, ...]) – The expected output path or paths.

• log (str | None) – Optional path to capture stdout/stderr.

Return type:

ShellDirective

Authoring Helpers

Template expansion and wildcard utilities for workflow authoring.

class ginkgo.wildcards.ExpandedTemplate

Expanded template strings that remember the template they came from.

expand() and zip_expand() return one string per wildcard combination, so the result is a column already aligned row-for-row with the values it was built from — never an independent axis to sweep. The list behaves exactly like list[str]; remembering the template lets .product_map() reject it by name instead of silently crossing it with the axes it was derived from.

The subclass must never reach yaml.safe_dump directly — the safe dumper represents only exact built-in types and raises on this one. Every code path that serialises task arguments already normalises sequences into plain lists first, and tests/core/test_helpers.py holds that normalisation in place.

__init__(values, *, template, function_name, placeholders)

Parameters:

• values (Iterable[str])

• template (str)

• function_name (str)

• placeholders (Sequence[str])

Return type:

None

unresolved_placeholders(names)

Return the placeholders that do not name one of names.

Resolution is by name, never by position: a template can be reused verbatim as a per_branch() template exactly when every placeholder already names an argument of the call. Positional correspondence between wildcards and arguments is the assumption that produced the mislabelling this type exists to prevent, so it is not assumed here either.

Parameters:

names (Sequence[str])

Return type:

tuple[str, …]

class ginkgo.wildcards.PerBranch

A template rendered once per fan-out branch from that branch’s values.

Parameters:

template (str) – Template whose placeholders name arguments of the fan-out call.

placeholder_names()

Return the argument names this template reads, in first-use order.

Return type:

list[str]

render(values)

Render the template from one branch’s argument values.

Parameters:

values (dict[str, Any])

Return type:

str

__init__(template)

Parameters:

template (str)

Return type:

None

ginkgo.wildcards.per_branch(template)

Derive one value per fan-out branch from that branch’s own arguments.

Use this for arguments that are a function of the branch — output paths above all — rather than an axis to sweep. Placeholders name other arguments of the same .map() / .product_map() call (or arguments fixed on the task call), and are rendered per branch, so the value can never drift out of step with the values it describes.

Parameters:

template (str) – Template containing named str.format placeholders, each naming an argument of the fan-out call.

Returns:

Marker consumed by .map() and .product_map().

Return type:

PerBranch

Examples

>>> per_branch("results/{temperature}_{defect_density}.json").template
'results/{temperature}_{defect_density}.json'

ginkgo.wildcards.expand(template, **wildcards)

Expand a string template across wildcard combinations.

The result is one string per combination, aligned row-for-row with the wildcard values, so it pairs with .map(). It is not an axis: passing it to .product_map() is rejected, since that would cross it with the very axes it was derived from. For a grid, use per_branch() instead.

Parameters:

• template (str) – Template containing named str.format placeholders.

• **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic Cartesian-product order.

Return type:

ExpandedTemplate

ginkgo.wildcards.zip_expand(template, **wildcards)

Expand a string template by zipping wildcard values positionally.

Like expand(), the result is a row-aligned column for .map(), not an axis for .product_map().

Parameters:

• template (str) – Template containing named str.format placeholders.

• **wildcards (collections.abc.Iterable[Any]) – Iterable values for each placeholder in template.

Returns:

Expanded strings in deterministic positional order.

Return type:

ExpandedTemplate

ginkgo.wildcards.slug(value)

Return a deterministic file-safe slug.

Parameters:

value (str) – Input text to normalize.

Returns:

Lowercased slug with non-alphanumeric runs collapsed to underscores.

Return type:

str

ginkgo.wildcards.flatten(items)

Flatten nested lists and tuples into a single list.

Parameters:

items (list[Any] | tuple[Any, ...]) – Nested list or tuple structure.

Returns:

Flat list preserving left-to-right order.

Return type:

list[Any]

Discovery of the project root directory.

A ginkgo project is rooted at the directory holding its configuration file (ginkgo.toml, or the YAML equivalents). Workflow files, .ginkgo/ runtime state, and config overrides are all located relative to that directory, so where it is needs one answer rather than one per caller.

find_project_root() is that answer: walk upward from a starting directory to the nearest project marker. project_root() is the user-facing form, walking up from the current working directory, so a workflow can name a path relative to the project rather than to wherever ginkgo happened to be invoked from.

ginkgo.project.find_project_root(start_dir)

Walk upward from start_dir to the nearest project root.

Parameters:

start_dir (Path) – Directory to start from. Searched before its parents, so a directory that is itself a project root is returned unchanged.

Returns:

The nearest ancestor of start_dir (inclusive) holding one of PROJECT_CONFIG_NAMES, or None if there is no such directory up to the filesystem root.

Return type:

Path | None

ginkgo.project.project_root()

Return the root directory of the project containing the current directory.

Walks upward from the current working directory to the nearest directory holding a ginkgo configuration file, so this resolves to the same directory whether a workflow is run from the project root or from a subdirectory such as workflow/.

The starting point is deliberately the working directory rather than the calling module’s location: a result that depended on which file asked would be harder to predict than one that depends on where the command was run.

Returns:

An absolute path to the project root. A project configuration file is optional, so when no marker is found this falls back to the current working directory — the same directory the rest of ginkgo already treats as the project root.

Return type:

Path