Python API¶
Building jobs¶
- class flux_batch.BatchJobV1(jobs=<factory>)[source]¶
Commands to be run within the batch wrapper.
This should mirror JobspecV1
- Parameters:
jobs (List[str])
- class flux_batch.BatchJobspecV1(attributes=None)[source]¶
A BatchJobspecV1 mirrors a JobspecV1. We need to:
Add some number of commands or a script
Add optional services (start/stop)
- Parameters:
attributes (BatchAttributesV1)
- classmethod from_jobs(batch, **kwargs)[source]¶
Generate the batch script from a set of jobs.
With more than one job, we assume we are waiting.
- Parameters:
batch (BatchJobV1)
- add_service(name, **kwargs)[source]¶
Add a service by name. Services can be started one off (a la-carte) separate from any kind of proper flux module. Keyword arguments become service attributes, on top of the batch attributes.
- Parameters:
name (str)
- add_module(name, **kwargs)[source]¶
A module typically has a service name. E.g., flux-scribe is added as a module, and the module starts a service. Keyword arguments become service attributes, e.g. add_module(“usernetes”, mode=”worker”, cluster_name=”lammps”).
- Parameters:
name (str)
- class flux_batch.models.BatchAttributesV1(logs_dir=None, container_tech=None, nslots=None, cores_per_slot=None, gpus_per_slot=None, nodes=None, exclusive=False, bank=None, queue=None, time_limit=None, urgency=None, job_name=None, cwd=None, setopt=<factory>, setattr=<factory>, add_file=<factory>, env=<factory>, env_remove=<factory>, env_file=<factory>, rlimit=<factory>, conf=<factory>, dependency=None, requires=None, begin_time=None, signal=None, broker_opts=None, dump=None, unbuffered=False, wrap=False, flags=None)[source]¶
Explicitly defined arguments allowed by flux batch for V1 spec
- Parameters:
logs_dir (bool | str | None)
container_tech (str | None)
nslots (int | None)
cores_per_slot (int | None)
gpus_per_slot (int | None)
nodes (int | None)
exclusive (bool)
bank (str | None)
queue (str | None)
time_limit (str | None)
urgency (int | None)
job_name (str | None)
cwd (str | None)
setopt (List[str])
setattr (List[str])
add_file (List[str])
env (List[str])
env_remove (List[str])
env_file (List[str])
rlimit (List[str])
conf (List[str])
dependency (str | None)
requires (str | None)
begin_time (str | None)
signal (str | None)
broker_opts (str | None)
dump (str | None)
unbuffered (bool)
wrap (bool)
flags (str | None)
Submitting¶
- flux_batch.submit(handle, spec, dry_run=False)[source]¶
Orchestrates the submission process: 1. Provisions any required user-space services. 2. Generates the wrapper shell script. 3. Uses ‘flux batch –dryrun’ to compile the Jobspec JSON. 4. Submits the Jobspec to the Flux instance.
- Parameters:
handle (flux.Flux)
spec (BatchJobV1)
- Return type:
int
- flux_batch.jobspec(spec)¶
Preview the jobspec.
- Parameters:
spec (BatchJobV1)
- Return type:
int
Modules¶
- class flux_batch.service.usernetes.service.UsernetesService(*args, **kwargs)[source]¶
Deploy user space Kubernetes (usernetes, hetchy branch) under a batch job.
Attributes are set on the jobspec attributes or passed to add_module:
- usernetes_root
Usernetes checkout whose service/ scripts do the per-node work (default: $USERNETES_ROOT in the submission environment, then /usr/workspace/usernetes/usernetes-wip).
- mode
“cluster” (default) or “worker” (burst: all ranks join an existing cluster).
- cluster_name
Name of the cluster; a burst job must use the name of the cluster it joins (default: “default”).
- shared_root
Shared-filesystem directory holding per-cluster state (default: ${HOME}/.usernetes/clusters).
- wait_timeout
Seconds to wait for each phase (default: 3600).
- container_tech
Rootless engine (default: podman).
- logs_dir
Directory for launcher/service logs on each node (default: /tmp/$USER).
- use_ssh
Launch through ‘ssh $(hostname) …’ so the node has a systemd user session, which the usernetes preflight check requires and tasks started by the broker lack (default: True). Needs passwordless ssh between nodes.
- usernetes_env
Dict of extra environment passed through to the usernetes service scripts (USERNETES_STORAGE_ROOT, USERNETES_CNI, CALICO_VXLAN_IIFNAME).