Batch jobs¶
A batch job is built in three steps: collect the commands, wrap them into a jobspec with resources and options, then preview or submit it.
Build and submit¶
import flux
import flux_batch
handle = flux.Flux()
# 1. Commands to run inside the batch job
batch = flux_batch.BatchJobV1()
batch.add_job(["echo", "Job 1 starting"])
batch.add_job(["sleep", "5"])
batch.add_job(["echo", "Job 2 finished"])
# 2. Resources and options (these mirror the flux batch flags)
spec = flux_batch.BatchJobspecV1.from_jobs(
batch,
nodes=1,
nslots=1,
time_limit="10m",
job_name="test-batch",
)
# Commands to run before and after the work
spec.add_prolog("echo 'Batch Wrapper Starting'")
spec.add_epilog("echo 'Batch Wrapper Finished'")
# 3. Preview the wrapper script, inspect the jobspec, then submit
print(flux_batch.submit(handle, spec, dry_run=True))
jobspec = flux_batch.jobspec(spec)
jobid = flux_batch.submit(handle, spec)
With more than one job, each is run with flux submit --wait and the wrapper
ends with flux job wait --all. A single command runs as-is. To run one
command directly, use BatchJobspecV1.from_command(["hostname"], nodes=1).
The preview above prints the generated wrapper:
#!/bin/bash
echo 'Batch Wrapper Starting'
flux submit --wait /bin/echo 'Job 1 starting'
flux submit --wait /bin/sleep 5
flux submit --wait /bin/echo 'Job 2 finished'
flux job wait --all
echo 'Batch Wrapper Finished'
What goes into the wrapper¶
The wrapper script is assembled in this order:
Prologs (
add_prolog)Service start commands (
add_service)Module gates, for example waiting for a cluster to be ready (
add_module)Your jobs or command
Module and service teardown, in reverse order
Epilogs (
add_epilog)Log saving, if
logs_diris set
Job attributes¶
Keyword arguments to from_jobs and from_command become
flux_batch.models.BatchAttributesV1 fields and map onto flux batch
flags:
Attribute |
Flag |
Attribute |
Flag |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
These take lists and repeat the flag once per value: setopt (-o),
setattr (-S), add_file, env, env_remove, env_file, rlimit, and
conf.
Saving logs¶
Set logs_dir and the wrapper saves the output, jobspec and event log of
every job in the instance to that directory before it exits:
spec = flux_batch.BatchJobspecV1.from_jobs(
batch, nodes=1, time_limit="10m", logs_dir="./logs"
)
See examples/save_logs.py in the repository.