Usernetes¶
The usernetes module deploys user space Kubernetes
across the nodes of a batch job. The per-node work (rootless podman storage,
image builds, kubeadm) is done by the service scripts of a usernetes checkout,
under its service/ directory. flux-batch orchestrates around them: which node
starts first, how workers find the join command, when your work may start, and
how nodes leave.
Modes¶
Cluster mode (the default) builds a new cluster from the whole job.
Rank 0 starts the control plane.
Every other rank starts a worker, which waits for this job’s join command.
Once the expected number of nodes has registered, rank 0 installs the CNI and syncs the external IPs of the nodes.
When every node is Ready, the batch wrapper runs your jobs, with
KUBECONFIGexported.
Worker mode bursts more nodes into a cluster that is already running. Every rank of the job, rank 0 included, starts a worker that joins the existing cluster named by cluster_name. The control plane of the original job notices the new nodes, re-syncs their external IPs, and restarts calico-node on any node that picked up the wrong address. When the burst job ends, its nodes are deleted from the cluster.
Deploy a cluster¶
Request an allocation with rabbit storage, then ssh to its first node:
flux alloc -q pall -N2 --setattr=dw="#DW jobdw capacity=400GiB type=xfs name=testfs" --bg -t 4h
flux jobs -a # note the job id and the first node in its node list
ssh <first node>
Note that we still need to ssh in because systemd with --user is broken. When flux shell can set that up, we don’t need to do that. On the node, connect to the allocation and launch:
# Submit into the allocation, not the system instance
# You can also flux proxy to the local:// socket path
export FLUX_URI=$(flux uri --local <jobid>)
flux resource list # should show the allocation's 2 nodes
You can target a usernetes clone of your choice. This is a Usernetes checkout visible on every node, this is my development branch that uses the rabbit for container storage (podman and containerd within). See “hetchy” on https://github.com/converged-computing/usernetes.
export USERNETES_ROOT=/usr/workspace/usernetes/usernetes-wip
# Quick
cd flux-batch
flux python examples/usernetes_module.py --nodes 2 --cluster-name demo
The cluster name is important as it will share the join key in your ~/.usernetes/demo/join-command for other jobs to find. You can tweak the number of nodes as you see fit. The example adds the module like this:
spec = flux_batch.BatchJobspecV1.from_jobs(
batch, nodes=2, nslots=2, job_name="usernetes-cluster"
)
spec.add_module("usernetes", cluster_name="demo")
The cluster lives only as long as the batch job. When the last job finishes, the instance shuts down and every node runs make down-v. The cluster stays up until you flux cancel the job or the allocation ends. To use the cluster from a shell on the control plane node:
cd /tmp/$USER/usernetes && . source_env.sh
kubectl get nodes
Burst workers into it¶
If we want to join nodes from different jobs, we can use bursting. While the cluster job is running, either from the same allocation (or a different one) again ssh to its first node, and connect to it:
export FLUX_URI=$(flux uri --local <second jobid>)
export USERNETES_ROOT=/usr/workspace/usernetes/usernetes-wip
cd flux-batch
flux python examples/usernetes_burst.py --nodes 1 --cluster-name demo
It will find the cluster join-command in the expected named path in your $HOME and use it to join to the cluster. This is how it does that - it adds the module, but joins as a worker node.
spec.add_module("usernetes", mode="worker", cluster_name="demo")
To try a burst without a second allocation, split one 2-node allocation into two 1-node jobs. The module always requests whole nodes (flux batch --exclusive), so each job gets its own node:
# control plane on one node
flux python examples/usernetes_module.py --nodes 1
# once kubectl get nodes shows it Ready, this is a worker on the other node
flux python examples/usernetes_burst.py --nodes 1
I have not yet tested this directly from a system instance, but I think it should work. Ideally we can do this without needing ssh, eventually. The cluster_name must match the running cluster. A burst worker waits until that cluster is ready, and fails after wait_timeout seconds if it never is.
Attributes¶
Pass these to add_module("usernetes", ...).
Attribute |
Default |
Description |
|---|---|---|
|
|
|
|
|
Names the cluster; a burst job joins by this name. |
|
|
Usernetes checkout whose |
|
|
Shared filesystem directory that holds the state of each cluster. |
|
|
Seconds to wait for each phase: join command, registration, Ready. |
|
|
Rootless container engine. |
|
|
Where launcher and service logs are written on each node. |
|
|
Launch through |
|
|
Extra environment for the usernetes service scripts, such as |
Where things are¶
You can always look at logs on each node (control plane or worker) to see what is going on.
What |
Where |
|---|---|
Cluster state: join command, launch id, |
|
Launcher logs |
|
Usernetes service logs |
|
Working copy and |
|
Rendered modprobe scripts |
|
How it stays safe across jobs¶
No stale join commands. Each submission gets a launch id, rendered into both the control plane and worker commands. The control plane clears the cluster directory and then writes the id; workers ignore any join command until they see their own id, so leftovers from an earlier cluster with the same name are never used.
No forgotten nodes. A worker that stops leaves a marker in
leaving/, and the control plane deletes that node from the cluster. Technically, we could skip this step and the node would register asNotReadybut I prefer the cleanup.One usernetes node per host. Two usernetes nodes on one host would share the working copy and podman state and tear each other down. The module submits with exclusive nodes, and each launcher holds a host lock (
/tmp/$USER/.flux-batch-usernetes.lock). A second launcher on a busy host exits with an error without touching the running node. We obviously use--exclusive. If we want to get wild and crazy we could test without and have multiple usernetes installs, but I don’t see the use case for that.Clean shutdown. When the job ends, each node runs
make down-v, and the control plane removes thereadysentinel so anything still waiting fails quickly instead of hanging.
Troubleshooting¶
Nothing happens on a node. Read the launcher log first; it names the phase it is waiting on and prints the tail of the service log if the service script died.
Workers wait forever. Check that
~/.usernetes/clusters/<name>/launch-idexists and that the control plane launcher log shows the join command was written.systemd not running? file not found: .../user@<uid>.service/cgroup.controllersin the service log: the node has no systemd user session. Keepuse_ssh=True(the default) and check thatssh $(hostname) trueworks without a password on every node.Pods on different nodes cannot reach each other. Run
<usernetes_root>/service/check-calico.sh --fixon the control plane.