Dynamic Scheduling Boosts Hybrid quantum-classical computing
Dynamic scheduling boosts hybrid quantum-classical computing efficiency.
European Researchers Improve Supercomputing-Quantum Integration by Addressing Resource Idle Time.
European researchers are investigating new methods to maximise supercomputer-quantum processor cooperation to overcome a major barrier to hybrid computing systems. A recent study found that dynamic scheduling, specifically “malleability scheduling,” can reduce idle time and speed up project completion in complex environments. Since quantum tasks are longer than conventional ones, this invention is crucial for sharing classical and quantum resources.
Hybrid Systems and Uneven Resources Challenge
Quantum computers may accelerate highly specialised computational workloads on classical HPC systems. Quantum processing units (QPUs) may help with optimisation, materials modelling, and quantum simulation. Merging these computational models is hindered by inefficient resource allocation.
Presently, hybrid computing jobs may reserve both a QPU and conventional HPC nodes, even while only one is being used. Due to CPU core idleness during quantum processes and vice versa, a lot of expensive hardware is wasted. QPUs are scarce, perhaps only one or two in a cluster, increasing the possibility of bottlenecks and wasted hardware.
Smart Solution: Dynamic Scheduling
Researchers recommend dynamic scheduling over static resource allocation to address this problem. Unlike static methods that assign resources for the length of a work, dynamic scheduling frees classical resources when quantum activities are offloaded and precisely reallocates them as needed. This strategy increases hybrid algorithm execution throughput, reduces idle time, and boosts task completion.
European study team members from E4 Computer Engineering, LINKS Foundation, Barcelona Supercomputing Centre, CINECA, and other universities examined two primary dynamic scheduling algorithms. These methods were workflow- and malleability-based.
Organising Task Dependencies: Workflow Management
Workflow management systems (WMS) break down complex operations into smaller, interrelated tasks. After then, resources are scheduled as needed for each task. By modelling a hybrid application as a three-step loop with StreamFlow WMS, the team tested parallel classical algorithms, quantum routine aggregation, and quality evaluation. This method frees up HPC nodes by only using quantum resources when needed.
Flexibility: Resize
Malleability offers a different yet effective dynamic scheduling method. This approach allows HPC workload resizing during quantum phases without re-queuing. Software can dynamically adjust the number of compute nodes it uses during runtime. In quantum computation, the researchers used the Dynamic Management of Resources (DMR) framework to lower the HPC footprint while keeping a minimum process. The footprint may rise again when classical computation resumes. This approach avoids re-queuing and frees up cores for other operations.
Experimental Hybrid Clustering Validation
To compare techniques, the researchers changed a clustering aggregation algorithm into a hybrid HPC-QC application. K-means, DBSCAN, and hierarchical clustering were run on HPC nodes during the classical phase. After graphing their outputs, the problem became a quadratic unconstrained binary optimisation (QUBO) problem and was resolved in the quantum phase.
Since their testbed lacked a quantum processing unit (QPU), simulated annealing was done using a “quantum emulator” node, which simulates quantum technology runtimes by adding customisable delays. Three setups were tested on a small SLURM-managed cluster:
Baseline: HPC and quantum resources are static throughout the job.
Workflow: Task-based scheduling that only uses quantum resources when needed.
Malleability: HPC allocation's quantum-phase dynamic resizing.
Key Findings and Implications
Dynamic scheduling's benefits became clear from experiments. For two-minute quantum phase simulations (like neutral-atom devices):
Despite using the least HPC node time, the workflow strategy took the longest to complete because to the scheduler's constant resource requests.
The baseline static allocation was the fastest for a single run but the least effective and idled resources.
Malleability saved resources without workflow scheduling delays.
When two jobs were done simultaneously, benefits rose. Both operations took longer and consumed more resources due to the baseline technique. However, workflow and malleability improved overlap. Malleability allowed computation to restart immediately after the quantum phase, even if not all HPC nodes were online. Finishing times dropped considerably. Malleability helps manage concurrent processes even during brief quantum phases (less than one second).
Static scheduling will not work well for hybrid workloads, especially when quantum processes are longer than conventional ones, the study found. Dynamic computing node reallocation in resource-constrained clusters could dramatically boost HPC gear consumption and reduce queue wait times. Workflow and malleability have pros and cons. Workflow systems require modular application design, yet malleability makes program state management with different resources harder, despite being easier to implement into present code.
Future Directions and “Maturity Gap”
Even though the tests were limited to simulated quantum workloads on a tiny test cluster, the discovery shows that hardware developments alone won't improve real-world performance until software and scheduling methods change. The researchers recommend more workloads, real conflict circumstances, and larger quantum experiments.
The “maturity gap” between HPC and quantum computing technologies makes orchestration difficult. Researchers can start to shrink this gap by applying techniques from decades of supercomputing research, such as quantum computing, to variable jobs. These dynamic scheduling solutions may let national supercomputing centres use petascale or exascale computers with quantum processors to run machines close to capacity without wasting time.
