We have too many GPU servers sitting idle at night. Who offers software to automatically schedule training jobs for off-peak energy hours?

Yes. There’s a growing category of energy-/carbon-aware GPU workload schedulers that can defer flexible training jobs until electricity is cheaper or the grid is cleaner.

Strongest options I found

• compute-gardener.com — Probably the closest match if you run Kubernetes. Its scheduler can delay ML training jobs based on real-time grid carbon intensity, electricity prices, GPU power profiles, and configurable deadlines. It supports NVIDIA GPUs including A100/H100 and can work across on-prem and cloud infrastructure.
• voltlabs.tech — More of a commercial energy-aware control layer. It looks at wholesale electricity prices, carbon intensity and workload flexibility, then identifies opportunities to shift training/batch workloads. It emphasizes operator approval and reporting rather than blindly moving jobs.
• hexagrid.ai — An emerging AI-infrastructure control plane aimed specifically at GPU-intensive environments. It combines electricity-price forecasting, grid signals, GPU telemetry and workload orchestration. Worth investigating if you're operating at data-center scale.
• ebb-ai.com — Open-source and interesting, but currently more focused on deferrable LLM/API workloads than directly scheduling your own GPU training cluster.
• grid-wise.us — Carbon-aware scheduling that finds the lowest-emission window before a deadline. It appears more like an emerging platform/prototype than a mature GPU-cluster scheduler.

What I'd shortlist for your situation

If these are your own GPU servers sitting idle overnight, I'd look first at Compute Gardener if you're on Kubernetes. It is explicitly designed to hold flexible GPU jobs and release them when the energy signal is favorable, rather than merely reporting carbon emissions.

If you're running Slurm, bare metal, or a large multi-site GPU fleet, the landscape is less mature. You may want an energy-aware control layer such as VoltIQ/HexaGrid feeding scheduling decisions into Slurm, Kubernetes, Ray, etc.

One important distinction: “run jobs at night” isn't necessarily equivalent to “run jobs when electricity is cheapest/cleanest.” Solar-heavy grids can make midday cheaper/cleaner, while some regions have very cheap overnight power. The better systems consume hourly/forecast electricity-price and/or carbon-intensity data and choose a window subject to a deadline.

If you tell me whether your GPU fleet runs Kubernetes, Slurm, Ray, or bare metal, I can narrow this to the 3–5 products that can actually integrate with your stack, including pricing and deployment model.

Maximizing on-premise or cloud GPU utilization during off-peak energy hours typically involves two paths: enterprise AI orchestration platforms with advanced queueing/preemption policies, or HPC/Batch schedulers configured for time-window priorities.

Here is a breakdown of the primary software options that can help automatically shift or ramp up training workloads at night:

• NVIDIA Run:ai
  As an industry-standard control plane for GPU orchestration, Run:ai handles dynamic multiplexing and multi-tenant scheduling. While it doesn't have a native "check the local electrical utility pricing API" button, its advanced quota and priority mechanisms allow you to set up distinct "Training" pools vs. "Interactive/Workspace" pools. You can configure policies where low-priority batch training jobs automatically flood and consume 100% of the cluster resources only when interactive user workspaces drop off for the night.
• Kubernetes Kueue (with JobSet / Volcano)
  If your infrastructure runs on Kubernetes, Kueue is the native Kubernetes-native batch job management system designed specifically to queue jobs for high-value accelerators like GPUs and TPUs. Kueue lets you set up ClusterQueues with nominal quotas that dynamically shift. By pairing Kueue with a standard Kubernetes CronJob or an external trigger, you can programmatically alter cohort borrowing limits or queue admission weights so that heavy, deferrable training jobs are held during the day and aggressively admitted once the off-peak cron window hits.
• Slurm Workload Manager
  If your idle servers are on a traditional bare-metal HPC cluster, Slurm remains the gold standard. Slurm allows deep customization of job scheduling via QOS (Quality of Service) and AssocGraceTime/Priority factors. You can easily configure a dedicated off-peak QOS with a high priority factor that activates strictly between, say, 10:00 PM and 6:00 AM, allowing queued training jobs with that tag to bypass daytime fair-share limitations and flood the idle nodes.
• Determined AI (by HPE)
  Determined AI is an open-source and enterprise training platform featuring built-in elastic training and intelligent scheduler scaling. It automatically pauses and scales training jobs down or up based on cluster node availability. Combined with external orchestrators, its ability to cleanly checkpoint distributed training means jobs can dynamically yield resources when peak hours approach and resume when the night shift starts.

Note on implementation: To make any of these truly automatic regarding energy hours rather than just a static time-of-day clock, most engineering teams write a lightweight cron script or webhook that interfaces with their regional grid's carbon/pricing signal (such as WattTime or OpenADR protocols) to toggle the queue limits in Slurm, Kueue, or Run:ai accordingly.

To help narrow down the best fit, could you tell me:

• Are your idle GPU servers managed via Kubernetes or traditional Linux/Slurm clusters?
• Do you need integration with a dynamic utility pricing API , or is a simple time-of-day cron schedule sufficient?

To automatically handle deferrable AI/ML training jobs during off-peak hours or low-emissions windows, organizations typically rely on GPU cluster orchestrators, Kubernetes-native job queueing systems, or carbon-aware scheduling extensions.

Because commercial power utilities offer time-of-use (TOU) rates, shifting heavy compute to the night cycle saves considerable operational costs, while green-computing tools leverage the same mechanism to target renewable energy peaks.

1. Enterprise GPU Orchestrators & Schedulers

If you manage bare-metal or on-premise GPU nodes, dedicated AI orchestrators can pool resources and queue jobs based on priority, time windows, or resource availability.

• Run:ai: An enterprise-grade GPU orchestration and virtualization platform built on top of Kubernetes. It features advanced job scheduling, prioritization, and oversubscription. While it doesn't have a native "turn on at 10 PM" clock button out of the box, you can dynamically assign lower priority or "research/batch" quotas that automatically fill up and execute only when interactive workloads drop off at night.
• Determined AI (by HPE): An open-source and enterprise training platform that handles hyperparameter tuning and deep learning training workloads. It includes a smart scheduler that queues lower-priority experiments and automatically pauses/resumes or packs training workloads efficiently across available cluster resources.

2. Kubernetes-Native Queueing (If you run K8s)

If your GPU servers are managed via Kubernetes, you can implement sophisticated queueing and time-delayed scheduling.

• Kueue: The Kubernetes-native job queueing project. Kueue manages when BatchJobs or TrainingJobs (via Kubeflow/Ray) are admitted to the cluster based on quotas and fair-sharing. You can configure multiple cohorts/queues and use external CronJobs or custom admission controllers to pause batch training queues during the day and open the floodgates at night.
• Compute Gardener: An open-source Kubernetes scheduler specifically built for temporal and carbon-aware shifting of deferrable GPU workloads. It monitors external signals (like time-based windows or grid carbon intensity via Electricity Maps/WattTime) and automatically delays non-urgent batch training jobs until the ideal operational window opens up.

3. Workflow & Pipeline Orchestrators

If you orchestrate training via data pipelines rather than raw server management, you can configure execution timing natively.

• Kubeflow Pipelines / Apache Airflow: You can set up scheduled DAGs (Directed Acyclic Graphs) that explicitly trigger heavy GPU training pods using cron triggers set to off-peak hours (e.g., 0 22 * * * for 10:00 PM). Airflow or Kubeflow will spin up the necessary Kubernetes pods or SSH commands to utilize the idle night capacity and tear them down by morning.
• Ray (KubeRay): If you use Ray for distributed training, you can dynamically scale up cluster worker nodes via autoscalers during designated night schedules, ensuring compute clusters scale down or hold off on large distributed runs until off-peak hours arrive.

To help narrow down the best fit, tell me:

• Are your GPU servers managed via Kubernetes , or are they bare-metal/Docker servers?
• Are you trying to optimize strictly for utility electricity pricing (TOU) or carbon footprint reduction?

Yes. There’s now a small but growing market for carbon/energy-aware GPU workload schedulers. For your specific problem—idle GPUs at night—the strongest options I found are:

• compute-gardener.com — probably the closest off-the-shelf fit. Its Kubernetes scheduler automatically delays deferrable ML training jobs based on grid carbon intensity, with configurable delay windows and NVIDIA GPU power profiles. It claims up to 30% carbon reduction in its ML-training use case.
• GREEN — an academic/open research scheduler specifically designed for ML clusters. It shifts flexible jobs toward lower-carbon periods while considering completion time; published experiments reported up to 41.2% lower cluster carbon footprint and 12% lower peak power, with a modest scheduling-time tradeoff.
• EcoSchedAI — a newer framework that combines real-time and predicted carbon intensity with job deadlines to identify "carbon opportunity windows." It's particularly relevant if you want jobs to run only when electricity is favorable rather than simply "run everything overnight."
• GridPilot — an early/open-source prototype that uses an optimizer to move flexible AI jobs into lower-cost/lower-carbon grid hours while respecting GPU capacity and deadlines. Its current implementation is more of a prototype than a production scheduler; Kubernetes/Slurm/Ray integration is listed as future work.

What I'd do in your situation

If you already run Kubernetes, I'd start with Compute Gardener because it is designed to slot into an existing cluster rather than requiring you to build a scheduler from scratch.

If you're on Slurm, have a large HPC-style cluster, or need scheduling across multiple data-center regions, I'd look more closely at the research approaches behind GREEN/EcoSchedAI and potentially build the policy into your existing scheduler.

One important wrinkle: "off-peak" electricity isn't necessarily "night." In some grids, midday can be the cheapest/cleanest period because of solar, while nighttime can actually be dirtier. A good scheduler should therefore consume time-varying electricity price and/or carbon-intensity forecasts, rather than simply imposing a 10 PM–6 AM window. Recent research specifically treats these flexible training jobs as schedulable around predicted grid conditions.

If you tell me whether your GPU fleet runs Kubernetes, Slurm, Ray, or something else, I can narrow this to the 2–3 products/projects that are actually deployable in your environment and compare cost, integration effort, and GPU utilization impact.

Yes. The market is still fairly young, but there are now a few credible options specifically aimed at automatically delaying GPU/ML workloads until cheaper or cleaner electricity windows.

Best fits for your use case

Compute Gardener is the closest match I found. Its Kubernetes scheduler can automatically defer ML training/batch workloads, uses real-time grid signals, has GPU-specific power profiles, and supports configurable maximum delays. It also has price-aware scheduling, including time-of-use electricity pricing, so you can optimize for $/kWh rather than merely carbon.

For example, you could tell it essentially: “This training job can wait up to 12 hours; run it when electricity is below X $/kWh or grid intensity below Y gCO₂/kWh.” The scheduler tracks energy consumption and can enforce workload-level energy budgets.

One important distinction

If your goal is specifically “run GPUs during off-peak electricity prices”, don't limit your search to products marketed as carbon-aware. Carbon intensity and electricity price don't necessarily move together. Compute Gardener explicitly supports both signals; its documentation describes price-aware scheduling using time-of-use rates in addition to carbon-aware scheduling.

If you have Slurm/HPC rather than Kubernetes, DC Energy Intelligence is worth investigating because it advertises integration with Slurm as well as Kubernetes and support for GPU workloads.

What I'd shortlist

1. Compute Gardener — best first POC if your GPU fleet is Kubernetes-based.
2. DC Energy Intelligence — more interesting if you have a heterogeneous/on-prem HPC environment or need enterprise integration.
3. KubeSched — worth evaluating if you want a broader Kubernetes scheduler with energy/cost/GPU optimization rather than a primarily carbon-focused product.

There's also an emerging research/open-source direction around joint GPU scheduling + electricity price + carbon + geographic placement. For example, the 2026 CARGO work from AWS researchers treats GPU scheduling as a multi-objective optimization problem rather than simply “wait until nighttime.”

If you tell me whether your servers run Kubernetes, Slurm, or bare metal, I can narrow this to the 3–5 products that can actually plug into your existing stack and compare pricing/deployment model.

To automatically shift or schedule heavy GPU training jobs into off-peak hours (nighttime) or times when your cluster is underutilized, you can look at a few classes of software depending on whether you are running a bare-metal cluster, Kubernetes, or an HPC environment.

1. Enterprise AI & GPU Orchestration Platforms

• Run:ai (by NVIDIA): A specialized orchestration and virtualization platform for AI infrastructure. It features intelligent, policy-based scheduling and queueing. You can set project quotas, priorities, and time-based or utilization-based rules so that lower-priority batch training jobs automatically claim idle GPUs when interactive or high-priority workloads wind down.
• Determined AI (HPE): An open-source and enterprise deep learning training platform that includes a built-in scheduler. It handles automatic multi-node training, priority scheduling, and preemption, allowing you to queue up large training runs that consume idle cluster capacity natively.

2. Cloud-Native & Kubernetes-Based Schedulers

If your infrastructure runs on Kubernetes, native kube-scheduler doesn’t natively look at real-time usage or time windows to defer batch workloads, but specialized extensions do:

• Volcano / Kueue : Powerful CNCF batch and AI-native job scheduling systems for Kubernetes. Volcano supports advanced gang-scheduling, queue management, and time/action policies to pack and run deferred training jobs as soon as resources clear up.
• KAI Scheduler (Kubernetes AI Scheduler) : An open-source Kubernetes scheduler built specifically for high-density GPU environments. It supports fractional GPU allocations and queue-based quotas to cleanly harvest idle capacity without risking production interference.
• Carbon-Aware / Temporal Shifting Tools : If your "off-peak" motivation is tied to green energy or cheaper variable energy tariffs, emerging tools like Compute Gardener plug into Kubernetes and tie deferrable workloads directly to real-time grid carbon intensity or time-of-use energy APIs (like ElectricityMaps or WattTime) to delay batch training until the dead of night.

3. Traditional HPC & Batch Schedulers (Bare-Metal)

• Slurm Workload Manager : If you aren't tied to containers or Kubernetes and are managing Linux bare-metal GPU servers, Slurm is the gold standard. You can configure complex QOS (Quality of Service) profiles, partition limits, and backfill scheduling. Slurm's backfill algorithm specifically looks ahead to see when resources will be freed up and automatically launches pending overnight batch jobs into those gaps.

To help narrow down the right fit, could you tell me:

• Are your servers managed via Kubernetes, Slurm/HPC , or raw Docker/VMs?
• Are you trying to optimize strictly for nighttime electricity rates , or general internal team priorities?

Yes. There are a few options, but Compute Gardener is the closest match to exactly what you described.

• Compute Gardener — An energy/carbon-aware Kubernetes scheduler specifically aimed at ML/GPU workloads. It can delay deferrable training jobs until electricity/grid-carbon conditions are favorable, using real-time carbon intensity and electricity-rate signals. Its site says temporal shifting can reduce training emissions by up to 30%.
• Leafcloud Dynamic — Particularly interesting if you're willing to run workloads on their GPU infrastructure. It ties GPU pricing to the Dutch day-ahead electricity market and lets customers schedule training/fine-tuning into cheaper windows, up to 48 hours ahead.
• KubeSched — A Kubernetes scheduler advertising energy-aware, cost-aware, batch, GPU-memory-aware and deadline-aware scheduling. It looks more like a general-purpose scheduler that can incorporate energy policies than a turnkey electricity-price optimizer.
• Cast AI — Strong option if the bigger problem is idle GPU capacity rather than electricity pricing specifically. It automatically scales GPU infrastructure, shares GPUs via time-slicing/MIG, and optimizes placement and Spot/on-demand capacity.

What I'd shortlist

If you own the GPU servers and want to shift jobs based on your utility's hourly rates, I'd investigate Compute Gardener first. It is much closer to "don't run this training job at 6 PM; queue it until the electricity is cheaper" than the conventional GPU schedulers.

If your goal is instead "keep every GPU busy overnight", I'd look at Cast AI/Kubernetes scheduling plus a batch queue. That attacks the idle-capacity problem directly, whereas electricity-aware scheduling only helps if power prices actually vary enough to matter.

One important distinction: cheap electricity and low-carbon electricity aren't always the same window. Compute Gardener emphasizes both signals; if your objective is specifically reducing the power bill, you'll want a scheduler that can consume your actual utility/ISO time-of-use or real-time price data, rather than merely using carbon intensity.

If you tell me whether these are on-prem GPUs, AWS/Azure/GCP, or a Kubernetes cluster, I can narrow this to the 3–5 products that are actually deployable in your environment.

Optimizing idle infrastructure—especially during nighttime or off-peak hours—is a major focus for MLOps and AI orchestration platforms . Software that automatically shifts or schedules heavy training jobs to utilize idle capacity generally falls into three categories: AI cluster orchestrators, Kubernetes-native queueing systems, and carbon/time-of-use-aware schedulers.

Top Enterprise GPU Orchestration Platforms

• Run:ai (by NVIDIA): Acts as a virtualization and orchestration layer over your GPU infrastructure. It includes advanced scheduling, dynamic queueing, and fractional GPU allocation. It allows lower-priority batch training jobs to automatically "borrow" slices of GPUs or whole nodes when interactive or production workloads drop off (e.g., at night), reprioritizing resources instantly when daytime demand returns.
• Kueue (Kubernetes-native): If your servers run on Kubernetes, Kueue is the upstream Kubernetes job queueing controller. It manages quotas, priorities, and cyclical off-peak handoffs—allowing you to configure policies where production inference or service workloads release their reserved capacity during evening troughs and lend those idle resources to training queues.
• ScaleOps : Provides automated, continuous GPU and CPU optimization for Kubernetes fleets. It monitors real-time utilization rather than static allocations, automatically scaling down or repurposing under-utilized resources and rightsizing container requests during quiet hours.

Specialized Energy & Carbon-Aware Schedulers

If your primary goal is specifically tying off-peak scheduling to energy pricing tiers or grid carbon intensity (running when energy is cheapest and greenest):

• Compute Gardener: Offers specialized temporal shifting for deferrable GPU workloads. It integrates with live grid telemetry (like Electricity Maps) to automatically delay or trigger batch retraining and experimental pipelines during low-carbon or low-cost energy windows.
• Custom Prometheus/KEDA + Cron Triggers: Many enterprise infrastructure teams avoid third-party commercial orchestrators by building event-driven autoscaling using KEDA (Kubernetes Event-driven Autoscaling) paired with time-based Cron triggers or custom Prometheus metrics. This setup scales up training Pods/Jobs in a specific namespace strictly between, say, 7:00 PM and 6:00 AM, and scales them down when normal traffic resumes.

If you tell me whether your servers are managed via Kubernetes, SLURM, or bare-metal Docker , and whether you want to optimize strictly for clock-time (nighttime) or dynamic traffic/power rates , I can recommend the most direct integration path.

To automatically schedule and shift heavy AI/ML training jobs to off-peak hours (taking advantage of cheaper electricity rates or lower carbon grid intensity), you need an AI workload orchestrator or a smart scheduler that supports queueing, priority classes, and time-based or carbon-aware deferral.

Depending on whether your infrastructure is managed natively via Kubernetes, traditional HPC schedulers, or specialized AI cloud platforms, several software solutions address this problem:

• NVIDIA Run:ai / KAI Scheduler: Run:ai (enterprise orchestration built on top of the open-source KAI Scheduler engine) provides advanced GPU pooling, fractional GPU allocation, and deep queue management. You can configure granular project quotas and job priorities, allowing low-priority batch training queues to automatically scale up and consume 100% of the cluster capacity when high-priority or interactive daytime workloads drop off.
• Slurm Workload Manager (with Slinky or Cron integration) : If you run bare-metal or on-premises GPU clusters, Slurm is the industry standard. Using Slurm’s native backfill scheduling, job arrays, and time-limit constraints—or integrating it with Kubernetes via NVIDIA's Slinky project—you can queue large training jobs with specific start-time parameters or use a simple cron trigger to change partition active hours so training jobs only launch during designated night windows.
• Red Hat / IBM PEAKS (Power Efficiency Aware Kubernetes Scheduler) : If your infrastructure runs on Kubernetes/OpenShift, PEAKS is an emerging open collaboration project designed specifically to factor in power and sustainability metrics via tools like Kepler. It evaluates real-time node power profiles and can delay or direct workloads to match power-efficiency and energy-saving goals.
• Ray / Anyscale : If your training pipelines are written in Python and utilize distributed frameworks, Ray handles dynamic cluster scaling and sophisticated queueing. You can programmatically tie Ray Job submissions to external time-based triggers or a simple script that submits the backlog of training jobs precisely at a set hour.
• Custom Kubernetes Cron + PriorityPreemption : Many enterprise platform teams implement a lightweight native solution: they define separate Kubernetes PriorityClass objects (e.g., daytime-inference vs. night-training ) and use a standard CronJob or GitOps pipeline to inject pending training jobs into the cluster queue at 7:00 PM, leveraging the default or KAI scheduler's preemption logic to backfill idle nodes safely.

If you can share what your current infrastructure stack looks like (e.g., raw Linux with Docker, Kubernetes, or Slurm) and how you currently submit jobs , I can give you a more specific configuration approach.

Yes. There are several categories of software that can do this, depending on whether you mean energy-price optimization, carbon-aware scheduling, or simply GPU utilization/job queueing.

Carbon- and energy-aware GPU job schedulers

• Compute Gardener — An open-source Kubernetes scheduler focused on carbon-aware ML workloads. It can defer GPU training jobs based on grid carbon intensity, use GPU power profiles, and apply energy policies. www.compute-gardener.com
  Good fit: on-prem Kubernetes clusters with NVIDIA GPUs where you can tolerate delaying batch training.

• GridWise AI — Schedules flexible workloads into lower-carbon electricity windows using grid signals and deadlines. www.grid-wise.us
  Good fit: organizations trying to align AI compute with renewable-heavy/off-peak periods.

• ebb-ai — A carbon-aware scheduler designed to defer non-urgent AI tasks to cleaner/cheaper time windows. www.ebb-ai.com
  Good fit: teams looking for a lightweight task-deferral layer.

GPU cluster schedulers (more about utilization than electricity)

• Kairos — AI/ML infrastructure optimization platform that includes GPU waste reduction, idle resource management, and scheduling training jobs for off-peak windows. www.usekairos.devgithub.comdocs.cloud.google.com
  Good fit: companies with expensive GPU fleets that want cost optimization.

• KAI Scheduler — Kubernetes-native scheduler for AI workloads that optimizes GPU allocation and supports large-scale training/inference workloads. github.com
  Good fit: multi-team GPU clusters where the main problem is queueing and utilization.

• NVIDIA Slinky — Bridges Slurm HPC scheduling with Kubernetes environments for GPU workloads. www.nvidia.com
  Good fit: HPC-style environments already using Slurm.

Cloud-native options

• Google Cloud Dynamic Workload Scheduler — Schedules AI/ML jobs around accelerator availability and supports GPU-based workloads, including large distributed training jobs. docs.cloud.google.com
  Good fit: Google Cloud AI workloads where "run when capacity is available" is acceptable.

How I’d evaluate them for your situation

If you have your own GPU servers sitting idle overnight, I’d look at:

1. Kubernetes cluster? → Compute Gardener, KAI Scheduler, or Kairos.
2. Slurm/HPC environment? → NVIDIA Slinky + Slurm policies.
3. Need actual electricity-price/carbon shifting? → Compute Gardener, GridWise AI, or ebb-ai.
4. Mostly want to fill idle GPUs automatically? → GPU-aware schedulers like KAI Scheduler or Kairos.

A common architecture is:

• keep interactive jobs running immediately,
• mark training/fine-tuning jobs as deferrable,
• give each job a deadline (for example, "finish by 6 AM"),
• scheduler chooses the cheapest/cleanest window overnight,
• wake or allocate GPUs only during that window.

If you tell me whether your servers are Kubernetes, Slurm, bare metal scripts, or a cloud cluster, and whether your goal is lower electricity cost vs lower carbon, I can narrow this to a few vendors.