Find out why a training job slowed down, why inference latency rose, or why a distributed run failed. Goodput brings workload and infrastructure evidence together so your team can decide what to fix—and verify the result.
Illustrative architecture
Low GPU activity can come from input stalls, CPU work, synchronization, or a failing node. Start with the affected job, then connect its execution profile to the resources and events around it.
A longer training step does not necessarily mean a slower GPU kernel. Separate time spent preparing inputs, executing kernels, and waiting for other ranks before choosing an optimization.
The GPU waits for the next batch.
Synthetic 400 ms example, not a measured customer result. CPU and GPU rows share one clock; overlapping work is not additive.
One rank fails. The rest stop making progress.
Synthetic incident. Timing alone does not establish causality; Xid interpretation depends on the code and surrounding evidence.
A distributed job can produce errors on every rank when the original failure happened on just one node. Bring job events and node health into the same investigation.
Use the workload’s own success metrics. GPU utilization alone cannot tell you whether a service meets its latency target or a training job makes useful progress.
Separate time waiting in the request queue from prefill and decode. Compare request lengths, concurrency, KV-cache pressure, and configuration before changing batch limits.
Illustrative categories, not an additive latency chart.
Distinguish application errors from CPU memory limits, GPU memory exhaustion, and node failures. A retry is useful only if the conditions that caused the failure have changed.
Illustrative investigation · requires access to job accounting and logs.
The proposed Goodput workflow keeps the evidence, recommended action, and validation together. Engineers decide which changes to approve.
A throughput regression, a failed job, or a node health event identifies where to look.
Connect profiles, job context, and infrastructure signals. State what is known and what still needs checking.
Inspect the evidence and proposed scope before a configuration change, restart, or escalation.
Repeat the measurement under comparable conditions. Confirm recovery and watch for regressions.
Choose one recurring performance or reliability problem. Agree on the data sources, access boundaries, and success criteria before connecting to your cluster.
An agent for investigating performance and reliability issues in AI clusters. The focus is connecting workload evidence with job and infrastructure context, then helping engineers choose and verify a fix.
No. The diagrams and investigations on this page illustrate the intended workflow. Example timings and events are synthetic; they are not customer measurements or evidence of deployed capabilities.
The proposed pilot uses the signals your environment already exposes. Integration scope is agreed with your team; Goodput is not a replacement scheduler.
The workflow shown here requires engineer approval for changes. Access and actions are scoped with each design partner.
Contact hello@goodput.place with your environment and a recurring slowdown or failure. We are looking for design partners to define a focused pilot. Availability, integrations, pricing, and terms are discussed together.
Let’s define the first investigation worth doing together.