Senior Eval Infrastructure Engineer

About the role

Every GONATURELLE gate is a distributed job: a suite of golden cases is sharded across isolated runners, each runner drives a customer's agent through its tool calls inside a sandbox, captures the trajectory, and hands the trace to deterministic assertions and a judge ensemble. The fast path has to finish inside a developer's patience budget on every push. The full suite has to finish overnight across several thousand cases without a single flaky verdict, because a flaky gate gets disabled and a disabled gate protects nobody.

You will join the team led by Vikram Choudhary that owns this path end to end: the scheduler, the runner images, the trace store, the result cache keyed on dataset version plus prompt hash plus model version, and the GitHub Actions, GitLab CI and CircleCI integrations that call it. The same code ships as our multi-tenant cloud, as a dedicated VPC deployment in a customer's AWS, GCP or Azure account, and as an air-gapped Kubernetes install with zero egress, so portability is a design constraint and not an afterthought.

This is a senior individual-contributor role. You will write a lot of Go and Python, review a lot of design documents, and be on the rotation that answers when a hospital network's nightly suite stalls at 2 a.m. IST.

What you will do

• Design and operate the shard scheduler that splits suites across runners, retries infrastructure failures without retrying genuine agent failures, and reports per-shard latency
• Harden runner sandboxes: network policy per suite, secrets injection for customer tool endpoints, resource ceilings that stop runaway tool loops from burning a customer's token budget
• Build the result cache so unchanged cases on an unchanged prompt and model are not re-run, and prove cache correctness with property tests
• Keep the Helm charts and Terraform modules for VPC and air-gapped installs at parity with the cloud release
• Instrument everything with OpenTelemetry and own the SLOs behind the status page
• Write runbooks and take part in an on-call rotation of roughly one week in six

What you bring

• 6+ years building distributed systems or build and test infrastructure in production
• Strong Go or Python; comfortable reading the other
• Hands-on Kubernetes: you have debugged scheduling, networking and storage problems in a real cluster, not only written manifests
• Experience with at least one CI system's internals (runners, caching, artifacts) at scale
• A habit of measuring before optimizing, and of writing down why a system behaves the way it does

Nice to have

• Experience with gVisor, Firecracker or other sandboxing runtimes
• Air-gapped or regulated deployments (banking, healthcare, defence)
• Familiarity with agent frameworks such as LangGraph or the OpenAI Agents SDK
• Open-source contributions to build or test tooling