Engineering Manager, Model Routing & Inference

About This RoleAI processing…

Key Responsibilities

• 1
  Building and evolving our inference gateway, a single abstraction over every provider's API semantics, so model onboarding becomes a config change.
• 2
  Building the systems that dynamically select the best model for each request based on cost, latency, and quality.
• 3
  Managing GPU cluster utilization and capacity planning across providers, optimizing for cost and performance.
• 4
  Designing routing backpressure and admission control so traffic spikes don't cascade into providers.
• 5
  Hiring and growing the team: sourcing, interviewing, and closing top inference and systems talent, while developing your engineers through coaching, mentorship, and high-leverage project assignments.

Requirements

• You have led engineering teams building high-throughput, low-latency distributed systems, especially in inference serving, traffic routing, or real-time data pipelines.
• You have strong software engineering fundamentals and enjoy shipping production systems that handle millions of requests.
• Experience with model serving frameworks (vLLM, TensorRT-LLM, TGI), load balancing, or building resilient multi-provider architectures is a plus.
• You may be a fit if You have led engineering teams building high-throughput, low-latency distributed systems, especially in inference serving, traffic routing, or real-time data pipelines.

Perks & BenefitsTypical for this role