Senior AI Research Engineer - LLM Training & Inference Infrastructure
Boost your chances at datadog
Tailor your resume to this exact job and generate a matching cover letter in about 60 seconds with JobEase — our AI application assistant.
- ATS-optimized resume
- Personalized cover letter
- Match score & keyword gaps
Free to start · no card required
Get more other jobs in your inbox
Verified daily — no ghost listings.
About This Role
Senior AI Research Engineer builds and operates distributed training and inference infrastructure for large language models at Datadog. The role partners with research scientists to design systems that schedule GPU workloads, optimize memory, and ensure reliable long training runs. It also focuses on improving researcher experience and maintaining high-performance execution environments for RL and inference.
Key ResponsibilitiesAI-extracted
- 1Design, build, and operate distributed training and inference infrastructure for large language models.
- 2Scale training with data, tensor, pipeline, and expert parallelism, and implement memory optimization and checkpointing.
- 3Develop inference and rollout systems that support evaluation, experimentation, and reinforcement learning without impacting training.
- 4Create execution environments and sandboxes for online reinforcement learning and agent post‑training.
- 5Own continuous performance validation, including benchmarks of rollout speed, training throughput, and learning behavior.
- 6Improve researcher experience by enhancing job submission, scheduling, quotas, GPU utilization, and observability.
- 7Collaborate with platform teams on cluster capacity, networking, and scheduling across GPU providers.
- 8Investigate performance and reliability issues across hardware, software, and distributed systems.
Requirements
- You have depth in distributed systems and ML infrastructure for training and inference at scale; experience with Ray, Kubernetes, Slurm, or similar frameworks is a plus
- You are proficient in Python and in at least one systems language (e.g., Rust, C++, or Go) ● You have experience operating large-scale training or serving infrastructure, or schedulers and orchestration systems for GPU or other high-performance workloads
- You understand what limits throughput/latency in distributed inference workloads: communication, memory, scheduling, and stragglers
- You have strong debugging, profiling, and incident-diagnosis skills, ideally including on-call experience
- You can explain design and performance trade-offs clearly to both technical and non-technical audiences
- Prior experience training machine learning models is valuable but not essential for engineers with strong distributed systems or HPC backgrounds
- You are familiar with distributed training frameworks such as Megatron-LM, TorchTitan, or DeepSpeed
- You have experience with LLM inference engines such as vLLM or SGLang, including batching and KV-cache management
- You have experience with RL training systems and rollout infrastructure
- You have experience with collective communication (NCCL), RDMA, or low-latency networking
- You have experience with GPU benchmarking, profiling, and cluster performance monitoring
- You have experience building sandboxed code execution or large-scale evaluation infrastructure
- You have contributed to open-source ML or distributed systems projects
Perks & Benefits
Apply to This Job in Minutes
Generate ATS-optimized resume + cover letter + interview prep with Jobease.ca AI. Complete your application faster.
75% of AI Resumes Get Rejected
Beat the ATS with Jobease.ca's AI Resume Builder. Optimized for real hiring systems.
Build My ResumeProfile Match
Loading…Checking your profile against this job…
Job Overview
Share This Job
Track All Your Applications
Never lose track again. Jobease.ca organizes every application, interview, and follow-up.
Organize My Search