Software Engineer, Data Infrastructure

About This RoleAI processing…

Key Responsibilities

• 1
  A core pipeline started as a pragmatic reuse of infrastructure built for something else. It works, but it cannot guarantee properties downstream consumers now need (for example, point-in-time consistency). You design and ship the replacement while keeping the existing system running.
• 2
  A new product surface ships without instrumentation. You talk to the team, define what needs to be captured, and wire it through before the absence becomes anyone else’s problem.
• 3
  Eval coverage drops. You trace it to an instrumentation gap introduced weeks ago by a product change nobody flagged. You fix the gap, add a contract so it cannot recur, and ship the dashboard that would have caught it earlier.
• 4
  Multiple consumers depend on overlapping data. You design schema evolution and validation so changes in one place do not silently degrade the others.
• 5
  Storage costs rise faster than usage. You decide what is worth keeping, implement retention and compression, and delete what is not.

Requirements

• Deep experience with Spark (Databricks or open-source Spark both count)
• Production experience with Ray Data
• Hands-on ownership of large data pipelines and storage systems
• Comfort debugging performance issues across client instrumentation, streaming, storage, and model-facing workflows, as well as, compute, storage, and networking layers
• Clear thinking about data modeling and long-term maintainability
• You have good judgment about when to patch and when to rebuild
• Experience running or scaling ClickHouse
• Familiarity with dbt, Dagster, or similar orchestration and modeling tools

Perks & BenefitsTypical for this role