Customer Reliability Engineer (Remote, APAC)

About the role

Andromeda is a market and infrastructure platform for buying, selling and operating compute, enabling fragmented compute to flow through one platform to model builders, research labs and inference providers. Its customers run large AI training and inference workloads on GPU clusters sourced from providers worldwide. The Customer Reliability Engineer is the point of contact when a node goes dark or a job fails, triaging and debugging issues at the Linux and Kubernetes layer, working with providers and building monitoring and scripts.

Key responsibilities

• Triage and fix customer issues from start to finish: reproduce, diagnose, fix or escalate, and close the loop.
• Debug Linux-layer problems such as processes, networking, storage, kernel logs, resource contention, systemd and journald.
• Investigate Kubernetes issues such as stuck pods, crash loops, node conditions, scheduling failures and resource limits.
• Work GPU failures: driver and device-plugin issues, XID errors, thermal throttling, and nodes needing cordoning or draining.
• Escalate beyond your depth with evidence already gathered.
• Take part in a 24/7 on-call rotation, respond to alerts and outages, keep customers updated, and turn incidents into runbooks, alerts or fixes.
• Identify provider-side faults, open tickets with compute providers and track recurring patterns.
• Write Python or Bash scripts to automate checks, and build and improve monitoring, cluster and node health checks, GPU telemetry and dashboards.

Required profile

• Comfortable in a Linux shell; no GPU or HPC experience needed, as most people learn it on the job.
• Strong writing: explaining technical problems clearly to frustrated customers.
• Good judgment under pressure, knowing what to check first and when to escalate.
• Willingness to join a 24/7 on-call rotation.

Required skills

• Linux troubleshooting from the command line
• Kubernetes: pods, nodes, deployments, services, scheduling
• Python or Bash scripting
• Nice to have: NVIDIA GPU hands-on (drivers, CUDA, DCGM, device plugin), InfiniBand/RoCE or NCCL, Slurm, Prometheus/Grafana/Datadog, Terraform/Ansible/Helm

What we offer

• High-growth environment at the centre of the AI infrastructure boom
• Competitive compensation with meaningful equity
• Benefits for you and your dependents, including healthcare, dental and vision coverage, 401(k) and unlimited PTO