GPU Cluster Reliability Engineer

About the RoleIshan Technologies is looking for an experienced Cluster Reliability Engineer to manage and maintain the compute infrastructure of Saksham AI Services' 1000+ GPU cluster.The role involves managing GPU servers, Linux infrastructure, Slurm and Kubernetes environments, cluster reliability, automation and MLOps operations. The selected candidate will play a key role in ensuring high availability, performance, scalability and reliability of AI training and inference workloads.The position will also serve as an L2/L3 technical escalation point, working closely with the 24x7 NOC team, platform engineers, customers and technology partners.Key Responsibilities1. GPU Infrastructure & Server ManagementReceive, rack, configure and maintain GPU servers.Manage BIOS, firmware, Linux OS images, GPU drivers and software libraries.Perform GPU server burn-in, hardware diagnostics and acceptance testing.Maintain consistent server configurations and manage firmware and security updates.Coordinate with OEM vendors for hardware faults, replacements and RMAs.2. Cluster Management – Slurm & KubernetesAdminister and maintain Slurm and Kubernetes environments for AI training and inference workloads.Manage Slurm controllers, partitions, QoS, fairshare and resource allocation.Configure Kubernetes GPU Operator, device plugins, Helm and related operators.Manage cluster upgrades, patching, tenant onboarding and scheduling.Ensure stable and efficient multi-tenant GPU cluster operations.3. Cluster Reliability & MonitoringDevelop and maintain GPU and node health checks.Monitor GPU errors, ECC events, RAS events and hardware failures.Implement automated fault detection and remediation mechanisms.Develop monitoring dashboards and alerts using Prometheus, Grafana, Loki or ELK.Maintain cluster availability in line with the 99.9% uptime commitment.Maintain operational logs and monitoring records in accordance with applicable compliance requirements.4. MLOps & AI Workload SupportBuild and maintain optimized framework images for PyTorch, JAX and vLLM.Develop ready-to-use job templates for AI training and inference.Conduct workload benchmarks and GPU performance testing.Support distributed AI workloads, including data, tensor and pipeline parallelism.Provide technical assistance to customers and solution architects on workload performance and tooling.5. Incident Management & AutomationHandle L2/L3 technical escalations from the 24x7 NOC team.Lead incident investigations, root cause analysis and post-incident reviews.Develop automation scripts using Python and Bash.Maintain operational runbooks and auto-remediation procedures.Track capacity, GPU utilisation, goodput and service-level performance.Identify recurring issues and implement long-term corrective actions.Required Technical SkillsStrong hands-on experience in Linux System Administration.Experience managing GPU servers, GPU drivers, firmware, BIOS and kernel configurations.Hands-on experience with multi-GPU accelerator servers.Strong knowledge of Slurm administration and workload scheduling.Experience with Kubernetes administration for GPU workloads.Knowledge of GPU Operator, Helm, Kubeflow and KubeRay.Familiarity with GPU monitoring, ECC/RAS events and vendor diagnostic tools.Experience with Prometheus, Grafana, Loki or ELK.Working knowledge of PyTorch, JAX, vLLM and GPU collective communication libraries.Experience with Docker, Apptainer and container image management.Automation experience using Python, Bash and Ansible or Terraform.Familiarity with bare-metal provisioning tools such as MAAS, Ironic, Metal3 or Warewulf.Educational QualificationsBachelor's degree in Engineering or Technology.Preferred specialisations: Computer Science, Information Technology, Electronics or related disciplines.Relevant certifications in Linux, Kubernetes or GPU computing will be an added advantage.Preferred Certifications:Certified Kubernetes Administrator (CKA).Red Hat RHCSA/RHCE or Linux Foundation Certified System Administrator (LFCS).GPU vendor training or certifications.Experience Requirements4–8 years of overall IT infrastructure or engineering experience.Minimum 2+ years of relevant experience in GPU/HPC cluster operations, Site Reliability Engineering (SRE) or MLOps.Hands-on experience with Linux GPU servers and Slurm and/or Kubernetes is essential.Experience supporting production environments with on-call and incident management responsibilities.Experience with multi-vendor data-centre GPU platforms will be an advantage.Exposure to distributed AI training and inference workloads is preferred.Preferred Additional ExposureNVIDIA GPU ecosystem and related management tools.ClusterMAX standards and CMAX audit tool.Public cloud platforms such as GCP or AWS.HPC infrastructure and large-scale AI compute environments.Key CompetenciesStrong troubleshooting and problem-solving abilities.Hands-on technical ownership and attention to detail.Ability to handle critical production incidents under pressure.Strong communication and coordination skills.Willingness to participate in a 24x7 on-call support rotation.Ability to learn and adapt to evolving GPU, HPC and AI technologies.