Network Performance Engineer – GPU/HPC Infrastructure

About the RoleIshan Technologies is looking for an experienced Network Performance Engineer to build, operate, optimise and maintain the high-performance network infrastructure supporting Saksham AI Services' 1000+ GPU cluster. The role will focus on backend GPU networking, high-speed Ethernet/RoCE and InfiniBand fabrics, front-end connectivity, storage and management networks. The ideal candidate should have strong hands-on experience in data centre networking, network performance optimisation, RDMA technologies, congestion control and collective communication benchmarking. The selected candidate will also act as an L2/L3 escalation point for network and performance-related issues raised by Ishan Technologies' 24x7 NOC team.Key Responsibilities1. GPU Network & Data Centre InfrastructureInstall, configure and operate high-speed backend GPU networks using RoCEv2/Ethernet or InfiniBand.Manage front-end, storage and out-of-band management networks.Configure and maintain switches, network interface cards (NICs), transceivers and fibre connectivity.Work on rail-optimised leaf-spine network architecture.Maintain network topology, cable mapping and infrastructure documentation.2. Network Performance & OptimisationConfigure and optimise BGP, EVPN-VXLAN and leaf-spine network architecture.Implement congestion control mechanisms, including PFC, ECN/DCQCN, adaptive routing and QoS.Monitor network bandwidth, latency, packet loss, link flaps and congestion.Troubleshoot GPU-to-GPU communication and collective performance issues.Ensure full-bandwidth network performance across different workload and cluster sizes.3. RoCE, InfiniBand & RDMA NetworkingManage RoCEv2 and InfiniBand fabrics for GPU/HPC environments.Work with NVIDIA UFM and InfiniBand management tools.Configure and troubleshoot RDMA drivers, SR-IOV and GPUDirect RDMA.Work with NVIDIA ConnectX and AMD Pensando Pollara networking technologies.Maintain VLAN/VRF segmentation, PKeys and fabric-level tenant isolation.4. AI Workload & Collective CommunicationPerform collective communication benchmarking using RCCL, NCCL, MPI and related testing tools.Conduct RDMA performance testing using perftest, ib_write_bw and ib_read_bw.Work with nccl-tests and rccl-tests to validate network performance.Understand how data, tensor and expert parallelism impact network utilisation.Investigate slow-running AI jobs, communication bottlenecks and straggler GPUs.Collaborate with compute and MLOps teams to optimise AI workload performance.5. Network Monitoring & AutomationDevelop network monitoring dashboards using Prometheus and Grafana.Monitor switch health, link errors, optics performance and network congestion.Automate network configurations, health checks and compliance activities using Python and Ansible.Maintain network performance baselines and conduct acceptance testing after infrastructure changes.Support Kubernetes RDMA networking using Multus, SR-IOV device plugins and network operators.6. Incident Management & OperationsHandle L2/L3 network and performance escalations from the 24x7 NOC team.Lead network incident investigations and root cause analysis.Coordinate with customers, solution architects, vendors and internal engineering teams.Plan and execute switch, NIC and optics firmware upgrades under change management.Maintain network runbooks, technical documentation and operational standards.Support internet edge connectivity, peering and private customer network links.Required Technical SkillsStrong hands-on experience in Data Centre Networking.Experience with RoCEv2 or InfiniBand networking.Strong understanding of BGP, EVPN-VXLAN and leaf-spine architecture.Knowledge of PFC, ECN/DCQCN, QoS and adaptive routing.Experience with RDMA, GPUDirect RDMA and SR-IOV.Hands-on experience with NVIDIA ConnectX, Spectrum-X or AMD Pensando Pollara.Knowledge of RCCL, NCCL, MPI and collective communication benchmarking.Experience with NVIDIA UFM and InfiniBand management.Understanding of VLAN, VRF, PKeys and network tenant isolation.Experience with 400G/800G networking, optics and fibre connectivity.Working knowledge of Python, Ansible and network automation.Experience with Prometheus, Grafana, SNMP and streaming telemetry.Educational QualificationsBachelor's degree in Engineering or Technology.Preferred specialisations: Computer Science, Information Technology, Electronics, Telecommunications or related disciplines.Relevant data centre networking certifications will be an added advantage.Preferred Certifications:Cisco CCNP/CCIE Data Center.Arista ACE.Juniper JNCIP-DC.NVIDIA Networking (InfiniBand/Spectrum-X) certifications.Experience Requirements6–10 years of overall professional experience.Minimum 4+ years of relevant experience in Data Centre Networking for GPU, HPC or hyperscale environments.Hands-on experience with RoCEv2 or InfiniBand fabrics is essential.Experience with large-scale leaf-spine networks, cabling and optics is essential.Experience in RCCL/NCCL or MPI performance optimisation is preferred.Exposure to AMD Pensando, NVIDIA Spectrum-X or ConnectX networking is preferred.Key CompetenciesStrong troubleshooting and analytical skills.Hands-on approach to network engineering and performance optimisation.Ability to handle critical production incidents under pressure.Strong ownership and root cause analysis capabilities.Effective communication with customers, NOC teams and technical stakeholders.Willingness to participate in a 24x7 on-call support rotation.Ability to learn and adapt to evolving AI and GPU networking technologies.Key Success OutcomesStable, high-performance GPU networking infrastructure.Full-bandwidth collective communication across cluster workloads.Verified network isolation between multiple customers and tenants.Improved network performance and reduced communication bottlenecks.Faster resolution of network incidents and performance escalations.Standardised, automated and well-documented network operations.