NVIDIA NCP-AIN Practice Exam Questions & Answers

RoCE Adaptive Routing is a key feature of NVIDIA Spectrum-X that ensures high performance and resiliency in the network, even in the event of a link failure. This technology dynamically reroutes traffic to the least congested and operational paths, effectively mitigating the impact of link failures. By continuously evaluating the network's egress queue loads and receiving status notifications from neighboring switches, Spectrum-X can adaptively select optimal paths for data transmission. This ensures that the network maintains high throughput and low latency, crucial for AI workloads, even when certain links are down.

Reference Extracts from NVIDIA Documentation:

'Spectrum-X employs global adaptive routing to quickly reroute traffic during link failures, minimizing disruptions and preserving optimal storage fabric utilization.'

'RoCE Adaptive Routing avoids congestion by dynamically routing large AI flows away from congestion points. This approach improves network resource utilization, leaf/spine efficiency, and performance.'

The NVIDIA Network Operator is designed to simplify and automate the deployment and management of networking components in Kubernetes clusters, particularly those utilizing NVIDIA Spectrum-X switches. It manages the installation and configuration of necessary drivers, plugins, and other networking resources to enable features like RDMA and GPUDirect RDMA, which are essential for high-performance AI workloads.

By leveraging Kubernetes Custom Resource Definitions (CRDs) and the Operator Framework, the Network Operator ensures that networking components are consistently and correctly configured across the cluster, reducing manual intervention and potential configuration errors.

To upgrade the MLNX-OS on InfiniBand switches, the recommended procedure is as follows:

Connect to the switch via SSH: Establish a secure shell connection to the switch using its management IP address.

Fetch the MLNX-OS software image: Obtain the appropriate MLNX-OS software image from the official source or repository.

Use the 'install' command to perform the upgrade: Execute the 'install' command on the switch to initiate the upgrade process with the fetched software image.

This method ensures a smooth and efficient upgrade without the need for physical intervention or service disruption.

Reference Extracts from NVIDIA Documentation:

'Click on Systems MLNX-OS Upgrade. Select the desired upgrade method (e.g. 'Install from local file'). Select your image and click 'Install Image'.'

EVPN Multi-homing enables active-active redundancy without inter-switch links by using overlay routing over VXLAN and distributed control plane using BGP EVPN.

From the official NVIDIA Cumulus Linux EVPN Multihoming Documentation:

'EVPN multihoming allows multiple Top-of-Rack (ToR) switches to connect to the same server while maintaining full layer-2 redundancy without the need for inter-switch links or traditional MLAG configuration.'

Key benefits:

Simplified topology (no ISL/peer-link needed)

BGP-based control plane

Fast convergence

Active-active links per host NIC

Incorrect Options:

MLAG requires ISL between switches and peer-link configuration.

VSS (Virtual Switching System) is a Cisco term, not supported in NVIDIA networking.

The ibnetdiscover utility is used to perform InfiniBand subnet discovery and outputs a human-readable topology file. GUIDs, node types, and port numbers are displayed, as well as port LIDs and node descriptions. All nodes and links are displayed, providing a full topology. This utility can also be used to list the current connected nodes. The output is printed to the standard output unless a topology file is specified.

InfiniBand is a high-performance, low-latency interconnect technology used in AI and HPC data centers, particularly for east-west traffic between compute nodes in large-scale fabrics. Ensuring seamless data flow requires tools to troubleshoot and monitor the network, including the ability to trace and discover network paths between nodes. The question asks for the specific tool used to trace and discover paths in an InfiniBand fabric, which is a key task in InfiniBand troubleshooting.

According to NVIDIA's official InfiniBand documentation, the ibnetdiscover tool is designed to discover and map the topology of an InfiniBand fabric, including the paths between nodes. It scans the fabric, queries the subnet manager, and generates a topology map that details the connections between switches, Host Channel Adapters (HCAs), and other devices. This tool is essential for verifying connectivity, identifying routing paths, and troubleshooting issues like misconfigured routes or link failures in large-scale InfiniBand fabrics.

Exact Extract from NVIDIA Documentation:

''The ibnetdiscover tool is used to discover the InfiniBand fabric topology and generate a map of the network. It queries the subnet manager to retrieve information about all nodes, switches, and links in the fabric, providing a detailed view of the paths between nodes. This tool is critical for troubleshooting connectivity issues and ensuring proper routing in InfiniBand networks.''

--- NVIDIA InfiniBand Networking Guide

This extract confirms that ibnetdiscover is the correct tool for discovering network paths in an InfiniBand east-west fabric. It provides a comprehensive view of the fabric's topology, enabling administrators to trace paths between compute nodes and ensure seamless data flow.

In multi-subnet InfiniBand environments, AI training clusters are segmented across network zones (or subnets). Direct GPU-to-GPU communication (especially for collective ops like AllReduce, Broadcast, etc.) requires inter-subnet reachability. NCCL supports this via the InfiniBand Router (IB Router) feature.

From the NCCL User Guide -- Environment Variables Section:

'NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers which are used in multi-subnet InfiniBand fabrics. When enabled, NCCL can traverse IB subnets using a properly configured IB router.'

This is critical because without IB Router support:

NCCL would be restricted to intra-subnet GPU collectives.

Multi-node training across subnets would fail or fall back to slower TCP fallback mechanisms.

Technical Explanation:

IB Routers use subnet managers (like OpenSM with routing tables) to bridge communication across different InfiniBand partitions.

NCCL queries the subnet topology, discovers routing paths, and uses RDMA CM (Connection Manager) to establish GPU transport over routers.

This capability is especially important in data center-scale AI clusters spanning multiple racks or zones, connected via IB routers like Mellanox SB7800 or QM8700 series.

When NCCL_IB_USE_IB_ROUTER=1 is set:

NCCL includes router-aware route resolution in its path selection logic.

Enables efficient zero-copy communication across GPUs in different IB domains, maintaining low latency.

Other Options Explained:

A . Lazy connection establishment -- controls when peer connections are made but does not enable cross-subnet reach.

B . GPU Direct RDMA -- enables intra-node direct memory access, not applicable for routing across subnets.

C . Static plugin linking -- affects how NCCL links plugins, not related to IB topology.

Exact Extract Reference:

Source: NVIDIA NCCL User Guide -- Environment Variables Section

Extract: 'NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers, required for multi-subnet InfiniBand configurations. Ensures proper routing of collectives over fabric-wide topologies.'

The ibstat command is utilized to display the operational status of InfiniBand Host Channel Adapters (HCAs). It provides detailed information, including the state (e.g., Active, Down), physical state (e.g., LinkUp, Polling), and link layer (e.g., InfiniBand, Ethernet) of each port on the HCA. This information is crucial for diagnosing connectivity issues and ensuring that the InfiniBand interfaces are functioning correctly.

Reference Extracts from NVIDIA Documentation:

'The ibstat command displays the status of the host channel adapters (HCAs) in your InfiniBand fabric. The status includes the HCAs' state, physical state, and link layer.'

'For proper operation, you are looking for 'State: Active' and 'Physical State: LinkUp'.'