Boost Jetson Xavier NX Performance: Implementing GCAPS on L4T R35.2.1

Want to optimize your Nvidia Jetson Xavier NX for better GPU performance? This guide walks you through implementing the GCAPS (GPU Compute and preemption API support) approach within the Tegra driver. By following these steps, you'll be able to leverage GCAPS for more efficient GPU utilization on your Jetson device.

This guide is specifically tailored for:

• Nvidia Jetson Xavier NX
• L4T R35.2.1 with Jetpack 5.0.2
• CUDA 11.4

Step-by-Step: GCAPS Driver Implementation for Jetson

We'll navigate through the process of patching, compiling, and installing the GCAPS-enabled driver.

1. Preparing Your System for GCAPS

First, we need to download the necessary source files and install crucial cross-compilation tools on your PC.

• Download Kernel and Driver Source: Get the BSP (Board Support Package) source from the Nvidia Developer website. Unzip the source package to access the kernel.
• Install Cross-Compilation Tools: This is essential for building the kernel on your PC for the Jetson's architecture. Use apt install build-essential bc to install the prerequisites. Then, download and extract the toolchain.

2. Applying the GCAPS Patches to the Driver

This step involves applying the provided patch files to the Tegra driver source code. These patches introduce the GCAPS functionality.

• Navigate to the nvgpu Driver Path: Go to the directory containing the nvgpu driver source code within the extracted BSP.
• Overwrite Files with Patches: Use the patch command to apply each patch file to its corresponding source file. These patch files are located in the gcaps-super-repo/gcaps_driver_patch directory. Specifically, you'll be patching:
  • drivers/gpu/nvgpu/os/linux/ioctl_ctrl.c
  • drivers/gpu/nvgpu/include/nvgpu/sched.h
  • drivers/gpu/nvgpu/os/linux/sched.c
  • include/uapi/linux/nvgpu-ctrl.h

3. Compiling and Installing the GCAPS Driver

Now, compile the kernel with the GCAPS patches and install the updated driver on your Jetson Xavier NX.

• Cross-Compile on PC: Set the necessary environment variables, including CROSS_COMPILE, LOCALVERSION, TEGRA_KERNEL_OUT, and INSTALL_MOD_PATH. These variables tell the build system where to find the cross-compiler and where to place the output files.
• Compile the Kernel and Modules: Use the make command with the tegra_defconfig target to create a default configuration file. Then, use make menuconfig to customize the kernel configuration. Finally, compile the kernel image and modules.
• Copy Files to the Target Platform: After successful compilation, copy the nvgpu.ko module, the Image file, and the nvgpu-ctrl.h header file to your Jetson Xavier NX.
• Install the Changes on the Target Platform (Jetson): Copy the new nvgpu.ko to /lib/modules/.../kernel/drivers/gpu/nvgpu/, the new Image to /boot/, and the new nvgpu-ctrl.h to /usr/src/linux-headers-.../include/uapi/linux/.

After copying the files, run sync and sudo reboot to apply the changes.

Important Note Regarding nvgpu-ctrl.h: After updating nvgpu-ctrl.h, remember to adjust the corresponding path in your userspace Makefile to ensure proper compilation of userspace applications.

4. Validating the GCAPS Implementation

With the new driver installed, how do you prove that all went well? After rebooting, verify the driver loaded correctly. Then, move on to testing GCAPS using the GCAPS userspace implementation. This confirms that the preemptive GPU approach is functioning as expected.

Benefit of GCAPS on Jetson Xavier NX:

By implementing GCAPS, you will see improved:

• GPU Utilization: Better scheduling leading resources used well.
• Responsiveness: Preemptive capabilities keep system more responsive.
• Workload Management: Efficient dealing with mixed graphic and compute tasks.