RapidIO Support for NVIDIA Jetson TK1
To enable RapidIO support to NVIDIA Jetson TK1 development board users have to patch and rebuild
board’s standard Linux kernel image using set of provided patches. This set of patches backports most
recent existing RapidIO functionality (as for mainline Linux kernel v3.17) to Linux kernel version currently
provided by NVIDIA for Jetson boards - v3.10.x.
Steps to build and install a customized Linux kernel image on Jetson board are well described on NVIDIA
embedded developers forum. For full details and discussions please follow the link belo:
https://devtalk.nvidia.com/default/topic/762653/embedded-systems/-howto-build-own-kernel-for-
jetson-tk1/1/ (Windows Explorer does not display the page properly, use Chrome to view this page)
For user convenience text from the HowTo guide referenced by the link is included into this document
with added specifics of enabling RapidIO support.
I. Update and rebuild Linux kernel
Before patching the kernel source code tree please unpack the latest available package
jetson_srio_kernel_patches_<release_date>.tbz2 into user home directory. For this example we will use
default user ubuntu as configured on fresh “out of the box” Jetson board.
ubuntu@jetson:~$ tar –xvf Jetson_srio_kernel_patches.tbz2
Follow the instructions below (steps added for RapidIO are highlighted in italic):
This method is for fastboot mode. If you use U-boot look to linuxdev post on this thread.
1. On Jetson-TK1
a) Install dependencies:
1. sudo apt-add-repository universe
2. sudo apt-get update
3. sudo apt-get install libncurses5-dev
b) Switch to root account:
1. sudo suc) Download, unpack & configure kernel source:
1. cd /usr/src/
2. wget
https://developer.nvidia.com/sites/default/files/akamai/mobile/files/L4T
/kernel_src.tbz2
3. tar -xvf kernel_src.tbz2
4. cd kernel
5. Apply patches from your home directory in their numbering order:
“patch –p1 < /home/ubuntu/SRIO_PATCHES/0001-rapidio-add-RapidIO...
“patch –p1 < /home/ubuntu/SRIO_PATCHES/0002-... and so on until the last
one.
6. Copy config file: cp /home/ubuntu/SRIO_PATCHES/.config .config
7. make menuconfig (if you want your own build options, by default all
RapidIO drivers will be built as loadable modules)
Go to: "General setup -> Local version" and set your version name eg.: -SRIO
After customize kernel options, save & exit. You should look "configuration written to .config"
d) Compile your kernel (about 10minutes)
1. make -j4
2. make modules
3. make modules_install
Please note that kernel re-flashing procedure may be changed over time and
apply steps specific for your specific version of Linux kernel package
provided by NVIDIA.
One of the nodes in a RapidIO network has to act as a network enumerator. To
enable this function kernel boot command line parameter “rapidio.hdi=x” where
x is number in 0 to 254 range should be added. See section 2d below.
e) Check modules was installed
1. ls /lib/modules/
You should look -SRIO modules directory:
3.10.24-gf455cd4 3.10.24-SRIO
f) Copy kernel zImage to USB or FTP from arch/arm/boot/ directory
-rwxr-xr-x 1 root 5721960 Jul 20 18:32 arch/arm/boot/zImageg) Turn off Jetson TK1, connect via USB cable to PC-host and turn Jetson in RECOVERY mode.
2. On PC host
a) On PC-Host you should look device 0955:7140:
1. lsusb
2. Bus 006 Device 010: ID 0955:7140 NVidia Corp.
b) Download L4T if you haven't yet:
1. wget
https://developer.nvidia.com/sites/default/files/akamai/mobile/files/L4T
/Tegra124_Linux_R19.3.0_armhf.tbz2
2. tar -xvf Tegra124_Linux_R19.3.0_armhf.tbz2
3. cd Linux_for_Tegra
c) Replace zImage in "kernel/" in your zImage file compiled before (1.f)
d) Flash kernel
1. sudo ./flash.sh -k 6 –C rapidio.hdid=1 jetson-tk1 mmcblk0p1 (enumerator)
2. sudo ./flash.sh -k 6 jetson-tk1 mmcblk0p1 (non-enumerating nodes)
You should look:
1. copying dtbfile(/root/Linux_for_Tegra/kernel/dtb/tegra124-pm375.dtb) to
tegra124-pm375.dtb... done.
2. Just updating kernel and boot device.
3. copying
bootloader(/root/Linux_for_Tegra/bootloader/ardbeg/fastboot.bin)...
done.
4. Bootloader(/root/Linux_for_Tegra/bootloader/ardbeg/fastboot.bin) used as
flasher.
5. Existing flash application(/root/Linux_for_Tegra/bootloader/nvflash)
reused.
6. making zero initrd...
7. done.
8. Making Boot image... done
9. *** Flashing kernel update started. ***
10. Nvflash 4.13.0000 started
11. chip uid from BR is: 0x340010017408f044100000000a020440
12. rcm version 0X400001
13. System Information:
14.
chip name: unknown
15.
chip id: 0x40 major: 1 minor: 116.
chip sku: 0x81
17.
chip uid: 0x000000017408f044100000000a020440
18.
macrovision: disabled
19.
hdcp: enabled
20.
jtag: enabled
21.
sbk burned: false
22.
board id: 375
23.
warranty fuse: 0
24.
dk burned: true
25.
boot device: emmc
26.
operating mode: 3
27.
device config strap: 0
28.
device config fuse: 0
29.
sdram config strap: 3
30.
31. RCM communication completed
32. downloading bootloader -- load address: 0x80108000 entry point:
0x80108000
33. sending file: fastboot.bin
34. - 900492/900492 bytes sent
35. fastboot.bin sent successfully
36. waiting for bootloader to initialize
37. bootloader downloaded successfully
38. sending file: boot.img
39. \ 5724160/5724160 bytes sent
40. boot.img sent successfully
41. Time taken for flashing 3 Secs
42. *** The kernel has been updated successfully. ***
IF FLASH FAILED and message is: "mapping system.img to loop device failed." edit flash.sh on line 284:
1. losetup /dev/loop0 system.img > /dev/null 2>&1
replace to:
1. losetup --find --show system.img
thanks linuxdev - more info
e) Reset Jetson (power off-on)
3. On Jetson-TK1
a) Check kernel version
1. uname -a1. Linux tegra-ubuntu 3.10.24-SRIO #1 SMP PREEMPT Sun Jul 20 18:31:40 CEST
2014 armv7l armv7l armv7l GNU/Linux
b) Copy extra modules from NVIDIA kernel
1. sudo cp -rv /lib/modules/3.10.24-gf455cd4/extra /lib/modules/`uname -r`/
2. sudo depmod -a
Well done. You have own kernel!
II. Build and Run Test Applications
To install provided sample test programs unpack a provided package jetson_srio_tests_<date>.tbz2 into
user’s home directory:
ubuntu@jetson:~$ tar –xvf jetson_srio_tests.tbz2
These tests will be unpacked into mcd_base directory and already include pre-built executable binary
files. To rebuild test programs do following steps:
1. cd mcd_base
2. make clean
3. make KDIR=/usr/src/kernel (if kernel source code directory path is set as in section I above).
Before running test applications make sure that RapidIO subsystem is initialized and all drivers are
loaded (See section III below).
Because test applications access to RapidIO specific device drivers, user must acquire the root privileges
before running them (“sudo su”).
DMA test application report data transfer performance number. To achieve better performance
numbers please make sure that RapidIO subsystem debug messages are disabled in kernel configuration
options. By default, the config file provided in the patch package enables RapidIO debug messages to
allow new users to verify their RapidIO system operations.
1. RapidIO Subsystem Initialization
To initialize RapidIO network in your Jetson cluster setup, first power up all participating nodes and
RapidIO switch(es) connecting them.
Identify enumerating node (one that was flashed with kernel boot command line parameter
“rapidio.hdid=” in the section I).Load RapidIO modules using following steps:
1.
2.
3.
4.
sudo su
modprobe rio-scan scan=1
modprobe rio_mport_cdev
modprobe rio_cm
To check result of RapidIO network enumeration you can do “ls /sys/bus/rapidio/devices” which will list
attached RapidIO devices (local node not shown). Example of such operation for system with two nodes
and one switch is shown below.
After RapidIO initialization steps have been performed on the enumerator node, steps listed above can
be performed on remaining nodes in a RapidIO network. As result each node should create list of
attached RapidIO devices (with different local nodes absent on each node) and create character device
files /dev/rio_cm and /dev/rio_mportN [N=0...(number_of_ local_SRIO_ports - 1)].2. Running Test Programs
To run test program a user must have root priveleges.
All programs are printing their usage information if started with “-h” option.
The “riodp_test_misc –q” can be used to check SRIO link/port status before running data transfer tests.
The program riodp_test_dma has two roles – it can act as a passive remote memory target or as active
DMA user that performs writes and reads to/from remote target memory. To run a DMA test between
two RapidIO nodes the remote target memory has to be initiated first and DMA transfers started after
that. An example below assumes that RapidIO network has two nodes with destination IDs 0 and 1 and
user selects a node with destID 1 to be an active DMA requester:
1. Initiate target memory space on the node with destID=0: “riodp_test_dma –i” will allocate
kernel space contiguous buffer and map it into RapidIO address (using default settings: buffer
size = 2MB and RapidIO base address = 0).
2. On the node with destID=1 start the DMA requestor: “riodp_test_dma –D0 –S0x200000” will
perform single DMA data exchange cycle (write data to remote memory, read them back and
compare). NOTES: A) Data transfer size must be equal or less than size of remote target
memory. B) “-k” option uses contiguous kernel space buffers and therefore allows to have
better data transfer performance compared to user space buffers that may require large
number of memory pages to be used.
The RapidIO messaging test performs messaging exchange using channelized message exchange similar
to sockets and is provided in classic form of client and server applications. The server application
riodp_test_server has to be started first followed by the client: riodp_test_client.