API tools faq
paste
Guest User

README

a guest
May 9th, 2024
38
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
text 112.53 KB | None | 0 0
raw download clone embed print report
  1. ice Linux* Base Driver for the Intel(R) Ethernet 800 Series
  2. ===============================================================================
  3.  
  4. June 23, 2023
  5.  
  6. ===============================================================================
  7.  
  8. Contents
  9. --------
  10.  
  11. - Overview
  12. - Identifying Your Adapter
  13. - Important Notes
  14. - Building and Installation
  15. - Command Line Parameters
  16. - Additional Features & Configurations
  17. - Performance Optimization
  18. - Known Issues/Troubleshooting
  19.  
  20.  
  21. Overview
  22. ========
  23. This driver supports kernel versions 3.10.0 and newer. However, some features
  24. may require a newer kernel version. The associated Virtual Function (VF) driver
  25. for this driver is iavf. The associated RDMA driver for this driver is irdma.
  26.  
  27. Driver information can be obtained using ethtool, devlink, lspci, and ip.
  28. Instructions on updating ethtool can be found in the section Additional
  29. Configurations later in this document.
  30.  
  31. This driver is only supported as a loadable module at this time. Intel is not
  32. supplying patches against the kernel source to allow for static linking of the
  33. drivers.
  34.  
  35. For questions related to hardware requirements, refer to the documentation
  36. supplied with your Intel adapter. All hardware requirements listed apply to use
  37. with Linux.
  38.  
  39. This driver supports XDP (Express Data Path) on kernel 4.14 and later and
  40. AF_XDP zero-copy on kernel 4.18 and later. Note that XDP is blocked for frame
  41. sizes larger than 3KB.
  42.  
  43.  
  44. Related Documentation
  45. =====================
  46.  
  47. See the "Intel(R) Ethernet Adapters and Devices User Guide" for additional
  48. information on features. It is available on the Intel website at either of the
  49. following:
  50.  
  51. - https://cdrdv2.intel.com/v1/dl/getContent/705831
  52. -
  53. https://www.intel.com/content/www/us/en/download/19373/adapter-user-guide-for-in
  54. tel-ethernet-adapters.html
  55.  
  56.  
  57. Identifying Your Adapter
  58. ========================
  59. The driver is compatible with devices based on the following:
  60. * Intel(R) Ethernet Controller E810-C
  61. * Intel(R) Ethernet Controller E810-XXV
  62. * Intel(R) Ethernet Connection E822-C
  63. * Intel(R) Ethernet Connection E822-L
  64. * Intel(R) Ethernet Connection E823-C
  65. * Intel(R) Ethernet Connection E823-L
  66.  
  67. For information on how to identify your adapter, and for the latest Intel
  68. network drivers, refer to the Intel Support website:
  69. http://www.intel.com/support
  70.  
  71.  
  72. Important Notes
  73. ===============
  74.  
  75. Configuring SR-IOV for improved network security
  76. ------------------------------------------------
  77. In a virtualized environment, on Intel(R) Ethernet Network Adapters that
  78. support SR-IOV or Intel(R) Scalable I/O Virtualization (Intel(R) Scalable IOV),
  79. the virtual function (VF) may be subject to malicious behavior.
  80. Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
  81. 802.1Qbb (priority based flow-control), and others of this type, are not
  82. expected and can throttle traffic between the host and the virtual switch,
  83. reducing performance. To resolve this issue, and to ensure isolation from
  84. unintended traffic streams, configure all SR-IOV or Intel Scalable IOV enabled
  85. ports for VLAN tagging from the administrative interface on the PF. This
  86. configuration allows unexpected, and potentially malicious, frames to be
  87. dropped.
  88.  
  89. See "Configuring VLAN Tagging on SR-IOV Enabled Adapter Ports" or "Intel(R)
  90. Scalable I/O Virtualization Support" later in this README for configuration
  91. instructions.
  92.  
  93.  
  94. Do not unload port driver if VF with active VM is bound to it
  95. -------------------------------------------------------------
  96. Do not unload a port's driver if a Virtual Function (VF) with an active Virtual
  97. Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
  98. Once the VM shuts down, or otherwise releases the VF, the command will complete.
  99.  
  100.  
  101. Firmware Recovery Mode
  102. ----------------------
  103. A device will enter Firmware Recovery mode if it detects a problem that
  104. requires the firmware to be reprogrammed. When a device is in Firmware Recovery
  105. mode it will not pass traffic or allow any configuration; you can only attempt
  106. to recover the device's firmware. Refer to the Intel(R) Ethernet Adapters and
  107. Devices User Guide for details on Firmware Recovery Mode and how to recover
  108. from it.
  109.  
  110.  
  111. Important notes for SR-IOV, RDMA, and Link Aggregation
  112. ------------------------------------------------------
  113. The VF driver will not block teaming/bonding/link aggregation, but this is not
  114. a supported feature. Do not expect failover or load balancing on the VF
  115. interface.
  116.  
  117. LAG and RDMA are compatible only in certain conditions. See the "RDMA (Remote
  118. Direct Memory Access)" section later in this README for more information.
  119.  
  120. Bridging and MACVLAN are also affected by this. If you wish to use bridging or
  121. MACVLAN with RDMA/SR-IOV, you must set up bridging or MACVLAN before enabling
  122. RDMA or SR-IOV. If you are using bridging or MACVLAN in conjunction with SR-IOV
  123. and/or RDMA, and you want to remove the interface from the bridge or MACVLAN,
  124. you must follow these steps:
  125. 1. Remove RDMA if it is active
  126. 2. Destroy SR-IOV VFs if they exist
  127. 3. Remove the interface from the bridge or MACVLAN
  128. 4. Reactivate RDMA and recreate SR-IOV VFs as needed
  129.  
  130.  
  131. Building and Installation
  132. =========================
  133. The ice driver requires the Dynamic Device Personalization (DDP) package file
  134. to enable advanced features (such as dynamic tunneling, Intel(R) Ethernet Flow
  135. Director, RSS, and ADQ, or others). The driver installation process installs
  136. the default DDP package file and creates a soft link ice.pkg to the physical
  137. package ice-x.x.x.x.pkg in the firmware root directory (typically
  138. /lib/firmware/ or /lib/firmware/updates/). The driver install process also puts
  139. both the driver module and the DDP file in the initramfs/initrd image.
  140.  
  141. NOTE: When the driver loads, it looks for intel/ice/ddp/ice.pkg in the firmware
  142. root. If this file exists, the driver will download it into the device. If not,
  143. the driver will go into Safe Mode where it will use the configuration contained
  144. in the device's NVM. This is NOT a supported configuration and many advanced
  145. features will not be functional. See "Dynamic Device Personalization" later for
  146. more information.
  147.  
  148.  
  149. To manually build the driver
  150. ----------------------------
  151. 1. Move the base driver tar file to the directory of your choice.
  152. For example, use '/home/username/ice' or '/usr/local/src/ice'.
  153.  
  154. 2. Untar/unzip the archive, where <x.x.x> is the version number for the
  155. driver tar file:
  156.  
  157. # tar zxf ice-<x.x.x>.tar.gz
  158.  
  159. 3. Change to the driver src directory, where <x.x.x> is the version number
  160. for the driver tar:
  161.  
  162. # cd ice-<x.x.x>/src/
  163.  
  164. 4. Compile the driver module:
  165.  
  166. # make install
  167.  
  168. The binary will be installed as:
  169. /lib/modules/<KERNEL VER>/updates/drivers/net/ethernet/intel/ice/ice.ko
  170.  
  171. The install location listed above is the default location. This may differ
  172. for various Linux distributions.
  173.  
  174. NOTE: To build the driver using the schema for unified ethtool statistics
  175. defined in https://sourceforge.net/p/e1000/wiki/Home/, use the following
  176. command:
  177.  
  178. # make CFLAGS_EXTRA='-DUNIFIED_STATS' install
  179.  
  180. NOTE: To compile the driver with ADQ (Application Device Queues) flags set,
  181. use the following command, where <nproc> is the number of logical cores:
  182.  
  183. # make -j<nproc> CFLAGS_EXTRA='-DADQ_PERF_COUNTERS' install
  184.  
  185. (This will also apply the above 'make install' command.)
  186.  
  187. NOTE: You may see warnings from depmod related to unknown RDMA symbols
  188. during the make of the OOT base driver. These warnings are normal and
  189. appear because the in-tree RDMA driver will not work with the OOT base
  190. driver. To address the issue, you need to install the latest OOT versions
  191. of the base and RDMA drivers.
  192.  
  193.  
  194. 5. Load the module using the modprobe command.
  195.  
  196. To check the version of the driver and then load it:
  197.  
  198. # modinfo ice
  199. # modprobe ice
  200.  
  201. Alternately, make sure that any older ice drivers are removed from the
  202. kernel before loading the new module:
  203.  
  204. # rmmod ice; modprobe ice
  205.  
  206. NOTE: To enable verbose debug messages in the kernel log, use the dynamic debug
  207. feature (dyndbg). See "Dynamic Debug" later in this README for more information.
  208.  
  209. 6. Assign an IP address to the interface by entering the following,
  210. where <ethX> is the interface name that was shown in dmesg after modprobe:
  211.  
  212. # ip address add <IP_address>/<netmask bits> dev <ethX>
  213.  
  214. 7. Verify that the interface works. Enter the following, where IP_address
  215. is the IP address for another machine on the same subnet as the interface
  216. that is being tested:
  217.  
  218. # ping <IP_address>
  219.  
  220.  
  221. To build a binary RPM package of this driver
  222. --------------------------------------------
  223. Note: RPM functionality has only been tested in Red Hat distributions.
  224.  
  225. 1. Run the following command, where <x.x.x> is the version number for the
  226. driver tar file.
  227.  
  228. # rpmbuild -tb ice-<x.x.x>.tar.gz
  229.  
  230. NOTE: For the build to work properly, the currently running kernel MUST
  231. match the version and configuration of the installed kernel sources. If
  232. you have just recompiled the kernel, reboot the system before building.
  233.  
  234. 2. After building the RPM, the last few lines of the tool output contain the
  235. location of the RPM file that was built. Install the RPM with one of the
  236. following commands, where <RPM> is the location of the RPM file:
  237.  
  238. # rpm -Uvh <RPM>
  239. or
  240. # dnf/yum localinstall <RPM>
  241.  
  242. 3. If your distribution or kernel does not contain inbox support for auxiliary
  243. bus, you must also install the auxiliary RPM:
  244.  
  245. # rpm -Uvh <ice RPM> <auxiliary RPM>
  246. or
  247. # dnf/yum localinstall <ice RPM> <auxiliary RPM>
  248.  
  249. NOTE: On some distributions, the auxiliary RPM may fail to install due to
  250. missing kernel-devel headers. To workaround this issue, specify '--excludepath'
  251. during installation. For example:
  252.  
  253. # rpm -Uvh auxiliary-1.0.0-1.x86_64.rpm
  254. --excludepath=/lib/modules/3.10.0-957.el7.x86_64/source/include/linux/auxiliary_
  255. bus.h
  256.  
  257. NOTES:
  258. - To compile the driver on some kernel/arch combinations, you may need to
  259. install a package with the development version of libelf (e.g. libelf-dev,
  260. libelf-devel, elfutils-libelf-devel).
  261. - When compiling an out-of-tree driver, details will vary by distribution.
  262. However, you will usually need a kernel-devel RPM or some RPM that provides the
  263. kernel headers at a minimum. The RPM kernel-devel will usually fill in the link
  264. at /lib/modules/'uname -r'/build.
  265.  
  266.  
  267. Command Line Parameters
  268. =======================
  269. The only command line parameter the ice driver supports is the debug parameter
  270. that can control the default logging verbosity of the driver. (Note: dyndbg
  271. also provides dynamic debug information.)
  272.  
  273. In general, use ethtool and other OS-specific commands to configure
  274. user-changeable parameters after the driver is loaded.
  275.  
  276.  
  277. Additional Features and Configurations
  278. ======================================
  279.  
  280. ethtool
  281. -------
  282. The driver utilizes the ethtool interface for driver configuration and
  283. diagnostics, as well as displaying statistical information. The latest ethtool
  284. version is required for this functionality. Download it at:
  285. https://kernel.org/pub/software/network/ethtool/
  286.  
  287.  
  288. Viewing Link Messages
  289. ---------------------
  290. Link messages will not be displayed to the console if the distribution is
  291. restricting system messages. In order to see network driver link messages on
  292. your console, set dmesg to eight by entering the following:
  293.  
  294. # dmesg -n 8
  295.  
  296. NOTE: This setting is not saved across reboots.
  297.  
  298.  
  299. Dynamic Device Personalization
  300. ------------------------------
  301. Dynamic Device Personalization (DDP) allows you to change the packet processing
  302. pipeline of a device by applying a profile package to the device at runtime.
  303. Profiles can be used to, for example, add support for new protocols, change
  304. existing protocols, or change default settings. DDP profiles can also be rolled
  305. back without rebooting the system.
  306.  
  307. The ice driver automatically installs the default DDP package file during
  308. driver installation. NOTE: It's important to do 'make install' during initial
  309. ice driver installation so that the driver loads the DDP package automatically.
  310.  
  311. The DDP package loads during device initialization. The driver looks for
  312. intel/ice/ddp/ice.pkg in your firmware root (typically /lib/firmware/ or
  313. /lib/firmware/updates/) and checks that it contains a valid DDP package file.
  314.  
  315. If the driver is unable to load the DDP package, the device will enter Safe
  316. Mode. Safe Mode disables advanced and performance features and supports only
  317. basic traffic and minimal functionality, such as updating the NVM or
  318. downloading a new driver or DDP package. Safe Mode only applies to the affected
  319. physical function and does not impact any other PFs. See the "Intel(R) Ethernet
  320. Adapters and Devices User Guide" for more details on DDP and Safe Mode.
  321.  
  322. NOTES:
  323. - If you encounter issues with the DDP package file, you may need to download
  324. an updated driver or DDP package file. See the log messages for more
  325. information.
  326.  
  327. - The ice.pkg file is a symbolic link to the default DDP package file installed
  328. by the Linux-firmware software package or the ice out-of-tree driver
  329. installation.
  330.  
  331. - You cannot update the DDP package if any PF drivers are already loaded. To
  332. overwrite a package, unload all PFs and then reload the driver with the new
  333. package.
  334.  
  335. - Only the first loaded PF per device can download a package for that device.
  336.  
  337. You can install specific DDP package files for different physical devices in
  338. the same system. To install a specific DDP package file:
  339.  
  340. 1. Download the DDP package file you want for your device.
  341.  
  342. 2. Rename the file ice-xxxxxxxxxxxxxxxx.pkg, where 'xxxxxxxxxxxxxxxx' is the
  343. unique 64-bit PCI Express device serial number (in hex) of the device you want
  344. the package downloaded on. The filename must include the complete serial number
  345. (including leading zeros) and be all lowercase. For example, if the 64-bit
  346. serial number is b887a3ffffca0568, then the file name would be
  347. ice-b887a3ffffca0568.pkg.
  348.  
  349. To find the serial number from the PCI bus address, you can use the following
  350. command:
  351.  
  352. # lspci -vv -s af:00.0 | grep -i Serial
  353. Capabilities: [150 v1] Device Serial Number b8-87-a3-ff-ff-ca-05-68
  354.  
  355. You can use the following command to format the serial number without the
  356. dashes:
  357.  
  358. # lspci -vv -s af:00.0 | grep -i Serial | awk '{print $7}' | sed s/-//g
  359. b887a3ffffca0568
  360.  
  361. 3. Copy the renamed DDP package file to /lib/firmware/updates/intel/ice/ddp/.
  362. If the directory does not yet exist, create it before copying the file.
  363.  
  364. 4. Unload all of the PFs on the device.
  365.  
  366. 5. Reload the driver with the new package.
  367.  
  368. NOTE: The presence of a device-specific DDP package file overrides the loading
  369. of the default DDP package file (ice.pkg).
  370.  
  371.  
  372. RDMA (Remote Direct Memory Access)
  373. ----------------------------------
  374. Remote Direct Memory Access, or RDMA, allows a network device to transfer data
  375. directly to and from application memory on another system, increasing
  376. throughput and lowering latency in certain networking environments.
  377.  
  378. The ice driver supports the following RDMA protocols:
  379. - iWARP (Internet Wide Area RDMA Protocol)
  380. - RoCEv2 (RDMA over Converged Ethernet)
  381. The major difference is that iWARP performs RDMA over TCP, while RoCEv2 uses
  382. UDP.
  383.  
  384. RDMA requires auxiliary bus support. Refer to "Auxiliary Bus" in this
  385. README for more information.
  386.  
  387. Devices based on the Intel(R) Ethernet 800 Series do not support RDMA when
  388. operating in multiport mode with more than 4 ports.
  389.  
  390. For detailed installation and configuration information for RDMA, see the
  391. README file in the irdma driver tarball.
  392.  
  393.  
  394. RDMA in the VF
  395. --------------
  396. Devices based on the Intel(R) Ethernet 800 Series support RDMA in a Linux VF,
  397. on supported Windows or Linux hosts.
  398.  
  399. The iavf driver supports the following RDMA protocols in the VF:
  400. - iWARP (Internet Wide Area RDMA Protocol)
  401. - RoCEv2 (RDMA over Converged Ethernet)
  402.  
  403. Refer to the README inside the irdma driver tarball for details on configuring
  404. RDMA in the VF.
  405.  
  406. NOTE: To support VF RDMA, load the irdma driver on the host before creating
  407. VFs. Otherwise VF RDMA support may not be negotiated between the VF and PF
  408. driver.
  409.  
  410.  
  411. Auxiliary Bus
  412. -------------
  413. Inter-Driver Communication (IDC) is the mechanism in which LAN drivers (such as
  414. ice) communicate with peer drivers (such as irdma). Starting in kernel 5.11,
  415. Intel LAN and RDMA drivers use an auxiliary bus mechanism for IDC.
  416.  
  417. RDMA functionality requires use of the auxiliary bus.
  418.  
  419. If your kernel supports the auxiliary bus, the LAN and RDMA drivers will use
  420. the inbox auxiliary bus for IDC. For kernels lower than 5.11, the base driver
  421. will automatically install an out-of-tree auxiliary bus module.
  422.  
  423.  
  424. NVM Express* (NVMe) over TCP and Fabrics
  425. ----------------------------------------
  426. RDMA provides a high throughput, low latency means to directly access NVM
  427. Express* (NVMe*) drives on a remote server.
  428.  
  429. Refer to the following for details on supported operating systems and how to
  430. set up and configure your server and client systems:
  431. - NVM Express over TCP for Intel(R) Ethernet Products Configuration Guide
  432. - NVM Express over Fabrics for Intel(R) Ethernet Products with RDMA
  433. Configuration Guide
  434.  
  435. Both guides are available on the Intel Technical Library at:
  436. https://www.intel.com/content/www/us/en/design/products-and-solutions/networking
  437. -and-io/ethernet-controller-e810/technical-library.html
  438.  
  439.  
  440. Link Aggregation and RDMA
  441. -------------------------
  442. Link aggregation (LAG) and RDMA are compatible only if all the following are
  443. true:
  444. - You are using an Intel Ethernet 810 Series device with the latest drivers and
  445. NVM installed.
  446. - RDMA technology is set to RoCEv2.
  447. - LAG configuration is active-backup.
  448. - Bonding is between two ports within the same device.
  449. - The QoS configuration of the two ports matches prior to the bonding of the
  450. devices.
  451.  
  452. If the above conditions are not met:
  453. - The PF driver will not enable RDMA.
  454. - RDMA peers will not be able to register with the PF.
  455.  
  456. NOTE: The first interface added to an aggregate (bond) is assigned as the
  457. "primary" interface for RDMA and LAG functionality. If LAN interfaces are
  458. assigned to the bond and you remove the primary interface from the bond, RDMA
  459. will not function properly over the bonded interface. To address the issue,
  460. remove all interfaces from the bond and add them again. Interfaces that are not
  461. assigned to the bond will operate normally.
  462.  
  463.  
  464. Application Device Queues (ADQ)
  465. -------------------------------
  466. Application Device Queues (ADQ) allow you to dedicate one or more queues to a
  467. specific application. This can reduce latency for the specified application,
  468. and allow Tx traffic to be rate limited per application.
  469.  
  470. The ADQ information contained here is specific to the ice driver. For more
  471. details, refer to the E810 ADQ Configuration Guide at:
  472. https://cdrdv2.intel.com/v1/dl/getContent/609008
  473.  
  474. Requirements:
  475. - Kernel version: Varies by feature. Refer to the E810 ADQ Configuration Guide
  476. for more information on required kernel versions for different ADQ features.
  477. - Operating system: Red Hat* Enterprise Linux* 7.5+ or SUSE* Linux Enterprise
  478. Server* 12+
  479. - The latest ice driver and NVM image (Note: You must compile the ice driver
  480. with the ADQ flag as shown in the "Building and Installation" section.)
  481. - The sch_mqprio, act_mirred and cls_flower modules must be loaded. For example:
  482. # modprobe sch_mqprio
  483. # modprobe act_mirred
  484. # modprobe cls_flower
  485. - The latest version of iproute2
  486. We recommend the following installation method:
  487. # cd iproute2
  488. # ./configure
  489. # make DESTDIR=/opt/iproute2 install
  490. # ln -s /opt/iproute2/sbin/tc /usr/local/sbin/tc
  491.  
  492. When ADQ is enabled:
  493. - You cannot change RSS parameters, the number of queues, or the MAC address in
  494. the PF or VF. Delete the ADQ configuration before changing these settings.
  495. - The driver supports subnet masks for IP addresses in the PF and VF. When you
  496. add a subnet mask filter, the driver forwards packets to the ADQ VSI instead of
  497. the main VSI.
  498. - When the PF adds or deletes a port VLAN filter for the VF, it will extend to
  499. all the VSIs within that VF.
  500. - The driver supports ADQ and GTP filters in the PF. Note: You must have a DDP
  501. package that supports GTP; the default OS package does not. Download the
  502. appropriate package from your hardware vendor and load it on your device.
  503. - ADQ allows tc ingress filters that include any destination MAC address.
  504. - You can configure up to 256 queue pairs (256 MSI-X interrupts) per PF.
  505.  
  506. See "Creating traffic class filters" in this README for more information on
  507. configuring filters, including examples. See the E810 ADQ Configuration Guide
  508. for detailed instructions.
  509.  
  510. ADQ KNOWN ISSUES:
  511. - The latest RHEL and SLES distros have kernels with back-ported support for
  512. ADQ. For all other Linux distributions, you must use LTS Linux kernel v4.19.58
  513. or higher to use ADQ. The latest out-of-tree driver is required for ADQ on all
  514. operating systems.
  515.  
  516. - You must clear ADQ configuration in the reverse order of the initial
  517. configuration steps. Issues may result if you do not execute the steps to clear
  518. ADQ configuration in the correct order.
  519.  
  520. - ADQ configuration is not supported on a bonded or teamed ice interface.
  521. Issuing the ethtool or tc commands to a bonded ice interface will result in
  522. error messages from the ice driver to indicate the operation is not supported.
  523.  
  524. - If the application stalls, the application-specific queues may stall for up
  525. to two seconds. Configuring only one application per Traffic Class (TC) channel
  526. may resolve the issue.
  527.  
  528. - DCB and ADQ cannot coexist. A switch with DCB enabled might remove the ADQ
  529. configuration from the device. To resolve the issue, do not enable DCB on the
  530. switch ports being used for ADQ. You must disable LLDP on the interface and
  531. stop the firmware LLDP agent using the following command:
  532. # ethtool --set-priv-flags <ethX> fw-lldp-agent off
  533.  
  534. - MACVLAN offloads and ADQ are mutually exclusive. System instability may occur
  535. if you enable l2-fwd-offload and then set up ADQ, or if you set up ADQ and then
  536. enable l2-fwd-offload.
  537.  
  538. - NOTE (unrelated to Intel drivers): The version 5.8.0 Linux kernel introduced
  539. a bug that broke the interrupt affinity setting mechanism, which breaks the
  540. ability to pin interrupts to ADQ hardware queues. Use an earlier or later
  541. version of the Linux kernel.
  542.  
  543. - A core-level reset of an ADQ-configured PF port (rare events usually
  544. triggered by other failures in the device or ice driver) results in loss of ADQ
  545. configuration. To recover, reapply the ADQ configuration to the PF interface.
  546.  
  547. - Commands such as 'tc qdisc add' and 'ethtool -L' will cause the driver to
  548. close the associated RDMA interface and reopen it. This will disrupt RDMA
  549. traffic for 3-5 seconds until the RDMA interface is available again for
  550. traffic.
  551.  
  552. - Commands such as 'tc qdisc add' and 'ethtool -L' will clear other tuning
  553. settings such as interrupt affinity. These tuning settings will need to be
  554. reapplied. When the number of queues are increased using 'ethtool -L', the new
  555. queues will have the same interrupt moderation settings as queue 0 (i.e., Tx
  556. queue 0 for new Tx queues and Rx queue 0 for new Rx queues). You can change
  557. this using the ethtool per-queue coalesce commands.
  558.  
  559. - TC filters may not get offloaded in hardware if you apply them immediately
  560. after issuing the 'tc qdisc add' command. We recommend you wait 5 seconds after
  561. issuing 'tc qdisc add' before adding TC filters. Dmesg will report the error if
  562. TC filters fail to add properly.
  563.  
  564.  
  565. Setting up ADQ
  566. --------------
  567. To set up the adapter for ADQ, where <ethX> is the interface in use:
  568. 1. Reload the ice driver to remove any previous TC configuration:
  569.  
  570. # rmmod ice
  571. # modprobe ice
  572.  
  573. 2. Enable hardware TC offload on the interface:
  574.  
  575. # ethtool -K <ethX> hw-tc-offload on
  576.  
  577. 3. Disable LLDP on the interface, if it isn't already:
  578.  
  579. # ethtool --set-priv-flags <ethX> fw-lldp-agent off
  580.  
  581. 4. Verify settings:
  582.  
  583. # ethtool -k <ethX> | grep "hw-tc"
  584. # ethtool --show-priv-flags <ethX>
  585.  
  586.  
  587. ADQ Configuration Script
  588. ------------------------
  589. Intel also provides a script to configure ADQ. This script allows you configure
  590. ADQ-specific parameters such as traffic classes, priority, filters, and ethtool
  591. parameters.
  592.  
  593. Refer to the README.md file in scripts/adqsetup inside the driver tarball for
  594. more information.
  595.  
  596. The script and README are also available as part of the Python Package Index:
  597. https://pypi.org/project/adqsetup
  598.  
  599.  
  600. Using ADQ with independent pollers
  601. ----------------------------------
  602. The ice driver supports ADQ acceleration using independent pollers. Independent
  603. pollers are kernel threads invoked by interrupts and are used for busy polling
  604. on behalf of the application.
  605.  
  606. You can configure the number of queues per poller and poller timeout per ADQ
  607. traffic class (TC) or queue group using the 'devlink dev param' interface.
  608.  
  609. To set the number of queue pairs per poller, use the following:
  610.  
  611. # devlink dev param set <pci/D:b:d.f> name tc<x>_qps_per_poller value <num>
  612. cmode runtime
  613.  
  614. Where:
  615. - <pci/D:b:d.f> is the PCI address of the device
  616. (pci/Domain:bus:device.function).
  617. - tc<x> is the traffic class number.
  618. - <num> is the number of queues of the corresponding traffic class that each
  619. poller would poll.
  620.  
  621. To set the timeout for the independent poller, use the following:
  622.  
  623. # devlink dev param set <pci/D:b:d.f> name tc<x>_poller_timeout value <num>
  624. cmode runtime
  625.  
  626. Where:
  627. - <pci/D:b:d.f> is the PCI address of the device
  628. (pci/Domain:bus:device.function).
  629. - tc<x> is the traffic class number.
  630. - <num> is a nonzero integer value in jiffies.
  631.  
  632. For example:
  633.  
  634. - To configure 3 queues of TC1 to be polled by each independent poller:
  635.  
  636. # devlink dev param set pci/0000:3b:00.0 name tc1_qps_per_poller value 3 cmode
  637. runtime
  638.  
  639. - To set the timeout value in jiffies for TC1 when no traffic is flowing:
  640.  
  641. # devlink dev param set pci/0000:3b:00.0 name tc1_poller_timeout value 1000
  642. cmode runtime
  643.  
  644.  
  645. Configuring ADQ flows per traffic class
  646. ---------------------------------------
  647. The ice OOT driver allows you to configure inline Intel(R) Ethernet Flow
  648. Director (Intel(R) Ethernet FD) filters per traffic class (TC) using the
  649. devlink interface. Inline Intel Ethernet FD allows uniform distribution of
  650. flows among queues in a TC.
  651.  
  652. NOTE:
  653. - This functionality requires Linux kernel version 5.6 or newer and is
  654. supported only with the OOT ice driver.
  655. - You must enable Transmit Packet Steering (XPS) using receive queues for this
  656. feature to work correctly.
  657. - Per-TC filters set with devlink are not compatible with Intel Ethernet FD
  658. filters set via ethtool.
  659.  
  660. Use the following to configure inline Intel Ethernet FD filters per TC:
  661.  
  662. # devlink dev param set <pci/D:b:d.f> name tc<x>_inline_fd value <setting>
  663. cmode runtime
  664.  
  665. Where:
  666. - <pci/D:b:d.f> is the PCI address of the device
  667. (pci/Domain:bus:device.function).
  668. - tc<x> is the traffic class number.
  669. - <setting> is true to enable inline per-TC Intel Ethernet FD, or false to
  670. disable it.
  671.  
  672. For example, to enable inline Intel Ethernet FD for TC1:
  673.  
  674. # devlink dev param set pci/0000:af:00.0 name tc1_inline_fd value true cmode
  675. runtime
  676.  
  677. To show the current inline Intel Ethernet FD setting:
  678.  
  679. # devlink dev param show <pci/D:b:d.f> name tc<x>_inline_fd
  680.  
  681. For example, to show the inline Intel Ethernet FD setting for TC2 for the
  682. specified device:
  683.  
  684. # devlink dev param show pci/0000:af:00.0 name tc2_inline_fd
  685.  
  686.  
  687. Creating traffic classes
  688. ------------------------
  689. NOTE: These instructions are not specific to ADQ configuration. Refer to the tc
  690. and tc-flower man pages for more information on creating traffic classes (TCs).
  691.  
  692. To create traffic classes on the interface:
  693.  
  694. 1. Use the tc command to create traffic classes. You can create a maximum of
  695. 16 TCs per interface.
  696.  
  697. # tc qdisc add dev <ethX> root mqprio num_tc <tcs> map <priorities>
  698. queues <count1@offset1 ...> hw 1 mode channel shaper bw_rlimit
  699. min_rate <min_rate1 ...> max_rate <max_rate1 ...>
  700. Where:
  701. num_tc <tcs>: The number of TCs to use.
  702. map <priorities>: The map of priorities to TCs. You can map up to
  703. 16 priorities to TCs.
  704. queues <count1@offset1 ...>: For each TC, <num queues>@<offset>. The max
  705. total number of queues for all TCs is the number of cores.
  706. hw 1 mode channel: 'channel' with 'hw' set to 1 is a new hardware offload
  707. mode in mqprio that makes full use of the mqprio options, the TCs,
  708. the queue configurations, and the QoS parameters.
  709. shaper bw_rlimit: For each TC, sets the minimum and maximum bandwidth
  710. rates. The totals must be equal to or less than the port speed. This
  711. parameter is optional and is required only to set up the Tx rates.
  712. min_rate <min_rate1>: Sets the minimum bandwidth rate limit for each TC.
  713. max_rate <max_rate1 ...>: Sets the maximum bandwidth rate limit for each
  714. TC. You can set a min and max rate together.
  715.  
  716. NOTE:
  717. - If you set max_rate to less than 50Mbps, then max_rate is rounded up to
  718. 50Mbps and a warning is logged in dmesg.
  719. - See the mqprio man page and the examples below for more information.
  720.  
  721. 2. Verify the bandwidth limit using network monitoring tools such as ifstat or
  722. sar -n DEV [interval] [number of samples]
  723.  
  724. NOTE: Setting up channels via ethtool (ethtool -L) is not supported when the
  725. TCs are configured using mqprio.
  726.  
  727. 3. Enable hardware TC offload on the interface:
  728.  
  729. # ethtool -K <ethX> hw-tc-offload on
  730.  
  731. 4. Add clsact qdisc to enable adding ingress/egress filters for Rx/Tx:
  732.  
  733. # tc qdisc add dev <ethX> clsact
  734.  
  735. 5. Verify successful TC creation after qdisc is created:
  736.  
  737. # tc qdisc show dev <ethX> ingress
  738.  
  739.  
  740. TRAFFIC CLASS EXAMPLES:
  741. See the tc and tc-flower man pages for more information on traffic control and
  742. TC flower filters.
  743.  
  744. - To set up two TCs (tc0 and tc1), with 16 queues each, priorities 0-3 for
  745. tc0 and 4-7 for tc1, and max Tx rate set to 1Gbit for tc0 and 3Gbit for tc1:
  746.  
  747. # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
  748. 16@0 16@16 hw 1 mode channel shaper bw_rlimit max_rate 1Gbit 3Gbit
  749. Where:
  750. map 0 0 0 0 1 1 1 1: Sets priorities 0-3 to use tc0 and 4-7 to use tc1
  751. queues 16@0 16@16: Assigns 16 queues to tc0 at offset 0 and 16 queues
  752. to tc1 at offset 16
  753.  
  754. - To create 8 TCs with 256 queues spread across all the TCs, when ADQ is
  755. enabled:
  756.  
  757. # tc qdisc add dev <ethX> root mqprio num_tc 8 map 0 1 2 3 4 5 6 7
  758. queues 2@0 4@2 8@6 16@14 32@30 64@62 128@126 2@254 hw 1 mode channel
  759.  
  760. - To set a minimum rate for a TC:
  761.  
  762. # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
  763. 4@0 8@4 hw 1 mode channel shaper bw_rlimit min_rate 25Gbit 50Gbit
  764.  
  765. - To set a maximum data rate for a TC:
  766.  
  767. # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
  768. 4@0 8@4 hw 1 mode channel shaper bw_rlimit max_rate 25Gbit 50Gbit
  769.  
  770. - To set both minimum and maximum data rates together:
  771.  
  772. # tc qdisc add dev ens4f0 root mqprio num_tc 2 map 0 0 0 0 1 1 1 1 queues
  773. 4@0 8@4 hw 1 mode channel shaper bw_rlimit min_rate 10Gbit 20Gbit
  774. max_rate 25Gbit 50Gbit
  775.  
  776.  
  777. Creating traffic class filters
  778. ------------------------------
  779. NOTE: These instructions are not specific to ADQ configuration.
  780.  
  781. After creating traffic classes, use the tc command to create filters for
  782. traffic. Refer to the tc and tc-flower man pages for more information.
  783.  
  784. To view all TC filters:
  785.  
  786. # tc filter show dev <ethX> ingress
  787. # tc filter show dev <ethX> egress
  788.  
  789. For detailed configuration information and example code for switchdev mode on
  790. Intel Ethernet 800 Series devices, refer to the configuration guide at
  791. https://cdrdv2.intel.com/v1/dl/getContent/645272.
  792.  
  793. TC FILTER EXAMPLES:
  794. To configure TCP TC filters, where:
  795. protocol: Encapsulation protocol (valid options are IP and 802.1Q).
  796. prio: Priority.
  797. flower: Flow-based traffic control filter.
  798. dst_ip: IP address of the device.
  799. ip_proto: IP protocol to use (TCP or UDP).
  800. dst_port: Destination port.
  801. src_port: Source port.
  802. skip_sw: Flag to add the rule only in hardware.
  803. hw_tc: Route incoming traffic flow to this hardware TC. The TC count
  804. starts at 0. For example, 'hw_tc 1' indicates that the filter
  805. is on the second TC.
  806. vlan_id: VLAN ID.
  807.  
  808. - TCP: Destination IP + L4 Destination Port
  809. To route incoming TCP traffic with a matching destination IP address and
  810. destination port to the given TC:
  811.  
  812. # tc filter add dev <ethX> protocol ip ingress prio 1 flower dst_ip
  813. <ip_address> ip_proto tcp dst_port <port_number> skip_sw hw_tc 1
  814.  
  815. - TCP: Source IP + L4 Source Port
  816. To route outgoing TCP traffic with a matching source IP address and
  817. source port to the given TC associated with the given priority:
  818.  
  819. # tc filter add dev <ethX> protocol ip egress prio 1 flower src_ip
  820. <ip_address> ip_proto tcp src_port <port_number> action skbedit priority 1
  821.  
  822.  
  823. - TCP: Destination IP + L4 Destination Port + VLAN Protocol
  824. To route incoming TCP traffic with a matching destination IP address and
  825. destination port to the given TC using the VLAN protocol (802.1Q):
  826.  
  827. # tc filter add dev <ethX> protocol 802.1Q ingress prio 1 flower
  828. dst_ip <ip address> eth_type ipv4 ip_proto tcp dst_port <port_number>
  829. vlan_id <vlan_id> skip_sw hw_tc 1
  830.  
  831.  
  832. - To add a GTP filter:
  833.  
  834. # tc filter add dev <ethX> protocol ip parent ffff: prio 1 flower
  835. src_ip 16.0.0.0/16 ip_proto udp dst_port 5678 enc_dst_port 2152
  836. enc_key_id <tunnel_id> skip_sw hw_tc 1
  837.  
  838. Where:
  839. dst_port: inner destination port of application (5678)
  840. enc_dst_port: outer destination port (for GTP user data tunneling occurs
  841. on UDP port 2152)
  842. enc_key_id: tunnel ID (vxlan ID)
  843.  
  844.  
  845. NOTE: You can add multiple filters to the device, using the same recipe (and
  846. requires no additional recipe resources), either on the same interface or on
  847. different interfaces. Each filter uses the same fields for matching, but can
  848. have different match values.
  849. # tc filter add dev <ethX> protocol ip ingress prio 1 flower ip_proto
  850. tcp dst_port <port_number> skip_sw hw_tc 1
  851. # tc filter add dev <ethX> protocol ip egress prio 1 flower ip_proto tcp
  852. src_port <port_number> action skbedit priority 1
  853.  
  854. For example:
  855.  
  856. # tc filter add dev ens4f0 protocol ip ingress prio 1 flower ip_proto
  857. tcp dst_port 5555 skip_sw hw_tc 1
  858. # tc filter add dev ens4f0 protocol ip egress prio 1 flower ip_proto
  859. tcp src_port 5555 action skbedit priority 1
  860.  
  861.  
  862. Using TC filters to forward to a queue
  863. --------------------------------------
  864. The ice driver supports directing traffic based on L2/L3/L4 fields in the
  865. packet to specific Rx queues, using the TC filter's class ID. Note: This
  866. functionality can be used with or without ADQ.
  867.  
  868. To add filters for the desired queue, use the following tc command:
  869.  
  870. # tc filter add dev <ethX> ingress prio 1 protocol all flower src_mac
  871. <mac_address> skip_sw classid ffff:<queue_id>
  872.  
  873. Where:
  874. - <mac_address> is the MAC address(es) you want to direct to the Rx queue
  875. - <queue_id> is the Rx queue ID number in hexadecimal
  876.  
  877. For example, to direct a single MAC address to queue 10:
  878.  
  879. # ethtool -K ens801 hw-tc-offload on
  880. # tc qdisc add dev ens801 clsact
  881. # tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
  882. 68:dd:ac:dc:19:00 skip_sw classid ffff:b
  883.  
  884. To direct 4 source MAC addresses to Rx queues 10-13:
  885.  
  886. # ethtool -K ens801 hw-tc-offload on
  887. # tc qdisc add dev ens801 clsact
  888. # tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
  889. 68:dd:ac:dc:19:00 skip_sw classid ffff:b
  890. # tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
  891. 68:dd:ac:dc:19:01 skip_sw classid ffff:c
  892. # tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
  893. 68:dd:ac:dc:19:02 skip_sw classid ffff:d
  894. # tc filter add dev ens801 ingress prio 1 protocol all flower src_mac
  895. 68:dd:ac:dc:19:03 skip_sw classid ffff:e
  896.  
  897.  
  898. Intel(R) Ethernet Flow Director
  899. -------------------------------
  900. The Intel(R) Ethernet Flow Director (Intel(R) Ethernet FD) performs the
  901. following tasks:
  902.  
  903. - Directs receive packets according to their flows to different queues
  904. - Enables tight control on routing a flow in the platform
  905. - Matches flows and CPU cores for flow affinity
  906.  
  907. NOTE: An included script (set_irq_affinity) automates setting the IRQ to CPU
  908. affinity.
  909.  
  910. NOTE: This driver supports the following flow types:
  911. - IPv4
  912. - TCPv4
  913. - UDPv4
  914. - SCTPv4
  915. - IPv6
  916. - TCPv6
  917. - UDPv6
  918. - SCTPv6
  919. Each flow type supports valid combinations of IP addresses (source or
  920. destination) and UDP/TCP/SCTP ports (source and destination). You can supply
  921. only a source IP address, a source IP address and a destination port, or any
  922. combination of one or more of these four parameters.
  923.  
  924. NOTE: This driver allows you to filter traffic based on a user-defined flexible
  925. two-byte pattern and offset by using the ethtool user-def and mask fields. Only
  926. L3 and L4 flow types are supported for user-defined flexible filters. For a
  927. given flow type, you must clear all Intel Ethernet Flow Director filters before
  928. changing the input set (for that flow type).
  929.  
  930. NOTE: Intel Ethernet Flow Director filters impact only LAN traffic. RDMA
  931. filtering occurs before Intel Ethernet Flow Director, so Intel Ethernet Flow
  932. Director filters will not impact RDMA.
  933.  
  934. The following table summarizes supported Intel Ethernet Flow Director features
  935. across Intel(R) Ethernet controllers.
  936.  
  937. ---------------------------------------------------------------------------
  938. Feature 500 Series 700 Series 800 Series
  939. ===========================================================================
  940. VF FLOW DIRECTOR Supported Routing to VF Not supported
  941. not supported
  942. ---------------------------------------------------------------------------
  943. IP ADDRESS RANGE Supported Not supported Field masking
  944. FILTER
  945. ---------------------------------------------------------------------------
  946. IPv6 SUPPORT Supported Supported Supported
  947. ---------------------------------------------------------------------------
  948. CONFIGURABLE Configured Configured Configured
  949. INPUT SET per port globally per port
  950. ---------------------------------------------------------------------------
  951. ATR Supported Supported Not supported
  952. ---------------------------------------------------------------------------
  953. FLEX BYTE FILTER Starts at Starts at Starts at
  954. beginning beginning of beginning
  955. of packet payload of packet
  956. ---------------------------------------------------------------------------
  957. TUNNELED PACKETS Filter matches Filter matches Filter matches
  958. outer header inner header inner header
  959. ---------------------------------------------------------------------------
  960.  
  961.  
  962. Intel Ethernet Flow Director Filters
  963. ------------------------------------
  964. Intel Ethernet Flow Director filters are used to direct traffic that matches
  965. specified characteristics. They are enabled through ethtool's ntuple interface.
  966. To enable or disable the Intel Ethernet Flow Director and these filters:
  967.  
  968. # ethtool -K <ethX> ntuple <off|on>
  969.  
  970. NOTE: When you disable ntuple filters, all the user programmed filters are
  971. flushed from the driver cache and hardware. All needed filters must be re-added
  972. when ntuple is re-enabled.
  973.  
  974. To display all of the active filters:
  975.  
  976. # ethtool -u <ethX>
  977.  
  978. To add a new filter:
  979.  
  980. # ethtool -U <ethX> flow-type <type> src-ip <ip> [m <ip_mask>] dst-ip <ip> [m
  981. <ip_mask>] src-port <port> [m <port_mask>] dst-port <port> [m <port_mask>]
  982. action <queue>
  983. Where:
  984. <ethX> - the Ethernet device to program
  985. <type> - can be ip4, tcp4, udp4, sctp4, ip6, tcp6, udp6, sctp6
  986. <ip> - the IP address to match on
  987. <ip_mask> - the IPv4 address to mask on
  988. NOTE: These filters use inverted masks. An inverted mask with 0
  989. means exactly match while with 0xF means DON'T CARE. Please
  990. refer to the examples for more details about inverted masks.
  991. <port> - the port number to match on
  992. <port_mask> - the 16-bit integer for masking
  993. NOTE: These filters use inverted masks.
  994. <queue> - the queue to direct traffic toward (-1 discards the
  995. matched traffic)
  996.  
  997. To delete a filter:
  998.  
  999. # ethtool -U <ethX> delete <N>
  1000. Where <N> is the filter ID displayed when printing all the active filters,
  1001. and may also have been specified using "loc <N>" when adding the filter.
  1002.  
  1003. EXAMPLES:
  1004. To add a filter that directs packet to queue 2:
  1005.  
  1006. # ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
  1007. 192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1]
  1008.  
  1009. To set a filter using only the source and destination IP address:
  1010.  
  1011. # ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
  1012. 192.168.10.2 action 2 [loc 1]
  1013.  
  1014. To set a filter based on a user-defined pattern and offset:
  1015.  
  1016. # ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
  1017. 192.168.10.2 user-def 0x4FFFF action 2 [loc 1]
  1018.  
  1019. where the value of the user-def field contains the offset (4 bytes) and
  1020. the pattern (0xffff).
  1021.  
  1022. To match TCP traffic sent from 192.168.0.1, port 5300, directed to 192.168.0.5,
  1023. port 80, and then send it to queue 7:
  1024.  
  1025. # ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5
  1026. src-port 5300 dst-port 80 action 7
  1027.  
  1028. To add a TCPv4 filter with a partial mask for a source IP subnet. Here the
  1029. matched src-ip is 192.*.*.* (inverted mask):
  1030.  
  1031. # ethtool -U <ethX> flow-type tcp4 src-ip 192.168.0.0 m 0.255.255.255 dst-ip
  1032. 192.168.5.12 src-port 12600 dst-port 31 action 12
  1033.  
  1034. NOTES:
  1035. For each flow-type, the programmed filters must all have the same matching
  1036. input set. For example, issuing the following two commands is acceptable:
  1037.  
  1038. # ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
  1039. # ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10
  1040.  
  1041. Issuing the next two commands, however, is not acceptable, since the first
  1042. specifies src-ip and the second specifies dst-ip:
  1043.  
  1044. # ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
  1045. # ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10
  1046.  
  1047. The second command will fail with an error. You may program multiple filters
  1048. with the same fields, using different values, but, on one device, you may not
  1049. program two tcp4 filters with different matching fields.
  1050.  
  1051. The ice driver does not support matching on a subportion of a field, thus
  1052. partial mask fields are not supported.
  1053.  
  1054.  
  1055. Flex Byte Intel Ethernet Flow Director Filters
  1056. ----------------------------------------------
  1057. The driver also supports matching user-defined data within the packet payload.
  1058. This flexible data is specified using the "user-def" field of the ethtool
  1059. command in the following way:
  1060. +----------------------------+--------------------------+
  1061. | 31 28 24 20 16 | 15 12 8 4 0 |
  1062. +----------------------------+--------------------------+
  1063. | offset into packet payload | 2 bytes of flexible data |
  1064. +----------------------------+--------------------------+
  1065.  
  1066. For example,
  1067. ... user-def 0x4FFFF ...
  1068.  
  1069. tells the filter to look 4 bytes into the payload and match that value against
  1070. 0xFFFF. The offset is based on the beginning of the payload, and not the
  1071. beginning of the packet. Thus
  1072.  
  1073. flow-type tcp4 ... user-def 0x8BEAF ...
  1074.  
  1075. would match TCP/IPv4 packets which have the value 0xBEAF 8 bytes into the
  1076. TCP/IPv4 payload.
  1077.  
  1078. Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload.
  1079. Thus to match the first byte of the payload, you must actually add 4 bytes to
  1080. the offset. Also note that ip4 filters match both ICMP frames as well as raw
  1081. (unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame.
  1082.  
  1083. The maximum offset is 64. The hardware will only read up to 64 bytes of data
  1084. from the payload. The offset must be even because the flexible data is 2 bytes
  1085. long and must be aligned to byte 0 of the packet payload.
  1086.  
  1087. The user-defined flexible offset is also considered part of the input set and
  1088. cannot be programmed separately for multiple filters of the same type. However,
  1089. the flexible data is not part of the input set and multiple filters may use the
  1090. same offset but match against different data.
  1091.  
  1092.  
  1093. RSS Hash Flow
  1094. -------------
  1095. Allows you to set the hash bytes per flow type and any combination of one or
  1096. more options for Receive Side Scaling (RSS) hash byte configuration.
  1097.  
  1098. # ethtool -N <ethX> rx-flow-hash <type> <option>
  1099.  
  1100. Where <type> is:
  1101. tcp4 signifying TCP over IPv4
  1102. udp4 signifying UDP over IPv4
  1103. tcp6 signifying TCP over IPv6
  1104. udp6 signifying UDP over IPv6
  1105. And <option> is one or more of:
  1106. s Hash on the IP source address of the Rx packet.
  1107. d Hash on the IP destination address of the Rx packet.
  1108. f Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet.
  1109. n Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet.
  1110.  
  1111. For example, to hash on the source and destination IP address for TCP IPv4
  1112. traffic, use the following:
  1113.  
  1114. # ethtool -N <ethX> rx-flow-hash tcp4 sd
  1115.  
  1116. To hash on the source and destination ports for UDP IPv6 traffic, use the
  1117. following:
  1118.  
  1119. # ethtool -N <ethX> rx-flow-hash udp6 sdfn
  1120.  
  1121.  
  1122. Accelerated Receive Flow Steering (aRFS)
  1123. ----------------------------------------
  1124. Devices based on the Intel(R) Ethernet 800 Series support Accelerated Receive
  1125. Flow Steering (aRFS) on the PF. aRFS is a load-balancing mechanism that allows
  1126. you to direct packets to the same CPU where an application is running or
  1127. consuming the packets in that flow.
  1128.  
  1129. NOTES:
  1130. - aRFS requires that ntuple filtering is enabled via ethtool.
  1131. - aRFS support is limited to the following packet types:
  1132. - TCP over IPv4 and IPv6
  1133. - UDP over IPv4 and IPv6
  1134. - Nonfragmented packets
  1135. - aRFS only supports Intel Ethernet Flow Director filters, which consist of the
  1136. source/destination IP addresses and source/destination ports.
  1137. - aRFS and ethtool's ntuple interface both use the device's Intel Ethernet Flow
  1138. Director. aRFS and ntuple features can coexist, but you may encounter
  1139. unexpected results if there's a conflict between aRFS and ntuple requests. See
  1140. "Intel(R) Ethernet Flow Director" for additional information.
  1141.  
  1142. To set up aRFS:
  1143. 1. Enable the Intel Ethernet Flow Director and ntuple filters using ethtool.
  1144.  
  1145. # ethtool -K <ethX> ntuple on
  1146.  
  1147. 2. Set up the number of entries in the global flow table. For example:
  1148.  
  1149. # NUM_RPS_ENTRIES=16384
  1150. # echo $NUM_RPS_ENTRIES > /proc/sys/net/core/rps_sock_flow_entries
  1151.  
  1152. 3. Set up the number of entries in the per-queue flow table. For example:
  1153.  
  1154. # NUM_RX_QUEUES=64
  1155. # for file in /sys/class/net/$IFACE/queues/rx-*/rps_flow_cnt; do
  1156. # echo $(($NUM_RPS_ENTRIES/$NUM_RX_QUEUES)) > $file;
  1157. # done
  1158.  
  1159. 4. Disable the IRQ balance daemon (this is only a temporary stop of the service
  1160. until the next reboot).
  1161.  
  1162. # systemctl stop irqbalance
  1163.  
  1164. 5. Configure the interrupt affinity.
  1165.  
  1166. # set_irq_affinity <ethX>
  1167.  
  1168.  
  1169. To disable aRFS using ethtool:
  1170.  
  1171. # ethtool -K <ethX> ntuple off
  1172.  
  1173. NOTE: This command will disable ntuple filters and clear any aRFS filters in
  1174. software and hardware.
  1175.  
  1176. Example Use Case:
  1177. 1. Set the server application on the desired CPU (e.g., CPU 4).
  1178.  
  1179. # taskset -c 4 netserver
  1180.  
  1181. 2. Use netperf to route traffic from the client to CPU 4 on the server with
  1182. aRFS configured. This example uses TCP over IPv4.
  1183.  
  1184. # netperf -H <Host IPv4 Address> -t TCP_STREAM
  1185.  
  1186.  
  1187. Enabling Virtual Functions (VFs) for SR-IOV
  1188. -------------------------------------------
  1189. Use sysfs to enable virtual functions (VF).
  1190.  
  1191. For example, you can create 4 VFs as follows:
  1192.  
  1193. # echo 4 > /sys/class/net/<ethX>/device/sriov_numvfs
  1194.  
  1195. To disable VFs, write 0 to the same file:
  1196.  
  1197. # echo 0 > /sys/class/net/<ethX>/device/sriov_numvfs
  1198.  
  1199. The maximum number of VFs for the ice driver is 256 total (all ports). To check
  1200. how many VFs each PF supports, use the following command:
  1201.  
  1202. # cat /sys/class/net/<ethX>/device/sriov_totalvfs
  1203. The VF driver will not block teaming/bonding/link aggregation, but this is not
  1204. a supported feature. Do not expect failover or load balancing on the VF
  1205. interface.
  1206.  
  1207.  
  1208. SR-IOV Live Migration
  1209. ---------------------
  1210. You can use VFIO Device Migration to move an active virtual machine (VM)
  1211. between different physical machines so it does not lose its network connection.
  1212. After migrating, the virtual function (VF) will continue most Ethernet
  1213. operations without further interruption. During migration, data and VIRTCHNL
  1214. operations are sent to a buffer so the can br recreated when the migration
  1215. completes. If the memory allocated for the command buffer is exceeded, the
  1216. system will drop the buffer and disable the live migration capability for the
  1217. VF. You must reset the VF for live migration to be re-enabled.
  1218.  
  1219. NOTES:
  1220. - Live migration requires kernel version 5.15 to 5.17
  1221. - You cannot migrate a VM if it has a VF that is using RDMA.
  1222. - You can only migrate the VF to a device in the same family with a similar
  1223. firmware version. For example, you can migrate a VF from one 810 device to
  1224. another, but not from an 810 device to an 820 device.
  1225. - Any VF properties that are set by the PF will not be migrated. Make sure that
  1226. both devices have the same PF-set properties.
  1227.  
  1228. Refer to https://qemu.readthedocs.io/en/latest/devel/vfio-migration.html for
  1229. more details.
  1230.  
  1231.  
  1232. Intel(R) Scalable I/O Virtualization Support
  1233. --------------------------------------------
  1234. Intel(R) Scalable I/O Virtualization (Intel(R) Scalable IOV) allows you to
  1235. share a physical device across multiple virtual machines and applications.
  1236. Intel Scalable IOV provides your system the ability to share device resources
  1237. with different address domains using different abstractions. For example,
  1238. application processes may access a device using system calls and VMs may access
  1239. a device through virtual device interfaces.
  1240.  
  1241. For more information, please refer to the Intel Scalable I/O Virtualization
  1242. Technical Specification
  1243. <https://software.intel.com/sites/default/files/managed/cc/0e/intel-scalable-io-
  1244. virtualization-technical-specification.pdf> (login required)
  1245.  
  1246. The VF driver will not block teaming/bonding/link aggregation, but this is not
  1247. a supported feature. Do not expect failover or load balancing on the VF
  1248. interface.
  1249.  
  1250. Intel Scalable IOV is not available in the kernel driver. Download and install
  1251. the current ice driver to use this feature. Refer to the SUPPORT section for
  1252. where to download the current driver.
  1253.  
  1254. Requirements
  1255. ------------
  1256.  
  1257. * Your system platform must support Intel Scalable IOV
  1258. * A network device based on an Intel(R) Ethernet 800 Series controller
  1259. * The host operating system must be a Linux distro using kernel version 5.12 -
  1260. 5.15
  1261. * The host PF driver must be version 1.9.0, or later
  1262. * The guest operating system must be Linux
  1263. * The guest iAVF driver must be version 4.5.0, or later
  1264.  
  1265. Enabling Intel Scalable IOV
  1266. ---------------------------
  1267.  
  1268. You can use Intel's Ethernet Port Configuration Tool (EPCT) to enable Intel
  1269. Scalable IOV. If the EPCT tool is not available, you can also enable Intel
  1270. Scalable IOV through your system's HII interface (if it has one). The
  1271. recommended method is to use the EPCT tool. To enable or disable Intel Scalable
  1272. IOV using the EPCT tool, use one of these commands:
  1273. #epct -nic=1 -set 'siov enable'
  1274. #epct -nic=1 -set 'siov disable'
  1275. Where -nic=1 specifies the Intel Ethernet device. See the EPCT tool
  1276. documentation for instructions on how to determine the NIC number of your
  1277. device.
  1278.  
  1279. If the EPCT tool is not available, and your system has an HII interface, you
  1280. can use the HII interface to enable/disable Intel Scalable IOV. Find the 'Intel
  1281. Scalable IOV (Scalable IOV)' setting and select your desired value.
  1282.  
  1283.  
  1284. Enabling Intel(R) Scalable I/O Virtualization Virtual Devices
  1285. -------------------------------------------------------------
  1286. Use the following steps to enable Intel(R) Scalable I/O Virtualization
  1287. (Intel(R) Scalable IOV) virtual devices (VDEVs):
  1288. Create an mdev
  1289. # echo "83b8f4f2-509f-382f-3c1e-e6bfe0fa1001" | sudo tee
  1290. /sys/class/mdev_bus/0000:38:00.0/mdev_supported_types/ice-ivdm/create
  1291. Use qemu to launch a VM with four processors
  1292. # sudo ./x86_64-softmmu/qemu-system-x86_64 \
  1293. -enable-kvm \
  1294. -m 1G \
  1295. -smp 4 \
  1296. -device
  1297. vfio-pci,sysfsdev=/sys/bus/mdev/devices/83b8f4f2-509f-382f-3c1e-e6bfe0fa
  1298. 1001 \
  1299. -drive file=../../img/test.qcow2 \
  1300. -nic user,hostfwd=tcp::5555-:22 \
  1301. -monitor stdio
  1302.  
  1303.  
  1304. Displaying VF Statistics on the PF
  1305. ----------------------------------
  1306. Use the following command to display the statistics for the PF and its VFs:
  1307.  
  1308. # ip -s link show dev <ethX>
  1309.  
  1310. NOTE: The output of this command can be very large due to the maximum number of
  1311. possible VFs.
  1312.  
  1313. The PF driver will display a subset of the statistics for the PF and for all
  1314. VFs that are configured. The PF will always print a statistics block for each
  1315. of the possible VFs, and it will show zero for all unconfigured VFs.
  1316.  
  1317.  
  1318. Configuring VLAN Tagging on SR-IOV Enabled Adapter Ports
  1319. --------------------------------------------------------
  1320. To configure VLAN tagging for the ports on an SR-IOV enabled adapter, use the
  1321. following command. The VLAN configuration should be done before the VF driver
  1322. is loaded or the VM is booted. The VF is not aware of the VLAN tag being
  1323. inserted on transmit and removed on received frames (sometimes called "port
  1324. VLAN" mode).
  1325.  
  1326. # ip link set dev <ethX> vf <id> vlan <vlan id>
  1327.  
  1328. For example, the following will configure PF eth0 and the first VF on VLAN 10:
  1329.  
  1330. # ip link set dev eth0 vf 0 vlan 10
  1331.  
  1332.  
  1333. Enabling a VF link if the port is disconnected
  1334. ----------------------------------------------
  1335. If the physical function (PF) link is down, you can force link up (from the
  1336. host PF) on any virtual functions (VF) bound to the PF. Note that this requires
  1337. kernel support (Red Hat kernel 3.10.0-327 or newer, upstream kernel 3.11.0 or
  1338. newer) and associated iproute2 user space support.
  1339.  
  1340. For example, to force link up on VF 0 bound to PF eth0:
  1341.  
  1342. # ip link set eth0 vf 0 state enable
  1343.  
  1344. Note: If the command does not work, it may not be supported by your system.
  1345.  
  1346.  
  1347. Setting the MAC Address for a VF
  1348. --------------------------------
  1349. To change the MAC address for the specified VF:
  1350.  
  1351. # ip link set <ethX> vf 0 mac <address>
  1352.  
  1353. For example:
  1354.  
  1355. # ip link set <ethX> vf 0 mac 00:01:02:03:04:05
  1356.  
  1357. This setting lasts until the PF is reloaded.
  1358.  
  1359. NOTE: For untrusted VFs, assigning a MAC address for a VF from the host will
  1360. disable any subsequent requests to change the MAC address from within the VM.
  1361. This is a security feature. The VM is not aware of this restriction, so if this
  1362. is attempted in the VM, it will trigger MDD events. Trusted VFs are allowed to
  1363. change the MAC address from within the VM.
  1364.  
  1365.  
  1366. Trusted VFs and VF Promiscuous Mode
  1367. -----------------------------------
  1368. This feature allows you to designate a particular VF as trusted and allows that
  1369. trusted VF to request selective promiscuous mode on the Physical Function (PF).
  1370.  
  1371. To set a VF as trusted or untrusted, enter the following command in the
  1372. Hypervisor:
  1373.  
  1374. # ip link set dev <ethX> vf 1 trust [on|off]
  1375.  
  1376. NOTE: It's important to set the VF to trusted before setting promiscuous mode.
  1377. If the VM is not trusted, the PF will ignore promiscuous mode requests from the
  1378. VF. If the VM becomes trusted after the VF driver is loaded, you must make a
  1379. new request to set the VF to promiscuous.
  1380.  
  1381. Once the VF is designated as trusted, use the following commands in the VM to
  1382. set the VF to promiscuous mode. For promiscuous all:
  1383.  
  1384. # ip link set <ethX> promisc on
  1385. Where <ethX> is a VF interface in the VM
  1386.  
  1387. For promiscuous Multicast:
  1388.  
  1389. # ip link set <ethX> allmulticast on
  1390. Where <ethX> is a VF interface in the VM
  1391.  
  1392. NOTE: By default, the ethtool private flag vf-true-promisc-support is set to
  1393. "off," meaning that promiscuous mode for the VF will be limited. To set the
  1394. promiscuous mode for the VF to true promiscuous and allow the VF to see all
  1395. ingress traffic, use the following command:
  1396.  
  1397. # ethtool --set-priv-flags <ethX> vf-true-promisc-support on
  1398.  
  1399. The vf-true-promisc-support private flag does not enable promiscuous mode;
  1400. rather, it designates which type of promiscuous mode (limited or true) you will
  1401. get when you enable promiscuous mode using the 'ip link' commands above. You
  1402. can toggle the vf-true-promisc-support flag separately for all PFs.
  1403.  
  1404. Next, add a VLAN interface on the VF interface. For example:
  1405.  
  1406. # ip link add link eth2 name eth2.100 type vlan id 100
  1407.  
  1408. Note that the order in which you set the VF to promiscuous mode and add the
  1409. VLAN interface does not matter (you can do either first). The result in this
  1410. example is that the VF will get all traffic that is tagged with VLAN 100.
  1411.  
  1412.  
  1413. Virtual Function (VF) Tx Rate Limit
  1414. -----------------------------------
  1415. Use the ip command to configure the maximum or minimum Tx rate limit for a VF
  1416. from the PF interface.
  1417.  
  1418. For example, to set a maximum Tx rate limit of 8000Mbps for VF 0:
  1419.  
  1420. # ip link set eth0 vf 0 max_tx_rate 8000
  1421.  
  1422. For example, to set a minimum Tx rate limit of 1000Mbps for VF 0:
  1423.  
  1424. # ip link set eth0 vf 0 min_tx_rate 1000
  1425.  
  1426. NOTE:
  1427. - If DCB or ADQ are enabled on a PF, you cannot set a minimum Tx rate on the
  1428. VFs associated with that PF.
  1429. - If both DCB and ADQ are disabled on a PF, then you can set a minimum Tx rate
  1430. on the VFs associated with that PF.
  1431. - If you set a minimum Tx rate limit on a PF for SR-IOV VFs and then apply a
  1432. DCB or ADQ configuration, the PF cannot guarantee the minimum Tx rate limits
  1433. for those VFs.
  1434. - If you set a minimum Tx rate on VFs across multiple ports that have an
  1435. aggregate bandwidth over 100Gbps, the PFs cannot guarantee the minimum Tx rate
  1436. set for the VFs.
  1437.  
  1438.  
  1439. Malicious Driver Detection (MDD) for VFs
  1440. ----------------------------------------
  1441. Some Intel Ethernet devices use Malicious Driver Detection (MDD) to detect
  1442. malicious traffic from the VF and disable Tx/Rx queues or drop the offending
  1443. packet until a VF driver reset occurs. You can view MDD messages in the PF's
  1444. system log using the dmesg command.
  1445.  
  1446. - If the PF driver logs MDD events from the VF, confirm that the correct VF
  1447. driver is installed.
  1448. - To restore functionality, you can manually reload the VF or VM or enable
  1449. automatic VF resets.
  1450. - When automatic VF resets are enabled, the PF driver will immediately reset
  1451. the VF and reenable queues when it detects MDD events on the receive path.
  1452. - If automatic VF resets are disabled, the PF will not automatically reset the
  1453. VF when it detects MDD events.
  1454.  
  1455. To enable or disable automatic VF resets, use the following command:
  1456.  
  1457. # ethtool --set-priv-flags <ethX> mdd-auto-reset-vf on|off
  1458.  
  1459.  
  1460. MAC and VLAN Anti-Spoofing Feature for VFs
  1461. ------------------------------------------
  1462. When a malicious driver on a Virtual Function (VF) interface attempts to send a
  1463. spoofed packet, it is dropped by the hardware and not transmitted.
  1464.  
  1465. NOTE: This feature can be disabled for a specific VF:
  1466.  
  1467. # ip link set <ethX> vf <vf id> spoofchk {off|on}
  1468.  
  1469.  
  1470. VLAN Pruning
  1471. ------------
  1472. The ice driver allows you to enable or disable VLAN pruning for the VF VSI
  1473. using the ethtool private flag vf-vlan-pruning.
  1474.  
  1475. NOTE:
  1476. - You cannot change this private flag while any VFs are active.
  1477. - If a port VLAN is configured, VLAN pruning will always be enabled.
  1478. - When VLAN pruning is enabled, the interface will:
  1479. - Discard all packets with a VLAN tag when Rx VLAN filtering is disabled.
  1480. - Discard untagged packets when Rx VLAN filtering is enabled.
  1481.  
  1482. To disable or enable VLAN pruning on all VFs, do the following:
  1483. 1. Deinitialize any VFs.
  1484. 2. On the PF, use the following command:
  1485.  
  1486. # ethtool --set-priv-flags <ethX> vf-vlan-pruning on|off
  1487. Where:
  1488. on - enables VLAN pruning
  1489. off - disables VLAN pruning (default)
  1490.  
  1491. 3. Initialize and configure any VFs.
  1492.  
  1493. VLAN pruning will then be disabled or enabled on any of these VFs, depending on
  1494. the flag you set.
  1495.  
  1496.  
  1497. Switchdev mode
  1498. --------------
  1499. The PF driver supports legacy and switchdev eSwitch modes. Switchdev mode
  1500. allows the driver to create additional port representor netdevs that enable a
  1501. control plane running on the host to configure filters for the VFs and also
  1502. handle default/exception traffic from the uplink and the VFs.
  1503.  
  1504. The driver loads in legacy mode by default. You can configure eSwitch modes
  1505. independently per physical port using the devlink command. You can change
  1506. between eSwitch modes only if no VFs have been created. If SR-IOV is enabled
  1507. and VFs are bound to the PF, you must do the following before changing between
  1508. switchdev and legacy mode:
  1509. - Unload all VFs that were bound
  1510. - Set the number of VFs on the PF to zero
  1511.  
  1512. NOTE:
  1513. - ADQ, trusted VFs, L2 forwarding, and S-IOV are not supported in switchdev
  1514. mode.
  1515. - Switchdev mode is not persistent across reboots or driver reloads.
  1516.  
  1517. To configure the device in switchdev mode, enter the following, where
  1518. <pci/0000:##:##.#> is the PCI address of the PF:
  1519.  
  1520. # devlink dev eswitch set <pci/0000:##:##.#> mode switchdev
  1521.  
  1522. For example:
  1523. # devlink dev eswitch set pci/0000:17:00.0 mode switchdev
  1524.  
  1525. To configure the device in legacy mode:
  1526.  
  1527. # devlink dev eswitch set <pci/0000:##:##.#> mode legacy
  1528.  
  1529. To check the current eSwitch mode:
  1530.  
  1531. # devlink dev eswitch show <pci/0000:##:##.#>
  1532.  
  1533. The ice driver supports the following hardware offloads in switchdev mode:
  1534. - Supported filter conditions:
  1535. L2: Source/Destination MAC addresses, VLAN ID
  1536. L3: Source/Destination IP addresses (IPv4, IPv6), IP protocol (TCP, UDP),
  1537. ToS (IPv4), Traffic Class (IPv6), TTL (IPv4)
  1538. L4: Source and Destination port
  1539. VXLAN/GRETAP/GENEVE: VNI/GRE Key, Outer Destination IP, Inner Source IP,
  1540. Inner Destination IP, Inner Destination MAC, TCP/UDP Source port and
  1541. Destination port
  1542. GTP: TEID, PDU type, QFI, Outer Destination IP, Outer Source IP
  1543. - Supported filter actions: redirect, drop
  1544.  
  1545. NOTE: GTP support requires kernel 5.18 and iproute2 5.18 or newer. On older
  1546. kernel versions, the DCF method provides the same functionality.
  1547.  
  1548. For detailed configuration information and example code for switchdev mode on
  1549. Intel Ethernet 800 Series devices, refer to the configuration guide at
  1550. https://cdrdv2.intel.com/v1/dl/getContent/645272.
  1551.  
  1552. At a high level, do the following to offload TC filters to the hardware and
  1553. create switch rules in switchdev mode:
  1554. 1. Verify that switchdev mode is enabled.
  1555. 2. Enable hw-tc-offload on the VF port representor (VF_PR).
  1556. 3. For tunnel interfaces: Use the ip link command to create the tunnel.
  1557. 4. Use the tc-flower command to create the switch rule.
  1558. 5. Verify the offloaded flow in hardware.
  1559.  
  1560. Switchdev mode supports the following ip link commands to configure the VF:
  1561. * mac
  1562. * vlan, vxlan, geneve, gre, nvgre, gtp, qos, proto
  1563. * max_tx_rate
  1564. * min_tx_rate
  1565. * spoofchk
  1566. * query_rss
  1567. * state
  1568. * node_guid
  1569. * port_guid
  1570. NOTE: 'trust' is not supported. 'rate' is supported but deprecated; use
  1571. 'max_tx_rate' instead.
  1572.  
  1573. To limit the VF's interrupt rate for Rx and Tx in switchdev mode, use the
  1574. following command, where <vf_pr> is the designated VF port representor and <N>
  1575. is the desired cap for the interrupt rate:
  1576.  
  1577. # ethtool -C <vf_pr> rx-usecs-high <N>
  1578.  
  1579.  
  1580. Jumbo Frames
  1581. ------------
  1582. Jumbo Frames support is enabled by changing the Maximum Transmission Unit (MTU)
  1583. to a value larger than the default value of 1500.
  1584.  
  1585. Use the ip command to increase the MTU size. For example, enter the following
  1586. where <ethX> is the interface number:
  1587.  
  1588. # ip link set mtu 9000 dev <ethX>
  1589. # ip link set up dev <ethX>
  1590.  
  1591. This setting is not saved across reboots.
  1592.  
  1593. Add 'MTU=9000' to the following file to make the setting change permanent:
  1594. /etc/sysconfig/network-scripts/ifcfg-<ethX> for RHEL
  1595. or
  1596. /etc/sysconfig/network/<config_file> for SLES
  1597.  
  1598. NOTE: The maximum MTU setting for jumbo frames is 9702. This corresponds to the
  1599. maximum jumbo frame size of 9728 bytes.
  1600.  
  1601. NOTE: This driver will attempt to use multiple page sized buffers to receive
  1602. each jumbo packet. This should help to avoid buffer starvation issues when
  1603. allocating receive packets.
  1604.  
  1605. NOTE: Packet loss may have a greater impact on throughput when you use jumbo
  1606. frames. If you observe a drop in performance after enabling jumbo frames,
  1607. enabling flow control may mitigate the issue.
  1608.  
  1609.  
  1610. Speed and Duplex Configuration
  1611. ------------------------------
  1612. You cannot set speed, duplex, or autonegotiation settings using ethtool.
  1613.  
  1614.  
  1615. To see the speed configurations your device supports, run the following:
  1616.  
  1617. # ethtool <ethX>
  1618.  
  1619. To have your device advertise supported speeds, use the following:
  1620.  
  1621. # ethtool -s <ethX> advertise N
  1622. Where N is a bitmask of the desired speeds.
  1623.  
  1624. For example, to have your device advertise 10000baseSR Full, use:
  1625.  
  1626. # ethtool -s <ethX> advertise 0x80000000000
  1627.  
  1628. For more details, please refer to the ethtool man page.
  1629.  
  1630.  
  1631. Data Center Bridging (DCB)
  1632. --------------------------
  1633. NOTE: The kernel assumes that TC0 is available, and will disable Priority Flow
  1634. Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is
  1635. enabled when setting up DCB on your switch.
  1636.  
  1637. DCB is a configuration Quality of Service implementation in hardware. It uses
  1638. the VLAN priority tag (802.1p) to filter traffic. That means that there are 8
  1639. different priorities that traffic can be filtered into. It also enables
  1640. priority flow control (802.1Qbb) which can limit or eliminate the number of
  1641. dropped packets during network stress. Bandwidth can be allocated to each of
  1642. these priorities, which is enforced at the hardware level (802.1Qaz).
  1643.  
  1644. DCB is normally configured on the network using the DCBX protocol (802.1Qaz), a
  1645. specialization of LLDP (802.1AB). The ice driver supports the following
  1646. mutually exclusive variants of DCBX support:
  1647. 1) Firmware-based LLDP Agent
  1648. 2) Software-based LLDP Agent
  1649.  
  1650. In firmware-based mode, firmware intercepts all LLDP traffic and handles DCBX
  1651. negotiation transparently for the user. In this mode, the adapter operates in
  1652. "willing" DCBX mode, receiving DCB settings from the link partner (typically a
  1653. switch). The local user can only query the negotiated DCB configuration. For
  1654. information on configuring DCBX parameters on a switch, please consult the
  1655. switch manufacturer's documentation.
  1656.  
  1657. In software-based mode, LLDP traffic is forwarded to the network stack and user
  1658. space, where a software agent can handle it. In this mode, the adapter can
  1659. operate in either "willing" or "nonwilling" DCBX mode and DCB configuration can
  1660. be both queried and set locally. This mode requires the FW-based LLDP Agent to
  1661. be disabled.
  1662.  
  1663. NOTE:
  1664. - You can enable and disable the firmware-based LLDP Agent using an ethtool
  1665. private flag. Refer to the "FW-LLDP (Firmware Link Layer Discovery Protocol)"
  1666. section in this README for more information.
  1667. - In software-based DCBX mode, you can configure DCB parameters using software
  1668. LLDP/DCBX agents that interface with the Linux kernel's DCB Netlink API. We
  1669. recommend using OpenLLDP as the DCBX agent when running in software mode. For
  1670. more information, see the OpenLLDP man pages and
  1671. https://github.com/intel/openlldp.
  1672. - The driver implements the DCB netlink interface layer to allow the user space
  1673. to communicate with the driver and query DCB configuration for the port.
  1674. - iSCSI with DCB is not supported.
  1675.  
  1676.  
  1677. L3 QoS mode
  1678. -----------
  1679. The ice driver supports setting DSCP-based Layer 3 Quality of Service (L3 QoS)
  1680. in the PF driver. The driver initializes in L2 QoS mode. L3 QoS mode is:
  1681. - Automatically enabled when the first DSCP/ToS to TC mapping is defined
  1682. - Automatically disabled when the last DSCP/ToS to TC mapping is removed
  1683.  
  1684. The following is an example of how to map a DSCP/ToS to a TC:
  1685.  
  1686. # lldptool -T -i <ethX> -V APP app=<prio>,<sel>,<pid>
  1687. where:
  1688. <prio>: The TC assigned to the DSCP/ToS code point
  1689. <sel>: 5 for DSCP to TC mapping
  1690. <pid>: The DSCP/ToS code point
  1691.  
  1692. For example, to map packets containing DSCP value 63 to traffic class 0 on
  1693. interface eth0:
  1694.  
  1695. # lldptool -T -i eth0 -V APP app=63,5,0
  1696.  
  1697. To remove a mapping, use the following:
  1698.  
  1699. # lldptool -T -I <ethX> -V APP -d app=<prio>,<sel>,<pid>
  1700.  
  1701. To view the currently configured mappings, use the following:
  1702.  
  1703. # lldptool -t -i <ethX> -V APP -c
  1704.  
  1705. NOTE:
  1706. - L3 QoS mode is not available when FW-LLDP is enabled. You also cannot enable
  1707. FW-LLDP if L3 QoS mode is active. Disable FW-LLDP before switching to L3 QoS
  1708. mode. Refer to the "FW-LLDP (Firmware Link Layer Discovery Protocol)" section
  1709. in this README for more information on disabling FW-LLDP.
  1710. - Once a mapping has been submitted for a DSCP value, another mapping for that
  1711. value will not be accepted until the first one has been deleted.
  1712.  
  1713.  
  1714. FW-LLDP (Firmware Link Layer Discovery Protocol)
  1715. ------------------------------------------------
  1716. Use ethtool to change FW-LLDP settings. The FW-LLDP setting is per port and
  1717. persists across boots.
  1718.  
  1719. To enable LLDP:
  1720.  
  1721. # ethtool --set-priv-flags <ethX> fw-lldp-agent on
  1722.  
  1723. To disable LLDP:
  1724.  
  1725. # ethtool --set-priv-flags <ethX> fw-lldp-agent off
  1726.  
  1727. To check the current LLDP setting:
  1728.  
  1729. # ethtool --show-priv-flags <ethX>
  1730.  
  1731. NOTE: You must enable the UEFI HII "LLDP Agent" attribute for this setting to
  1732. take effect. If "LLDP AGENT" is set to disabled, you cannot enable it from the
  1733. OS.
  1734.  
  1735.  
  1736. Forward Error Correction (FEC)
  1737. ------------------------------
  1738. Allows you to set the Forward Error Correction (FEC) mode. FEC improves link
  1739. stability, but increases latency. Many high quality optics, direct attach
  1740. cables, and backplane channels provide a stable link without FEC.
  1741.  
  1742. NOTE:
  1743. For devices to benefit from this feature, link partners must have FEC enabled.
  1744.  
  1745. If you enable the flag 'allow-no-fec-modules-in-auto', Auto FEC negotiation
  1746. will include 'No FEC' in case your link partner does not have FEC enabled or is
  1747. not FEC capable.
  1748. # ethtool --set-priv-flags <ethX> allow-no-fec-modules-in-auto on
  1749.  
  1750. NOTE:
  1751.  
  1752. On kernels older than 4.14, use the following private flags to disable
  1753. FEC modes:
  1754. rs-fec (0 to disable, 1 to enable)
  1755. base-r-fec (0 to disable, 1 to enable)
  1756. On kernel 4.14 or later, use ethtool to get/set the following FEC modes:
  1757. No FEC
  1758. Auto FEC
  1759. BASE-R FEC
  1760. RS-FEC
  1761.  
  1762.  
  1763. Link-Level Flow Control (LFC)
  1764. -----------------------------
  1765. Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enable
  1766. receiving and transmitting pause frames for ice. When transmit is enabled,
  1767. pause frames are generated when the receive packet buffer crosses a predefined
  1768. threshold. When receive is enabled, the transmit unit will halt for the time
  1769. delay specified when a pause frame is received.
  1770.  
  1771. NOTE: You must have a flow control capable link partner.
  1772.  
  1773. Flow Control is disabled by default.
  1774.  
  1775. Use ethtool to change the flow control settings.
  1776.  
  1777. To enable or disable Rx or Tx Flow Control:
  1778.  
  1779. # ethtool -A <ethX> rx <on|off> tx <on|off>
  1780.  
  1781. Note: This command only enables or disables Flow Control if auto-negotiation is
  1782. disabled. If auto-negotiation is enabled, this command changes the parameters
  1783. used for auto-negotiation with the link partner.
  1784.  
  1785. Note: Flow Control auto-negotiation is part of link auto-negotiation. Depending
  1786. on your device, you may not be able to change the auto-negotiation setting.
  1787.  
  1788. NOTE:
  1789. - The ice driver requires flow control on both the port and link partner. If
  1790. flow control is disabled on one of the sides, the port may appear to hang on
  1791. heavy traffic.
  1792. - You may encounter issues with link-level flow control (LFC) after disabling
  1793. DCB. The LFC status may show as enabled but traffic is not paused. To resolve
  1794. this issue, disable and reenable LFC using ethtool:
  1795.  
  1796. # ethtool -A <ethX> rx off tx off
  1797. # ethtool -A <ethX> rx on tx on
  1798.  
  1799.  
  1800. Limiting the Maximum Bitrate for a Transmit Queue
  1801. -------------------------------------------------
  1802. The ice driver supports limiting the transmit queue bit rate with the
  1803. tx_maxrate sysfs entry. Use this entry to set a maximum bitrate in Mbps. A
  1804. value of zero means no limiting.
  1805. Setting the bit rate for transmit queue 1 to 300 Mbps:
  1806. # echo 300 > /sys/class/<ethx>/queues/tx-1/tx_maxrate
  1807. Removing the limit:
  1808. # echo 0 > /sys/class/<ethx>/queues/tx-1/tx_maxrate
  1809. NAPI
  1810. ----
  1811. This driver supports NAPI (Rx polling mode). For more information on NAPI, see
  1812. https://docs.kernel.org/networking/napi.html.
  1813.  
  1814.  
  1815. MACVLAN
  1816. -------
  1817. This driver supports MACVLAN. Kernel support for MACVLAN can be tested by
  1818. checking if the MACVLAN driver is loaded. You can run 'lsmod | grep macvlan' to
  1819. see if the MACVLAN driver is loaded or run 'modprobe macvlan' to try to load
  1820. the MACVLAN driver.
  1821.  
  1822. NOTE:
  1823. - In passthru mode, you can only set up one MACVLAN device. It will inherit the
  1824. MAC address of the underlying PF (Physical Function) device.
  1825.  
  1826. ice devices support L2 Forwarding Offload. This will offload the processing
  1827. required for L2 Forwarding from the system processors to the ice device.
  1828. Perform the following steps to enable L2 Forwarding Offload:
  1829.  
  1830. 1. Enable L2 Forwarding offload:
  1831.  
  1832. # ethtool -K <ethX> l2-fwd-offload on
  1833.  
  1834. 2. Create the MACVLAN netdevs and bind them to the PF.
  1835.  
  1836. 3. Bring up/enable the MACVLAN netdevs.
  1837.  
  1838. NOTE: MACVLAN offloads and ADQ are mutually exclusive. System instability may
  1839. occur if you enable l2-fwd-offload and then set up ADQ, or if you set up ADQ
  1840. and then enable l2-fwd-offload.
  1841.  
  1842.  
  1843. IEEE 802.1ad (QinQ) Support
  1844. ---------------------------
  1845. The IEEE 802.1ad standard, informally known as QinQ, allows for multiple VLAN
  1846. IDs within a single Ethernet frame. VLAN IDs are sometimes referred to as
  1847. "tags," and multiple VLAN IDs are thus referred to as a "tag stack." Tag stacks
  1848. allow L2 tunneling and the ability to separate traffic within a particular VLAN
  1849. ID, among other uses.
  1850.  
  1851. The following are examples of how to configure 802.1ad (QinQ):
  1852.  
  1853. # ip link add link eth0 eth0.24 type vlan proto 802.1ad id 24
  1854. # ip link add link eth0.24 eth0.24.371 type vlan proto 802.1Q id 371
  1855. Where "24" and "371" are example VLAN IDs.
  1856.  
  1857. NOTES:
  1858. - 802.1ad (QinQ) is supported in 3.19 and later kernels.
  1859. - VLAN protocols use the following EtherTypes:
  1860. 802.1Q = EtherType 0x8100
  1861. 802.1ad = EtherType 0x88A8
  1862. - For QinQ traffic to work at MTU 1500, the L2 peer (switch port or another
  1863. NIC) should be able to receive Ethernet frames of 1526 bytes. Some third-party
  1864. NICs support a maximum Ethernet frame size of 1522 bytes at MTU 1500, which
  1865. will cause QinQ traffic to fail. To work around this issue, restrict the MTU on
  1866. the Intel Ethernet device to 1496.
  1867.  
  1868.  
  1869. Double VLANs
  1870. ------------
  1871. Devices based on the Intel(R) Ethernet 800 Series can process up to two VLANs
  1872. in a packet when all the following are installed:
  1873. - ice driver version 1.4.0 or later
  1874. - NVM version 2.4 or later
  1875. - ice DDP package version 1.3.21 or later
  1876. If you don't use the versions above, the only supported VLAN configuration is
  1877. single 802.1Q VLAN traffic.
  1878.  
  1879. When two VLAN tags are present in a packet, the outer VLAN tag can be either
  1880. 802.1Q or 802.1ad. The inner VLAN tag must always be 802.1Q.
  1881.  
  1882. Note the following limitations:
  1883. - For each VF, the PF can only allow VLAN hardware offloads (insertion and
  1884. stripping) of one type, either 802.1Q or 802.1ad.
  1885. - You can't enable or disable outer or single 802.1Q or 802.1ad filtering
  1886. separately. They are either both on or both off.
  1887. - In SR-IOV mode, the VF may not receive all network traffic based on the inner
  1888. VLAN header when VF true promiscuous mode (vf-true-promisc-support) and double
  1889. VLANs are enabled.
  1890.  
  1891. To enable outer or single 802.1Q VLAN insertion and stripping and disable
  1892. 802.1ad VLAN insertion and stripping:
  1893. # ethtool -K <ethX> rxvlan on txvlan on rx-vlan-stag-hw-parse off
  1894. tx-vlan-stag-hw-insert off
  1895.  
  1896. To enable outer or single 802.1ad VLAN insertion and stripping and disable
  1897. 802.1Q VLAN insertion and stripping:
  1898. # ethtool -K <ethX> rxvlan off txvlan off rx-vlan-stag-hw-parse on
  1899. tx-vlan-stag-hw-insert on
  1900.  
  1901. To enable outer or single VLAN filtering:
  1902. # ethtool -K <ethX> rx-vlan-filter on rx-vlan-stag-filter on
  1903.  
  1904. To disable outer or single VLAN filtering:
  1905. # ethtool -K <ethX> rx-vlan-filter off rx-vlan-stag-filter off
  1906.  
  1907.  
  1908. Combining QinQ with SR-IOV VFs
  1909. ------------------------------
  1910. We recommend you always configure a port VLAN for the VF from the PF. If a port
  1911. VLAN is not configured, the VF driver may only offload VLANs via software. The
  1912. PF allows all VLAN traffic to reach the VF, and the VF manages all VLAN traffic.
  1913.  
  1914. When the device is configured for double VLANs and the PF has configured a port
  1915. VLAN:
  1916. - The VF can only offload guest VLANs for 802.1Q traffic.
  1917. - The VF can only configure VLAN filtering rules for guest VLANs using 802.1Q
  1918. traffic.
  1919.  
  1920. However, when the device is configured for double VLANs and the PF has NOT
  1921. configured a port VLAN:
  1922. - You must use iavf driver version 4.1.0 or later to offload and filter VLANs.
  1923. - The PF turns on VLAN pruning and antispoof in the VF's VSI by default. The VF
  1924. will not transmit or receive any tagged traffic until the VF requests a VLAN
  1925. filter.
  1926. - The VF can offload (insert and strip) the outer VLAN tag of 802.1Q or 802.1ad
  1927. traffic.
  1928. - The VF can create filter rules for the outer VLAN tag of both 802.1Q and
  1929. 802.1ad traffic.
  1930.  
  1931. If the PF does not support double VLANs, the VF can hardware offload single
  1932. 802.1Q VLANs without a port VLAN.
  1933.  
  1934. When the PF is enabled for double VLANs, for iavf drivers before version 4.1.x:
  1935. - VLAN hardware offloads and filtering are supported only when the PF has
  1936. configured a port VLAN.
  1937. - VLAN filtering, insertion, and stripping will be software offloaded when no
  1938. port VLAN is configured.
  1939.  
  1940. To see VLAN filtering and offload capabilities, use the following command:
  1941.  
  1942. # ethtool -k <ethX> | grep vlan
  1943.  
  1944.  
  1945. IEEE 1588 Precision Time Protocol (PTP) Hardware Clock (PHC)
  1946. ------------------------------------------------------------
  1947. Precision Time Protocol (PTP) is used to synchronize clocks in a computer
  1948. network. PTP support varies among Intel devices that support this driver. Use
  1949. 'ethtool -T <ethX>' to get a definitive list of PTP capabilities supported by
  1950. the device.
  1951.  
  1952. A detailed user guide is available for the following devices. Refer to it for
  1953. advanced configuration of this feature.
  1954. - Intel(R) Ethernet Network Adapter E810-XXV-4T:
  1955. https://cdrdv2.intel.com/v1/dl/getContent/646265
  1956.  
  1957. - Intel(R) Ethernet Network Adapter E810-C-Q2T:
  1958. https://cdrdv2.intel.com/v1/dl/getContent/722960
  1959.  
  1960.  
  1961. Some devices support hardware-generated timestamps. The ice driver uses these
  1962. timestamps to synchronize clocks on the platform and report precise timestamps
  1963. on packets. Use the following hwstamp_ctl command, which is available in the
  1964. linuxptp utility, to enable this setting:
  1965.  
  1966. # hwstamp_ctl -i <ethX> -t 1 -r 1
  1967.  
  1968.  
  1969. SyncE Support
  1970. -------------
  1971. On hardware that supports Synchronous Ethernet (SyncE), the ice driver has
  1972. interfaces that allow you to synchronize frequencies with other SyncE-supported
  1973. ports. After you manually configure SyncE, the device dynamically selects the
  1974. best quality signal from the ones that are available. Then, once the signal is
  1975. locked, it synchronizes its frequency clock to it. The best quality signal is
  1976. determined based on the topology configured with the ice SyncE interfaces.
  1977.  
  1978. A detailed user guide is available for the following devices. Refer to it for
  1979. advanced configuration of this feature.
  1980. - Intel(R) Ethernet Network Adapter E810-XXV-4T:
  1981. https://cdrdv2.intel.com/v1/dl/getContent/646265
  1982.  
  1983. - Intel(R) Ethernet Network Adapter E810-C-Q2T:
  1984. https://cdrdv2.intel.com/v1/dl/getContent/722960
  1985.  
  1986.  
  1987. Tunnel/Overlay Stateless Offloads
  1988. ---------------------------------
  1989. Supported tunnels and overlays include VXLAN, GENEVE, and others depending on
  1990. hardware and software configuration. Stateless offloads are enabled by default.
  1991.  
  1992. To view the current state of all offloads:
  1993.  
  1994. # ethtool -k <ethX>
  1995.  
  1996.  
  1997. UDP Segmentation Offload
  1998. ------------------------
  1999. Allows the adapter to offload transmit segmentation of UDP packets with
  2000. payloads up to 64K into valid Ethernet frames. Because the adapter hardware is
  2001. able to complete data segmentation much faster than operating system software,
  2002. this feature may improve transmission performance.
  2003. In addition, the adapter may use fewer CPU resources.
  2004.  
  2005. NOTE:
  2006. - UDP transmit segmentation offload requires Linux kernel 4.18 or later.
  2007. - The application sending UDP packets must support UDP segmentation offload.
  2008.  
  2009. To enable/disable UDP Segmentation Offload, issue the following command:
  2010.  
  2011. # ethtool -K <ethX> tx-udp-segmentation [off|on]
  2012.  
  2013.  
  2014. Runtime Control of CRC/FCS Stripping
  2015. ------------------------------------
  2016. The frame check sequence (FCS) is a four-octet cyclic redundancy check (CRC)
  2017. that allows the driver to detect corrupted data within a received Ethernet
  2018. frame.
  2019.  
  2020. The ice driver allows you to disable or enable FCS/CRC stripping using the
  2021. ethtool command.
  2022.  
  2023. NOTE:
  2024. - FCS/CRC stripping is enabled by default.
  2025. - The driver enforces valid combinations of FCS/CRC and VLAN stripping. You can
  2026. only disable FCS/CRC stripping if VLAN stripping is also disabled on the PF.
  2027. - Disabling FCS/CRC stripping may help when debugging issues. XDP programs can
  2028. also use FCS/CRC for their purposes.
  2029.  
  2030. Use the following ethtool command to enable or disable FCS/CRC stripping:
  2031.  
  2032. # ethtool -K <ethX> rx-fcs on|off
  2033.  
  2034. To check the status of FCS/CRC stripping, look for the 'rx-fcs' information
  2035. reported from ethtool:
  2036.  
  2037. # ethtool -k <ethX>
  2038.  
  2039.  
  2040. Using Devlink to update a device's NVM
  2041. --------------------------------------
  2042. When you update the NVM on some devices, the update may use the devlink
  2043. interface, rather than the ethtool interface. This will happen if the following
  2044. are true:
  2045. - You are updating an Intel Ethernet 800 Series device.
  2046. - Your system is running a distro that supports the "devlink dev flash" command.
  2047. - The firmware currently installed on the device supports it.
  2048. - The new NVM conforms to the correct PLDM format.
  2049.  
  2050. Most of the functionality and commands are the same with the following
  2051. exceptions:
  2052. - You cannot update a device in Recovery Mode. (To update a device in recovery
  2053. mode, you must download and install the Intel Ethernet driver set.)
  2054. - You cannot update the OROM or Netlist as a separate update, only as part of a
  2055. full NVM update.
  2056. - If you specified a preservation level of PRESERVE_ALL, the system will
  2057. immediately perform an EMPR reset after the NVM update.
  2058.  
  2059. On devices that support it, you can also use the devlink command line directly
  2060. to update the device NVM. However, we recommend that you use NVMUpdate.
  2061.  
  2062. # devlink dev flash pci/0000:3b:00.0 file filename.bin
  2063.  
  2064. Where :
  2065. - pci/0000:3b:00.0 – The device you wish to update. You can get a list of
  2066. devices with the "devlink dev info" command.
  2067. - filename.bin – The file that contains the new NVM image.
  2068.  
  2069.  
  2070. Port Split Configuration Using Devlink
  2071. --------------------------------------
  2072. Most CVL devices support changing their port split configuration to suit your
  2073. needs. For example, a dual port device may support two 100Gbps links, two 50
  2074. Gbps links, and (with the correct cables) four 25 Gbps links, etc. The
  2075. supported port split configurations are defined in the device's NVM. You can
  2076. use a tool like Intel's Ethernet Port Configuration Tool (EPCT) to query and
  2077. set this configuration. If no such tool is available, you can use devlink to
  2078. cycle through a device's possible prt split configurations. If you use devlink
  2079. to change the configuration, you must check the log to determine which
  2080. configuration was selected. If you use devlink, you specify the number of ports
  2081. you want configured on the device. Each time you call devlink with that port
  2082. count, the driver will check the device's current configuration and then move
  2083. to the next configuration with the specified number of ports. For example, if
  2084. your device has two four-port configurations defined in its NVM, the first time
  2085. you called devlink, it would select the first configuration. The second time
  2086. you called devlink, it would select the second configuration. If you called
  2087. devlink again, it would select the first configuration. There is no direct
  2088. feedback mechanism; you must check the log to determine which configuration was
  2089. set. Use the following command:
  2090.  
  2091. # devlink port split <pci/D:b:d.f>/0 count <num>
  2092.  
  2093. Where:
  2094. - <pci/D:b:d.f>/0 is the PCI address of the device
  2095. (pci/Domain:bus:device.function). /0 is the PORT_INDEX
  2096. - <num> is the desired port split count.
  2097.  
  2098. NOTES:
  2099. - If you successfully change a port's configuration, the driver logs an
  2100. information message: "Reboot required to finish port split" and the port split
  2101. configuration selected. This is the only indication of success.
  2102. - If you request an unsupported count value parameter in devlink port split,
  2103. the driver logs an information message: "Port split requested unsupported port
  2104. config".
  2105. - If you try to change the configuration on a PF that is not PF 0, the driver
  2106. returns the error "Port cannot be split."
  2107.  
  2108. For example, if your device had the following configurations defined in its NVM:
  2109.  
  2110. ice 0000:16:00.0: Status Split Quad 0 _ Quad 1
  2111. ice 0000:16:00.0: count L0 L1 L2 L3 L4 L5 L6 L7
  2112. ice 0000:16:00.0: Active 2 100 - - - 100 - - -
  2113. ice 0000:16:00.0: 2 50 - 50 - - - - -
  2114. ice 0000:16:00.0: 4 25 25 25 25 - - - -
  2115. ice 0000:16:00.0: 4 25 25 - - 25 25 - -
  2116. ice 0000:16:00.0: 8 10 10 10 10 10 10 10 10
  2117. ice 0000:16:00.0: 1 100 - - - - - - -
  2118.  
  2119. If you call
  2120.  
  2121. # devlink port split pci/0000.16:00.0/0 count 4
  2122.  
  2123. Your device will be configured for
  2124.  
  2125. ice 0000:16:00.0: 4 25 25 25 25 - - - -
  2126.  
  2127. If you call the same command again, your device will be configured for
  2128.  
  2129. ice 0000:16:00.0: 4 25 25 - - 25 25 - -
  2130.  
  2131. If you call the same command a third time, your device will cycle back to the
  2132. top of its 4-port configurations (because there are only two 4-port
  2133. configurations defined it its NVM) and will be set to
  2134.  
  2135. ice 0000:16:00.0: 4 25 25 25 25 - - - -
  2136.  
  2137.  
  2138. Firmware Logs
  2139. -------------
  2140. The ice driver allows you to generate firmware logs for supported categories of
  2141. events, to help debug issues with Customer Support. Firmware logs are enabled
  2142. by default. Refer to the Intel(R) Ethernet Adapters and Devices User Guide for
  2143. an overview of this feature and additional tips.
  2144.  
  2145. At a high level, you must do the following to capture a firmware log:
  2146.  
  2147. 1. Set the configuration for the firmware log.
  2148. 2. Perform the necessary steps to generate the issue you are trying to debug.
  2149. 3. Capture the firmware log.
  2150. 4. Stop capturing the firmware log.
  2151. 5. Reset your firmware log settings as needed.
  2152. 6. Work with Customer Support to debug your issue.
  2153.  
  2154. NOTE: Firmware logs are generated in a binary format and must be decoded by
  2155. Customer Support. Information collected is related only to firmware and
  2156. hardware for debug purposes.
  2157.  
  2158. Firmware logs are printed to dmesg. The driver groups these events into
  2159. categories, called "modules." Supported modules include:
  2160. * 00000001 - General (Bit 0)
  2161. * 00000002 - Control (Bit 1)
  2162. * 00000004 - Link Management (Bit 2)
  2163. * 00000008 - Link Topology Detection (Bit 3)
  2164. * 00000010 - Link Control Technology (Bit 4)
  2165. * 00000020 - I2C (Bit 5)
  2166. * 00000040 - SDP (Bit 6)
  2167. * 00000080 - MDIO (Bit 7)
  2168. * 00000100 - Admin Queue (Bit 8)
  2169. * 00000200 - Host DMA (Bit 9)
  2170. * 00000400 - LLDP (Bit 10)
  2171. * 00000800 - DCBx (Bit 11)
  2172. * 00001000 - DCB (Bit 12)
  2173. * 00002000 - XLR (function-level resets; Bit 13)
  2174. * 00004000 - NVM (Bit 14)
  2175. * 00008000 - Authentication (Bit 15)
  2176. * 00010000 - VPD (Vital Product Data; Bit 16)
  2177. * 00020000 - IOSF (Intel On-Chip System Fabric, Bit 17)
  2178. * 00040000 - Parser (Bit 18)
  2179. * 00080000 - Switch (Bit 19)
  2180. * 00100000 - Scheduler (Bit 20)
  2181. * 00200000 - TX Queue Management (Bit 21)
  2182. * 00400000 - ACL (Access Control List; Bit 22)
  2183. * 00800000 - Post (Bit 23)
  2184. * 01000000 - Watchdog (Bit 24)
  2185. * 02000000 - Task Dispatcher (Bit 25)
  2186. * 04000000 - Manageability (Bit 26)
  2187. * 08000000 - SyncE (Bit 27)
  2188. * 10000000 - Health (Bit 28)
  2189. * 20000000 - Time Sync (Bit 29)
  2190. * 40000000 - PF Registration (Bit 30)
  2191. * 80000000 - Module Version (Bit 31)
  2192.  
  2193. You can change the verbosity level of the firmware logs. You can set only one
  2194. log level per module, and each level includes the verbosity levels lower than
  2195. it. For instance, setting the level to "normal" will also log warning and error
  2196. messages. Available verbosity levels are:
  2197. 0 = none
  2198. 1 = error
  2199. 2 = warning
  2200. 3 = normal
  2201. 4 = verbose
  2202.  
  2203. NOTE:
  2204. - Firmware logs can overrun the dmesg buffer. Before loading the driver,
  2205. redirect dmesg to a file.
  2206. - Use a bitmap to set the desired verbosity level for the module(s). NOTE: You
  2207. must have dynamic debug enabled in the kernel.
  2208. - You cannot change firmware log parameters at runtime. You must reload the
  2209. driver for changes to take effect.
  2210.  
  2211. Do the following to capture a firmware log in Linux:
  2212.  
  2213. 1. Remove the driver:
  2214.  
  2215. # rmmod ice
  2216.  
  2217. 2. Redirect the firmware log from dmesg to a file:
  2218.  
  2219. # dmesg -w > filename.log
  2220.  
  2221. 3. Load the driver using the following command, changing the events and level
  2222. values as needed:
  2223.  
  2224. # sudo insmod ice.ko dyndbg="+p" fwlog_events=<bitmask> fwlog_level=<level 0-4>
  2225.  
  2226. 4. Perform the necessary steps to generate the issue you are trying to debug.
  2227.  
  2228. 5. Work with Customer Support to decode your firmware log file and debug the
  2229. issue.
  2230.  
  2231. NOTE: To disable firmware logging completely, remove the driver and reload it.
  2232. Firmware logging will remain disabled until you enable it again.
  2233.  
  2234. CODE EXAMPLES:
  2235. To set all events to log warning messages, use the following command:
  2236.  
  2237. # sudo insmod ice.ko dyndbg="+p" fwlog_events=0x0FFFFFFF fwlog_level=2
  2238.  
  2239. To log verbose, normal, warning, and error messages for the ACL (Bit 22),
  2240. Switch (Bit 19), and Parser (Bit 18) modules, for example, use the following:
  2241.  
  2242. # sudo insmod ice.ko dyndbg="+p" fwlog_events=0x4C0000 fwlog_level=4
  2243.  
  2244. To dump the firmware logging configuration to dmesg, use the following commands:
  2245.  
  2246. # echo dump fwlog > command
  2247. # dmesg
  2248.  
  2249.  
  2250. Hierarchical QoS (HQoS) Transmit Scheduler
  2251. ------------------------------------------
  2252. You can configure a custom transmit scheduler tree structure to shape transmit
  2253. traffic for specific needs. You change the tree structure by creating parent
  2254. nodes on the device and then assigning child nodes (VFs)to the parent node. You
  2255. can also change the transmit rate management configuration for each node.
  2256.  
  2257. NOTES:
  2258. - Reconfiguring the scheduler topology should only be done by an expert.
  2259. Modifying the scheduler topology may adversly impact your device's network
  2260. availability and throughput. Do not do this unless you are willing to take
  2261. these risks. After modifying the scheduler topology, if your device does not
  2262. perform as expected, you should return the device to the default topology.
  2263. - Modifying the Hierarchical QoS (HQoS) Transmit Scheduler requires Kernel 6.2,
  2264. or later.
  2265. - Modifying the Hierarchical QoS (HQoS) Transmit Scheduler is not compatible
  2266. with ADQ, DCB, RDMA, or other custom scheduler tree features.
  2267.  
  2268. To create a devlink-rate parent group:
  2269. # devlink port function rate add <dev/port>/<group>
  2270. where <dev/port> is the pci bus:device:function of the device
  2271. <group> is a new parent group
  2272.  
  2273. For example:
  2274. # devlink port function rate add pci/0000:03:00.0/operators
  2275. creates the "operators" group on the specified device
  2276.  
  2277. To create a new child node in a parent group:
  2278. # devlink port function rate add <dev/port>/<child> parent <group>
  2279. where <dev/port> is the pci bus:device:function of the device
  2280. <child> is a new child node
  2281. <group> is an existing parent group
  2282. For example:
  2283. # devlink port function rate add pci/0000:03:00.0/class_1 parent operators
  2284. creates the "class_1" child node in the "operators" parent group.
  2285.  
  2286. To display a device's current tree structure:
  2287. # devlink port function rate show <dev/port>
  2288. where <dev/port> is the pci bus:device:function of the device
  2289. For example:
  2290. # devlink port function rate show pci/0000:03:00.0
  2291.  
  2292. Example output:
  2293. pci/0000:03:00.0/node_0 type node (root)
  2294. pci/0000:03:00.0/operators type node tx_share 20Mbit tx_max 100Mbit tx_priority
  2295. 2 tx_weight 5
  2296. pci/0000:03:00.0/class_1 type node parent operators
  2297. pci/0000:03:00.0/1 type leaf parent class_1
  2298. Refer to the devlink-rate MAN page and other documentation for details.
  2299.  
  2300.  
  2301.  
  2302. Performance Optimization
  2303. ========================
  2304. Driver defaults are meant to fit a wide variety of workloads, but if further
  2305. optimization is required, we recommend experimenting with the following
  2306. settings.
  2307.  
  2308.  
  2309. Transmit/Receive Queue Allocation
  2310. ---------------------------------
  2311. The driver allocates a number of transmit/receive queue pairs equal to the
  2312. number of local node CPU threads with the following constraints:
  2313. * The driver will allocate a minimum of 8 queue pairs, or the total number of
  2314. CPUs, whichever is lower
  2315. * The driver will allocate a maximum of 64 queue pairs. Or 256 for the iavf
  2316. driver.
  2317.  
  2318. You can set the number of queues symmetrical or asymmetrical using the ethtool
  2319. -L command. For example:
  2320. Setting 16 queue pairs for the interface
  2321. # ethtool -L <ethX> combined 16
  2322. # ethtool -L <ethX> tx 16 rx 16
  2323.  
  2324. Setting 16 Tx queues and 8 Rx queues
  2325. # ethtool -L <ethX> tx 16 rx 8
  2326.  
  2327. NOTE: You cannot configure less than 1 Rx or 1 Tx queue. Attempts to do so will
  2328. be rejected by the driver.
  2329. NOTE: You cannot configure more Tx/Rx queues than there are MSI-X interrupts
  2330. available. Attempts to do so will be rejected by the driver.
  2331.  
  2332.  
  2333. IRQ to Adapter Queue Alignment
  2334. ------------------------------
  2335. Pin the adapter's IRQs to specific cores by disabling the irqbalance service
  2336. and using the included set_irq_affinity script. Please see the script's help
  2337. text for further options.
  2338.  
  2339. - The following settings will distribute the IRQs across all the cores
  2340. evenly:
  2341.  
  2342. # scripts/set_irq_affinity -x all <interface1> , [ <interface2>, ... ]
  2343.  
  2344. - The following settings will distribute the IRQs across all the cores that
  2345. are local to the adapter (same NUMA node):
  2346.  
  2347. # scripts/set_irq_affinity -x local <interface1> ,[ <interface2>, ... ]
  2348.  
  2349. - For very CPU-intensive workloads, we recommend pinning the IRQs to all
  2350. cores.
  2351.  
  2352.  
  2353. Rx Descriptor Ring Size
  2354. -----------------------
  2355. To reduce the number of Rx packet discards, increase the number of Rx
  2356. descriptors for each Rx ring using ethtool.
  2357.  
  2358. - Check if the interface is dropping Rx packets due to buffers being full
  2359. (rx_dropped.nic can mean that there is no PCIe bandwidth):
  2360.  
  2361. # ethtool -S <ethX> | grep "rx_dropped"
  2362.  
  2363. - If the previous command shows drops on queues, it may help to increase
  2364. the number of descriptors using 'ethtool -G':
  2365.  
  2366. # ethtool -G <ethX> rx <N>
  2367. Where <N> is the desired number of ring entries/descriptors
  2368.  
  2369. This can provide temporary buffering for issues that create latency while
  2370. the CPUs process descriptors.
  2371.  
  2372.  
  2373.  
  2374. Interrupt Rate Limiting
  2375. -----------------------
  2376. This driver supports an adaptive interrupt throttle rate (ITR) mechanism that
  2377. is tuned for general workloads. The user can customize the interrupt rate
  2378. control for specific workloads, via ethtool, adjusting the number of
  2379. microseconds between interrupts.
  2380.  
  2381. To set the interrupt rate manually, you must disable adaptive mode:
  2382.  
  2383. # ethtool -C <ethX> adaptive-rx off adaptive-tx off
  2384.  
  2385. For lower CPU utilization:
  2386. - Disable adaptive ITR and lower Rx and Tx interrupts. The examples below
  2387. affect every queue of the specified interface.
  2388.  
  2389. - Setting rx-usecs and tx-usecs to 80 will limit interrupts to about
  2390. 12,500 interrupts per second per queue:
  2391.  
  2392. # ethtool -C <ethX> adaptive-rx off adaptive-tx off rx-usecs 80
  2393. tx-usecs 80
  2394.  
  2395. For reduced latency:
  2396. - Disable adaptive ITR and ITR by setting rx-usecs and tx-usecs to 0
  2397. using ethtool:
  2398.  
  2399. # ethtool -C <ethX> adaptive-rx off adaptive-tx off rx-usecs 0
  2400. tx-usecs 0
  2401.  
  2402. Per-queue interrupt rate settings:
  2403. - The following examples are for queues 1 and 3, but you can adjust other
  2404. queues.
  2405.  
  2406. - To disable Rx adaptive ITR and set static Rx ITR to 10 microseconds or
  2407. about 100,000 interrupts/second, for queues 1 and 3:
  2408.  
  2409. # ethtool --per-queue <ethX> queue_mask 0xa --coalesce adaptive-rx off
  2410. rx-usecs 10
  2411.  
  2412. - To show the current coalesce settings for queues 1 and 3:
  2413.  
  2414. # ethtool --per-queue <ethX> queue_mask 0xa --show-coalesce
  2415.  
  2416. Bounding interrupt rates using rx-usecs-high:
  2417. - Valid Range: 0-236 (0=no limit)
  2418.  
  2419. The range of 0-236 microseconds provides an effective range of 4,237 to
  2420. 250,000 interrupts per second. The value of rx-usecs-high can be set
  2421. independently of rx-usecs and tx-usecs in the same ethtool command, and is
  2422. also independent of the adaptive interrupt moderation algorithm. The
  2423. underlying hardware supports granularity in 4-microsecond intervals, so
  2424. adjacent values may result in the same interrupt rate.
  2425.  
  2426. - The following command would disable adaptive interrupt moderation, and allow
  2427. a maximum of 5 microseconds before indicating a receive or transmit was
  2428. complete. However, instead of resulting in as many as 200,000 interrupts per
  2429. second, it limits total interrupts per second to 50,000 via the rx-usecs-high
  2430. parameter.
  2431.  
  2432. # ethtool -C <ethX> adaptive-rx off adaptive-tx off rx-usecs-high 20
  2433. rx-usecs 5 tx-usecs 5
  2434.  
  2435.  
  2436. Virtualized Environments
  2437. ------------------------
  2438. In addition to the other suggestions in this section, the following may be
  2439. helpful to optimize performance in VMs.
  2440.  
  2441. - Using the appropriate mechanism (vcpupin) in the VM, pin the CPUs to
  2442. individual LCPUs, making sure to use a set of CPUs included in the
  2443. device's local_cpulist: /sys/class/net/<ethX>/device/local_cpulist.
  2444.  
  2445. - Configure as many Rx/Tx queues in the VM as available. (See the iavf driver
  2446. documentation for the number of queues supported.) For example:
  2447.  
  2448. # ethtool -L <virt_interface> rx <max> tx <max>
  2449.  
  2450.  
  2451. Transmit Balancing
  2452. ------------------
  2453. Some Intel(R) Ethernet 800 Series devices allow you to enable a transmit
  2454. balancing feature to improve transmit performance under certain conditions.
  2455. When the feature is enabled, you should experience more consistent transmit
  2456. performance across queues and/or PFs and VFs.
  2457.  
  2458. By default, transmit balancing is disabled in the NVM. To enable this feature,
  2459. use one of the following to persistently change the setting for the device:
  2460. - Use the Ethernet Port Configuration Tool (EPCT) to enable the tx_balancing
  2461. option. Refer to the EPCT readme for more information.
  2462. - Enable the Transmit Balancing device setting in UEFI HII.
  2463. - Enable transmit balancing via Linux devlink (see below).
  2464.  
  2465. When the driver loads, it reads the transmit balancing setting from the NVM and
  2466. configures the device accordingly.
  2467.  
  2468. NOTE:
  2469. - The user selection for transmit balancing in EPCT, HII, or Linux devlink is
  2470. persistent across reboots. You must reboot the system for the selected setting
  2471. to take effect.
  2472. - This setting is device wide.
  2473. - The driver, NVM, and DDP package must all support this functionality to
  2474. enable the feature.
  2475.  
  2476. To set the transmit balancing feature via devlink:
  2477.  
  2478. # devlink dev param set <pci/D:b:d.f> name txbalancing value <setting> cmode
  2479. permanent
  2480.  
  2481. Where:
  2482. - <pci/D:b:d.f> is the PCI address of the PF.
  2483. - <setting> is true to enable transmit balancing, or false to disable transmit
  2484. balancing.
  2485.  
  2486. To show the current transmit balancing setting:
  2487.  
  2488. # devlink dev param show [ <pci> name txbalancing ]
  2489.  
  2490.  
  2491. MSI-X Vector Allocation
  2492. -----------------------
  2493. The ice driver automatically allocates MSI-X vectors for PF, VF, and RDMA from
  2494. a pool of 2048 vectors. If there are 8, or fewer, local node CPU threads, the
  2495. driver will automatically allocate 8 vectors for each PF. This scales up by
  2496. allocating one vector per local node CPU thread, up to 64 vectors. The driver
  2497. will not automatically allocate more than 64 MSI-X vectors for each PF. RDMA
  2498. requires one more MSI-X vector than the PF allocation, so the driver will
  2499. automatically allocation 9-65 MSI-X vectors for RDMA.
  2500.  
  2501. Setting MSI-X Vector Allocation
  2502. -------------------------------
  2503. You can use sysfs to override the automatic MSI-X vector allocation for a
  2504. particular PF or RDMA function, or for the pool of vectors used by the VFs
  2505. bound to a PF.
  2506.  
  2507. # devlink resource set <pci/D:b:d.f> msix/<parameter> size <num>
  2508.  
  2509. Where:
  2510. - <pci/D:b:d.f> is the PCI address of the device
  2511. (pci/Domain:bus:device.function)
  2512. - <parameter> is one of the following:
  2513. * For a PF use the msix_eth parameter
  2514. * For an RDMA function use the msix_rdma parameter
  2515. * For the pool of vectors used by the VFs use the msix_vf parameter
  2516. - <num> is the number of MSI-X vector to assign to the function
  2517.  
  2518. For example, to set a PF to use 320 MSI-X vectors:
  2519.  
  2520. # devlink resource set pci/0000:31:00.1 msix/msix_eth size 320
  2521.  
  2522. NOTE: For this change to take affect you must reinitialize the driver after you
  2523. make this change. Reinitializing the driver may drop some netdev
  2524. configurations, including reset or downtime. Refer to the Devlink Reload
  2525. documentation for more information.
  2526.  
  2527. You can set the allocation for a particular VF with the sriov_vf_msix_count
  2528. sysfs parameter.
  2529.  
  2530. # echo <num> > /sys/bus/pci/devices/D:b:d.f/sriov_vf_msix_count
  2531.  
  2532. Where:
  2533. - <D:b:d.f> is the PCI address of the device (Domain:bus:device.function)
  2534. - <num> is the number of MSI-X vector allocate to the particular VF
  2535.  
  2536. For example, to set a VF to 64 MSI-X vectors, use
  2537.  
  2538. # echo 64 > /sys/bus/pci/devices/0000:31:00.2/sriov_vf_msix_count
  2539.  
  2540. Current MSI-X Allocation
  2541. ------------------------
  2542. You can check the current MSI-X vector allocation by using the devlink resource
  2543. show parameter. For example:
  2544.  
  2545. # devlink resource show pci/0000:31:00.1
  2546.  
  2547. Might return:
  2548.  
  2549. name: msix size 520 occ 262 unit entry dpipe_tables none
  2550. resources:
  2551. name msix_misc size 4 unit entry dpipe_tables none
  2552. name: msix_eth size 48 occ 24 unit
  2553. name: msix_vf size 48 occ 24 unit
  2554. name: msix_rdma size 48 occ 24 unit
  2555.  
  2556. Increasing the automatic allocation limit
  2557. -----------------------------------------
  2558. The ice driver supports changing the automatic MSI-X vector allocation for PFs
  2559. and VFs to spread the RSS load across more cores. Each PF has its own LUT,
  2560. while all VFs use the global LUT. Each PF LUT allows for 2048 MSI-X vectors.
  2561. The VF default is a limit of 64 MSI-X vectors, but you can increase this to 512
  2562. vectors if there are enough resources in the global LUT. You can also assign a
  2563. PF's LUT to a bound VF, increasing the VF's MSI-X vector limit to 2048, but
  2564. decreasing the PF's limit to 512. Use the rss_lut_pf_attr and rss_lut_vf_attr
  2565. sysfs parameters to manage this.
  2566.  
  2567. NOTES:
  2568. - Before changing rss_lut_vf_attr, you must first set sriov_drivers_autoprobe
  2569. to zero. After changing rss_lut_vf_attr, you can set sriov_drivers_autoprobe
  2570. back to 1.
  2571. - You must reload the iavf driver after making these changes.
  2572.  
  2573. Set a VF's limit to 512, using the global LUT:
  2574. # echo 0 > /sys/bus/pci/devices/<ethx>/sriov_drivers_autoprobe
  2575. # echo 512 > /sys/bus/pci/devices/<ethx>/rss_lut_vf_attr
  2576.  
  2577. Set a VF to use its PF's LUT:
  2578. # echo 0 > /sys/bus/pci/devices/<ethx>/sriov_drivers_autoprobe
  2579. # echo 512 > /sys/bus/pci/devices/<ethx>/rss_lut_pf_attr
  2580. # echo 2048 > /sys/bus/pci/devices/<ethx>/rss_lut_vf_attr
  2581.  
  2582. Set a PF back to using its PF LUT.
  2583. # echo 0 > /sys/bus/pci/devices/<ethx>/sriov_drivers_autoprobe
  2584. # echo 512 > /sys/bus/pci/devices/<ethx>/rss_lut_vf_attr
  2585. # echo 2048 > /sys/bus/pci/devices/<ethx>/rss_lut_pf_attr
  2586.  
  2587.  
  2588. Known Issues/Troubleshooting
  2589. ============================
  2590.  
  2591. Receive Error counts may be higher than the actual packet error count
  2592. ---------------------------------------------------------------------
  2593. When a packet is received with more than one error, two bad packets may be
  2594. reported. This affects all devices based on 10G, or faster, controllers.
  2595.  
  2596.  
  2597. Dynamic Debug
  2598. -------------
  2599. If you encounter unexpected issues during driver load, some of the most useful
  2600. information for developers to receive in a bug report can include driver
  2601. logging. This logging uses a kernel feature called Dynamic Debug, which is
  2602. generally enabled in most kernel configurations (CONFIG_DYNAMIC_DEBUG=y).
  2603.  
  2604. To load the driver with dynamic debug enabled, run modprobe with the dyndbg
  2605. parameter:
  2606.  
  2607. # modprobe ice dyndbg=+p
  2608.  
  2609. The driver will then load and print debugging information into the kernel log
  2610. (dmesg) and is usually logged into the system log viewable by journalctl or in
  2611. /var/log/messages. Saving this information to a file and attaching it to any
  2612. bug report can help shorten the reproduction and debugging time for a developer.
  2613.  
  2614. To enable dynamic debug during runtime operation of the driver, use this
  2615. command:
  2616.  
  2617. # echo "module ice +p" > /sys/kernel/debug/dynamic_debug/control
  2618.  
  2619. For more details, see the Dynamic Debug documentation included in the Linux
  2620. kernel instructions.
  2621.  
  2622.  
  2623. PF Message Queue Overflow
  2624. -------------------------
  2625. The device driver can detect some types of anomalous behavior. When it does, it
  2626. will log the VF MAC address and associated PF MAC address. Using this
  2627. information, you can check the virtual machine (VM) that is using the VF MAC
  2628. address to ensure that the VM is operating correctly.
  2629.  
  2630.  
  2631. 'ethtool -S' does not display Tx/Rx packet statistics
  2632. -----------------------------------------------------
  2633. Issuing the command 'ethtool -S' does not display Tx/Rx packet statistics. This
  2634. is by convention. Use other tools (such as the 'ip' command) that display
  2635. standard netdev statistics such as Tx/Rx packet statistics.
  2636.  
  2637.  
  2638. 'ethtool -S' rx_bytes and ip stats rx_bytes don't match statistics
  2639. ------------------------------------------------------------------
  2640. The rx_bytes value of ethtool does not match the rx_bytes value of Netdev, due
  2641. to the 4-byte CRC being stripped by the device. The difference between the two
  2642. rx_bytes values will be 4 x the number of Rx packets. For example, if Rx
  2643. packets are 10 and Netdev (software statistics) displays rx_bytes as "X", then
  2644. ethtool (hardware statistics) will display rx_bytes as "X+40" (4 bytes CRC x 10
  2645. packets).
  2646.  
  2647.  
  2648. Unexpected Issues when the device driver and DPDK share a device
  2649. ----------------------------------------------------------------
  2650. Unexpected issues may result when an ice device is in multi driver mode and the
  2651. kernel driver and DPDK driver are sharing the device. This is because access to
  2652. the global NIC resources is not synchronized between multiple drivers. Any
  2653. change to the global NIC configuration (writing to a global register, setting
  2654. global configuration by AQ, or changing switch modes) will affect all ports and
  2655. drivers on the device. Loading DPDK with the "multi-driver" module parameter
  2656. may mitigate some of the issues.
  2657.  
  2658.  
  2659. Fiber optics and auto-negotiation
  2660. ---------------------------------
  2661. Modules based on 100GBASE-SR4, active optical cable (AOC), and active copper
  2662. cable (ACC) do not support auto-negotiation per the IEEE specification. To
  2663. obtain link with these modules, you must turn off auto-negotiation on the link
  2664. partner's switch ports.
  2665.  
  2666.  
  2667. 'ethtool -a' autonegotiate result may vary between drivers
  2668. ----------------------------------------------------------
  2669. For kernel versions 4.6 or higher, 'ethtool -a' will show the advertised and
  2670. negotiated autoneg settings. For kernel versions below 4.6, ethtool will only
  2671. report the negotiated link status.
  2672.  
  2673. The issue is cosmetic and does not affect functionality. Installing the latest
  2674. ice driver and upgrading your kernel to version 4.6 or higher will resolve the
  2675. issue.
  2676.  
  2677.  
  2678. AF_XDP fails to allocate buffers
  2679. --------------------------------
  2680. On kernels older than 5.3, you may see an undesirable CPU load during packet
  2681. processing if you enable AF_XDP in native mode and the Rx ring size is larger
  2682. than the UMEM fill queue. This is due to a known issue in the kernel and was
  2683. fixed in 5.3. To address the issue, upgrade your kernel to 5.3 or newer.
  2684.  
  2685.  
  2686. SCTP checksum offloads aren't indicated on Geneve tunnel
  2687. --------------------------------------------------------
  2688. For SCTP traffic over a Geneve tunnel, the SCTP checksum isn't offloaded to the
  2689. device, even when tx-checksum-sctp is on. This is due to a limitation in the
  2690. Linux kernel. However, for Rx traffic, the SCTP checksum is verified if
  2691. rx-checksumming is on. For both Tx and Rx traffic, you can offload the outer
  2692. UDP checksum to the device.
  2693.  
  2694.  
  2695. CentOS 7.2 Issues
  2696. -----------------
  2697. The following issues are specific to CentOS 7.2. Upgrading to the latest
  2698. version of the operating system will resolve these issues.
  2699. - base-r-fec mode is supposed to be on by default. On CentOS 7.2, Ethtool
  2700. --show-priv-flags shows that it is off, instead of on.
  2701. - ethtool -m <ethX> does not display optical module information as expected.
  2702. - You cannot create an ipv6 Intel(R) Ethernet Flow Director rule. For example:
  2703. # ethtool -U p1p1 flow-type tcp6 src-ip 3001:1::2:1:1 dst-ip 3001:1::1:1:1
  2704. src-port 22 dst-port 23 action 10
  2705. Returns a bad syntax error.
  2706.  
  2707.  
  2708. Incorrect link speed reported on older VF drivers
  2709. -------------------------------------------------
  2710. Linux distributions with older iavf or i40evf drivers (including Red Hat
  2711. Enterprise Linux 8) may show an incorrect link speed on VF interfaces. This
  2712. issue is cosmetic and does not affect VF functionality. To resolve the issue,
  2713. download the latest iavf driver.
  2714.  
  2715.  
  2716. Older VF drivers on Intel Ethernet 800 Series adapters
  2717. ------------------------------------------------------
  2718. Some Windows* VF drivers from Release 22.9 or older may encounter errors when
  2719. loaded on a PF based on the Intel Ethernet 800 Series on Linux KVM. You may see
  2720. errors and the VF may not load. This issue does not occur starting with the
  2721. following Windows VF drivers:
  2722. - v40e64, v40e65: Version 1.5.65.0 and newer
  2723.  
  2724. To resolve this issue, download and install the latest iavf driver.
  2725.  
  2726.  
  2727. 'VF X failed opcode 24' error message in dmesg on host
  2728. ------------------------------------------------------
  2729. With a Microsoft Windows Server 2019 guest machine running on a Linux host, you
  2730. may see 'VF <vf_number> failed opcode 24' error messages in dmesg on the host.
  2731. This error is benign and does not affect traffic. Installing the latest iavf
  2732. driver in the guest will resolve the issue.
  2733.  
  2734.  
  2735. Windows guest OSs on a Linux host may not pass traffic across VLANs
  2736. -------------------------------------------------------------------
  2737. The VF is not aware of the VLAN configuration if you use Load Balancing and
  2738. Failover (LBFO) to configure VLANs in a Windows guest. VLANs configured using
  2739. LBFO on a VF driver may result in failure to pass traffic.
  2740.  
  2741.  
  2742. SR-IOV virtual functions have identical MAC addresses
  2743. -----------------------------------------------------
  2744. When you create multiple SR-IOV virtual functions, the VFs may have identical
  2745. MAC addresses. Only one VF will pass traffic, and all traffic on other VFs with
  2746. identical MAC addresses will fail. This is related to the
  2747. "MACAddressPolicy=persistent" setting in
  2748. /usr/lib/systemd/network/99-default.link.
  2749.  
  2750. To resolve this issue, edit the /usr/lib/systemd/network/99-default.link file
  2751. and change the MACAddressPolicy line to "MACAddressPolicy=none". For more
  2752. information, see the systemd.link man page.
  2753.  
  2754.  
  2755. MDD events in dmesg when creating maximum number of VLANs on the VF
  2756. -------------------------------------------------------------------
  2757. When you create the maximum number of VLANs on the VF, you may see MDD events
  2758. in dmesg on the host. This is due to the asynchronous design of the iavf
  2759. driver. It always reports success to any VLAN requests, but the requests may
  2760. fail later. The guest OS could try to send traffic on a VLAN that is not
  2761. configured on the VF, which will cause a Malicious Driver Detection (MDD) event
  2762. in dmesg on the host.
  2763.  
  2764. This issue is cosmetic. You do not need to reload the PF driver.
  2765.  
  2766.  
  2767. 'ip address' or 'ip link' command displays an error on a single-port NIC
  2768. with 245+ VFs
  2769. ------------------------------------------------------------------------
  2770. When you use the 'ip address' or 'ip link' command on a Linux host configured
  2771. with 245 or more VFs on a single-port adapter, you may encounter a "Buffer too
  2772. small for object" error. This is due to a known issue in the iproute2 tools.
  2773. Please use ifconfig instead of iproute2. You can install ifconfig via the
  2774. net-tools-deprecated package.
  2775.  
  2776.  
  2777. Support
  2778. =======
  2779. For general information, go to the Intel support website at:
  2780. http://www.intel.com/support/
  2781.  
  2782. or the Intel Wired Networking project hosted by Sourceforge at:
  2783. http://sourceforge.net/projects/e1000
  2784.  
  2785. If an issue is identified with the released source code on a supported kernel
  2786. with a supported adapter, email the specific information related to the issue
  2787. to [email protected].
  2788.  
  2789.  
  2790. License
  2791. =======
  2792. This program is free software; you can redistribute it and/or modify it under
  2793. the terms and conditions of the GNU General Public License, version 2, as
  2794. published by the Free Software Foundation.
  2795.  
  2796. This program is distributed in the hope it will be useful, but WITHOUT ANY
  2797. WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
  2798. PARTICULAR PURPOSE. See the GNU General Public License for more details.
  2799.  
  2800. You should have received a copy of the GNU General Public License along with
  2801. this program; if not, write to the Free Software Foundation, Inc., 51 Franklin
  2802. St - Fifth Floor, Boston, MA 02110-1301 USA.
  2803.  
  2804. The full GNU General Public License is included in this distribution in the
  2805. file called "COPYING".
  2806.  
  2807. Copyright(c) 2017 - 2023 Intel Corporation.
  2808.  
  2809.  
  2810. Trademarks
  2811. ==========
  2812. Intel is a trademark or registered trademark of Intel Corporation or its
  2813. subsidiaries in the United States and/or other countries.
  2814.  
  2815. * Other names and brands may be claimed as the property of others.
  2816.  
  2817.  
  2818.  
Advertisement
Add Comment
Please, Sign In to add comment
Public Pastes
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!