forked from kghost/i40e-dkms
-
Notifications
You must be signed in to change notification settings - Fork 3
/
README
2264 lines (1708 loc) · 88.9 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
i40e Linux* Base Driver for the Intel(R) Ethernet 700 Series
============================================================
February 24, 2022
Copyright(c) 2014 - 2022 Intel Corporation.
Contents
========
- Overview
- Identifying Your Adapter
- Important Notes
- Building and Installation
- Command Line Parameters
- Additional Features & Configurations
- Performance Optimization
- Known Issues/Troubleshooting
Overview
========
This driver supports kernel versions 2.6.32 and newer. However, some features
may require a newer kernel version. The associated Virtual Function (VF) driver
for this driver is iavf. The associated RDMA driver for this driver is irdma.
Driver information can be obtained using ethtool, lspci, and ip. Instructions
on updating ethtool can be found in the section Additional Configurations later
in this document.
This driver is only supported as a loadable module at this time. Intel is not
supplying patches against the kernel source to allow for static linking of the
drivers.
For questions related to hardware requirements, refer to the documentation
supplied with your Intel adapter. All hardware requirements listed apply to use
with Linux.
This driver supports XDP (Express Data Path) on kernel 4.14 and later and
AF_XDP zero-copy on kernel 4.18 and later. Note that XDP is blocked for frame
sizes larger than 3KB.
NOTE: 1 Gb devices based on the Intel(R) Ethernet Network Connection X722 do
not support the following features:
* Data Center Bridging (DCB)
* QOS
* VMQ
* SR-IOV
* Task Encapsulation offload (VXLAN, NVGRE)
* Energy Efficient Ethernet (EEE)
* Auto-media detect
Identifying Your Adapter
========================
This driver is compatible with devices based on the following:
* Intel(R) Ethernet Controller I710
* Intel(R) Ethernet Controller X710
* Intel(R) Ethernet Controller XL710
* Intel(R) Ethernet Network Connection X722
* Intel(R) Ethernet Controller XXV710
* Intel(R) Ethernet Controller V710
For the best performance, make sure the latest NVM/FW is installed on your
device and that you are using the newest drivers.
For information on how to identify your adapter, and for the latest NVM/FW
images and Intel network drivers, refer to the Intel Support website:
http://www.intel.com/support
SFP+ and QSFP+ Devices:
-----------------------
For information about supported media, refer to this document:
http://www.intel.com/content/dam/www/public/us/en/documents/release-notes/xl710-
ethernet-controller-feature-matrix.pdf
NOTE: Some adapters based on the Intel(R) Ethernet 700 Series only support
Intel Ethernet Optics modules. On these adapters, other modules are not
supported and will not function.
NOTE: For connections based on Intel(R) Ethernet 700 Series, support is
dependent on your system board. Please see your vendor for details.
NOTE: In all cases Intel recommends using Intel Ethernet Optics; other modules
may function but are not validated by Intel. Contact Intel for supported media
types.
NOTE: In systems that do not have adequate airflow to cool the adapter and
optical modules, you must use high temperature optical modules.
Important Notes
===============
TC0 must be enabled when setting up DCB on a switch
---------------------------------------------------
The kernel assumes that TC0 is available, and will disable Priority Flow
Control (PFC) on the device if TC0 is not available. To fix this, ensure TC0 is
enabled when setting up DCB on your switch.
Enabling a VF link if the port is disconnected
----------------------------------------------
If the physical function (PF) link is down, you can force link up (from the
host PF) on any virtual functions (VF) bound to the PF. Note that this requires
kernel support (Red Hat kernel 3.10.0-327 or newer, upstream kernel 3.11.0 or
newer) and associated iproute2 user space support.
For example, to force link up on VF 0 bound to PF eth0:
# ip link set eth0 vf 0 state enable
Note: If the command does not work, it may not be supported by your system.
Do not unload port driver if VF with active VM is bound to it
-------------------------------------------------------------
Do not unload a port's driver if a Virtual Function (VF) with an active Virtual
Machine (VM) is bound to it. Doing so will cause the port to appear to hang.
Once the VM shuts down, or otherwise releases the VF, the command will complete.
Configuring SR-IOV for improved network security
------------------------------------------------
In a virtualized environment, on Intel(R) Ethernet Network Adapters that
support SR-IOV, the virtual function (VF) may be subject to malicious behavior.
Software-generated layer two frames, like IEEE 802.3x (link flow control), IEEE
802.1Qbb (priority based flow-control), and others of this type, are not
expected and can throttle traffic between the host and the virtual switch,
reducing performance. To resolve this issue, and to ensure isolation from
unintended traffic streams, configure all SR-IOV enabled ports for VLAN tagging
from the administrative interface on the PF. This configuration allows
unexpected, and potentially malicious, frames to be dropped.
See "Configuring VLAN Tagging on SR-IOV Enabled Adapter Ports" later in this
README for configuration instructions.
Firmware Recovery Mode
----------------------
A device will enter Firmware Recovery mode if it detects a problem that
requires the firmware to be reprogrammed. When a device is in Firmware Recovery
mode it will not pass traffic or allow any configuration; you can only attempt
to recover the device's firmware. Refer to the Intel(R) Ethernet Adapters and
Devices User Guide for details on Firmware Recovery Mode and how to recover
from it.
Building and Installation
=========================
To manually build the driver
----------------------------
1. Move the base driver tar file to the directory of your choice.
For example, use '/home/username/i40e' or '/usr/local/src/i40e'.
2. Untar/unzip the archive, where <x.x.x> is the version number for the
driver tar file:
# tar zxf i40e-<x.x.x>.tar.gz
3. Change to the driver src directory, where <x.x.x> is the version number
for the driver tar:
# cd i40e-<x.x.x>/src/
4. Compile the driver module:
# make install
The binary will be installed as:
/lib/modules/<KERNEL VER>/updates/drivers/net/ethernet/intel/i40e/i40e.ko
The install location listed above is the default location. This may differ
for various Linux distributions.
NOTE: To gather and display additional statistics, use the
I40E_ADD_PROBES pre-processor macro:
# make CFLAGS_EXTRA=-DI40E_ADD_PROBES
Please note that this additional statistics gathering can impact
performance.
5. Load the module using the modprobe command.
To check the version of the driver and then load it:
# modinfo i40e
# modprobe i40e [parameter=port1_value,port2_value]
Alternately, make sure that any older i40e drivers are removed from the
kernel before loading the new module:
# rmmod i40e; modprobe i40e
6. Assign an IP address to the interface by entering the following,
where <ethX> is the interface name that was shown in dmesg after modprobe:
# ip address add <IP_address>/<netmask bits> dev <ethX>
7. Verify that the interface works. Enter the following, where IP_address
is the IP address for another machine on the same subnet as the interface
that is being tested:
# ping <IP_address>
Note: For certain distributions like (but not limited to) Red Hat Enterprise
Linux 7 and Ubuntu, once the driver is installed, you may need to update the
initrd/initramfs file to prevent the OS loading old versions of the i40e driver.
For Red Hat distributions:
# dracut --force
For Ubuntu:
# update-initramfs -u
To build a binary RPM package of this driver
--------------------------------------------
Note: RPM functionality has only been tested in Red Hat distributions.
1. Run the following command, where <x.x.x> is the version number for the
driver tar file.
# rpmbuild -tb i40e-<x.x.x>.tar.gz
NOTE: For the build to work properly, the currently running kernel MUST
match the version and configuration of the installed kernel sources. If
you have just recompiled the kernel, reboot the system before building.
2. After building the RPM, the last few lines of the tool output contain the
location of the RPM file that was built. Install the RPM with one of the
following commands, where <RPM> is the location of the RPM file:
# rpm -Uvh <RPM>
or
# dnf/yum localinstall <RPM>
3. If your distribution or kernel does not contain inbox support for auxiliary
bus, you must also install the auxiliary RPM:
# rpm -Uvh <i40e RPM> <auxiliary RPM>
or
# dnf/yum localinstall <i40e RPM> <auxiliary RPM>
NOTES:
- To compile the driver on some kernel/arch combinations, you may need to
install a package with the development version of libelf (e.g. libelf-dev,
libelf-devel, elfutilsl-libelf-devel).
- When compiling an out-of-tree driver, details will vary by distribution.
However, you will usually need a kernel-devel RPM or some RPM that provides the
kernel headers at a minimum. The RPM kernel-devel will usually fill in the link
at /lib/modules/'uname -r'/build.
Command Line Parameters
=======================
If the driver is built as a module, enter optional parameters on the command
line with the following syntax:
# modprobe i40e [<option>=<VAL1>]
There needs to be a <VAL#> for each network port in the system supported by
this driver. The values will be applied to each instance, in function order.
For example:
# modprobe i40e max_vfs=7
- The i40e driver only supports the max_vfs kernel parameter on older kernels
that do not have the standard sysfs interface.
- The only other module parameter supported is the debug parameter that can
control the default logging verbosity of the driver.
- In general, use ethtool and other OS-specific commands to configure
user-changeable parameters after the driver is loaded.
- The default value for each parameter is generally the recommended setting,
unless otherwise noted.
max_vfs
-------
This parameter adds support for SR-IOV. It causes the driver to spawn up to
max_vfs worth of virtual functions.
Valid Range:
1-32 (Intel Ethernet Controller X710 based devices)
1-64 (Intel Ethernet Controller XXV710/XL710 based devices)
NOTE: This parameter is only used on kernel 3.7.x and below. On kernel 3.8.x
and above, use sysfs to enable VFs. Use sysfs for Red Hat distributions.
For example, you can create 4 VFs as follows:
# echo 4 > /sys/class/net/<ethX>/device/sriov_numvfs
To disable VFs, write 0 to the same file:
# echo 0 > /sys/class/net/<ethX>/device/sriov_numvfs
The parameters for the driver are referenced by position. Thus, if you have a
dual port adapter, or more than one adapter in your system, and want N virtual
functions per port, you must specify a number for each port with each parameter
separated by a comma. For example:
# modprobe i40e max_vfs=4
This will spawn 4 VFs on the first port.
# modprobe i40e max_vfs=2,4
This will spawn 2 VFs on the first port and 4 VFs on the second port.
NOTE: Caution must be used in loading the driver with these parameters.
Depending on your system configuration, number of slots, etc., it is impossible
to predict in all cases where the positions would be on the command line.
NOTE: Neither the device nor the driver control how VFs are mapped into config
space. Bus layout will vary by operating system. On operating systems that
support it, you can check sysfs to find the mapping.
Some hardware configurations support fewer SR-IOV instances, as the whole Intel
Ethernet Controller XL710 (all functions) is limited to 128 SR-IOV interfaces
in total.
NOTE: When SR-IOV mode is enabled, hardware VLAN filtering and VLAN tag
stripping/insertion will remain enabled. Please remove the old VLAN filter
before the new VLAN filter is added. For example:
# ip link set eth0 vf 0 vlan 100 // set vlan 100 for VF 0
# ip link set eth0 vf 0 vlan 0 // Delete vlan 100
# ip link set eth0 vf 0 vlan 200 // set a new vlan 200 for VF 0
Additional Features and Configurations
======================================
ethtool
-------
The driver utilizes the ethtool interface for driver configuration and
diagnostics, as well as displaying statistical information. The latest ethtool
version is required for this functionality. Download it at:
https://kernel.org/pub/software/network/ethtool/
Viewing Link Messages
---------------------
Link messages will not be displayed to the console if the distribution is
restricting system messages. In order to see network driver link messages on
your console, set dmesg to eight by entering the following:
# dmesg -n 8
NOTE: This setting is not saved across reboots.
Configuring the Driver on Different Distributions
-------------------------------------------------
Configuring a network driver to load properly when the system is started is
distribution dependent. Typically, the configuration process involves adding an
alias line to /etc/modules.conf or /etc/modprobe.conf as well as editing other
system startup scripts and/or configuration files. Many popular Linux
distributions ship with tools to make these changes for you. To learn the
proper way to configure a network device for your system, refer to your
distribution documentation. If during this process you are asked for the driver
or module name, the name for the Base Driver is i40e.
Displaying VF Statistics on the PF
----------------------------------
Use the following command to display the statistics for all VFs on the PF:
# ethtool -S <ethX>
NOTE: The output of this command can be very large due to the large number of
VF statistics and the maximum number of possible VFs.
The PF driver will display a subset of the statistics for the PF and for all
VFs that are configured. The PF will always print a statistics block for each
of the possible VFs, and it will show zero for all unconfigured VFs.
VF stats are listed in a single block at the end of the PF statistics, using
the following naming convention:
vf<XXX>.<statistic name>
Where:
<XXX> is the VF number (for example, vf008).
<statistic name> is the name of the statistic as supplied by the VF driver.
For example:
vf008.rx_bytes: 0
vf008.rx_unicast: 0
vf008.rx_multicast: 0
vf008.rx_broadcast: 0
vf008.rx_discards: 0
vf008.rx_unknown_protocol: 0
vf008.tx_bytes: 0
vf008.tx_unicast: 0
vf008.tx_multicast: 0
vf008.tx_broadcast: 0
vf008.tx_discards: 0
vf008.tx_errors: 0
Configuring VLAN Tagging on SR-IOV Enabled Adapter Ports
--------------------------------------------------------
To configure VLAN tagging for the ports on an SR-IOV enabled adapter, use the
following command. The VLAN configuration should be done before the VF driver
is loaded or the VM is booted. The VF is not aware of the VLAN tag being
inserted on transmit and removed on received frames (sometimes called "port
VLAN" mode).
# ip link set dev <ethX> vf <id> vlan <vlan id>
For example, the following will configure PF eth0 and the first VF on VLAN 10:
# ip link set dev eth0 vf 0 vlan 10
Setting the MAC Address for a VF
--------------------------------
To change the MAC address for the specified VF:
# ip link set <ethX> vf 0 mac <address>
For example:
# ip link set <ethX> vf 0 mac 00:01:02:03:04:05
This setting lasts until the PF is reloaded.
NOTE: For untrusted VFs, assigning a MAC address for a VF from the host will
disable any subsequent requests to change the MAC address from within the VM.
This is a security feature. The VM is not aware of this restriction, so if this
is attempted in the VM, it will trigger MDD events. Trusted VFs are allowed to
change the MAC address from within the VM.
Trusted VFs and VF Promiscuous Mode
-----------------------------------
This feature allows you to designate a particular VF as trusted and allows that
trusted VF to request selective promiscuous mode on the Physical Function (PF).
To set a VF as trusted or untrusted, enter the following command in the
Hypervisor:
# ip link set dev <ethX> vf 1 trust [on|off]
NOTE: It's important to set the VF to trusted before setting promiscuous mode.
If the VM is not trusted, the PF will ignore promiscuous mode requests from the
VF. If the VM becomes trusted after the VF driver is loaded, you must make a
new request to set the VF to promiscuous.
Once the VF is designated as trusted, use the following commands in the VM to
set the VF to promiscuous mode. For promiscuous all:
# ip link set <ethX> promisc on
Where <ethX> is a VF interface in the VM
For promiscuous Multicast:
# ip link set <ethX> allmulticast on
Where <ethX> is a VF interface in the VM
NOTE: By default, the ethtool private flag vf-true-promisc-support is set to
"off," meaning that promiscuous mode for the VF will be limited. To set the
promiscuous mode for the VF to true promiscuous and allow the VF to see all
ingress traffic, use the following command:
# ethtool --set-priv-flags <ethX> vf-true-promisc-support on
The vf-true-promisc-support private flag does not enable promiscuous mode;
rather, it designates which type of promiscuous mode (limited or true) you will
get when you enable promiscuous mode using the ip link commands above. Note
that this is a global setting that affects the entire device. However, the
vf-true-promisc-support private flag is only exposed to the first PF of the
device. The PF remains in limited promiscuous mode (unless it is in MFP mode)
regardless of the vf-true-promisc-support setting.
Next, add a VLAN interface on the VF interface. For example:
# ip link add link eth2 name eth2.100 type vlan id 100
Note that the order in which you set the VF to promiscuous mode and add the
VLAN interface does not matter (you can do either first). The result in this
example is that the VF will get all traffic that is tagged with VLAN 100.
Virtual Function (VF) Tx Rate Limit
-----------------------------------
Use the ip command to configure the Tx rate limit for a VF from the PF
interface.
For example, to set a Tx rate limit of 1000Mbps for VF 0:
# ip link set eth0 vf 0 rate 1000
Malicious Driver Detection (MDD) for VFs
----------------------------------------
Some Intel Ethernet devices use Malicious Driver Detection (MDD) to detect
malicious traffic from the VF and disable Tx/Rx queues or drop the offending
packet until a VF driver reset occurs. You can view MDD messages in the PF's
system log using the dmesg command.
- If the PF driver logs MDD events from the VF, confirm that the correct VF
driver is installed.
- To restore functionality, you can manually reload the VF or VM.
MAC and VLAN Anti-Spoofing Feature for VFs
------------------------------------------
When a malicious driver on a Virtual Function (VF) interface attempts to send a
spoofed packet, it is dropped by the hardware and not transmitted.
NOTE: This feature can be disabled for a specific VF:
# ip link set <ethX> vf <vf id> spoofchk {off|on}
VLAN Pruning
------------
The i40e driver allows you to enable or disable VLAN pruning for the VF VSI
using the ethtool private flag vf-vlan-pruning.
NOTE:
- You cannot change this private flag while any VFs are active.
- If a port VLAN is configured, VLAN pruning will always be enabled.
- When VLAN pruning is enabled, the interface will:
- Discard all packets with a VLAN tag when Rx VLAN filtering is disabled.
- Discard untagged packets when Rx VLAN filtering is enabled.
To disable or enable VLAN pruning on all VFs, do the following:
1. Deinitialize any VFs.
2. On the PF, use the following command:
# ethtool --set-priv-flags <ethX> vf-vlan-pruning on|off
Where:
on - enables VLAN pruning
off - disables VLAN pruning (default)
3. Initialize and configure any VFs.
VLAN pruning will then be disabled or enabled on any of these VFs, depending on
the flag you set.
Intel(R) Ethernet Flow Director
-------------------------------
NOTE: Intel Ethernet Flow Director parameters are only supported on kernel
versions 2.6.30 or newer.
The Intel Ethernet Flow Director performs the following tasks:
- Directs receive packets according to their flows to different queues
- Enables tight control on routing a flow in the platform
- Matches flows and CPU cores for flow affinity
- Supports multiple parameters for flexible flow classification and load
balancing (in SFP mode only)
NOTE: An included script (set_irq_affinity) automates setting the IRQ to CPU
affinity.
NOTE: This driver supports the following flow types:
- IPv4
- TCPv4
- UDPv4
- SCTPv4
- IPv6
- TCPv6
- UDPv6
- SCTPv6
Each flow type supports valid combinations of IP addresses (source or
destination) and UDP/TCP ports (source and destination). You can supply only a
source IP address, a source IP address and a destination port, or any
combination of one or more of these four parameters.
NOTE: This driver allows you to filter traffic based on a user-defined flexible
two-byte pattern and offset by using the ethtool user-def and mask fields. Only
L3 and L4 flow types are supported for user-defined flexible filters. For a
given flow type, you must clear all Intel Ethernet Flow Director filters before
changing the input set (for that flow type).
The following table summarizes supported Intel Ethernet Flow Director features
across Intel(R) Ethernet controllers.
---------------------------------------------------------------------------
Feature 500 Series 700 Series 800 Series
===========================================================================
VF FLOW DIRECTOR Supported Routing to VF Not supported
not supported
---------------------------------------------------------------------------
IP ADDRESS RANGE Supported Not supported Field masking
FILTER
---------------------------------------------------------------------------
IPv6 SUPPORT Supported Supported Supported
---------------------------------------------------------------------------
CONFIGURABLE Configured Configured Configured
INPUT SET per port globally per port
---------------------------------------------------------------------------
ATR Supported Supported Not supported
---------------------------------------------------------------------------
FLEX BYTE FILTER Starts at Starts at Starts at
beginning beginning of beginning
of packet payload of packet
---------------------------------------------------------------------------
TUNNELED PACKETS Filter matches Filter matches Filter matches
outer header inner header inner header
---------------------------------------------------------------------------
Application Targeted Routing (ATR) Perfect Filters
--------------------------------------------------
Intel Ethernet Flow Director ATR is enabled by default when the kernel is in
multiple transmit queue mode. A rule is added when a TCP flow starts and is
deleted when the flow ends. Because this would interfere with sideband TCP
rules, the driver automatically disables ATR when a TCP rule is added via
ethtool (sideband). ATR is automatically re-enabled when all TCP sideband rules
are deleted or when sideband is disabled.
You can disable or enable ATR using the ethtool private flags interface. To
view the current setting:
# ethtool --show-priv-flags <ethX>
To change the setting:
# ethtool --set-priv-flags <ethX> flow-director-atr [off|on]
Packets that match the ATR rules will increment the port.fdir_atr_match stat in
ethtool. The current operational state of ATR is reflected by the stat
port.fdir_atr_status.
Sideband Perfect Filters
------------------------
Sideband Perfect Filters are used to direct traffic that matches specified
characteristics. They are enabled through ethtool's ntuple interface. To enable
or disable the Intel Ethernet Flow Director and these filters:
# ethtool -K <ethX> ntuple <off|on>
NOTE: When you disable ntuple filters, all the user programmed filters are
flushed from the driver cache and hardware. All needed filters must be re-added
when ntuple is re-enabled.
To display all of the active filters:
# ethtool -u <ethX>
To add a new filter:
# ethtool -U <ethX> flow-type <type> src-ip <ip> [m <ip_mask>] dst-ip <ip> [m
<ip_mask>] src-port <port> [m <port_mask>] dst-port <port> [m <port_mask>]
action <queue>
Where:
<ethX> - the Ethernet device to program
<type> - can be ip4, tcp4, udp4, sctp4, ip6, tcp6, udp6, sctp6
<ip> - the IP address to match on
<ip_mask> - the IPv4 address to mask on
NOTE: These filters use inverted masks. Address masks can be
either all 0 (to ignore a match) or all F (for a full match).
<port> - the port number to match on
<port_mask> - the 16-bit integer for masking
NOTE: These filters use inverted masks. Port masks can be
either all 0 (to ignore a match) or all F (for a full match).
<queue> - the queue to direct traffic toward (-1 discards the
matched traffic)
To delete a filter:
# ethtool -U <ethX> delete <N>
Where <N> is the filter ID displayed when printing all the active filters,
and may also have been specified using "loc <N>" when adding the filter.
EXAMPLES:
To add a filter that directs packet to queue 2:
# ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
192.168.10.2 src-port 2000 dst-port 2001 action 2 [loc 1]
To set a filter using only the source and destination IP address:
# ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
192.168.10.2 action 2 [loc 1]
To set a filter based on a user-defined pattern and offset:
# ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
192.168.10.2 user-def 0x4FFFF action 2 [loc 1]
where the value of the user-def field contains the offset (4 bytes) and
the pattern (0xffff).
To match TCP traffic sent from 192.168.0.1, port 5300, directed to 192.168.0.5,
port 80, and then send it to queue 7:
# ethtool -U enp130s0 flow-type tcp4 src-ip 192.168.0.1 dst-ip 192.168.0.5
src-port 5300 dst-port 80 action 7
To add a TCPv4 filter with a partial mask for a source IP subnet:
# ethtool -U <ethX> flow-type tcp4 src-ip 192.168.0.0 m 0.255.255.255 dst-ip
192.168.5.12 src-port 12600 dst-port 31 action 12
NOTES:
For each flow-type, the programmed filters must all have the same matching
input set. For example, issuing the following two commands is acceptable:
# ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
# ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.5 src-port 55 action 10
Issuing the next two commands, however, is not acceptable, since the first
specifies src-ip and the second specifies dst-ip:
# ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 src-port 5300 action 7
# ethtool -U enp130s0 flow-type ip4 dst-ip 192.168.0.5 src-port 55 action 10
The second command will fail with an error. You may program multiple filters
with the same fields, using different values, but, on one device, you may not
program two tcp4 filters with different matching fields.
The i40e driver does not support matching on a subportion of a field, thus
partial mask fields are not supported.
Flex Byte Flow Director Filters
-------------------------------
The driver also supports matching user-defined data within the packet payload.
This flexible data is specified using the "user-def" field of the ethtool
command in the following way:
+----------------------------+--------------------------+
| 31 28 24 20 16 | 15 12 8 4 0 |
+----------------------------+--------------------------+
| offset into packet payload | 2 bytes of flexible data |
+----------------------------+--------------------------+
For example,
... user-def 0x4FFFF ...
tells the filter to look 4 bytes into the payload and match that value against
0xFFFF. The offset is based on the beginning of the payload, and not the
beginning of the packet. Thus
flow-type tcp4 ... user-def 0x8BEAF ...
would match TCP/IPv4 packets which have the value 0xBEAF 8 bytes into the
TCP/IPv4 payload.
Note that ICMP headers are parsed as 4 bytes of header and 4 bytes of payload.
Thus to match the first byte of the payload, you must actually add 4 bytes to
the offset. Also note that ip4 filters match both ICMP frames as well as raw
(unknown) ip4 frames, where the payload will be the L3 payload of the IP4 frame.
The maximum offset is 64. The hardware will only read up to 64 bytes of data
from the payload. The offset must be even because the flexible data is 2 bytes
long and must be aligned to byte 0 of the packet payload.
The user-defined flexible offset is also considered part of the input set and
cannot be programmed separately for multiple filters of the same type. However,
the flexible data is not part of the input set and multiple filters may use the
same offset but match against different data.
Filters to Direct Traffic to a Specific VF
------------------------------------------
It is possible to create filters that direct traffic to a specific Virtual
Function. For older versions of ethtool, this depends on the "action"
parameter. Specify the action as a 64-bit value, where the lower 32 bits
represent the queue number, while the next 8 bits represent the VF ID. Note
that 0 is the PF, so the VF identifier is offset by 1. For example:
# ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
192.168.10.2 src-port 2000 dst-port 2001 action 0x800000002 [loc 1]
The action field specifies to direct traffic to Virtual Function 7 (8 minus 1)
into queue 2 of that VF.
Newer versions of ethtool (version 4.11 and later) use "vf" and "queue"
parameters instead of the "action" parameter. Note that using the new ethtool
"vf" parameter does not require the value to be offset by 1. This command is
equivalent to the above example:
# ethtool -U <ethX> flow-type tcp4 src-ip 192.168.10.1 dst-ip \
192.168.10.2 src-port 2000 dst-port 2001 vf 7 queue 2 [loc 1]
Note that these filters will not break internal routing rules, and will not
route traffic that otherwise would not have been sent to the specified VF.
Cloud Filter Support
--------------------
On a complex network that supports multiple types of traffic (such as for
storage as well as cloud), cloud filter support allows you to send one type of
traffic (for example, the storage traffic) to the Physical Function (PF) and
another type (say, the cloud traffic) to a Virtual Function (VF). Because cloud
networks are typically VXLAN/GENEVE-based, you can define a cloud filter to
identify VXLAN/GENEVE packets and send them to a queue in the VF to be
processed by the virtual machine (VM). Similarly, other cloud filters can be
designed for various other traffic tunneling.
NOTE: Cloud filters are only supported when the underlying device is in Single
Function per Port mode.
NOTE: The "action -1" option, which drops matching packets in regular Intel
Ethernet Flow Director filters, is not available to drop packets when used with
cloud filters.
NOTE: For IPv4 and ether flow-types, cloud filters cannot be used for TCP or
UDP filters.
NOTE: Cloud filters can be used as a method for implementing queue splitting in
the PF.
NOTE: Queue 0xffff, set through either 'queue 0xffff' or 'action 0x1000ffff',
is used for RSS.
The following filters are supported:
Cloud Filters
Inner MAC, Inner VLAN (for NVGRE, VXLAN or GENEVE packets)
Inner MAC, Inner VLAN, Tenant ID (for NVGRE, VXLAN or GENEVE packets)
Inner MAC, Tenant ID (NVGRE packet or VXLAN/GENEVE packets)
Outer MAC L2 filter
Inner MAC filter
Outer MAC, Tenant ID, Inner MAC
Application Destination IP
Application Source-IP, Inner MAC (see NOTES below)
Destination Port: TCP, UDP, or both, depending on module parameter
ToQueue: Use MAC, VLAN to point to a queue
L3 filters
Application Destination IP
NOTES:
- Cloud filters are not compatible with ADQ.
- The Destination Port cloud filter is a load time option; use the 'l4mode'
module parameter to enable it. The l4mode module parameter supports the
following settings:
parameter not present = destination port filters disabled
0 = UDP cloud filter mode enabled; destination port applies to UDP
1 = TCP cloud filter mode enabled; destination port applies to TCP
2 = both TCP and UDP filters are enabled; destination port applies to
both UDP and TCP protocols
Note: When the l4mode parameter specifies TCP (1) or both (2), the i40e
driver will disable Application Targeted Routing (ATR). The driver will
reenable ATR when the last Destination Port cloud filter rule is removed.
- The Application Source-IP, Inner MAC filter is not available when the
Destination Port cloud filter is selected.
- To change back to default mode (which supports the Application Source-IP,
Inner MAC filter listed above), you must reboot the system.
Cloud filters are specified using ethtool's ntuple interface, but the driver
uses the "user-def" field to determine whether to treat the filter as a cloud
filter or a regular filter. To enable a cloud filter, set the highest bit of
the user-def field, "user-def 0x8000000000000000" to enable the cloud features
described below. This specifies to the driver to treat the filter specially and
not treat it like the regular filters described above. Note that cloud filters
also read the other bits in the user-def field separately so you cannot use the
flexible data feature described above.
For regular Intel Ethernet Flow Director filters:
- No user-def specified or highest bit (bit 63) is 0. For example:
# ethtool -U enp130s0 flow-type ip4 src-ip 192.168.0.1 dst-ip 192.168.0.109
action 6 loc <N>
For L3 filters (non-tunneled packets):
- "user-def 0x8000000000000000" (no Tenant ID/VNI specified in remaining
bits of the user-def field)
- Only L3 parameters (src-IP, dst-IP) are considered.
- For example, to redirect traffic coming from 192.168.42.13 with destination
192.168.42.33 into VF id 1, and call this "rule 3":
# ethtool -U enp130s0 flow-type ip4 src-ip 192.168.42.13 dst-ip 192.168.42.33
user-def 0x8000000000000000 action 0x200000000 loc 3
For cloud filters (tunneled packets):
- All other filters, including where Tenant ID/VNI is specified.
The lower 32 bits of the user-def field can carry the tenant ID/VNI
if required.
- The 'loc' parameter specifies the rule number of the filter as being
stored in the base driver.
- The VF can be specified using the "action" field, just as regular filters
described in the "Sideband Perfect Filters" section above.
- To forward tunneled GRE packets directly to the VF, set bit 24 of the
"user-def" field. (Note: This feature is not supported on Intel(R) Ethernet
Network Connection X722 devices.)
- Cloud filters can be defined with inner MAC, outer MAC, inner IP address,
inner VLAN, and VNI as part of the cloud tuple. Cloud filters filter on
destination (not source) MAC and IP. The destination and source MAC
address fields in the ethtool command are overloaded as dst = outer,
src = inner MAC address to facilitate tuple definition for a cloud filter.
- Examples:
To redirect traffic on VXLAN using tunnel id 34 (hex 0x22) coming from
outer MAC address 8b:9d:ed:6a:ce:43 and inner MAC address
1d:44:9d:54:da:de into VF id 1 and call this "rule 38":
# ethtool -U enp130s0 flow-type ether dst 8b:9d:ed:6a:ce:43 \
src 1d:44:9d:54:da:de user-def 0x8000000000000022 loc 38 \
action 0x200000000
To forward tunneled GRE packets to VF 0:
# ethtool -U enp130s0 flow-type ether user-def 0x8000000001000000
action 0x10000ffff
To redirect traffic to L4 destination port 4789 to VF 0, when the l4mode
module parameter is set to UDP:
# ethtool -U enp130s0 flow-type udp4 dst-port 4789 action 0xffffffff00000000
Link-Level Flow Control (LFC)
-----------------------------
Ethernet Flow Control (IEEE 802.3x) can be configured with ethtool to enable
receiving and transmitting pause frames for i40e. When transmit is enabled,
pause frames are generated when the receive packet buffer crosses a predefined
threshold. When receive is enabled, the transmit unit will halt for the time
delay specified when a pause frame is received.
NOTE: You must have a flow control capable link partner.
Flow Control is disabled by default.
Use ethtool to change the flow control settings.
To enable or disable Rx or Tx Flow Control:
# ethtool -A <ethX> rx <on|off> tx <on|off>
Note: This command only enables or disables Flow Control if auto-negotiation is
disabled. If auto-negotiation is enabled, this command changes the parameters
used for auto-negotiation with the link partner.
To enable or disable auto-negotiation:
# ethtool -s <ethX> autoneg <on|off>
Note: Flow Control auto-negotiation is part of link auto-negotiation. Depending
on your device, you may not be able to change the auto-negotiation setting.
NOTE:
- The i40e driver requires flow control on both the port and link partner. If
flow control is disabled on one of the sides, the port may appear to hang on
heavy traffic.
RSS Hash Flow
-------------
Allows you to set the hash bytes per flow type and any combination of one or
more options for Receive Side Scaling (RSS) hash byte configuration.
# ethtool -N <ethX> rx-flow-hash <type> <option>
Where <type> is:
tcp4 signifying TCP over IPv4
udp4 signifying UDP over IPv4
tcp6 signifying TCP over IPv6
udp6 signifying UDP over IPv6
And <option> is one or more of:
s Hash on the IP source address of the Rx packet.
d Hash on the IP destination address of the Rx packet.
f Hash on bytes 0 and 1 of the Layer 4 header of the Rx packet.
n Hash on bytes 2 and 3 of the Layer 4 header of the Rx packet.
For example, to hash on the source and destination IP address for TCP IPv4
traffic, use the following:
# ethtool -N <ethX> rx-flow-hash tcp4 sd
To hash on the source and destination ports for UDP IPv6 traffic, use the
following:
# ethtool -N <ethX> rx-flow-hash udp6 sdfn
Application Device Queues (ADQ)
-------------------------------
Application Device Queues (ADQ) allow you to dedicate one or more queues to a
specific application. This can reduce latency for the specified application,
and allow Tx traffic to be rate limited per application.
Requirements:
- Kernel version 4.19.58 or later
- The sch_mqprio, act_mirred and cls_flower modules must be loaded. For example:
# modprobe sch_mqprio
# modprobe act_mirred
# modprove cls_flower
- The latest version of iproute2
# cd iproute2
# ./configure
# make DESTDIR=/opt/iproute2 install
- NVM version 6.01 or later
- ADQ cannot be enabled when the following features are enabled: Data Center
Bridging (DCB), Multiple Functions per Port (MFP), or Sideband Filters.
- If another driver (for example, DPDK) has set cloud filters, you cannot
enable ADQ.
Creating traffic classes
------------------------
Run all TC commands from the ../iproute2/tc/ directory.
If you set max_rate to less than 50Mbps, then max_rate is rounded up to 50Mbps