Scaling the network interface driver framework (iflib)

You can scale iflib for larger systems by setting the number of PCI interrupts and queues.

Setting the number of PCI interrupts and queues

By default, the iflib framework used by many PCI drivers allocates a PCI MSI-X interrupt and queue for each CPU core to implement receive-side scaling (RSS) on each interface. In a system with many such interfaces, this mechanism can result in large numbers of interrupts allocated and multiple queues running on each core.

Because there are a limited number of MSI-X interrupts available in a system, high core and interface counts can lead to the system running out of interrupts. These errors are reported in slog2info. For example:

Jan 10 23:55:14.235 io_sock.184334 main_buffer 131 ixl2: Failed to enable MSI-X on 3:0:2 :The Requested Number And Type Of Interrupts Are Unavailable Or Could Not Be Reserved With Current Request Type [(PCI_ERR_IRQ_NOT_AVAIL)]

In addition, each queue is implemented as a thread, and multiple queues running on each core can lead to poor performance with excessive context switching between the threads.

You can resolve both these issues by using a sysctl tunable to set the number of queues per interface to an appropriate value rather than letting iflib use the default (the number of CPUs). The tunable uses the format: dev.device_type.device_instance.iflib.override_nrxqs.

For example, in a system with 24 CPUs and eight interfaces, the default total number of interrupts and queue processing threads is 192. To reduce it to 24, first create the configuration file /etc/io-sock.conf with the following contents:

dev.ix.0.iflib.override_nrxqs="3"
dev.ix.1.iflib.override_nrxqs="3"
dev.ix.2.iflib.override_nrxqs="3"
dev.ix.3.iflib.override_nrxqs="3"
dev.ix.4.iflib.override_nrxqs="3"
dev.ix.5.iflib.override_nrxqs="3"
dev.ix.6.iflib.override_nrxqs="3"
dev.ix.7.iflib.override_nrxqs="3"
dev.ix.0.iflib.override_ntxqs="3"
dev.ix.1.iflib.override_ntxqs="3"
dev.ix.2.iflib.override_ntxqs="3"
dev.ix.3.iflib.override_ntxqs="3"
dev.ix.4.iflib.override_ntxqs="3"
dev.ix.5.iflib.override_ntxqs="3"
dev.ix.6.iflib.override_ntxqs="3"
dev.ix.7.iflib.override_ntxqs="3"

Then, pass the file to io-sock when you start it. For example:

io-sock -o config=/etc/iosock.conf -d ixl
Page updated: