Commit 74f0862c authored by Bjorn Helgaas's avatar Bjorn Helgaas
Browse files

Merge branch 'pci/interrupts' 

  - Extend boot interrupt quirk to cover several Xeon chipsets (Sean V
    Kelley)

  - Add documentation about boot interrupts (Sean V Kelley)

* pci/interrupts:
  Documentation: PCI: Add background on Boot Interrupts
  PCI: Add boot interrupt quirk mechanism for Xeon chipsets
parents 30386853 06320ced

Documentation/PCI/boot-interrupts.rst

0 → 100644
+155 −0
Original line number Diff line number Diff line 
.. SPDX-License-Identifier: GPL-2.0



===============

Boot Interrupts

===============



:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>



Overview

========



On PCI Express, interrupts are represented with either MSI or inbound

interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a

given Core IO converts the legacy interrupt messages from PCI Express to

MSI interrupts.  If the IO-APIC is disabled (via the mask bits in the

IO-APIC table entries), the messages are routed to the legacy PCH. This

in-band interrupt mechanism was traditionally necessary for systems that

did not support the IO-APIC and for boot. Intel in the past has used the

term "boot interrupts" to describe this mechanism. Further, the PCI Express

protocol describes this in-band legacy wire-interrupt INTx mechanism for

I/O devices to signal PCI-style level interrupts. The subsequent paragraphs

describe problems with the Core IO handling of INTx message routing to the

PCH and mitigation within BIOS and the OS.





Issue

=====



When in-band legacy INTx messages are forwarded to the PCH, they in turn

trigger a new interrupt for which the OS likely lacks a handler. When an

interrupt goes unhandled over time, they are tracked by the Linux kernel as

Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it

reaches a specific count with the error "nobody cared". This disabled IRQ

now prevents valid usage by an existing interrupt which may happen to share

the IRQ line.



  irq 19: nobody cared (try booting with the "irqpoll" option)

  CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1

  Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020

  Call Trace:

  <IRQ>

   ? dump_stack+0x46/0x5e

   ? __report_bad_irq+0x2e/0xb0

   ? note_interrupt+0x242/0x290

   ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]

   ? handle_irq_event_percpu+0x55/0x70

   ? handle_irq_event+0x4f/0x80

   ? handle_fasteoi_irq+0x81/0x180

   ? handle_irq+0x1c/0x30

   ? do_IRQ+0x41/0xd0

   ? common_interrupt+0x84/0x84

  </IRQ>



  handlers:

  irq_default_primary_handler threaded usb_hcd_irq

  Disabling IRQ #19





Conditions

==========



The use of threaded interrupts is the most likely condition to trigger

this problem today. Threaded interrupts may not be reenabled after the IRQ

handler wakes. These "one shot" conditions mean that the threaded interrupt

needs to keep the interrupt line masked until the threaded handler has run.

Especially when dealing with high data rate interrupts, the thread needs to

run to completion; otherwise some handlers will end up in stack overflows

since the interrupt of the issuing device is still active.



Affected Chipsets

=================



The legacy interrupt forwarding mechanism exists today in a number of

devices including but not limited to chipsets from AMD/ATI, Broadcom, and

Intel. Changes made through the mitigations below have been applied to

drivers/pci/quirks.c



Starting with ICX there are no longer any IO-APICs in the Core IO's

devices.  IO-APIC is only in the PCH.  Devices connected to the Core IO's

PCIe Root Ports will use native MSI/MSI-X mechanisms.



Mitigations

===========



The mitigations take the form of PCI quirks. The preference has been to

first identify and make use of a means to disable the routing to the PCH.

In such a case a quirk to disable boot interrupt generation can be

added.[1]



  Intel® 6300ESB I/O Controller Hub

  Alternate Base Address Register:

   BIE: Boot Interrupt Enable

	  0 = Boot interrupt is enabled.

	  1 = Boot interrupt is disabled.



  Intel® Sandy Bridge through Sky Lake based Xeon servers:

  Coherent Interface Protocol Interrupt Control

   dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:

	  When this bit is set. Local INTx messages received from the

	  Intel® Quick Data DMA/PCI Express ports are not routed to legacy

	  PCH - they are either converted into MSI via the integrated IO-APIC

	  (if the IO-APIC mask bit is clear in the appropriate entries)

	  or cause no further action (when mask bit is set)



In the absence of a way to directly disable the routing, another approach

has been to make use of PCI Interrupt pin to INTx routing tables for

purposes of redirecting the interrupt handler to the rerouted interrupt

line by default.  Therefore, on chipsets where this INTx routing cannot be

disabled, the Linux kernel will reroute the valid interrupt to its legacy

interrupt. This redirection of the handler will prevent the occurrence of

the spurious interrupt detection which would ordinarily disable the IRQ

line due to excessive unhandled counts.[2]



The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or

disable) the redirection of the interrupt handler to the PCH interrupt

line. The option can be overridden by either pci=ioapicreroute or

pci=noioapicreroute.[3]





More Documentation

==================



There is an overview of the legacy interrupt handling in several datasheets

(6300ESB and 6700PXH below). While largely the same, it provides insight

into the evolution of its handling with chipsets.



Example of disabling of the boot interrupt

------------------------------------------



Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)

	5.7.3 Boot Interrupt

	https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf



Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families

Datasheet - Volume 2: Registers (Document # 330784-003)

	6.6.41 cipintrc Coherent Interface Protocol Interrupt Control

	https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf



Example of handler rerouting

----------------------------



Intel® 6700PXH 64-bit PCI Hub (Document # 302628)

	2.15.2 PCI Express Legacy INTx Support and Boot Interrupt

	https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf





If you have any legacy PCI interrupt questions that aren't answered, email me.



Cheers,

    Sean V Kelley

    sean.v.kelley@linux.intel.com



[1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/

[2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/

[3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/

Documentation/PCI/index.rst

+1 −0
Original line number Diff line number Diff line
@@ -16,3 +16,4 @@ Linux PCI Bus Subsystem 
   pci-error-recovery

   pcieaer-howto

   endpoint/index

   boot-interrupts

drivers/pci/quirks.c

+73 −7
Original line number Diff line number Diff line
@@ -1970,26 +1970,92 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_80332_1, quirk 
/*

 * IO-APIC1 on 6300ESB generates boot interrupts, see Intel order no

 * 300641-004US, section 5.7.3.

 */

#define INTEL_6300_IOAPIC_ABAR		0x40

 *

 * Core IO on Xeon E5 1600/2600/4600, see Intel order no 326509-003.

 * Core IO on Xeon E5 v2, see Intel order no 329188-003.

 * Core IO on Xeon E7 v2, see Intel order no 329595-002.

 * Core IO on Xeon E5 v3, see Intel order no 330784-003.

 * Core IO on Xeon E7 v3, see Intel order no 332315-001US.

 * Core IO on Xeon E5 v4, see Intel order no 333810-002US.

 * Core IO on Xeon E7 v4, see Intel order no 332315-001US.

 * Core IO on Xeon D-1500, see Intel order no 332051-001.

 * Core IO on Xeon Scalable, see Intel order no 610950.

 */

#define INTEL_6300_IOAPIC_ABAR		0x40	/* Bus 0, Dev 29, Func 5 */

#define INTEL_6300_DISABLE_BOOT_IRQ	(1<<14)



#define INTEL_CIPINTRC_CFG_OFFSET	0x14C	/* Bus 0, Dev 5, Func 0 */

#define INTEL_CIPINTRC_DIS_INTX_ICH	(1<<25)



static void quirk_disable_intel_boot_interrupt(struct pci_dev *dev)

{

	u16 pci_config_word;

	u32 pci_config_dword;



	if (noioapicquirk)

		return;



	pci_read_config_word(dev, INTEL_6300_IOAPIC_ABAR, &pci_config_word);

	switch (dev->device) {

	case PCI_DEVICE_ID_INTEL_ESB_10:

		pci_read_config_word(dev, INTEL_6300_IOAPIC_ABAR,

				     &pci_config_word);

		pci_config_word |= INTEL_6300_DISABLE_BOOT_IRQ;

	pci_write_config_word(dev, INTEL_6300_IOAPIC_ABAR, pci_config_word);



		pci_write_config_word(dev, INTEL_6300_IOAPIC_ABAR,

				      pci_config_word);

		break;

	case 0x3c28:	/* Xeon E5 1600/2600/4600	*/

	case 0x0e28:	/* Xeon E5/E7 V2		*/

	case 0x2f28:	/* Xeon E5/E7 V3,V4		*/

	case 0x6f28:	/* Xeon D-1500			*/

	case 0x2034:	/* Xeon Scalable Family		*/

		pci_read_config_dword(dev, INTEL_CIPINTRC_CFG_OFFSET,

				      &pci_config_dword);

		pci_config_dword |= INTEL_CIPINTRC_DIS_INTX_ICH;

		pci_write_config_dword(dev, INTEL_CIPINTRC_CFG_OFFSET,

				       pci_config_dword);

		break;

	default:

		return;

	}

	pci_info(dev, "disabled boot interrupts on device [%04x:%04x]\n",

		 dev->vendor, dev->device);

}

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,   PCI_DEVICE_ID_INTEL_ESB_10,	quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,   PCI_DEVICE_ID_INTEL_ESB_10,	quirk_disable_intel_boot_interrupt);

/*

 * Device 29 Func 5 Device IDs of IO-APIC

 * containing ABAR—APIC1 Alternate Base Address Register

 */

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	PCI_DEVICE_ID_INTEL_ESB_10,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,	PCI_DEVICE_ID_INTEL_ESB_10,

		quirk_disable_intel_boot_interrupt);



/*

 * Device 5 Func 0 Device IDs of Core IO modules/hubs

 * containing Coherent Interface Protocol Interrupt Control

 *

 * Device IDs obtained from volume 2 datasheets of commented

 * families above.

 */

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	0x3c28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	0x0e28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	0x2f28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	0x6f28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL,	0x2034,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,	0x3c28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,	0x0e28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,	0x2f28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,	0x6f28,

		quirk_disable_intel_boot_interrupt);

DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL,	0x2034,

		quirk_disable_intel_boot_interrupt);



/* Disable boot interrupts on HT-1000 */

#define BC_HT1000_FEATURE_REG		0x64