iommu: add documentation

Signed-off-by: Aditya Basu <mitthu@google.com>
Signed-off-by: Barret Rhoden <brho@cs.berkeley.edu>

diff --git a/Documentation/iommu.md b/Documentation/iommu.md
new file mode 100644 (file)
index 0000000..4a6416d
--- /dev/null
+++ b/Documentation/iommu.md
@@ -0,0 +1,172 @@
+I/O MMU
+=======
+**2019-08-15** Aditya Basu (`mitthu`)
+
+Contents
+-------------------------------
++ Acronyms
++ I/O MMU Workings
+    - IOTLB shootdowns
++ Running Akaros in QEMU
++ Example Interaction via sysfs
+
+Acronyms
+-------------------------------
++ IEC:  Interrupt Entry Cache
++ RWBF: Required Write-Buffer Flushing
++ IVT:  Invalidate IOTLB
+
+I/O MMU Workings
+-------------------------------
+
+### IOTLB Shootdowns / flushing
+
+* If RWBF set in capability, then perform write buffer flushing.
+* If IOTLB is present, then we can perform one of the following:
+    + global shootdown
+    + DID specific shootdown (We perform this!)
+    + page-specific shootdown for a specific DID
+
+Running Akaors in QEMU
+-------------------------------
+* Currently running in QEMU requires a recompilation. This is to allow 4-level
+  paging for the IOMMU. Do the following:
+    - For qemu version 4.0.50, we need to modify line 52 of
+      "include/hw/i386/intel_iommu.h".
+    - The macro `VTD_HOST_ADDRESS_WIDTH` will be `VTD_HOST_AW_39BIT`. Change it
+      to `VTD_HOST_AW_48BIT` and recompile.
+
+```bash
+# Prepare device for passthrough (on Linux host)
+# - unbind from existing driver
+# - make sure the VFIO module is loaded
+# - make sure all virtual functions are released by the driver
+# - bind to VFIO driver for passthrough
+$ PCIDEVICE_BDF=0000:00:04.*
+$ echo $(PCIDEVICE_BDF) | sudo tee /sys/bus/pci/devices/$(PCIDEVICE_BDF)/driver/unbind
+$ sudo modprobe vfio-pci
+$ sudo rmmod ioatdma
+$ echo $(PCIDEVICE_ID) | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
+
+# Standard run
+$ sudo $(QEMU) \
+    -enable-kvm \
+    -cpu host \
+    -smp 8 \
+    -m 4096 \
+    -nographic \
+    -net nic,model=e1000 \
+    -net user,hostfwd=tcp::5555-:22 \
+    -machine q35,accel=kvm,kernel-irqchip=split \
+    -device intel-iommu,intremap=off,caching-mode=on,device-iotlb=on \
+    -device vfio-pci,host=00:04.0 \
+    -kernel obj/kern/akaros-kernel
+
+# Trace QEMU calls (for debugging)
+$ echo -n '' >/tmp/qemu-trace
+$ echo 'vfio_pci_read_config' >>/tmp/qemu-trace
+$ echo 'vfio_pci_write_config' >>/tmp/qemu-trace
+$ echo 'vfio_region_read' >>/tmp/qemu-trace
+$ echo 'vfio_region_write' >>/tmp/qemu-trace
+$ echo 'pci_data_read' >>/tmp/qemu-trace
+
+$ sudo $(QEMU) \
+    -trace events=/tmp/qemu-trace \
+    -enable-kvm \
+    -cpu host \
+    -smp 8 \
+    -m 4096 \
+    -nographic \
+    -net nic,model=e1000 \
+    -net user,hostfwd=tcp::5555-:22 \
+    -device vfio-pci,host=00:04.0 \
+    -kernel obj/kern/akaros-kernel
+```
+
+Example Interaction via sysfs
+-------------------------------
+```bash
+# Mount the device
+bash-4.3$ mkdir -p /sys/iommu
+bash-4.3$ /bin/bind \#iommu /sys/iommu
+bash-4.3$ cd /sys/iommu/
+
+# All files
+bash-4.3$ ls
+attach    detach    info      mappings  power
+
+# Attach devices
+bash-4.3$ echo 0:3.0 1 >attach
+bash-4.3$ echo 0:0.0 22 >attach
+bash-4.3$ echo 0:1:0 22 >attach
+bash-4.3$ echo 0:2.0 26 >attach
+
+# Display mappings
+bash-4.3$ cat mappings
+Mappings for iommu@0xffff8001090acce8
+        pid = 1
+                device = 0:3.0
+        pid = 22
+                device = 0:0.0
+                device = 0:1.0
+        pid = 26
+                device = 0:2.0
+
+# Detach devices
+bash-4.3$ echo 0:2.0 >detach
+bash-4.3$ echo 0:1:0 >detach
+bash-4.3$ cat mappings
+Mappings for iommu@0xffff8001090acce8
+        pid = 1
+                device = 0:3.0
+        pid = 22
+                device = 0:0.0
+
+# Display IOMMU information
+bash-4.3$ cat info
+driver info:
+        default did = 1
+        status = enabled
+
+iommu@0xffff8001090ab0e8
+        rba = 0x00000000fed90000
+        supported = yes
+        num_assigned_devs = 0
+        regspace = 0xfffffff000000000
+        host addr width (dmar) = 48
+        host addr width (cap[mgaw]) = 48
+        version = 0x10
+        capabilities = 0x00d2008c222f0686
+                mgaw: 48
+                sagaw (paging level): 0x6
+                caching mode: yes (1)
+                zlr: 0x0
+                rwbf: not required
+                num domains: 65536
+                supports protected high-memory region: no
+                supports Protected low-memory region: no
+        ext. capabilities = 0x0000000000000f46
+                pass through: yes
+                device iotlb: yes
+                iotlb register offset: 0xf0
+                snoop control: no
+                coherency: no
+                queue invalidation support: yes
+                interrupt remapping support: no
+                extended interrupt mode: 0x0
+        global status = 0xc0000000
+                translation: enabled
+                root table: set
+        root entry table = 0x00000001090aa000 (phy) or 0xffff8001090aa000 (vir)
+
+# Turn off IOMMU (global)
+bash-4.3$ cat power
+IOMMU status: enabled
+Write 'enable' or 'disable' OR 'on' or 'off' to change status
+
+bash-4.3$ echo -n off >power
+
+bash-4.3$ cat power
+IOMMU status: disabled
+Write 'enable' or 'disable' OR 'on' or 'off' to change status
+```