1*618f2068SAndy Fiddaman# Chelsio T4 Factory Default configuration file. 2*618f2068SAndy Fiddaman# 3*618f2068SAndy Fiddaman# Copyright (C) 2010-2014 Chelsio Communications. All rights reserved. 4*618f2068SAndy Fiddaman# 5*618f2068SAndy Fiddaman# DO NOT MODIFY THIS FILE UNDER ANY CIRCUMSTANCES. MODIFICATION OF 6*618f2068SAndy Fiddaman# THIS FILE WILL RESULT IN A NON-FUNCTIONAL T4 ADAPTER AND MAY RESULT 7*618f2068SAndy Fiddaman# IN PHYSICAL DAMAGE TO T4 ADAPTERS. 8*618f2068SAndy Fiddaman 9*618f2068SAndy Fiddaman# This file provides the default, power-on configuration for 4-port T4-based 10*618f2068SAndy Fiddaman# adapters shipped from the factory. These defaults are designed to address 11*618f2068SAndy Fiddaman# the needs of the vast majority of T4 customers. The basic idea is to have 12*618f2068SAndy Fiddaman# a default configuration which allows a customer to plug a T4 adapter in and 13*618f2068SAndy Fiddaman# have it work regardless of OS, driver or application except in the most 14*618f2068SAndy Fiddaman# unusual and/or demanding customer applications. 15*618f2068SAndy Fiddaman# 16*618f2068SAndy Fiddaman# Many of the T4 resources which are described by this configuration are 17*618f2068SAndy Fiddaman# finite. This requires balancing the configuration/operation needs of 18*618f2068SAndy Fiddaman# device drivers across OSes and a large number of customer application. 19*618f2068SAndy Fiddaman# 20*618f2068SAndy Fiddaman# Some of the more important resources to allocate and their constaints are: 21*618f2068SAndy Fiddaman# 1. Virtual Interfaces: 128. 22*618f2068SAndy Fiddaman# 2. Ingress Queues with Free Lists: 1024. PCI-E SR-IOV Virtual Functions 23*618f2068SAndy Fiddaman# must use a power of 2 Ingress Queues. 24*618f2068SAndy Fiddaman# 3. Egress Queues: 128K. PCI-E SR-IOV Virtual Functions must use a 25*618f2068SAndy Fiddaman# power of 2 Egress Queues. 26*618f2068SAndy Fiddaman# 4. MSI-X Vectors: 1088. A complication here is that the PCI-E SR-IOV 27*618f2068SAndy Fiddaman# Virtual Functions based off of a Physical Function all get the 28*618f2068SAndy Fiddaman# same umber of MSI-X Vectors as the base Physical Function. 29*618f2068SAndy Fiddaman# Additionally, regardless of whether Virtual Functions are enabled or 30*618f2068SAndy Fiddaman# not, their MSI-X "needs" are counted by the PCI-E implementation. 31*618f2068SAndy Fiddaman# And finally, all Physical Funcations capable of supporting Virtual 32*618f2068SAndy Fiddaman# Functions (PF0-3) must have the same number of configured TotalVFs in 33*618f2068SAndy Fiddaman# their SR-IOV Capabilities. 34*618f2068SAndy Fiddaman# 5. Multi-Port Support (MPS) TCAM: 336 entries to support MAC destination 35*618f2068SAndy Fiddaman# address matching on Ingress Packets. 36*618f2068SAndy Fiddaman# 37*618f2068SAndy Fiddaman# Some of the important OS/Driver resource needs are: 38*618f2068SAndy Fiddaman# 6. Some OS Drivers will manage all resources through a single Physical 39*618f2068SAndy Fiddaman# Function (currently PF0 but it could be any Physical Function). Thus, 40*618f2068SAndy Fiddaman# this "Unified PF" will need to have enough resources allocated to it 41*618f2068SAndy Fiddaman# to allow for this. And because of the MSI-X resource allocation 42*618f2068SAndy Fiddaman# constraints mentioned above, this probably means we'll either have to 43*618f2068SAndy Fiddaman# severely limit the TotalVFs if we continue to use PF0 as the Unified PF 44*618f2068SAndy Fiddaman# or we'll need to move the Unified PF into the PF4-7 range since those 45*618f2068SAndy Fiddaman# Physical Functions don't have any Virtual Functions associated with 46*618f2068SAndy Fiddaman# them. 47*618f2068SAndy Fiddaman# 7. Some OS Drivers will manage different ports and functions (NIC, 48*618f2068SAndy Fiddaman# storage, etc.) on different Physical Functions. For example, NIC 49*618f2068SAndy Fiddaman# functions for ports 0-3 on PF0-3, FCoE on PF4, iSCSI on PF5, etc. 50*618f2068SAndy Fiddaman# 51*618f2068SAndy Fiddaman# Some of the customer application needs which need to be accommodated: 52*618f2068SAndy Fiddaman# 8. Some customers will want to support large CPU count systems with 53*618f2068SAndy Fiddaman# good scaling. Thus, we'll need to accommodate a number of 54*618f2068SAndy Fiddaman# Ingress Queues and MSI-X Vectors to allow up to some number of CPUs 55*618f2068SAndy Fiddaman# to be involved per port and per application function. For example, 56*618f2068SAndy Fiddaman# in the case where all ports and application functions will be 57*618f2068SAndy Fiddaman# managed via a single Unified PF and we want to accommodate scaling up 58*618f2068SAndy Fiddaman# to 8 CPUs, we would want: 59*618f2068SAndy Fiddaman# 60*618f2068SAndy Fiddaman# 4 ports * 61*618f2068SAndy Fiddaman# 3 application functions (NIC, FCoE, iSCSI) per port * 62*618f2068SAndy Fiddaman# 8 Ingress Queue/MSI-X Vectors per application function 63*618f2068SAndy Fiddaman# 64*618f2068SAndy Fiddaman# for a total of 96 Ingress Queues and MSI-X Vectors on the Unified PF. 65*618f2068SAndy Fiddaman# (Plus a few for Firmware Event Queues, etc.) 66*618f2068SAndy Fiddaman# 67*618f2068SAndy Fiddaman# 9. Some customers will want to use T4's PCI-E SR-IOV Capability to allow 68*618f2068SAndy Fiddaman# Virtual Machines to directly access T4 functionality via SR-IOV 69*618f2068SAndy Fiddaman# Virtual Functions and "PCI Device Passthrough" -- this is especially 70*618f2068SAndy Fiddaman# true for the NIC application functionality. (Note that there is 71*618f2068SAndy Fiddaman# currently no ability to use the TOE, FCoE, iSCSI, etc. via Virtual 72*618f2068SAndy Fiddaman# Functions so this is in fact solely limited to NIC.) 73*618f2068SAndy Fiddaman# 74*618f2068SAndy Fiddaman 75*618f2068SAndy Fiddaman 76*618f2068SAndy Fiddaman# Global configuration settings. 77*618f2068SAndy Fiddaman# 78*618f2068SAndy Fiddaman[global] 79*618f2068SAndy Fiddaman rss_glb_config_mode = basicvirtual 80*618f2068SAndy Fiddaman rss_glb_config_options = tnlmapen,hashtoeplitz,tnlalllkp 81*618f2068SAndy Fiddaman 82*618f2068SAndy Fiddaman # The following Scatter Gather Engine (SGE) settings assume a 4KB Host 83*618f2068SAndy Fiddaman # Page Size and a 64B L1 Cache Line Size. It programs the 84*618f2068SAndy Fiddaman # EgrStatusPageSize and IngPadBoundary to 64B and the PktShift to 2. 85*618f2068SAndy Fiddaman # If a Master PF Driver finds itself on a machine with different 86*618f2068SAndy Fiddaman # parameters, then the Master PF Driver is responsible for initializing 87*618f2068SAndy Fiddaman # these parameters to appropriate values. 88*618f2068SAndy Fiddaman # 89*618f2068SAndy Fiddaman # Notes: 90*618f2068SAndy Fiddaman # 1. The Free List Buffer Sizes below are raw and the firmware will 91*618f2068SAndy Fiddaman # round them up to the Ingress Padding Boundary. 92*618f2068SAndy Fiddaman # 2. The SGE Timer Values below are expressed below in microseconds. 93*618f2068SAndy Fiddaman # The firmware will convert these values to Core Clock Ticks when 94*618f2068SAndy Fiddaman # it processes the configuration parameters. 95*618f2068SAndy Fiddaman # 96*618f2068SAndy Fiddaman reg[0x1008] = 0x40810/0x21c70 # SGE_CONTROL 97*618f2068SAndy Fiddaman reg[0x100c] = 0x22222222 # SGE_HOST_PAGE_SIZE 98*618f2068SAndy Fiddaman reg[0x10a0] = 0x01040810 # SGE_INGRESS_RX_THRESHOLD 99*618f2068SAndy Fiddaman reg[0x1044] = 4096 # SGE_FL_BUFFER_SIZE0 100*618f2068SAndy Fiddaman reg[0x1048] = 65536 # SGE_FL_BUFFER_SIZE1 101*618f2068SAndy Fiddaman reg[0x104c] = 1536 # SGE_FL_BUFFER_SIZE2 102*618f2068SAndy Fiddaman reg[0x1050] = 9024 # SGE_FL_BUFFER_SIZE3 103*618f2068SAndy Fiddaman reg[0x1054] = 9216 # SGE_FL_BUFFER_SIZE4 104*618f2068SAndy Fiddaman reg[0x1058] = 2048 # SGE_FL_BUFFER_SIZE5 105*618f2068SAndy Fiddaman reg[0x105c] = 128 # SGE_FL_BUFFER_SIZE6 106*618f2068SAndy Fiddaman reg[0x1060] = 8192 # SGE_FL_BUFFER_SIZE7 107*618f2068SAndy Fiddaman reg[0x1064] = 16384 # SGE_FL_BUFFER_SIZE8 108*618f2068SAndy Fiddaman reg[0x10a4] = 0xa000a000/0xf000f000 # SGE_DBFIFO_STATUS 109*618f2068SAndy Fiddaman reg[0x10a8] = 0x2000/0x2000 # SGE_DOORBELL_CONTROL 110*618f2068SAndy Fiddaman sge_timer_value = 5, 10, 20, 50, 100, 200 # SGE_TIMER_VALUE* in usecs 111*618f2068SAndy Fiddaman 112*618f2068SAndy Fiddaman # enable TP_OUT_CONFIG.IPIDSPLITMODE 113*618f2068SAndy Fiddaman reg[0x7d04] = 0x00010000/0x00010000 114*618f2068SAndy Fiddaman 115*618f2068SAndy Fiddaman # disable TP_PARA_REG3.RxFragEn 116*618f2068SAndy Fiddaman reg[0x7d6c] = 0x00000000/0x00007000 117*618f2068SAndy Fiddaman 118*618f2068SAndy Fiddaman reg[0x7dc0] = 0x0e2f8849 # TP_SHIFT_CNT 119*618f2068SAndy Fiddaman 120*618f2068SAndy Fiddaman # TP_VLAN_PRI_MAP to select filter tuples 121*618f2068SAndy Fiddaman # filter tuples : fragmentation, mpshittype, macmatch, ethertype, 122*618f2068SAndy Fiddaman # protocol, tos, vlan, vnic_id, port, fcoe 123*618f2068SAndy Fiddaman # valid filterModes are described the Terminator 4 Data Book 124*618f2068SAndy Fiddaman filterMode = fragmentation, mpshittype, protocol, vlan, port, fcoe 125*618f2068SAndy Fiddaman 126*618f2068SAndy Fiddaman # filter tuples enforced in LE active region (equal to or subset of filterMode) 127*618f2068SAndy Fiddaman filterMask = protocol, fcoe 128*618f2068SAndy Fiddaman 129*618f2068SAndy Fiddaman # Percentage of dynamic memory (in either the EDRAM or external MEM) 130*618f2068SAndy Fiddaman # to use for TP RX payload 131*618f2068SAndy Fiddaman tp_pmrx = 34 132*618f2068SAndy Fiddaman 133*618f2068SAndy Fiddaman # TP RX payload page size 134*618f2068SAndy Fiddaman tp_pmrx_pagesize = 64K 135*618f2068SAndy Fiddaman 136*618f2068SAndy Fiddaman # TP number of RX channels 137*618f2068SAndy Fiddaman tp_nrxch = 0 # 0 (auto) = 1 138*618f2068SAndy Fiddaman 139*618f2068SAndy Fiddaman # Percentage of dynamic memory (in either the EDRAM or external MEM) 140*618f2068SAndy Fiddaman # to use for TP TX payload 141*618f2068SAndy Fiddaman tp_pmtx = 32 142*618f2068SAndy Fiddaman 143*618f2068SAndy Fiddaman # TP TX payload page size 144*618f2068SAndy Fiddaman tp_pmtx_pagesize = 64K 145*618f2068SAndy Fiddaman 146*618f2068SAndy Fiddaman # TP number of TX channels 147*618f2068SAndy Fiddaman tp_ntxch = 0 # 0 (auto) = equal number of ports 148*618f2068SAndy Fiddaman 149*618f2068SAndy Fiddaman # TP OFLD MTUs 150*618f2068SAndy Fiddaman tp_mtus = 88, 256, 512, 576, 808, 1024, 1280, 1488, 1500, 2002, 2048, 4096, 4352, 8192, 9000, 9600 151*618f2068SAndy Fiddaman 152*618f2068SAndy Fiddaman # ULPRX iSCSI Page Sizes 153*618f2068SAndy Fiddaman reg[0x19168] = 0x04020100 # 64K, 16K, 8K and 4K 154*618f2068SAndy Fiddaman 155*618f2068SAndy Fiddaman# Some "definitions" to make the rest of this a bit more readable. We support 156*618f2068SAndy Fiddaman# 4 ports, 3 functions (NIC, FCoE and iSCSI), scaling up to 8 "CPU Queue Sets" 157*618f2068SAndy Fiddaman# per function per port ... 158*618f2068SAndy Fiddaman# 159*618f2068SAndy Fiddaman# NMSIX = 1088 # available MSI-X Vectors 160*618f2068SAndy Fiddaman# NVI = 128 # available Virtual Interfaces 161*618f2068SAndy Fiddaman# NMPSTCAM = 336 # MPS TCAM entries 162*618f2068SAndy Fiddaman# 163*618f2068SAndy Fiddaman# NPORTS = 4 # ports 164*618f2068SAndy Fiddaman# NCPUS = 8 # CPUs we want to support scalably 165*618f2068SAndy Fiddaman# NFUNCS = 3 # functions per port (NIC, FCoE, iSCSI) 166*618f2068SAndy Fiddaman 167*618f2068SAndy Fiddaman# Breakdown of Virtual Interface/Queue/Interrupt resources for the "Unified 168*618f2068SAndy Fiddaman# PF" which many OS Drivers will use to manage most or all functions. 169*618f2068SAndy Fiddaman# 170*618f2068SAndy Fiddaman# Each Ingress Queue can use one MSI-X interrupt but some Ingress Queues can 171*618f2068SAndy Fiddaman# use Forwarded Interrupt Ingress Queues. For these latter, an Ingress Queue 172*618f2068SAndy Fiddaman# would be created and the Queue ID of a Forwarded Interrupt Ingress Queue 173*618f2068SAndy Fiddaman# will be specified as the "Ingress Queue Asynchronous Destination Index." 174*618f2068SAndy Fiddaman# Thus, the number of MSI-X Vectors assigned to the Unified PF will be less 175*618f2068SAndy Fiddaman# than or equal to the number of Ingress Queues ... 176*618f2068SAndy Fiddaman# 177*618f2068SAndy Fiddaman# NVI_NIC = 4 # NIC access to NPORTS 178*618f2068SAndy Fiddaman# NFLIQ_NIC = 32 # NIC Ingress Queues with Free Lists 179*618f2068SAndy Fiddaman# NETHCTRL_NIC = 32 # NIC Ethernet Control/TX Queues 180*618f2068SAndy Fiddaman# NEQ_NIC = 64 # NIC Egress Queues (FL, ETHCTRL/TX) 181*618f2068SAndy Fiddaman# NMPSTCAM_NIC = 16 # NIC MPS TCAM Entries (NPORTS*4) 182*618f2068SAndy Fiddaman# NMSIX_NIC = 32 # NIC MSI-X Interrupt Vectors (FLIQ) 183*618f2068SAndy Fiddaman# 184*618f2068SAndy Fiddaman# NVI_OFLD = 0 # Offload uses NIC function to access ports 185*618f2068SAndy Fiddaman# NFLIQ_OFLD = 16 # Offload Ingress Queues with Free Lists 186*618f2068SAndy Fiddaman# NETHCTRL_OFLD = 0 # Offload Ethernet Control/TX Queues 187*618f2068SAndy Fiddaman# NEQ_OFLD = 16 # Offload Egress Queues (FL) 188*618f2068SAndy Fiddaman# NMPSTCAM_OFLD = 0 # Offload MPS TCAM Entries (uses NIC's) 189*618f2068SAndy Fiddaman# NMSIX_OFLD = 16 # Offload MSI-X Interrupt Vectors (FLIQ) 190*618f2068SAndy Fiddaman# 191*618f2068SAndy Fiddaman# NVI_RDMA = 0 # RDMA uses NIC function to access ports 192*618f2068SAndy Fiddaman# NFLIQ_RDMA = 4 # RDMA Ingress Queues with Free Lists 193*618f2068SAndy Fiddaman# NETHCTRL_RDMA = 0 # RDMA Ethernet Control/TX Queues 194*618f2068SAndy Fiddaman# NEQ_RDMA = 4 # RDMA Egress Queues (FL) 195*618f2068SAndy Fiddaman# NMPSTCAM_RDMA = 0 # RDMA MPS TCAM Entries (uses NIC's) 196*618f2068SAndy Fiddaman# NMSIX_RDMA = 4 # RDMA MSI-X Interrupt Vectors (FLIQ) 197*618f2068SAndy Fiddaman# 198*618f2068SAndy Fiddaman# NEQ_WD = 128 # Wire Direct TX Queues and FLs 199*618f2068SAndy Fiddaman# NETHCTRL_WD = 64 # Wire Direct TX Queues 200*618f2068SAndy Fiddaman# NFLIQ_WD = 64 ` # Wire Direct Ingress Queues with Free Lists 201*618f2068SAndy Fiddaman# 202*618f2068SAndy Fiddaman# NVI_ISCSI = 4 # ISCSI access to NPORTS 203*618f2068SAndy Fiddaman# NFLIQ_ISCSI = 4 # ISCSI Ingress Queues with Free Lists 204*618f2068SAndy Fiddaman# NETHCTRL_ISCSI = 0 # ISCSI Ethernet Control/TX Queues 205*618f2068SAndy Fiddaman# NEQ_ISCSI = 4 # ISCSI Egress Queues (FL) 206*618f2068SAndy Fiddaman# NMPSTCAM_ISCSI = 4 # ISCSI MPS TCAM Entries (NPORTS) 207*618f2068SAndy Fiddaman# NMSIX_ISCSI = 4 # ISCSI MSI-X Interrupt Vectors (FLIQ) 208*618f2068SAndy Fiddaman# 209*618f2068SAndy Fiddaman# NVI_FCOE = 4 # FCOE access to NPORTS 210*618f2068SAndy Fiddaman# NFLIQ_FCOE = 34 # FCOE Ingress Queues with Free Lists 211*618f2068SAndy Fiddaman# NETHCTRL_FCOE = 32 # FCOE Ethernet Control/TX Queues 212*618f2068SAndy Fiddaman# NEQ_FCOE = 66 # FCOE Egress Queues (FL) 213*618f2068SAndy Fiddaman# NMPSTCAM_FCOE = 32 # FCOE MPS TCAM Entries (NPORTS) 214*618f2068SAndy Fiddaman# NMSIX_FCOE = 34 # FCOE MSI-X Interrupt Vectors (FLIQ) 215*618f2068SAndy Fiddaman 216*618f2068SAndy Fiddaman# Two extra Ingress Queues per function for Firmware Events and Forwarded 217*618f2068SAndy Fiddaman# Interrupts, and two extra interrupts per function for Firmware Events (or a 218*618f2068SAndy Fiddaman# Forwarded Interrupt Queue) and General Interrupts per function. 219*618f2068SAndy Fiddaman# 220*618f2068SAndy Fiddaman# NFLIQ_EXTRA = 6 # "extra" Ingress Queues 2*NFUNCS (Firmware and 221*618f2068SAndy Fiddaman# # Forwarded Interrupts 222*618f2068SAndy Fiddaman# NMSIX_EXTRA = 6 # extra interrupts 2*NFUNCS (Firmware and 223*618f2068SAndy Fiddaman# # General Interrupts 224*618f2068SAndy Fiddaman 225*618f2068SAndy Fiddaman# Microsoft HyperV resources. The HyperV Virtual Ingress Queues will have 226*618f2068SAndy Fiddaman# their interrupts forwarded to another set of Forwarded Interrupt Queues. 227*618f2068SAndy Fiddaman# 228*618f2068SAndy Fiddaman# NVI_HYPERV = 16 # VMs we want to support 229*618f2068SAndy Fiddaman# NVIIQ_HYPERV = 2 # Virtual Ingress Queues with Free Lists per VM 230*618f2068SAndy Fiddaman# NFLIQ_HYPERV = 40 # VIQs + NCPUS Forwarded Interrupt Queues 231*618f2068SAndy Fiddaman# NEQ_HYPERV = 32 # VIQs Free Lists 232*618f2068SAndy Fiddaman# NMPSTCAM_HYPERV = 16 # MPS TCAM Entries (NVI_HYPERV) 233*618f2068SAndy Fiddaman# NMSIX_HYPERV = 8 # NCPUS Forwarded Interrupt Queues 234*618f2068SAndy Fiddaman 235*618f2068SAndy Fiddaman# Adding all of the above Unified PF resource needs together: (NIC + OFLD + 236*618f2068SAndy Fiddaman# RDMA + ISCSI + FCOE + EXTRA + HYPERV) 237*618f2068SAndy Fiddaman# 238*618f2068SAndy Fiddaman# NVI_UNIFIED = 28 239*618f2068SAndy Fiddaman# NFLIQ_UNIFIED = 106 240*618f2068SAndy Fiddaman# NETHCTRL_UNIFIED = 32 241*618f2068SAndy Fiddaman# NEQ_UNIFIED = 124 242*618f2068SAndy Fiddaman# NMPSTCAM_UNIFIED = 40 243*618f2068SAndy Fiddaman# 244*618f2068SAndy Fiddaman# The sum of all the MSI-X resources above is 74 MSI-X Vectors but we'll round 245*618f2068SAndy Fiddaman# that up to 128 to make sure the Unified PF doesn't run out of resources. 246*618f2068SAndy Fiddaman# 247*618f2068SAndy Fiddaman# NMSIX_UNIFIED = 128 248*618f2068SAndy Fiddaman# 249*618f2068SAndy Fiddaman# The Storage PFs could need up to NPORTS*NCPUS + NMSIX_EXTRA MSI-X Vectors 250*618f2068SAndy Fiddaman# which is 34 but they're probably safe with 32. 251*618f2068SAndy Fiddaman# 252*618f2068SAndy Fiddaman# NMSIX_STORAGE = 32 253*618f2068SAndy Fiddaman 254*618f2068SAndy Fiddaman# Note: The UnifiedPF is PF4 which doesn't have any Virtual Functions 255*618f2068SAndy Fiddaman# associated with it. Thus, the MSI-X Vector allocations we give to the 256*618f2068SAndy Fiddaman# UnifiedPF aren't inherited by any Virtual Functions. As a result we can 257*618f2068SAndy Fiddaman# provision many more Virtual Functions than we can if the UnifiedPF were 258*618f2068SAndy Fiddaman# one of PF0-3. 259*618f2068SAndy Fiddaman# 260*618f2068SAndy Fiddaman 261*618f2068SAndy Fiddaman# All of the below PCI-E parameters are actually stored in various *_init.txt 262*618f2068SAndy Fiddaman# files. We include them below essentially as comments. 263*618f2068SAndy Fiddaman# 264*618f2068SAndy Fiddaman# For PF0-3 we assign 8 vectors each for NIC Ingress Queues of the associated 265*618f2068SAndy Fiddaman# ports 0-3. 266*618f2068SAndy Fiddaman# 267*618f2068SAndy Fiddaman# For PF4, the Unified PF, we give it an MSI-X Table Size as outlined above. 268*618f2068SAndy Fiddaman# 269*618f2068SAndy Fiddaman# For PF5-6 we assign enough MSI-X Vectors to support FCoE and iSCSI 270*618f2068SAndy Fiddaman# storage applications across all four possible ports. 271*618f2068SAndy Fiddaman# 272*618f2068SAndy Fiddaman# Additionally, since the UnifiedPF isn't one of the per-port Physical 273*618f2068SAndy Fiddaman# Functions, we give the UnifiedPF and the PF0-3 Physical Functions 274*618f2068SAndy Fiddaman# different PCI Device IDs which will allow Unified and Per-Port Drivers 275*618f2068SAndy Fiddaman# to directly select the type of Physical Function to which they wish to be 276*618f2068SAndy Fiddaman# attached. 277*618f2068SAndy Fiddaman# 278*618f2068SAndy Fiddaman# Note that the actual values used for the PCI-E Intelectual Property will be 279*618f2068SAndy Fiddaman# 1 less than those below since that's the way it "counts" things. For 280*618f2068SAndy Fiddaman# readability, we use the number we actually mean ... 281*618f2068SAndy Fiddaman# 282*618f2068SAndy Fiddaman# PF0_INT = 8 # NCPUS 283*618f2068SAndy Fiddaman# PF1_INT = 8 # NCPUS 284*618f2068SAndy Fiddaman# PF2_INT = 8 # NCPUS 285*618f2068SAndy Fiddaman# PF3_INT = 8 # NCPUS 286*618f2068SAndy Fiddaman# PF0_3_INT = 32 # PF0_INT + PF1_INT + PF2_INT + PF3_INT 287*618f2068SAndy Fiddaman# 288*618f2068SAndy Fiddaman# PF4_INT = 128 # NMSIX_UNIFIED 289*618f2068SAndy Fiddaman# PF5_INT = 32 # NMSIX_STORAGE 290*618f2068SAndy Fiddaman# PF6_INT = 32 # NMSIX_STORAGE 291*618f2068SAndy Fiddaman# PF7_INT = 0 # Nothing Assigned 292*618f2068SAndy Fiddaman# PF4_7_INT = 192 # PF4_INT + PF5_INT + PF6_INT + PF7_INT 293*618f2068SAndy Fiddaman# 294*618f2068SAndy Fiddaman# PF0_7_INT = 224 # PF0_3_INT + PF4_7_INT 295*618f2068SAndy Fiddaman# 296*618f2068SAndy Fiddaman# With the above we can get 17 VFs/PF0-3 (limited by 336 MPS TCAM entries) 297*618f2068SAndy Fiddaman# but we'll lower that to 16 to make our total 64 and a nice power of 2 ... 298*618f2068SAndy Fiddaman# 299*618f2068SAndy Fiddaman# NVF = 16 300*618f2068SAndy Fiddaman 301*618f2068SAndy Fiddaman# For those OSes which manage different ports on different PFs, we need 302*618f2068SAndy Fiddaman# only enough resources to support a single port's NIC application functions 303*618f2068SAndy Fiddaman# on PF0-3. The below assumes that we're only doing NIC with NCPUS "Queue 304*618f2068SAndy Fiddaman# Sets" for ports 0-3. The FCoE and iSCSI functions for such OSes will be 305*618f2068SAndy Fiddaman# managed on the "storage PFs" (see below). 306*618f2068SAndy Fiddaman# 307*618f2068SAndy Fiddaman[function "0"] 308*618f2068SAndy Fiddaman nvf = 16 # NVF on this function 309*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 310*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 311*618f2068SAndy Fiddaman nvi = 1 # 1 port 312*618f2068SAndy Fiddaman niqflint = 8 # NCPUS "Queue Sets" 313*618f2068SAndy Fiddaman nethctrl = 8 # NCPUS "Queue Sets" 314*618f2068SAndy Fiddaman neq = 16 # niqflint + nethctrl Egress Queues 315*618f2068SAndy Fiddaman nexactf = 8 # number of exact MPSTCAM MAC filters 316*618f2068SAndy Fiddaman cmask = all # access to all channels 317*618f2068SAndy Fiddaman pmask = 0x1 # access to only one port 318*618f2068SAndy Fiddaman 319*618f2068SAndy Fiddaman[function "1"] 320*618f2068SAndy Fiddaman nvf = 16 # NVF on this function 321*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 322*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 323*618f2068SAndy Fiddaman nvi = 1 # 1 port 324*618f2068SAndy Fiddaman niqflint = 8 # NCPUS "Queue Sets" 325*618f2068SAndy Fiddaman nethctrl = 8 # NCPUS "Queue Sets" 326*618f2068SAndy Fiddaman neq = 16 # niqflint + nethctrl Egress Queues 327*618f2068SAndy Fiddaman nexactf = 8 # number of exact MPSTCAM MAC filters 328*618f2068SAndy Fiddaman cmask = all # access to all channels 329*618f2068SAndy Fiddaman pmask = 0x2 # access to only one port 330*618f2068SAndy Fiddaman 331*618f2068SAndy Fiddaman[function "2"] 332*618f2068SAndy Fiddaman nvf = 16 # NVF on this function 333*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 334*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 335*618f2068SAndy Fiddaman nvi = 1 # 1 port 336*618f2068SAndy Fiddaman niqflint = 8 # NCPUS "Queue Sets" 337*618f2068SAndy Fiddaman nethctrl = 8 # NCPUS "Queue Sets" 338*618f2068SAndy Fiddaman neq = 16 # niqflint + nethctrl Egress Queues 339*618f2068SAndy Fiddaman nexactf = 8 # number of exact MPSTCAM MAC filters 340*618f2068SAndy Fiddaman cmask = all # access to all channels 341*618f2068SAndy Fiddaman pmask = 0x4 # access to only one port 342*618f2068SAndy Fiddaman 343*618f2068SAndy Fiddaman[function "3"] 344*618f2068SAndy Fiddaman nvf = 16 # NVF on this function 345*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 346*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 347*618f2068SAndy Fiddaman nvi = 1 # 1 port 348*618f2068SAndy Fiddaman niqflint = 8 # NCPUS "Queue Sets" 349*618f2068SAndy Fiddaman nethctrl = 8 # NCPUS "Queue Sets" 350*618f2068SAndy Fiddaman neq = 16 # niqflint + nethctrl Egress Queues 351*618f2068SAndy Fiddaman nexactf = 8 # number of exact MPSTCAM MAC filters 352*618f2068SAndy Fiddaman cmask = all # access to all channels 353*618f2068SAndy Fiddaman pmask = 0x8 # access to only one port 354*618f2068SAndy Fiddaman 355*618f2068SAndy Fiddaman# Some OS Drivers manage all application functions for all ports via PF4. 356*618f2068SAndy Fiddaman# Thus we need to provide a large number of resources here. For Egress 357*618f2068SAndy Fiddaman# Queues we need to account for both TX Queues as well as Free List Queues 358*618f2068SAndy Fiddaman# (because the host is responsible for producing Free List Buffers for the 359*618f2068SAndy Fiddaman# hardware to consume). 360*618f2068SAndy Fiddaman# 361*618f2068SAndy Fiddaman[function "4"] 362*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 363*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 364*618f2068SAndy Fiddaman nvi = 28 # NVI_UNIFIED 365*618f2068SAndy Fiddaman niqflint = 170 # NFLIQ_UNIFIED + NLFIQ_WD 366*618f2068SAndy Fiddaman nethctrl = 100 # NETHCTRL_UNIFIED + NETHCTRL_WD 367*618f2068SAndy Fiddaman neq = 256 # NEQ_UNIFIED + NEQ_WD 368*618f2068SAndy Fiddaman nexactf = 40 # NMPSTCAM_UNIFIED 369*618f2068SAndy Fiddaman cmask = all # access to all channels 370*618f2068SAndy Fiddaman pmask = all # access to all four ports ... 371*618f2068SAndy Fiddaman nethofld = 1024 # number of user mode ethernet flow contexts 372*618f2068SAndy Fiddaman nroute = 32 # number of routing region entries 373*618f2068SAndy Fiddaman nclip = 32 # number of clip region entries 374*618f2068SAndy Fiddaman nfilter = 496 # number of filter region entries 375*618f2068SAndy Fiddaman nserver = 496 # number of server region entries 376*618f2068SAndy Fiddaman nhash = 12288 # number of hash region entries 377*618f2068SAndy Fiddaman protocol = nic_vm, ofld, rddp, rdmac, iscsi_initiator_pdu, iscsi_target_pdu 378*618f2068SAndy Fiddaman tp_l2t = 3072 379*618f2068SAndy Fiddaman tp_ddp = 3 380*618f2068SAndy Fiddaman tp_ddp_iscsi = 2 381*618f2068SAndy Fiddaman tp_stag = 3 382*618f2068SAndy Fiddaman tp_pbl = 10 383*618f2068SAndy Fiddaman tp_rq = 13 384*618f2068SAndy Fiddaman 385*618f2068SAndy Fiddaman# We have FCoE and iSCSI storage functions on PF5 and PF6 each of which may 386*618f2068SAndy Fiddaman# need to have Virtual Interfaces on each of the four ports with up to NCPUS 387*618f2068SAndy Fiddaman# "Queue Sets" each. 388*618f2068SAndy Fiddaman# 389*618f2068SAndy Fiddaman[function "5"] 390*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 391*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 392*618f2068SAndy Fiddaman nvi = 4 # NPORTS 393*618f2068SAndy Fiddaman niqflint = 34 # NPORTS*NCPUS + NMSIX_EXTRA 394*618f2068SAndy Fiddaman nethctrl = 32 # NPORTS*NCPUS 395*618f2068SAndy Fiddaman neq = 64 # NPORTS*NCPUS * 2 (FL, ETHCTRL/TX) 396*618f2068SAndy Fiddaman nexactf = 4 # NPORTS 397*618f2068SAndy Fiddaman cmask = all # access to all channels 398*618f2068SAndy Fiddaman pmask = all # access to all four ports ... 399*618f2068SAndy Fiddaman nserver = 16 400*618f2068SAndy Fiddaman nhash = 2048 401*618f2068SAndy Fiddaman tp_l2t = 1020 402*618f2068SAndy Fiddaman protocol = iscsi_initiator_fofld 403*618f2068SAndy Fiddaman tp_ddp_iscsi = 2 404*618f2068SAndy Fiddaman iscsi_ntask = 2048 405*618f2068SAndy Fiddaman iscsi_nsess = 2048 406*618f2068SAndy Fiddaman iscsi_nconn_per_session = 1 407*618f2068SAndy Fiddaman iscsi_ninitiator_instance = 64 408*618f2068SAndy Fiddaman 409*618f2068SAndy Fiddaman[function "6"] 410*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 411*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 412*618f2068SAndy Fiddaman nvi = 4 # NPORTS 413*618f2068SAndy Fiddaman niqflint = 34 # NPORTS*NCPUS + NMSIX_EXTRA 414*618f2068SAndy Fiddaman nethctrl = 32 # NPORTS*NCPUS 415*618f2068SAndy Fiddaman neq = 66 # NPORTS*NCPUS * 2 (FL, ETHCTRL/TX) + 2 (EXTRA) 416*618f2068SAndy Fiddaman nexactf = 32 # NPORTS + adding 28 exact entries for FCoE 417*618f2068SAndy Fiddaman # which is OK since < MIN(SUM PF0..3, PF4) 418*618f2068SAndy Fiddaman # and we never load PF0..3 and PF4 concurrently 419*618f2068SAndy Fiddaman cmask = all # access to all channels 420*618f2068SAndy Fiddaman pmask = all # access to all four ports ... 421*618f2068SAndy Fiddaman nhash = 2048 422*618f2068SAndy Fiddaman tp_l2t = 4 423*618f2068SAndy Fiddaman protocol = fcoe_initiator 424*618f2068SAndy Fiddaman tp_ddp = 1 425*618f2068SAndy Fiddaman fcoe_nfcf = 16 426*618f2068SAndy Fiddaman fcoe_nvnp = 32 427*618f2068SAndy Fiddaman fcoe_nssn = 1024 428*618f2068SAndy Fiddaman 429*618f2068SAndy Fiddaman# The following function, 1023, is not an actual PCIE function but is used to 430*618f2068SAndy Fiddaman# configure and reserve firmware internal resources that come from the global 431*618f2068SAndy Fiddaman# resource pool. 432*618f2068SAndy Fiddaman# 433*618f2068SAndy Fiddaman[function "1023"] 434*618f2068SAndy Fiddaman wx_caps = all # write/execute permissions for all commands 435*618f2068SAndy Fiddaman r_caps = all # read permissions for all commands 436*618f2068SAndy Fiddaman nvi = 4 # NVI_UNIFIED 437*618f2068SAndy Fiddaman cmask = all # access to all channels 438*618f2068SAndy Fiddaman pmask = all # access to all four ports ... 439*618f2068SAndy Fiddaman nexactf = 8 # NPORTS + DCBX + 440*618f2068SAndy Fiddaman nfilter = 16 # number of filter region entries 441*618f2068SAndy Fiddaman 442*618f2068SAndy Fiddaman# For Virtual functions, we only allow NIC functionality and we only allow 443*618f2068SAndy Fiddaman# access to one port (1 << PF). Note that because of limitations in the 444*618f2068SAndy Fiddaman# Scatter Gather Engine (SGE) hardware which checks writes to VF KDOORBELL 445*618f2068SAndy Fiddaman# and GTS registers, the number of Ingress and Egress Queues must be a power 446*618f2068SAndy Fiddaman# of 2. 447*618f2068SAndy Fiddaman# 448*618f2068SAndy Fiddaman[function "0/*"] # NVF 449*618f2068SAndy Fiddaman wx_caps = 0x82 # DMAQ | VF 450*618f2068SAndy Fiddaman r_caps = 0x86 # DMAQ | VF | PORT 451*618f2068SAndy Fiddaman nvi = 1 # 1 port 452*618f2068SAndy Fiddaman niqflint = 4 # 2 "Queue Sets" + NXIQ 453*618f2068SAndy Fiddaman nethctrl = 2 # 2 "Queue Sets" 454*618f2068SAndy Fiddaman neq = 4 # 2 "Queue Sets" * 2 455*618f2068SAndy Fiddaman nexactf = 4 456*618f2068SAndy Fiddaman cmask = all # access to all channels 457*618f2068SAndy Fiddaman pmask = 0x1 # access to only one port ... 458*618f2068SAndy Fiddaman 459*618f2068SAndy Fiddaman[function "1/*"] # NVF 460*618f2068SAndy Fiddaman wx_caps = 0x82 # DMAQ | VF 461*618f2068SAndy Fiddaman r_caps = 0x86 # DMAQ | VF | PORT 462*618f2068SAndy Fiddaman nvi = 1 # 1 port 463*618f2068SAndy Fiddaman niqflint = 4 # 2 "Queue Sets" + NXIQ 464*618f2068SAndy Fiddaman nethctrl = 2 # 2 "Queue Sets" 465*618f2068SAndy Fiddaman neq = 4 # 2 "Queue Sets" * 2 466*618f2068SAndy Fiddaman nexactf = 4 467*618f2068SAndy Fiddaman cmask = all # access to all channels 468*618f2068SAndy Fiddaman pmask = 0x2 # access to only one port ... 469*618f2068SAndy Fiddaman 470*618f2068SAndy Fiddaman[function "2/*"] # NVF 471*618f2068SAndy Fiddaman wx_caps = 0x82 # DMAQ | VF 472*618f2068SAndy Fiddaman r_caps = 0x86 # DMAQ | VF | PORT 473*618f2068SAndy Fiddaman nvi = 1 # 1 port 474*618f2068SAndy Fiddaman niqflint = 4 # 2 "Queue Sets" + NXIQ 475*618f2068SAndy Fiddaman nethctrl = 2 # 2 "Queue Sets" 476*618f2068SAndy Fiddaman neq = 4 # 2 "Queue Sets" * 2 477*618f2068SAndy Fiddaman nexactf = 4 478*618f2068SAndy Fiddaman cmask = all # access to all channels 479*618f2068SAndy Fiddaman pmask = 0x4 # access to only one port ... 480*618f2068SAndy Fiddaman 481*618f2068SAndy Fiddaman[function "3/*"] # NVF 482*618f2068SAndy Fiddaman wx_caps = 0x82 # DMAQ | VF 483*618f2068SAndy Fiddaman r_caps = 0x86 # DMAQ | VF | PORT 484*618f2068SAndy Fiddaman nvi = 1 # 1 port 485*618f2068SAndy Fiddaman niqflint = 4 # 2 "Queue Sets" + NXIQ 486*618f2068SAndy Fiddaman nethctrl = 2 # 2 "Queue Sets" 487*618f2068SAndy Fiddaman neq = 4 # 2 "Queue Sets" * 2 488*618f2068SAndy Fiddaman nexactf = 4 489*618f2068SAndy Fiddaman cmask = all # access to all channels 490*618f2068SAndy Fiddaman pmask = 0x8 # access to only one port ... 491*618f2068SAndy Fiddaman 492*618f2068SAndy Fiddaman# MPS features a 196608 bytes ingress buffer that is used for ingress buffering 493*618f2068SAndy Fiddaman# for packets from the wire as well as the loopback path of the L2 switch. The 494*618f2068SAndy Fiddaman# folling params control how the buffer memory is distributed and the L2 flow 495*618f2068SAndy Fiddaman# control settings: 496*618f2068SAndy Fiddaman# 497*618f2068SAndy Fiddaman# bg_mem: %-age of mem to use for port/buffer group 498*618f2068SAndy Fiddaman# lpbk_mem: %-age of port/bg mem to use for loopback 499*618f2068SAndy Fiddaman# hwm: high watermark; bytes available when starting to send pause 500*618f2068SAndy Fiddaman# frames (in units of 0.1 MTU) 501*618f2068SAndy Fiddaman# lwm: low watermark; bytes remaining when sending 'unpause' frame 502*618f2068SAndy Fiddaman# (in inuits of 0.1 MTU) 503*618f2068SAndy Fiddaman# dwm: minimum delta between high and low watermark (in units of 100 504*618f2068SAndy Fiddaman# Bytes) 505*618f2068SAndy Fiddaman# 506*618f2068SAndy Fiddaman# 507*618f2068SAndy Fiddaman 508*618f2068SAndy Fiddaman[port "0"] 509*618f2068SAndy Fiddaman dcb = ppp, dcbx # configure for DCB PPP and enable DCBX offload 510*618f2068SAndy Fiddaman bg_mem = 25 511*618f2068SAndy Fiddaman lpbk_mem = 25 512*618f2068SAndy Fiddaman hwm = 30 513*618f2068SAndy Fiddaman lwm = 15 514*618f2068SAndy Fiddaman dwm = 30 515*618f2068SAndy Fiddaman dcb_app_tlv[0] = 0x8906, ethertype, 3 516*618f2068SAndy Fiddaman dcb_app_tlv[1] = 0x8914, ethertype, 3 517*618f2068SAndy Fiddaman dcb_app_tlv[2] = 3260, socketnum, 5 518*618f2068SAndy Fiddaman 519*618f2068SAndy Fiddaman[port "1"] 520*618f2068SAndy Fiddaman dcb = ppp, dcbx 521*618f2068SAndy Fiddaman bg_mem = 25 522*618f2068SAndy Fiddaman lpbk_mem = 25 523*618f2068SAndy Fiddaman hwm = 30 524*618f2068SAndy Fiddaman lwm = 15 525*618f2068SAndy Fiddaman dwm = 30 526*618f2068SAndy Fiddaman dcb_app_tlv[0] = 0x8906, ethertype, 3 527*618f2068SAndy Fiddaman dcb_app_tlv[1] = 0x8914, ethertype, 3 528*618f2068SAndy Fiddaman dcb_app_tlv[2] = 3260, socketnum, 5 529*618f2068SAndy Fiddaman 530*618f2068SAndy Fiddaman[port "2"] 531*618f2068SAndy Fiddaman dcb = ppp, dcbx 532*618f2068SAndy Fiddaman bg_mem = 25 533*618f2068SAndy Fiddaman lpbk_mem = 25 534*618f2068SAndy Fiddaman hwm = 30 535*618f2068SAndy Fiddaman lwm = 15 536*618f2068SAndy Fiddaman dwm = 30 537*618f2068SAndy Fiddaman dcb_app_tlv[0] = 0x8906, ethertype, 3 538*618f2068SAndy Fiddaman dcb_app_tlv[1] = 0x8914, ethertype, 3 539*618f2068SAndy Fiddaman dcb_app_tlv[2] = 3260, socketnum, 5 540*618f2068SAndy Fiddaman 541*618f2068SAndy Fiddaman[port "3"] 542*618f2068SAndy Fiddaman dcb = ppp, dcbx 543*618f2068SAndy Fiddaman bg_mem = 25 544*618f2068SAndy Fiddaman lpbk_mem = 25 545*618f2068SAndy Fiddaman hwm = 30 546*618f2068SAndy Fiddaman lwm = 15 547*618f2068SAndy Fiddaman dwm = 30 548*618f2068SAndy Fiddaman dcb_app_tlv[0] = 0x8906, ethertype, 3 549*618f2068SAndy Fiddaman dcb_app_tlv[1] = 0x8914, ethertype, 3 550*618f2068SAndy Fiddaman dcb_app_tlv[2] = 3260, socketnum, 5 551*618f2068SAndy Fiddaman 552*618f2068SAndy Fiddaman[fini] 553*618f2068SAndy Fiddaman version = 0x1425001c 554*618f2068SAndy Fiddaman checksum = 0x5ceab41e 555*618f2068SAndy Fiddaman 556*618f2068SAndy Fiddaman# Total resources used by above allocations: 557*618f2068SAndy Fiddaman# Virtual Interfaces: 104 558*618f2068SAndy Fiddaman# Ingress Queues/w Free Lists and Interrupts: 526 559*618f2068SAndy Fiddaman# Egress Queues: 702 560*618f2068SAndy Fiddaman# MPS TCAM Entries: 336 561*618f2068SAndy Fiddaman# MSI-X Vectors: 736 562*618f2068SAndy Fiddaman# Virtual Functions: 64 563