*******************************************************************************
***************** Mellanox Performance Monitoring Counters ********************
*******************************************************************************
The performance data of the BlueField hardware is collected using two
mechanisms:
 1. Programming hardware counters to monitor specific events
 2. Reading registers that hold performance/event statistics

The performance modules in BlueField are present in several hardware blocks and
each block has a certain set of supported events. The hardware blocks that
include a performance module are:
 * Tile (block containing 2 cores and a shared L2 cache)
 * TRIO (PCIe root complex)
 * MSS (Memory Sub-system containing the Memory Controller and L3 cache)

The number of tiles depends on the system. There are a maximum of 8 Tiles,
3 TRIOs and 2 MSS blocks in BlueField.
The performance module in each Tile has 4 hardware counters that can be
simultaneously programmed to monitor specific events. There are 2 performance
modules in each TRIO and are referred to by the driver as "trio" and "triogen"
performance modules. The TRIO performance module (trio) has 4 counters while
the generic module (triogen) has only 1.
The MSS blocks have a generic performance module (mss) with 4 hardware
counters, an l3cache block which is split between the 2 MSSs, and each of these
is further divided into 2 halves (and there are 2 banks per half) where each
half has 5 performance counters (called l3cacheXhalf0 and l3cacheXhalf1) and
an "ecc" block that exposes a set of registers that hold ECC statistics.
The PCIE TLR statistics for each trio are under the "pcie" block.

The mlx_pmc driver provides access to all of these performance modules through
a sysfs interface. The driver creates a hwmon0 directory under /sys/class/hwmon
under which each of the blocks explained above has a sub-directory. Please note
that all directories under /sys/class/hwmon are named as "hwmonN" where N is
the hwmon device number corresponding to the device. This is assigned by linux
and could change with the addition of more devices to the hwmon class. Each
hwmon directory has a "name" node which can be used to identify the correct
device. In this case, reading the "name" file should return "bfperf".
For blocks that use hardware counters (mechanism 1) to collect this info, each
counter present in the block is represented by "eventN" and "counterN" sysfs
files. An eventN and counterN pair can be used to program and monitor events.
There is also an "event_list" sysfs file that displays the list of events
supported by that block along with the event numbers.

Programming a counter to monitor an event:
-----------------------------------------
Let's call the /sys/class/hwmon/hwmonN folder corresponding to this driver as
"BFPERF_DIR".
In order to program a counter to monitor one of the events from the event list,
the event name or number needs to be written to the corresponding event file.
For example, in order to monitor the event HNF_REQUESTS (0x45) on tile2 using
counter 3,
  echo 0x45 > <BFPERF_DIR>/tile2/event3
  (or)
  echo HNF_REQUESTS > <BFPERF_DIR>/tile2/event3
Once this is done, counter3 will reset the counter and start monitoring the
number of HNF_REQUESTS.
To read the counter value, just do:
  cat <BFPERF_DIR>/tile2/counter3

In order to see what event is currently being monitored by a counter, just read the corresponding event file to get the event name and number.
  cat <BFPERF_DIR>/tile2/event3
In this case, reading the event3 file will return "0x45: HNF_REQUESTS".

In order to clear the counter, write 0 to the counter file.
  echo 0 > <BFPERF_DIR>/tile2/counter3
This will reset the accumulator and the counter will continue monitoring the
same event that was previously programmed, but will start counting from 0
again. Writing non-zero values to the counter files is not allowed.
To stop monitoring an event, write 0xff to the corresponding event file.

This is slightly different for the l3cache blocks due to the restriction that
all counters can only be enabled, disabled or reset together. So once the
event is written to the event file, the counters will have to be enabled to
start monitoring their respective events by writing "1" to the "enable" file.
And writing "0" to this file will stop all the counters. The most reliable way
to get accurate counter values would be by disabling the counters after a
certain time period and then proceed to read the counter values. Please note
that programming a counter to monitor a new event will automatically stop all
the counters. Also, enabling the counters resets the counters to 0 first.

For blocks that have performance statistics registers (mechanism 2), all
of these statistics are directly made available to be read or reset.
For example, in order to read the number of incoming posted packets to TRIO2,
  cat <BFPERF_DIR>/pcie2/IN_P_PKT_CNT
The count can be reset to 0 by writing 0 to the same file. Again, non-zero
writes to these files are not allowed.

*******************************************************************************
List of supported events:
*******************************************************************************

TRIO generic performance module:
-------------------------------
Value	Name		Meaning

0x0	AW_REQ		The number of axi4 write requests
0x1	AW_BEATS	The number of axi4 write beats
0x2	AW_TRANS	The number of axi4 write transactions
0x3	AW_RESP		The number of axi4 write responses
0x4	AW_STL		The number of axi4 write stalls
0x5	AW_LAT		Finish one axi4 write transaction latency measurement
0x6	AW_REQ_TBU	The number of axi4 write requests after tbu
0x8	AR_REQ		The number of axi4 read requests
0x9	AR_BEATS	The number of axi4 read beats
0xa	AR_TRANS	The number of axi4 read transactions
0xb	AR_STL		The number of axi4 read stalls
0xc	AR_LAT		Finsih one axi4 read transaction latency measurement
0xd	AR_REQ_TBU	The number of axi4 read requests after TBU
0xe	TBU_MISS	The number of TBU miss
0xf	TX_DAT_AF	DDN channel write fifo almost full
0x10	RX_DAT_AF	DDN channel read fifo almost full
0x11	RETRYQ_CRED	RetryQ has availabel credits

Tile Performance Module:
------------------------
Value	Name		Meaning

0x45	HNF_REQUESTS	Number of REQs that were processed in HNF.
0x46	HNF_REJECTS	Number of REQs that were deferred due to an index match
			collision with another REQ being processed. Note that a
			given REQ can be rejected multiple times.
0x47	ALL_BUSY	Number of cycles that all MAFs in the directory were
			busy at the same time.
0x48	MAF_BUSY	Number of cycles that the MAF selected by HNF_PERF_CTL
			register MAF_SEL field was busy. MAFs are assigned in
			fixed order, so lower number MAFs will be busy more
			often than higher number MAFs. This information, along
			with the BUSY event, can be used to determine how
			loaded the Directory is.
0x49	MAF_REQUESTS	Number of REQs processed by the MAF selected by
			HNF_PERF_CTL register MAF_SEL field. This information,
			along with the number of cycles taken (MAF_BUSY) can be
			used to measure the average processing time for a REQ.
0x4a	RNF_REQUESTS	Number of REQs sent by the RN-F selected by
			HNF_PERF_CTL register RNF_SEL field.
0x4b	REQUEST_TYPE	Number of REQs that are of the type selected by
			HNF_PERF_CTL register OPCODE_SEL field.
0x4c	MEMORY_READS	Number of reads to MSS.
0x4d	MEMORY_WRITES	Number of writes to MSS.
0x4e	VICTIM_WRITE	Number of victim lines written to memory.
0x4f	POC_FULL	Number of times that all of the trackers in POC Monitor
			were ARMED when an exculsive load to a new address
			arrived.
0x50	POC_FAIL	Number of times that the POC Monitor sent RespErr Okay
			status to an Exclusive WriteNoSnp or CleanUnique REQ.
0x51	POC_SUCCESS	Number of times that the POC Monitor sent RespErr
			ExOkay status to an Exclusive WriteNoSnp or
			CleanUnique REQ.
0x52	POC_WRITES	Number of Exclusive WriteNoSnp or CleanUnique REQs
			processed by POC Monitor
0x53	POC_READS	Number of Exclusive ReadClean/ReadShared REQs processed
			by POC Monitor
0x54	FORWARD		Number of times data in "Forward" cache state was sent
			from one RN-F to another.
0x55	RXREQ_HNF	CHI Requests into HN-F.
0x56	RXRSP_HNF	CHI Responses into HN-F.
0x57	RXDAT_HNF	CHI Data flits into HN-F.
0x58	TXREQ_HNF	CHI Requests sent from HN-F.
0x59	TXRSP_HNF	CHI Responses sent from HN-F.
0x5a	TXDAT_HNF	CHI Data flits sent from HN-F.
0x5b	TXSNP_HNF	CHI Snoops sent from HN-F.
0x5c	INDEX_MATCH	Requests rejected due to index match.
0x5d	A72_ACCESS	Access Requests (Reads, Writes, CopyBack, CMO, DVM)
			from A72 clusters.
0x5e	IO_ACCESS	Accesse Requests (Reads, Writes) from DMA IO devices.
0x5f	TSO_WRITE	Total Store Order write Requests from DMA IO devices.
0x60	TSO_CONFLICT	Number of times TSO writes arrived to cacheline that
			already had TSO write in progress.
0x61	DIR_HIT		Requests that hit in directory.
0x62	HNF_ACCEPTS	Number of REQs that were accepted by HNF.
0x63	REQ_BUF_EMPTY	Number of cycles when request buffer is empty.
0x64	REQ_BUF_IDLE_MAF	Number of cycles when there are idle MAFs and
			the request buffer is not empty.
0x65	TSO_NOARB	Number of times that TSO anti-starvation was activated.
0x66	TSO_NOARB_CYCLES	Number of cycles that TSO anti-starvation
			blocked TSO requests.
0x67	MSS_NO_CREDIT	Number of cycles that a Request could not be sent to
			MSS due to lack of credits.
0x68	TXDAT_NO_LCRD	Number of cycles that TxDat has no link credits.
0x69	TXSNP_NO_LCRD	Number of cycles that TxSnp has no link credits.
0x6a	TXRSP_NO_LCRD	Number of cycles that TxRsp has no link credits.
0x6b	TXREQ_NO_LCRD	Number of cycles that TxReq has no link credits.
0x6c	TSO_CL_MATCH	Number of TSO writes that matched another TSO write in
			flight to same cacheline.
0x6d	MEMORY_READS_BYPASS	Number of reads to MSS that bypass Home Node.
0x6e	TSO_NOARB_TIMEOUT	Number of times that TSO anti-starvation was
			exited due to timeout.
0x6f	ALLOCATE	Number of times that Directory entry was allocated.
0x70	VICTIM		Number of times that Directory entry allocation did not
			find an Invalid way in the set.
0x71	A72_WRITE	Write Requests from A72 clusters.
0x72	A72_Read	Read Requests from A72 clusters.
0x73	IO_WRITE	Write Requests from DMA IO devices.
0x74	IO_Reads	Read Requests from DMA IO devices.
0x75	TSO_Reject	Number of TSO writes that were rejected due to match to
			another TSO write in flight to same cacheline.
0x80	TXREQ_RN	CHI Requests sent from RN-F/D.
0x81	TXRSP_RN	CHI Respones sent from RN-F/D.
0x82	TXDAT_RN	CHI Data sent from RN-F/D.
0x83	RXSNP_RN	CHI Snoops sent into RN-F/D.
0x84	RXRSP_RN	CHI Respones sent into RN-F/D.
0x85	RXDAT_RN	CHI Data sent into RN-F/D.

TRIO Performance Module:
------------------------
Value	Name			Meaning

0xa0	TPIO_DATA_BEAT		Data beats received from tpio source
0xa1	TDMA_DATA_BEAT		Data beats received from tdma source
0xa2	MAP_DATA_BEAT		Data beats received from map source
0xa3	TXMSG_DATA_BEAT		Data beats received from txmsg source
0xa4	TPIO_DATA_PACKET	Data packets received from tpio source
0xa5	TDMA_DATA_PACKET	Data packets received from tdma source
0xa6	MAP_DATA_PACKET		Data packets received from map source
0xa7	TXMSG_DATA_PACKET	Data packets received from txmsg source
0xa8	TDMA_RT_AF		Response tracker almost full
0xa9	TDMA_PBUF_MAC_AF	The indicator of the ring is too full
0xaa	TRIO_MAP_WRQ_BUF_EMPTY	The buffer of write transaction is empty
0xab	TRIO_MAP_CPL_BUF_EMPTY	The buffer of complete transaction is empty
0xac	TRIO_MAP_RDQ0_BUF_EMPTY	The buffer of MAC0's read transaction is empty
0xad	TRIO_MAP_RDQ1_BUF_EMPTY	The buffer of MAC1's read transaction is empty
0xae	TRIO_MAP_RDQ2_BUF_EMPTY	The buffer of MAC2's read transaction is empty
0xaf	TRIO_MAP_RDQ3_BUF_EMPTY	The buffer of MAC3's read transaction is empty
0xb0	TRIO_MAP_RDQ4_BUF_EMPTY	The buffer of MAC4's read transaction is empty
0xb1	TRIO_MAP_RDQ5_BUF_EMPTY	The buffer of MAC5's read transaction is empty
0xb2	TRIO_MAP_RDQ6_BUF_EMPTY	The buffer of MAC6's read transaction is empty
0xb3	TRIO_MAP_RDQ7_BUF_EMPTY	The buffer of MAC7's read transaction is empty

MSS Generic Performance Module:
-------------------------------
Value	Name		Meaning

0xc0	RXREQ_MSS	CHI Requests sent into memory.
0xc1	RXDAT_MSS	CHI Data sent into memory.
0xc2	TXRSP_MSS	CHI Responses sent from memory.
0xc3	TXDAT_MSS	CHI Data sent from memory.

L3 Cache Performance Module:
----------------------------
Value	Name				Meaning
0x00	DISABLE				Counter is disabled
0x01	CYCLES				Timestamp counter
0x02	TOTAL_RD_REQ_IN			CHI CDN Read Transactions
0x03	TOTAL_WR_REQ_IN			CHI CDN Write Transactions
0x04	TOTAL_WR_DBID_ACK		CHI NDN ACK Transactions
0x05	TOTAL_WR_DATA_IN		CHI DDN Write Transactions
0x06	TOTAL_WR_COMP			CHI NDN WrComp Transactions
0x07	TOTAL_RD_DATA_OUT		CHI DDN RData Transactions
0x08	TOTAL_CDN_REQ_IN_BANK0		CHI CDN Transactions Bank 0
0x09	TOTAL_CDN_REQ_IN_BANK1		CHI CDN Transactions Bank 1
0x0a	TOTAL_DDN_REQ_IN_BANK0		CHI DDN Transactions Bank 0
0x0b	TOTAL_DDN_REQ_IN_BANK1		CHI DDN Transactions Bank 1
0x0c	TOTAL_EMEM_RD_RES_IN_BANK0	Total EMEM Read Res Bank 0
0x0d	TOTAL_EMEM_RD_RES_IN_BANK1	Total EMEM Read Res Bank 1
0x0e	TOTAL_CACHE_RD_RES_IN_BANK0	Total Cache Read Res Bank 0
0x0f	TOTAL_CACHE_RD_RES_IN_BANK1	Total Cache Read Res Bank 1
0x10	TOTAL_EMEM_RD_REQ_BANK0		Total EMEM Read Req Bank 0
0x11	TOTAL_EMEM_RD_REQ_BANK1		Total EMEM Read Req Bank 1
0x12	TOTAL_EMEM_WR_REQ_BANK0		Total EMEM Write Req Bank 0
0x13	TOTAL_EMEM_WR_REQ_BANK1		Total EMEM Write Req Bank 1
0x14	TOTAL_RD_REQ_OUT		EMEM Read Transactions Out
0x15	TOTAL_WR_REQ_OUT		EMEM Write Transactions Out
0x16	TOTAL_RD_RES_IN			EMEM Read Transactions In
0x17	HITS_BANK0			Number of Hits Bank 0
0x18	HITS_BANK1			Number of Hits Bank 1
0x19	MISSES_BANK0			Number of Misses Bank 0
0x1a	MISSES_BANK1			Number of Misses Bank 1
0x1b	ALLOCATIONS_BANK0		Number of Allocations Bank 0
0x1c	ALLOCATIONS_BANK1		Number of Allocations Bank 1
0x1d	EVICTIONS_BANK0			Number of Evictions Bank 0
0x1e	EVICTIONS_BANK1			Number of Evictions Bank 1
0x1f	DBID_REJECT			Full usage and request
0x20	WRDB_REJECT_BANK0		Full usage and request
0x21	WRDB_REJECT_BANK1		Full usage and request
0x22	CMDQ_REJECT_BANK0		Full usage and request
0x23	CMDQ_REJECT_BANK1		Full usage and request
0x24	COB_REJECT_BANK0		Full usage and request
0x25	COB_REJECT_BANK1		Full usage and request
0x26	TRB_REJECT_BANK0		Full usage and request
0x27	TRB_REJECT_BANK1		Full usage and request
0x28	TAG_REJECT_BANK0		Full usage and request
0x29	TAG_REJECT_BANK1		Full usage and request
0x2a	ANY_REJECT_BANK0		Any reject Bank 0
0x2b	ANY_REJECT_BANK1		Any reject Bank 1

PCIe TLR Statistics:
--------------------
Value	Name				Meaning

0x0	PCIE_TLR_IN_P_PKT_CNT		Incoming Posted Packets.
0x10	PCIE_TLR_IN_NP_PKT_CNT		Incoming Non-Posted Packets.
0x18	PCIE_TLR_IN_C_PKT_CNT		Incoming Completion Packets.
0x20	PCIE_TLR_OUT_P_PKT_CNT		Outgoing Posted Packets.
0x28	PCIE_TLR_OUT_NP_PKT_CNT		Outgoing Non-Posted Packets.
0x30	PCIE_TLR_OUT_C_PKT_CNT		Outgoing Completion Packets.
0x38	PCIE_TLR_IN_P_BYTE_CNT		Incoming Posted bytes.
0x40	PCIE_TLR_IN_NP_BYTE_CNT		Incoming Non-POsted bytes.
0x48	PCIE_TLR_IN_C_BYTE_CNT		Incoming Completion bytes.
0x50	PCIE_TLR_OUT_P_BYTE_CNT		Outgoing Posted bytes.
0x58	PCIE_TLR_OUT_NP_BYTE_CNT	Outgoing Non-Posted bytes.
0x60	PCIE_TLR_OUT_C_BYTE_CNT		Outgoing Completion bytes.
