########################################
Author: Sangamesh Ragate
Date  : Nov 12th 2015
email : sragate@vols.utk.edu
INNOVATIVE COMPUTING LABORATORY, UTK
#######################################


Descripttion :	This Utility helps in configuring CUPTI for performing PC_SAMPLING
				on MAXWELL GPUS and those which have PC_SAMPLING support. It is a 
				standalone tool that works like nvprof and can be used to get the 
				PC sampling result on any cuda application without re building 
				the application. 

************************************************************************************************************
To Compile:

	This utility get compiled automatically when the cuda component for PAPI is compiled
	The *papi_sampling_cuda executable is generated in the src/utils diretory of PAPI

************************************************************************************************************
To run the utility:

	./papi_sampling_cuda [-d <GPU Device ID>] [-s <sampling period>] cuda_app [its arguments]

	-d : This switch is optional and is used to supply GPU device ID, should 
		 be integer > 0, Default is GPU device ID 0
	-s : This switch is optional and is used to supply PC sampling period. 
	     Range from 0 to 5, refer "enum CUpti_ActivityPCSamplingPeriod" in
		 CUPTI user manual. Default is set to 5.

    cuda_app : this is the cuda applicationf for which PC SAMPLING is performed
	All the arguments that come next belong to the cuda_app.
	

************************************************************************************************************
Example to try:
	
	After successful compilation of PAPI with cuda component
	> cd src/components/cuda/sampling/test


	NOTE: Make sure papi and cuda shared libraries are in the LD_LIBRARY_PATH before you run the test.
	matmul is a cuda application which performs 512x512 matrix multiplication

	Try:
	> ./papi_sampling_cuda matmul
	> ./papi_sampling_cuda -d 0 -s 0 matmul
	> ./papi_sampling_cuda -d 0 -s 5 matmul
	> ./papi_sampling_cuda -d 0 matmul
	> ./papi_sampling_cuda -s 2 matmul


************************************************************************************************************
Output:

	>Kernel activity record : This gives information about the cuda kernel that was launched for PC SAMPLING
	>Activity Kind record   : This gives information about the cuda kernel that was launched for PC SAMPLING
	>PC_SAMPLING record     : Kernel identification, PC value, samples, stall reason
	>Source locator record  : This is generated if cuda_app is compiled using "-lineinfo" in nvcc
	>STALL SUMMARY          : This gives the histogram of Stall reason Vs Number of samples due to the 
							  corresponding stall.

	NOTE: To better understand the output generated, the user should be familiar with the "Activity API" 
	Records of cupti, more specifically KERNEL,SOURCE_LOACTOR,PC_SAMPLING activity records mentioned in the 
	CUPT manual.
************************************************************************************************************
Additional Feature:

	The utility also generates SASS dump that can be used to trace the stall to the source code line in the 
	CUDA application. To get the source code line info, recompile only your cuda_app using "-lineinfo" flag in the
	nvcc.


************************************************************************************************************