====== Documentation for CMS Pixel Phase II simulation ======

===== Links =====
   * CMS Central Twiki page for Pixel Phase II simulation [[https://twiki.cern.ch/twiki/bin/view/CMS/TrackerPhase2Upgrade|Twiki]]
   * CMS Pixel Phase I Example Twiki [[https://twiki.cern.ch/twiki/bin/view/CMS/ExamplePhaseI|Twiki]]
   * [[vlsi:projects:cms|Torino Pixel Phase II Hardware]]
   * CMS Pixel Offline Twiki [[https://twiki.cern.ch/twiki/bin/viewauth/CMS/PixelOfflineSoftware|Twiki]]
   * CMS USSTF Twiki [[https://twiki.cern.ch/twiki/bin/viewauth/CMS/USSTF|Twiki]] 
      * runTheMatrix recipes for 620_SLHC [[https://twiki.cern.ch/twiki/bin/viewauth/CMS/Recipes620SLHC|Twiki]]
   

====== Code and recipes ======


==== Installation on lxplus ====

++++ GitHub configuration |

(most of them should be already included in ~/.gitconfig)

<code>
git config --global user.name "Ernesto Migliore"
git config --global user.email migliore@to.infn.it
git config --global user.github emiglior
git config --global http.sslVerify false
git config --global color.ui true  # pretty printing!
</code>

Instructions to generate and register the ssh key [[https://help.github.com/articles/generating-ssh-keys|link]]\\

**Note:** enter an empty passphrase to log in automatically\\

Check the following [[https://hypernews.cern.ch/HyperNews/CMS/get/git/31/2.html|HN]] in case of problems.\\ 
Next block of instructions likely obsolete for CMSSW_6_2_X_SLHC:
<code>
# for bash
# git clone git://github.com/fwyzard/cms-git-tools 
# PATH=${PWD}/cms-git-tools:$PATH 
# git cms-init --ssh
</code>

See also these links [[http://cms-sw.github.io/cmssw/index.html|CMSSW and GitHub]] and [[https://github.com/settings/ssh|ssh key and GitHub]]

++++

++++ Instructions for CMSSW_6_1_2_SLHC8 (obsolete!)|
<code>
# NOT NEEDED ON LXPLUS5 source /opt/exp_soft/cms/cmsset_default.sh
# NOT NEEDED ON LXPLUS5 export SCRAM_ARCH=slc5_amd64_gcc472
scram p CMSSW CMSSW_6_1_2_SLHC8_patch3 
cd CMSSW_6_1_2_SLHC8_patch3/src/
eval `scram r -sh`
git clone -b 612_slhc8_phase1 git://github.com/emiglior/usercode.git . 
or 
git clone -b 612_slhc8_phase2 git://github.com/emiglior/usercode.git .
# to "push" changes to the remote 
#git clone -b 612_slhc8_phase1/612_slhc8_phase2 git@github.com:emiglior/usercode.git .
</code>
++++

++++ Instructions for CMSSW_6_2_0_SLHC11 (obsolete!)|
<code>
scram p CMSSW CMSSW_6_2_0_SLHC11
cd CMSSW_6_2_0_SLHC11/src/
eval `scram r -sh`
git clone -b 620_slhc11_phase1 git://github.com/emiglior/usercode.git . 
# to "push" changes to the remote 
#git clone -b 620_slhc11_phase1 git@github.com:emiglior/usercode.git .
</code>
++++

++++ Instructions for CMSSW_6_2_0_SLHC17_patch1 (current baseline for pixel local reco studies)|
<code>
scram p CMSSW CMSSW_6_2_0_SLHC17_patch1
cd CMSSW_6_2_0_SLHC17_patch1/src/
eval `scram r -sh`
git clone -b 620_slhc17_patch1_phase2_fwd git://github.com/emiglior/usercode.git . 
# to "push" changes to the remote later
# option 1. 
# git remote set-url origin git@github.com:emiglior/usercode.git 
# option 2. clone with SSH protocol
# git clone -b 620_slhc17_patch1_phase2_fwd git@github.com:emiglior/usercode.git .
</code>
++++

<code>
scram b -j 8
</code>   

The Python scripts are:
  * ''PixelLocalRecoBuildAndRun.py'' For studies of the pixel local reconstruction (RecHit resolution, CPE production, occupancy) 
  * <del>''LoopCMSSWBuildAndRun.py''	DIGI->DQM steps for MultiTrackValidation (not yet available in 620_slhc17_patch1_phase2)</del>

Issue ''python <script_name> -h'' for online help.\\
e.g. ''PixelLocalRecoBuildAndRun.py -s -i MyTestConfiguration.ini'' it will create and submit jobs based on directives in the ''MyTestConfiguration.ini'' configuration file.

<del>**Note:**  in ''LoopCMSSWBuildAndRun.py'' the lines 
<code> os.chdir(os.path.join(HOME,"MyWorkSpace/public","SLHCSimPhase2",CMSSW_VER,"src")) </code> and
<code> fout.write("cd "+os.path.join(HOME,"MyWorkSpace/public","SLHCSimPhase2","${cmssw_ver}","src")+"\n") </code>
should be adapted to the path of your local installation.
</del>


==== Plots for pixel occupancy studies ====

''PixelLocalRecoBuildAndRun.py '' runs the ''DPGAnalysis/​SiStripTools/​plugins/​OccupancyPlots.cc'' EDAnalyzer to produce plots for the analysis of the occupancy of the pixels.

To produce the occupancy plots a set of macros collected in ''DPGAnalysis/SiStripTools/bin'' are used.
To build the shared object required to make the plots 

  * **On lxplus or on a machine with a CMSSW installation** (skip the first instructions if CMSSW is already installed):
  - ''scram p CMSSW CMSSW_6_2_0_SLHC17_patch1''
  - ''cd CMSSW_6_2_0_SLHC17_patch1/src''
  - ''eval `scram r -sh`''
  - ''git cms-addpkg DPGAnalysis/​SiStripTools''
  - ''scram b -r''
  * After the shared object is built, from an interactive ROOT session:
<code>
.L libDPGAnalysisSiStripToolsMacros.so # or whatever path to the shared object
TFile *_file0 = TFile::Open("MyOccupancyPlotsOnly.root","READ") # or whatever name of the root file
PlotOccupancyMapPhase2(_file0, "pixeloccupancyplots/run_1", 0.00001, 10., 0.00001, 10.,1)
</code>

  * **On your lapotop** (the macro will open several canvas, so it could be useful when working in remote). From any where: 
      * method #1: Using git and GitHub
          * ''git init''
          * ''git config core.sparsecheckout true''
          * ''git remote add -f official-cmssw %%https://github.com/cms-sw/cmssw.git%%'' 
          * ''echo %%"%%DPGAnalysis/SiStripTools%%"%% %%>>%% .git/info/sparse-checkout''
          * ''git pull official-cmssw CMSSW_6_2_0_SLHC17_patch1''  
      * method #2:copy to you laptop the CMSSW module DPGAnalysis/SiStripTools from an already existing installation (e.g. scp from lxplus) 
   - copy in ''DPGAnalysis/SiStripTools/bin'' this {{dpganalysissistriptools_makefile.zip|Makefile}}
   - ''cd DPGAnalysis/SiStripTools/bin''
   - ''make all'' 
   * After the shared object is built, from an interactive ROOT session:
<code>
.L libDPGAnalysisSiStripToolsMacros.so # or whatever path to the shared object
TFile *_file0 = TFile::Open("MyOccupancyPlotsOnly.root","READ") # or whatever name of the root file
PlotOccupancyMapPhase2(_file0, "pixeloccupancyplots/run_1", 0.00001, 10., 0.00001, 10.,1)
</code>


==== Change sensor thickness in GEN-SIM step ====
<code>
scram p CMSSW CMSSW_6_2_0_SLHC17_patch1
cd CMSSW_6_2_0_SLHC17_patch1/src
eval `scram r -sh`
git cms-addpkg Geometry/TrackerCommonData
# we need to clone usercode in a separate directory as a .git area has been already produced by the cms-addpkg command
git clone -b 620_slhc17_patch1_phase2_fwd git://github.com/emiglior/usercode.git
mv usercode/AuxCode . 
rm -fr usercode
cd AuxCode/SLHCSimPhase2/test/GenSimStep
./ChangeThickness.sh
</code>

===== EXPERT USE =====

Next sections are intented for expert use only !

==== Update/Create a branch on your my-cmssw repository  ====
Not needed unless you need to modify some CMSSW pakckage 
<code>
scramv1 p CMSSW CMSSW_6_2_0_SLHC17_patch1
cd CMSSW_6_2_0_SLHC17_patch1/src/
cmsenv
git cms-addpkg CalibTracker/SiPixelESProducers
git cms-addpkg CondFormats/SiPixelObjects
git cms-addpkg CondTools/SiPixel
git cms-addpkg Geometry/TrackerCommonData
git cms-addpkg SLHCUpgradeSimulations/Geometry
git cms-addpkg SLHCUpgradeSimulations/Configuration
git cms-addpkg RecoLocalTracker/SiPixelRecHits
git cms-addpkg RecoLocalTracker/SiPixelClusterizer
git cms-addpkg DPGAnalysis/SiStripTools
git cms-addpkg DataFormats/SiPixelCluster
git pull https://github.com/mmusich/cmssw ChangePitch_on620_SLHC17_patch1
git branch
git remote
git checkout -b ChangePitch_on620_SLHC17_patch1_32bit

###########
# Qui ho portato a mano i file cambiati per funzionare a 32 bit:
#
# DataFormats/SiPixelCluster/interface/SiPixelCluster.h
# DataFormats/SiPixelCluster/src/SiPixelCluster.cc
# DataFormats/SiPixelCluster/src/classes_def.xml
# DataFormats/SiPixelCluster/test/SiPixelCluster_t.cpp
# RecoLocalTracker/SiPixelClusterizer/src/PixelThresholdClusterizer.cc
# RecoLocalTracker/SiPixelRecHits/src/PixelCPEGeneric.cc
#
############

git commit -m " Changing SiPixelCluster data member to uint32" .
git push my-cmssw ChangePitch_on620_SLHC17_patch1_32bit
</code>

==== Pixel RecHit resolution and CPE production (to be updated for the NewStdHitNtuplizer) ====
  - run particle gun + nutuplizer to store hits summary information \\ [[https://github.com/emiglior/usercode/blob/master/AuxCode/SLHCSimPhase2/test/TenMuE_0_200_cff_py_GEN_TO_RECO_TO_PixelCPE_NTUPLE.py | link ]] to the pset to be used (on github) \\ (1k evts  10 mu particle-gun  -> about 5h wall clock on T2_IT_Bari, size of output ntuple 9 MB)
  - grab the ROOT script \\ ''cvs co -A -d Brownson  UserCode/Brownson/SLHCUpgradeSimulations/test/resolutionPlotter'' 
  - ''cd Brownson'' \\ ''make'' \\ ''ln -fs <path_to_your_ntuple> ./stdgrechitfullph1g_ntuple.root'' \\ ''./res'' 

==== Recipe for running to DIGI-to-DQM steps ====

  * run CMSSW

<code> cmsRun step_digitodqm_DIGI_L1_DIGI2RAW_RAW2DIGI_L1Reco_RECO_VALIDATION_DQM.py </code>

  * harvesting

<code>
cmsDriver.py step4  --geometry Extended2017 --customise SLHCUpgradeSimulations/Configuration/phase1TkCustoms.customise --conditions auto:upgrade2017 --mc  -s HARVESTING:validationHarvesting+dqmHarvesting -n 100  --filein file:step_digitodqm_inDQM.root --fileout file:step4.root  > step4_FourMuPt1_200_UPG2017+FourMuPt1_200_UPG2017+DIGIUP17+RECOUP17+HARVESTUP17.log  2>&1
</code>

  * run tracking validation macros
<code>
cd $CMSSW_BASE/src/SLHCUpgradeSimulations/Geometry/test
mkdir plots

# copy the output DQM files in plots
# edit slhcHPIterPerfValidation.csh
# NB: do not use blank chars in the labels

# execute the macro (FIXME 2013.09.04 script is crashing after making building3.pdf)
./slhcHPIterPerfValidation.csh
</code>

==== Production of the pset ====

This step should be done only when preparing the files to be committed to GitHub for a new version of CMSSW

   * log in to lxplus.cern.ch 
   * install CMSSW
 
<code>

scram p CMSSW CMSSW_6_2_0_SLHC17_patch1

cd CMSSW_6_2_0_SLHC17_patch1/src

eval `scram r -sh`

git cms-addpkg SLHCUpgradeSimulations/Configuration

git pull https://github.com/mmusich/cmssw ChangePitch_on620_SLHC17_patch1
 
# git based recipe to get the generation fragment
# https://twiki.cern.ch/twiki/bin/viewauth/CMS/GitRepositoryForGenProduction

mkdir -p Configuration/GenProduction/

git clone git@github.com:cms-sw/genproductions.git Configuration/GenProduction/ 

git cms-checkdeps -a 

scram b -j 8

</code>
 
   * runTheMatrix and follow the instructions documented in  [[https://hypernews.cern.ch/HyperNews/CMS/get/slhc-trackersim/569.html|HN]]

<code> 
mkdir MyTest

cd MyTest

runTheMatrix.py --what upgrade -l 10000 
</code> 

which produces the cmsDriver output for the four steps (GEN-SIM, DIGI-RAW, RAW2DIGI-VALIDATION, harvesting)

<code>
Preparing to run 10000 FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017

# in: /tmp/emiglior/CMSSW_6_2_0_SLHC17_patch1/src/MyTest going to execute cd 10000_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017
 cmsDriver.py FourMuPt_1_200_cfi  --conditions auto:upgrade2017 -n 10 --eventcontent FEVTDEBUG --relval 10000,100 -s GEN,SIM --datatier GEN-SIM --beamspot Gauss --customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017 --geometry Extended2017 --magField 38T_PostLS1 --fileout file:step1.root  > step1_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1
 
# in: /tmp/emiglior/CMSSW_6_2_0_SLHC17_patch1/src/MyTest going to execute cd 10000_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017
 cmsDriver.py step2  --conditions auto:upgrade2017 -n 10 --eventcontent FEVTDEBUGHLT -s DIGI:pdigi_valid,L1,DIGI2RAW --datatier GEN-SIM-DIGI-RAW --customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017 --geometry Extended2017 --magField 38T_PostLS1 --filein file:step1.root  --fileout file:step2.root  > step2_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1

# in: /tmp/emiglior/CMSSW_6_2_0_SLHC17_patch1/src/MyTest going to execute cd 10000_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017
 cmsDriver.py step3  --conditions auto:upgrade2017 -n 10 --eventcontent FEVTDEBUGHLT,DQM -s RAW2DIGI,L1Reco,RECO,VALIDATION,DQM --datatier GEN-SIM-RECO,DQM --customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017 --geometry Extended2017 --magField 38T_PostLS1 --filein file:step2.root  --fileout file:step3.root  > step3_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1

# in: /tmp/emiglior/CMSSW_6_2_0_SLHC17_patch1/src/MyTest going to execute cd 10000_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017
 cmsDriver.py step4  --conditions auto:upgrade2017 --scenario pp -s HARVESTING:validationHarvesting+dqmHarvesting --customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017 --geometry Extended2017 --mc  --magField 38T_PostLS1 -n 100  --filein file:step3_inDQM.root --fileout file:step4.root  > step4_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1

</code>
 
The steps above should be merged to produce the cmsDriver.py commands required to produce the different psets

 * GEN-RECO (for pixel CPE study)
<code> cmsDriver.py FourMuPt_1_200_cfi --no_exec --conditions auto:upgrade2017 -n 10 --eventcontent FEVTDEBUG --relval 10000,100 \
 -s GEN,SIM,DIGI:pdigi_valid,L1,DIGI2RAW,RAW2DIGI,L1Reco,RECO --datatier GEN-SIM-RECO --beamspot NoSmear --geometry Extended2017 --magField 38T_PostLS1 \
 --customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017 \
 --fileout file:step1.root  > step1_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1
</code> 

   * GEN-SIM
<code>
 cmsDriver.py FourMuPt_1_200_cfi --no_exec --conditions auto:upgrade2017 -n 10 --eventcontent FEVTDEBUG --relval 10000,100 \
-s GEN,SIM --datatier GEN-SIM --beamspot Gauss --geometry Extended2017 --magField 38T_PostLS1 \
--customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017  \
--fileout file:step1.root  > step1_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1
</code>

   * DIGI-RECO 
<code>
 cmsDriver.py step2 --no_exec --conditions auto:upgrade2017 -n 10 --eventcontent FEVTDEBUGHLT,DQM \
-s DIGI:pdigi_valid,L1,DIGI2RAW,RAW2DIGI,L1Reco,RECO,VALIDATION,DQM  --datatier GEN-SIM-DIGI-RAW-RECO,DQM \
--geometry Extended2017 --magField 38T_PostLS1
--customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017,AuxCode/SLHCSimPhase2/TkOnlyValidationCustoms.customise_tkonly  \
--filein file:step1.root  --fileout file:step2.root  > step2_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1
</code>

   * harvesting
<code>
 cmsDriver.py step4 --no_exec --conditions auto:upgrade2017 --scenario pp \
-s HARVESTING:validationHarvesting+dqmHarvesting \
--geometry Extended2017 --mc  --magField 38T_PostLS1 -n 100 \
--customise SLHCUpgradeSimulations/Configuration/combinedCustoms.cust_2017,AuxCode/SLHCSimPhase2/TkOnlyValidationCustoms.customise_tkonly  \
--filein file:step3_inDQM.root --fileout file:step4.root  > step4_FourMuPt1_200+FourMuPt_1_200_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log  2>&1
</code>



++++ Old Recipe|
<code>

scram p CMSSW CMSSW_6_2_0_SLHC11

cd CMSSW_6_2_0_SLHC11/src

eval `scram r -sh`

#### obsolete 
# CVS based recipe to get the generation fragment
# cvs co Configuration/GenProduction/python/FourteenTeV
# (this works also in CT, provided you have authenticated via cvs login, see https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookChapter1)
##############################

#### NEW

git cms-addpkg SLHCUpgradeSimulations/Configuration

git pull https://github.com/mmusich/cmssw ChangePitch_on620_SLHC11
 
# git based recipe to get the generation fragment
# https://twiki.cern.ch/twiki/bin/viewauth/CMS/GitRepositoryForGenProduction

mkdir -p Configuration/GenProduction/

git clone git@github.com:cms-sw/genproductions.git Configuration/GenProduction/ 

git cms-checkdeps -a 

scram b -j 8

mkdir MyTest

cd MyTest
</code>
++++

<code>
# edit step_digitodqm_DIGI_L1_DIGI2RAW_RAW2DIGI_L1Reco_RECO_VALIDATION_DQM.py to modify the parameters of the pixel digitizer
# changes should go at the end of the pset

---%<----
# Automatic addition of the customisation function from SLHCUpgradeSimulations.Configuration.phase1TkCustoms
from SLHCUpgradeSimulations.Configuration.phase1TkCustoms import customise

#call to customisation function customise imported from SLHCUpgradeSimulations.Configuration.phase1TkCustoms
process = customise(process)
# Uncomment next two lines to change pixel DIGI threshold
#process.mix.digitizers.pixel.ThresholdInElectrons_BPix = cms.double(2000.0)
#process.mix.digitizers.pixel.ThresholdInElectrons_BPix_L1 = cms.double(2000.0)

# End of customisation functions
--->%---

</code>

==== Running with PU ====
   * Generate Minimum bias events to mix:    

<code>
cmsDriver.py Configuration/GenProduction/python/FourteenTeV/MinBias_TuneZ2star_14TeV_pythia6_cff.py -s GEN,SIM --conditions auto:upgrade2017 --eventcontent FEVTDEBUG --beamspot Gauss  --geometry Extended2017 --relval 10000,100  --datatier GEN-SIM -n 5  --customise SLHCUpgradeSimulations/Configuration/postLS1Customs.customisePostLS1,SLHCUpgradeSimulations/Configuration/phase1TkCustoms.customise --fileout file:MinBias_TuneZ2star_14TeV_pythia6_cff_py_GEN_SIM.root  > step1_MinBiasTuneZ2star_5evts.log  2>&1
</code>

   * Run the DIGI,L1,DIGI2RAW,RAW2DIGI,L1Reco,RECO,DQM step

==== Tools and links ====

   * {{:cms:analysis:slhchpiterperfvalidation_csh.tgz|macro for running Tracking validation}} FIXME
  
   * CMSSW tools
      * cmsDriver [[https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCmsDriver|Twiki]]
      * global tags for upgrade studies [[https://twiki.cern.ch/twiki/bin/view/CMS/GlobalTagsForUpgrade|Twiki]] 
   * CMSSW frequently referenced files
      * [[http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/CMSSW/SLHCUpgradeSimulations/Configuration/python/phase1TkCustoms.py?revision=1.25&view=markup| CVS ]] standard values for configuration of the strip and pixel digitizer, ... 
   * Outer Tracker layout by S.Mersi [[http://mersi.web.cern.ch/mersi/layouts | link]]
   * Guide to EOS at CERN [[https://twiki.cern.ch/twiki/bin/viewauth/CMS/T2CHCERN | link]]
   * Guide to the T2_IT_Bari usage [[https://www.ba.infn.it/pagine-utenti.html?task=viewpage&user_id=111&pageid=96|link]]

===== Talks =====
   * Morris Swartz              [[http://www.pha.jhu.edu/~morris/dj_seminar4.pdf| Seminar on Irradiated Pixel Sensors]]
   * Morris Swartz   2013.03.21 [[https://indico.cern.ch/contributionDisplay.py?contribId=0&confId=241570| indico]]
   * Carrillo Moreno 2013.03.19 [[https://indico.cern.ch/contributionDisplay.py?contribId=12&confId=238001| indico]]
   * Alessia Tricomi 2013.01.13 [[https://indico.cern.ch/contributionDisplay.py?contribId=8&confId=224471| indico]]