Some basic queries regarding object-level selections and addition of new columns

[gsaha009]

Hello CF Experts,

This new issue is again with some basic questions:

1. If I run cf.ProduceColumns, how come it always runs on at least two files, even after mentioning branch 0? Is there any way to constrain it to one? like cf.SelectEvents task?
2. I wanted to save the object level cutflow similar to the event level cutflow. Is there anyway to save it in the stats, maybe?
3. I am struggling to add new columns to the events array. Here you can see my main selection and in Line242, another selector is called. There I explicitly mentioned a new column to be added. I can see this newly added column at the end of selection, but in the production step, it is no longer available. The variables to be kept at different levels are mentioned here. Does the new column need to be mentioned somewhere else?

Edited:
I would like to add one more question:
To reweight the events using genweight, is the EvtSumWeight from the "Runs" tree used? If not, is the nEvents in cmsdb campaings used for that?

Thank you :)

• Gourab, @oponcet

[mafrahm]

Hi @gsaha009, I took the freedom and converted this to a discussion. Regarding your questions:

1. I suppose it runs over multiple files for cf.SelectEvents etc., correct? There is a difference in the branch map between the selection tasks and everyting after the reduction because we perform a merging after reducing: https://github.com/columnflow/columnflow/blob/master/columnflow/tasks/reduction.py#L410
   If you do not want to merge your reduced files, there is an open PR (Refactor dynamic requirements of reduced events. #388) that'll allow you to skip the merging step.
2. Sure, you could e.g. store the number of jets before and after an object definition in the stats dict. Do you have anything more concrete in mind?
3. I do not see an issue with the code. Did you already check at which point the column is not stored anymore? I'd suggest to use the cf_inspect script to take a quick look into an exemplary output file after each task where this column should be present (cf_inspect fname.parquet opens a debugger where you get your stored array via objects[0])
4. For normalization we calculate the sum of weights ourselves (typically done as part of the event selection and stored in the stats). The sum_mc_weight_per_process is then used in our normalization_weight Producer: https://github.com/columnflow/columnflow/blob/master/columnflow/production/normalization.py#L110

I hope these answers are helpful.

Best
Mathis

[gsaha009]

Hi Mathis,
Thank you for this reply. A few followups are in the following

1. For cf.SelectEvents, I can mention --branch 0 to consider one file only, but in cf.ProduceColumns task, even if I mention branch 0, it starts from the cf.CalibrateEvents on the second file. But as you clarified, it is the difference in the branch map after reduction.
2. Not yet, but I will try to include object level selections in the stats dictionary.
3. I have used cf_inspect and I came to see that the events array didn't contain the new column with hcand object after the Selection task. But, for the time being, I bypassed it with saving some columns with float datatype instead. The problem I am having now is the following
   When I am going to execute cf.PlotVariables1D with --categories etau,mutau,tautau and --variables hcand_dr,hcand_invm, I see that the 1st variable is getting plotted on the 1st category given in the command line only. hcand_dr looks exactly the same in inclusive and etau category. When I tried to plot it separately for mutau and tautau, it appeared to be empty, but in the stats dict after selection, I saw similar number of events were selected in each of the three categories. It would be really helpful if you can have a look at here and let me know if you find any obvious mistake. Thanks :)
4. Nice. Thanks

Best,
Gourab
adding Stepan as well (@hephysicist)

[gsaha009]

Thank you guys. This issue has been resolved now. I am closing the discussion.