ValueError: SelectionResult event mask must be of type N * bool, but got {ak.type(self.event)}

[gsaha009]

Hi CF Users and Developers,

I am trying to build an analysis using columnflow framework. The problem I am facing is the following:

I am trying to produce a new column which would contain the pairs of leptons of any flavor, like, [ [e,tau], [mu,tau], [tau,tau], ... ].
I do also want to apply some event level selection based on the number of pairs per event.
First, I kept some common fields of the leptons by reconstructing a coffea.vector LorentzVector
e.g.

hcand_pair_concat.pt = 
[
[[]],
 [[29.92832374572754, 20.86834716796875]],
[[]],
 [[26.30992889404297, 36.008033752441406]],
 [[]],
...
]  # (printing the pt only. there can be multiple pairs per event)

Now the way I am creating the event level mask is the following:
ak.num(ak.firsts(hcand_pair_concat.pt, axis=1), axis=1) == 2 to make sure there is at least one pair per event.
If I add this boolean array in "steps" dict in SelectionResults, the error I am getting is the following

ValueError: SelectionResult event mask must be of type N * bool, but got {ak.type(self.event)}

I added that particular result to my main SelectioResults(), and just before event_sel = reduce(and_, results.steps.values()), I got this error.

ANY IDEA!!!

Thank you very much :)

[gsaha009]

Btw,

These are the steps in my analysis

In [4]: results.steps
Out[4]: 
DotDict([('trigger',
          <Array [False, False, False, ..., False, False, False] type='100000 * bool'>),
         ('met_filter',
          <Array [True, True, True, True, ..., True, True, True] type='100000 * bool'>),
         ('b_veto',
          <Array [True, True, True, True, ..., True, True, True] type='100000 * bool'>),
         ('At least two leptons of any sign',
          <Array [False, False, False, ..., False, False, False] type='100000 * bool'>),
         ('dilepton_veto',
          <Array [True, True, True, True, ..., True, True, True] type='100000 * bool'>),
         ('atleast_one_higgs_cand_per_event',
          <Array [False, False, False, ..., False, False, False] type='100000 * ?bool'>)])
In [5]:

Perhaps, the type of the last one is ?bool is creating the problem!!!

[gsaha009]

I bypassed it somehow using

ak.fill_none(slection_mask, False)

Probably, this is not the best way to do this.
If anyone of you has any comments, let me know
Thanks

[pkausw]

Hi @gsaha009 ,

This is just a quick feedback: Indeed, I think the ?bool in your array type indicates that you don't have only boolean values in your array. The and_ operator in your reduce does allow that, but this doesn't make sense in terms of selection masks of course, which is why the SelectionResult object explicitely checks for such cases and raises the error you see. It might be that ak.num returns a None value if you don't have any pairs to start with, such that your ak.firsts operation can't return anything (which I guess would also return a None in this case). Have you checked the return type ak.firsts in these cases?

Anyhow, in the cases where you have a None value in your array, it is indeed the best way to fill those with False using the way you already figured out 👍 Let us know if it's really because of the return type of the awkward operations or something else!

[pkausw]

That being said, I think we should look into the error that is raised, I think the print out is malformed

[gsaha009]

Thanks @pkausw
It is working now.
I would like to add one additional query here.

** ALERT: Long Error message ahead !!! :|

After selecting the events, in the selectEvents task, I was going to save the newly produced concatenated array of LorentzVector in the events array. The error I am getting is the following:

ERROR: luigi-interface - [pid 29850] Worker Worker(salt=5076733547, workers=1, host=sbgui3.in2p3.fr, username=gsaha, pid=28486) failed    cf.SelectEvents(effective_workflow=local, branch=0, analysis=httcp.config.analysis_httcp.analysis_httcp, version=v1, config=run2_2017_nano_local_v10, shift=nominal, local_shift=nominal, dataset=h_ggf_tautau_powheg, calibrators=main, selector=main)
Traceback (most recent call last):
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_dispatch.py", line 60, in dispatch
    next(gen_or_result)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/operations/ak_to_parquet.py", line 309, in to_parquet
    with pyarrow_parquet.ParquetWriter(
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/pyarrow/parquet/core.py", line 1016, in __init__
    self.writer = _parquet.ParquetWriter(
  File "pyarrow/_parquet.pyx", line 1776, in pyarrow._parquet.ParquetWriter.__cinit__
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: dense_union<0: large_list<item: large_list<item: struct<charge: int32 not null, decayMode: int16 not null, eta: float not null, mass: float not null, phi: float not null, pt: float not null> not null> not null> not null=0, 1: fixed_size_list<item: fixed_size_list<item: large_list<item: struct<charge: int32 not null, decayMode: int16 not null, eta: float not null, mass: float not null, phi: float not null, pt: float not null> not null> not null>[0] not null>[1] not null=1>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/luigi/worker.py", line 203, in run
    new_deps = self._run_get_new_deps()
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/luigi/worker.py", line 138, in _run_get_new_deps
    task_gen = self.task.run()
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 272, in log
    return fn(task, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/columnflow/util.py", line 303, in call
    return fn(task, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 244, in wrapper
    return decorator(fn, _opts, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 540, in localize
    return fn(task, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 231, in wrapper
    result = call(state)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/decorator.py", line 303, in call
    return fn(task, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/columnflow/tasks/selection.py", line 163, in run
    for (events, *cols), pos in self.iter_chunked_io(
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/columnflow/tasks/framework/mixins.py", line 1933, in iter_chunked_io
    for obj in self.iter_progress(handler, max(handler.n_chunks, 1), msg=msg):
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/modules/law/law/task/base.py", line 565, in iter_progress
    for i, val in enumerate(iterable):
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/columnflow/columnar_util.py", line 3388, in __iter__
    yield from self._iter_impl()
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/columnflow/columnar_util.py", line 3319, in _iter_impl
    result_obj = result.get()
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/conda/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/conda/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/modules/columnflow/columnflow/columnar_util.py", line 1041, in sorted_ak_to_parquet
    ak.to_parquet(ak_array, *args, **kwargs)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_dispatch.py", line 68, in dispatch
    return gen_or_result
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_errors.py", line 67, in __exit__
    self.handle_exception(exception_type, exception_value)
  File "/home/gsaha/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_errors.py", line 82, in handle_exception
    raise self.decorate_exception(cls, exception)
pyarrow.lib.ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: dense_union<0: large_list<item: large_list<item: struct<charge: int32 not null, decayMode: int16 not null, eta: float not null, mass: float not null, phi: float not null, pt: float not null> not null> not null> not null=0, 1: fixed_size_list<item: fixed_size_list<item: large_list<item: struct<charge: int32 not null, decayMode: int16 not null, eta: float not null, mass: float not null, phi: float not null, pt: float not null> not null> not null>[0] not null>[1] not null=1>

See if this has been reported at https://github.com/scikit-hep/awkward/issues
INFO: luigi-interface - This progress looks :( because there were failed tasks

By any chance, do you think that this is due to the ? in type?
If I check ak.num(ak.firsts(pair.pt, axis=1), axis=1), it returns type='100000 * ?int64'

If you think that I should raise a different discussion for this, let me know
:)

[gsaha009]

Without the hcand array, it is running smoothly. I didn't use this ak.to_parquet method, but I checked with cf_inspect, and all looked good without the hcand.
Yes, the array is really nested like nEvents * var * var * LV. Ah, okay... Then I will choose to keep the array simple for the time being and will check if it is working.
Thanks btw :)
will let you know shortly

[gsaha009]

Hi again,
I made the array simple
Now if I want to write hcand.pt which looks like

[
 [],
 [63.29899597167969, 45.53937911987305],
 [],
 [38.401275634765625, 56.79755401611328],
 [],
 [],
 [],
 [],
 [],
 [51.12352752685547, 49.83754348754883],
 ...,
]

I can't even write it to parquet
Looks strange

any idea?

[gsaha009]

Perhaps, long story short

I am not using the coffea LorentzVector now. However, I noticed the difference between the layout of Jet.pt and hcand.pt

# Jet.pt
<ListOffsetArray len='100000'>
    <offsets><Index dtype='int64' len='100001'>
        [     0      2      8 ... 533498 533505 533508]
    </Index></offsets>
    <content><NumpyArray dtype='float32' len='533508'>
        [31.162498 32.81094  31.387499 ... 31.264063 25.385939 17.058594]
    </NumpyArray></content>
</ListOffsetArray>

&

# hcand.pt
<ListArray len='100000'>
    <starts><Index dtype='int64' len='100000'>
        [39588 39588 39588 ... 39588 39588 39588]
    </Index></starts>
    <stops><Index dtype='int64' len='100000'>
        [39588 39588 39588 ... 39588 39588 39588]
    </Index></stops>
    <content><UnionArray len='39588'>
        <tags><Index dtype='int8' len='39588'>[0 0 0 ... 0 0 0]</Index></tags>
        <index><Index dtype='int64' len='39588'>
            [    21  67981     98 ... 676112 524119 676185]
        </Index></index>
        <content index='0'>
            <NumpyArray dtype='float32' len='726294'>
                [22.018867 38.186386 14.039998 ... 27.71415  29.300837
                 10.972406]
            </NumpyArray>
        </content>
        <content index='1'>
            <ListArray len='0'>
                <starts><Index dtype='int64' len='0'>
                    []
                </Index></starts>
                <stops><Index dtype='int64' len='0'>
                    []
                </Index></stops>
                <content><NumpyArray dtype='float32' len='13196'>
                    [34.344795 47.639206 34.660675 ... 44.429535 52.219776
                     50.008633]
                </NumpyArray></content>
            </ListArray>
        </content>
    </UnionArray></content>
</ListArray>

Maybe something is hidden there, and that is why, it still can't write this array in parquet... showing the following error

ArrowNotImplementedError                  Traceback (most recent call last)
Cell In[9], line 1
----> 1 ak.to_parquet(events.hcand.pt, "temp.parquet")

File ~/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_dispatch.py:68, in named_high_level_function.<locals>.dispatch(*args, **kwargs)
     63     else:
     64         raise AssertionError(
     65             "high-level functions should only implement a single yield statement"
     66         )
---> 68 return gen_or_result

File ~/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_errors.py:67, in ErrorContext.__exit__(self, exception_type, exception_value, traceback)
     60 try:
     61     # Handle caught exception
     62     if (
     63         exception_type is not None
     64         and issubclass(exception_type, Exception)
     65         and self.primary() is self
     66     ):
---> 67         self.handle_exception(exception_type, exception_value)
     68 finally:
     69     # `_kwargs` may hold cyclic references, that we really want to avoid
     70     # as this can lead to large buffers remaining in memory for longer than absolutely necessary
     71     # Let's just clear this, now.
     72     self._kwargs.clear()

File ~/Work/ColumnflowAnalyses/CPinHToTauTau/data/software/venvs/venv_columnar_dev_3cbb5aff/lib/python3.9/site-packages/awkward/_errors.py:82, in ErrorContext.handle_exception(self, cls, exception)
     80     self.decorate_exception(cls, exception)
     81 else:
---> 82     raise self.decorate_exception(cls, exception)

ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: dense_union<0: extension<awkward<AwkwardArrowType>> not null=0, 1: extension<awkward<AwkwardArrowType>> not null=1>

I saw this issue was discussed here
Btw, If I convert the whole ak.Array hcand_pair to list and then change it back to ak.Array, everything gets fixed
No union array issue and probably the pyarrow table conversion gets fixed.
But, it does not seem like a smart way, is it?

[pkausw]

Very nice, thanks for following up @gsaha009 ! This is very useful indeed, especially Jim's ak.enforce_type option might be good for future use cases (if the user is really sure about the type of course).

Regarding your hot fix which first casts the array to a list and back again: This is of course not very nice. But I guess that the route fixes the issue of the union types that is introduced with ak.concatenate somehow. As was already discussed in the discussion that is linked above, I hope that this gets fixed eventually. Until then, I guess there is nothing much to do except for either avoid using the concatenate operation altogether or manually casting the type by either your's or Jim's solution. We should keep track of this, as I'm sure that other users will try to generate nested fields in the future. Thanks a lot for summarizing your findings here!

[gsaha009]

Hi @pkausw
Thank you for the all your suggestions. I was converting awkward -> list -> awkward, but now I am using enforce_type what Jim suggested and it is working great. :)

[pkausw]

Great, sounds like this issue is resolved (within the context of columnflow). I'll close the discussion for now, but if there are new developments we should open it up again.