Issue Ingesting Sequence Data into Transformers4Rec Model for Next-Item Prediction (NvTabular Workflow Output)

[seneg0id]

Hi team,

We are currently working on training a model using Transformers4Rec for a next-item prediction task.
For preprocessing and generating sequential input data, we are using NVTabular with a workflow that outputs the required parquet files.

However, while ingesting this processed sequence data into the model, we are encountering an error. Below is the stack trace:

**** Launch training for day 1: *****
/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  warnings.warn(
  0%|                                                                                                                                                 | 0/10 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/projects/merlin/nba-merchants/recommender/model/model_training.py", line 221, in <module>
    trainer.run(strategy=strategy, input_dir=input_dir)
  File "/projects/merlin/nba-merchants/recommender/model/model_training.py", line 118, in run
    metrics = self._train_day_wise(input_dir)
  File "/projects/merlin/nba-merchants/recommender/model/model_training.py", line 159, in _train_day_wise
    self.recsys_trainer.train()
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1543, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1791, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2539, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/trainer.py", line 323, in compute_loss
    outputs = model(inputs, targets=targets, training=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 560, in forward
    head_output = head(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/model/base.py", line 382, in forward
    body_outputs = self.body(body_outputs, training=training, testing=testing, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/config/schema.py", line 50, in __call__
    return super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/block/base.py", line 256, in forward
    input = module(input, training=training, testing=testing)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/config/schema.py", line 50, in __call__
    return super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/tabular/base.py", line 392, in __call__
    outputs = super().__call__(inputs, *args, **kwargs)  # noqa
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/features/sequence.py", line 253, in forward
    outputs = super(TabularSequenceFeatures, self).forward(inputs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/tabular/base.py", line 604, in forward
    outputs.update(layer(inputs))
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/config/schema.py", line 50, in __call__
    return super().__call__(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/tabular/base.py", line 392, in __call__
    outputs = super().__call__(inputs, *args, **kwargs)  # noqa
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers4rec/torch/features/embedding.py", line 240, in forward
    embedded_outputs[name] = self.embedding_tables[name](val)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/sparse.py", line 163, in forward
    return F.embedding(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 2264, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

schema:

<style> </style>

name	tags	dtype	is_list	is_ragged	properties.num_buckets	properties.freq_threshold	properties.max_size	properties.cat_path	properties.embedding_sizes.cardinality	properties.embedding_sizes.dimension	properties.domain.min	properties.domain.max	properties.domain.name	properties.value_count.min	properties.value_count.max
sessn_id	(Tags.CATEGORICAL)	DType(name='int64', element_type=<ElementType....	FALSE	FALSE	None	0	0	.//categories/unique.sessn_id.parquet	198073	512	0	198072	sessn_id	NaN	NaN
cust_id	(Tags.USER, Tags.CATEGORICAL, Tags.ID)	DType(name='int64', element_type=<ElementType....	FALSE	FALSE	None	0	0	.//categories/unique.cust_id.parquet	169267	512	0	169266	cust_id	NaN	NaN
item_len	(Tags.CATEGORICAL)	DType(name='int32', element_type=<ElementType....	FALSE	FALSE	None	0	0	.//categories/unique.payload_val.parquet	29	16	0	28	payload_val	NaN	NaN
item_id-list	(Tags.LIST, Tags.ITEM, Tags.CATEGORICAL, Tags.ID)	DType(name='int64', element_type=<ElementType....	TRUE	TRUE	None	0	0	.//categories/unique.payload_val.parquet	29	16	0	28	payload_val	0	300
loss_mask-list	(Tags.LIST, Tags.CATEGORICAL)	DType(name='int64', element_type=<ElementType....	TRUE	TRUE	None	0	0	.//categories/unique.loss_mask.parquet	5	16	0	4	loss_mask	0	300
event_type-list	(Tags.LIST, Tags.CATEGORICAL)	DType(name='int64', element_type=<ElementType....	TRUE	TRUE	None	0	0	.//categories/unique.event_type.parquet	5	16	0	4	event_type	0	300

Some more context:

• The sequence data is created correctly via NVTabular workflows (using SequenceGroupby and other related ops).
• The model setup follows the standard Transformers4Rec NextItemPredictionTask.
• The issue seems to arise at the point where the model tries to load or process the data, not during data creation.

We would like to know:

• Are there any known compatibility requirements or additional steps needed when using NVTabular-processed sequence data for Transformers4Rec models?
• Any recommended way to debug or verify the format expected by the model vs. the actual data generated?
• Could this be an issue related to schema mismatch, padding, or data types in the parquet files?

Versions:
transformers4rec: 23.12.0
nvtabular: 23.8.0
transformers: 4.26.0

Please let me know if you need further details — I can share the workflow pipeline, schema, and model code if helpful.

Thank you!