Allow custom dimensions of any size (custom_dimension_X fields of unlimited length column (TEXT type) instead of VARCHAR 255)

[mattab]

Currently custom dimension values are limited to 255 characters, but in some cases we want to track more than 255 characters in a custom dimension.

Currently, if trying to issue a request with a dimension value longer than 255 chars, the tracker API request will fail with an error like

Exception: Error query: SQLSTATE[22001]: String data, right truncated: 1406 Data too long for column 'custom_dimension_2' at row 1 In query: INSERT INTO piwik_log_visit (idvisitor, config_id, location_ip, idsite, visit_first_action_time, visit_goal_buyer, visit_goal_converted, visit_last_action_time, visitor_days_since_first, visitor_days_since_order, visitor_returning, visitor_count_visits, visit_entry_idaction_name, visit_entry_idaction_url, [...], campaign_content, campaign_id, campaign_keyword, campaign_medium, campaign_name, campaign_source, custom_dimension_1, custom_dimension_2)

When manually changing the column type from VARCHAR 255 to TEXT, then the data is tracked properly, and displayed correct in the reports.

---> For users who need to track more than 255 chars, can we recommend them to manually change the DB schema column type to TEXT field? is there any possible problem with this solution?

[tsteur]

Yes would definitely not making this the default. And would also possibly trim after 255 characters to avoid errors and rather not fully tracking it instead of failing.

[mattab]

Maybe the columns could be to TEXT type in the Custom Dimensions release for Piwik 4.0.0 in the future.

[sgiehl]

@tsteur @mattab is this change something that should be considered for Matomo 4?

[tsteur]

Not sure. Personally would keep things the way they are maybe. Eg TEXT can have performance problems creating temporary tables etc. Having an FAQ on how to change this be good though.

[mattab]

Btw before closing we should also update the link in the page: https://matomo.org/docs/data-limits/
which currently link to this issue. It could link to the FAQ explaining how to make the alter table columns to TEXT.

[cah-erinblake]

Is there any update on this issue? Is this change going to be included in Matomo 4?

[tsteur]

It's not included in Matomo 4 but you can change the type manually in your database if needed.

[luflow]

@tsteur Is there already an FAQ entry for this? Could not find anything until now. Is it just an ALTER TABLE where the type gets changed to TEXT on all custom dimension fields that should get this change?

What happens if a future migration of piwik changes this type back?

[tsteur]

@luflow there isn't. AFAIK it's only an alter table query to change field to text like below:

ALTER TABLE matomo_log_visit MODIFY COLUMN custom_dimension_1 TEXT, MODIFY COLUMN custom_dimension_2 TEXT, MODIFY COLUMN custom_dimension_3 TEXT, MODIFY COLUMN custom_dimension_4 TEXT, MODIFY COLUMN custom_dimension_5 TEXT;

I'm not sure what would happen if we were to change it back. I'm assuming the content would be truncated but the historical archived reporting data would still include the full names.

[ashutosh-hs]

I am using matomo 4.6.2
Even after manually changing the column type to text in matomo DB for custom dimension columns, the values are still being truncated to 255 characters.
Is there anyway to get around this limitation at this time?

[ashutosh-hs]

@mattab @tsteur could you please help me with this? We have now upgraded to version 4.10.1 and still have this issue.
Even after manually changing the column type to text in matomo DB for custom dimension columns, the values are still being truncated to 255 characters.
Is there anything else that I need to do after changing column type?
Is there anyway to get around this limit at this time?

[justinvelluppillai]

Hi @ashutosh-hs we haven't got this on the list of currently prioritised work. You may have a better chance of getting help with this over on our forums https://forum.matomo.org/

[ashutosh-hs]

@justinvelluppillai thanks.
I have already created a topic on the form : https://forum.matomo.org/t/manually-changing-custom-dimension-x-fields-to-be-unlimited-length-column/46287
haven't received any replies on this yet

[tsteur]

Hi @ashutosh-hs the custom dimension value is hard coded to 250 characters: https://github.com/matomo-org/matomo/blob/4.11.0-rc1/plugins/CustomDimensions/Tracker/CustomDimensionsRequestProcessor.php#L175

It was added in Matomo 3.1.8 matomo-org/plugin-CustomDimensions@823e8e9#diff-bbc47867ddfd7fe85fe96b6a846a35ca5064315fb8a12ec8615a4ec69adf6fd3R132-R136 meaning the workaround to use TEXT columns doesn't work anymore since that release.

[ashutosh-hs]

oh, okay. @tsteur thanks for the confirmation

[Sonofendor]

Hi @tsteur
Maybe you know if other variables are also hard coded to be specific size? For example 'Event Name/Action/Category/Value'?
Your answer would help me a lot
Thanks

[tsteur]

@Sonofendor I may be wrong but I believe event name/action/category are limited to 4000 characters as per the log_action.name field definition. I don't think it's limited in the code though (unless I missed it). I can't really say for any other fields right now

[Sonofendor]

@tsteur Thanks for your answer!

[Webmasterei]

@tsteur as time moves on and data collections are getting bigger, I see a larger demand for this to be happening.
I have a usecase now where we want to push all "ecommerce items" on a page/cart/checkout to a dimension but due to this restriction it's not possible. So a big +1 from my side to bring this into priorisation. The Plans to do so are long-time announced.

[Eismann82]

Hello @tsteur tsteur and the Matomo team,

I would like to make a quick plea for the implementation of unlimited text lengths for the custom_dimension_X fields. In our projects, we regularly hit the limits of the current 255 character limit, which prevents us from utilizing the full depth of our data. An extension to the TEXT type would make us much more flexible and improve the quality of our analyses. We hope for a positive development in this matter.

[9joshua]

A customer has an issue with the 255 byte limit. The have faced several problems...

1. The limit is a byte not character limit. Therefore using multi-byte characters can substantially reduce the custom dimension value limit below 255 characters.
2. When sending a string like this 'test', 'character limit' URL encoding adds additional bytes to the size...
   'test', 'character limit' = 27 bytes
   %27test%27%2C%20%27character%20limit%27 = 39 bytes

The only solution presently is to:

• Split these data between multiple Custom Dimensions
• Compress and Base64 encode the data, then decode and decompress after downloading the data from Matomo
  Neither of these solutions are ideal.

@tsteur Perhaps this could be solved more easily now. I don't believe the SQL data type needs to change to "TEXT"...

The VARCHAR length can be specified as a value from 0 to 255 before MySQL 5.0.3, and 0 to 65,535 in 5.0.3 and later versions.

[jorgeuos]

Hi,

I have a client who requested a fix for this issue, and we are looking to contribute it back to the core. The idea is that the size of a custom dimension should be configurable based on the needs of the implementer of a Matomo installation, while keeping a default limit of 250 characters.

Similar to the existing configs:

•	datatable_archiving_maximum_rows_custom_dimensions
•	datatable_archiving_maximum_rows_subtable_custom_dimensions

I would like to introduce a new config option:

•	custom_dimensions_max_length

This would allow for flexibility in defining the length of custom dimensions, with the default set to 250 characters, but configurable up to 65,535 (the MySQL TEXT field limit).

I have submitted a PR for review here: #22582

Looking forward to your feedback!

Thanks!