Bounding box grouping into lines based on font similarity or bbox shape similarity

[ajkdrag]

Currently the process of merging word bboxes into lines uses spacing to decide which boxes to merge. It would be really nice if there was a flag/option to do merging based on different algorithms/clustering techniques, i.e. cluster based on font similarity and proximity, or say, cluster based on bbox shape. This is useful in some documents where there are multiple fonts and the merging isn't very precise. Although I understand that this doesn't happen very happen, but I think it would be a nice addition.

[felixdittrich92]

Hi @ajkdrag 👋,

So based on the font, it's very special 😅

As far as clustering is concerned, I would be open to suggestions, additions and optimizations.
Do you perhaps have a specific example or code snippet, that we can structure the problem a little more clearly ? :)

[ajkdrag]

Sure.
In case of certain documents, especially such as bank cheques, or filling a form by hand. In these cases, the handwritten "font" is different compared to the printed "fields". Regular bbox merging logic can merge the handwritten text with the printed background text if they happen to be close. That's why I felt like it would be a nice addition. I understand this sounds like a very niche use-case, but this is useful in industry related ml projects (in my company we are trying to do this).
If clustering by font is challenging, i also thought about clustering by bbox. (I don't have code yet), if a bbox is very tiny in height (i.e. smaller font) and right next to a taller bbox (i.e larger/different font), it's very likely they shouldn't be merged.

[felixdittrich92]

Mh yes, I agree that solving for lines and blocks can be optimized. If you have a specific approach there, I would suggest opening a PR that we can iterate on. :)

[ajkdrag]

Sure I would love that!