Not building extra impls by default

[re-xyr]

The extra implementations of TypeRepMap generally has worse performance and adds dependency footprint, especially the vector dependency that takes a decade to build.

Could we not build them by default?