In a custom-generated world where the terrain is uniformly below Y0 at the border, crossing the border teleports the player to world spawn.
It looks to me like the reason for this is that WorldBorder contains a hard-wired assumption that no safe (to stand upon) blocks can be generated below Y0, as initialised here.
That variable is used to seek a safe place to stand here. Note that local variable Y is set to limBot (both 0) and therefore the first if test in the for loop is if (0 > 0) i.e. always false. It denies the possibility of searching for a safe negative Y.
I believe the way to fix it is to initialise limBot to WorldInfo.getMinHeight(), as implemented by World.