Pathfinder runs out of memory for large models

[velochy]

I am fitting a factor model of 5 factors on ~7000 respondents, leading to the number of variables being optimized close to 35000.

This leads to an interesting error:
MemoryError: Unable to allocate 168. GiB for an array with shape (19,35000,35000) and data type float64

[velochy]

I am not an expert on numerical optimization, but a quick glance at wikipedia tells me L-BFGS was designed to avoid materializing the N x N hessian to avoid exactly this kind of blowup.

My friendly AI future overlord tells me this is due to pt.diag() usage which can materialize the matrix, and recommends a fix that looks rather plausible.

After that, memory complexity comes down to (L, N, 2J) where L is number of L-BFGS iterations and J is the number of trace steps used in the hessian approximation (maxcor). Which is a lot better than N x N but still concerning as L can be rather large (hundreds) for complex models.
So i asked the AI to also propose a fix via batching L into L' and had it put together a spec for the refactor:
PATHFINDER_MEMORY_REFACTOR_SPEC.md

I'm happy to try to have AI implement it, review the code for bloat and make sure it creates and passes reasonable tests for result similarity. But it would be good if someone more knowledgeable about pathfinder looked over the spec just to make sure the approach actually make sense, before I end up running head-first into a brick wall.

[ricardoV94]

Sounds great. The T @ diag(X) -> Mul thing is something I wanted to have in PyTensor for a while, so if you want to tackle that in parallel as well, that's also great.

I don't know about the Scan / python loop part of the discussion. I didn't spend time tracking it down to see how it looks now. Either way you already have a sure win, so it's just a question of whether you have yet another win.