Incorporate initial guesses into residual for iterative solvers.

[shakedregev]

Summary

We should have an option to add an initial guess (for iterative refinement or regular iterative solve).
If there is an initial guess $x$ to the solution of $Ax=b$, we can compute $r=b-Ax$.
We then solve $Ay=r$ and return $x+y$ as the solution.
An ideal implementation is one where we have the same function name and it can take either two arguments ($A,b$) or three ($A,b,x$). The first is the current implementation, and the second will add a wrapper that stores the initial guess, and add it to the solution obtained from solving $Ay=r$.

Rationale

Right now, when we use an iterative solver directly, we have no option of supplying an initial guess. When we use it for iterative refinement, we use an initial guess in a way that is less numerically accurate.

Description

Some features to make this complete:

1. Check if $|b|_2</code>&lt;<code>|b-Ax|_2$. If it is, warn the user that their initial guess is worse than all 0s and ignore that initial guess.
2. If a nonzero guess was supplied, check if $|b-Ax-Ay|_2</code>&gt;<code>|b-Ax|_2$. Note that the order of subtraction is important for the left side. If it is, warn the user that the iterative solver was not able to find a solution better than the initial guess, and output $x$ (not $x+y$). For iterative refinement, this corresponds to not improving upon the direct solve.

Additional information

No response

[ASunil-17]

Hello Shaked, I have interpreted this to updating the LinSolverIterativeFGRMES member to align with the implementation that you are describing. Specifically, I am thinking of about treating the KLU solution as our initial guess and setting up the solver to compute an update. If the update plus the initial guess is offering a better estimation to the solution than the initial guess, I will return that quantity. Let me know if I should implement this change to the other iterative solvers as well

[shakedregev]

Yes, do it for all iterative solvers. Your interpretation is correct.

[ASunil-17]

Hello Shaked! I have implemented the changes where the KLU solution is treated as an initial guess and an update is computed with respect to the residual from this initial guess. I have also included the error check on the initial guess too. The function adds the update to the initial guess if the residual from this combination is lower than the initial residual. There is added space complexity from two new arrays: vec_Y_ (storing the update) and vec_R_ (storing the residual). The change in time complexity involves an additional matvec after the loop has terminated (to compute the final residual), two dot products (one before the loop to compute the norm of the RHS, and the other at the end to compute the norm of the final residual), and one axpy (also computed after the loop has terminated). Should I push these changes to the ReSolve repo?

[shakedregev]

There is no need for a special invite to push your code. Push it to your own branch and open a PR. Your standard workflow should have all changes on the remote repo on your own branch.

[ASunil-17]

Hello Shaked! Thank you for the suggestions. I have created a new branch called feature/cusolver-rf-config. This is the first time I have pushed to ReSolve github, so there are a lot of changes from the past few months in this version. The relevant files can be found in resolve sub-directory (the files I have updates are LinSolverIterativeFGMRES and LinSolverIterativeRandFGMRES). Let me know if you have any questions?

[nkoukpaizan]

@ASunil-17 One suggestion would be to rebase your branch to develop (git rebase develop), so your changes can be easily seen. You'd have to resolve conflict if there are any.

It will also be helpful to create a (draft) pull request once you're confident in your proposed solution.

[shakedregev]

I support everything Nicholson said. Please read about how to solve conflicts when rebasing. In the future, you can save yourself a lot of pain by pulling from the repo before branching off to start your work. You shouldn't have months of changes for a PR that took one week to write.

[ASunil-17]

I will pull from the repo before starting off my own work in the future. Thank you for the suggestion @nkoukpaizan