Skip to content

Commit

Permalink
fix: use float64 to avoid truncation (#100)
Browse files Browse the repository at this point in the history
* fix: use float64 to avoid truncation

* explanation
  • Loading branch information
jokasimr authored Jan 24, 2025
1 parent fb643e2 commit 10535f1
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions src/ess/nmx/scaling.py
Original file line number Diff line number Diff line change
Expand Up @@ -359,6 +359,10 @@ def polyval_wavelength(
"""
# We need to use float64 precision because
# the curve fit routine depends on finite difference
# estimates of the derivative.
wavelength = wavelength.to(dtype='float64')
out = sc.zeros_like(wavelength)
out.unit = out_unit
xk = sc.ones_like(wavelength)
Expand Down

0 comments on commit 10535f1

Please sign in to comment.