New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TF-Keras mixed precision training leads to autograph errors #66374
Labels
Comments
sushreebarsa
added
the
stat:awaiting response
Status - Awaiting response from author
label
Apr 25, 2024
@sushreebarsa As mentioned in the issue this is an issue with TF-Keras, I added |
google-ml-butler
bot
removed
the
stat:awaiting response
Status - Awaiting response from author
label
Apr 25, 2024
Fixed in #67189 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
binary
TensorFlow version
2.16.1, 2.17.0.dev20240423
Custom code
No
OS platform and distribution
Linux, Colab
Mobile device
No response
Python version
3.10, 3.11
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
12.3 / 8.9.7.29
GPU model and memory
NVIDIA Tesla T4
Current behavior?
Since TF version 2.16 mixed precision training is fails to compile with autograph and throws warnings when running a minimal mixed precision training example:
TF Autograph doesn't transform the
create_autocast_variable
function and throws the following warnings:Autograph should be able to transform the function and should not throw any warnings like it did in TF 2.15.
Unfortunately we're currently unable to upgrade to Keras 3 due to other issues, so it would be good to be able to get this patched in TF-Keras as well.
I'm not sure if this issue is caused by Autograph or by TF-Keras.
Standalone code to reproduce the issue
See the following notebooks: This wasn't an issue in TF 2.15, but fails in TF 2.16 and still fails in TF Nightly.
Relevant log output
No response
The text was updated successfully, but these errors were encountered: