-
Notifications
You must be signed in to change notification settings - Fork 691
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tfjs_next_page_prediction example failed #6023
Comments
This error looks like the pipeline fails at the tf model to tfjs model conversion step. I was able to replicate the issue. Both test case and pipeline fails with "AssertionError: Identity is not in graph" error. Please find the attached gist. @briron, Could you please have a look why the tfjs_next_page_prediction example fails. Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
System information
pip freeze
output):absl-py==1.4.0
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1663850771047/work
aiohttp-cors==0.7.0
aiorwlock==1.3.0
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
ansiwrap==0.8.4
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1685486400275/work/dist
apache-beam==2.46.0
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work
argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1649500320262/work
astunparse==1.6.3
async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1640026696943/work
asynctest==0.13.0
attrs==21.4.0
Babel==2.12.1
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backoff==2.2.1
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
beatrix-jupyterlab @ file:///home/kbuilder/miniconda3/conda-bld/dlenv-tf-2-11-gpu_1686189831411/work/packages/beatrix_jupyterlab-2023.62.145605.tar.gz#sha256=dc0618129624f08938e5de9508880ef81dba8b272d097fd0b0c61f77af4da3f5
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1680888073205/work
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1674535352125/work
blessed==1.20.0
brotlipy==0.7.0
cached-property==1.5.2
cachetools==4.2.4
certifi==2023.5.7
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1666183775483/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1661170624537/work
chex==0.1.5
click==7.1.2
cloud-tpu-client==0.10
cloud-tpu-profiler==2.4.0
cloudpickle==2.2.1
colorama @ file:///tmp/build/80754af9/colorama_1607707115595/work
colorful==0.5.5
conda==22.9.0
conda-content-trust @ file:///tmp/build/80754af9/conda-content-trust_1617045594566/work
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1669907009957/work
conda_package_streaming @ file:///home/conda/feedstock_root/build_artifacts/conda-package-streaming_1685101166527/work
crcmod==1.7
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1666563371538/work
cycler==0.11.0
Cython==0.29.35
dacite==1.8.1
db-dtypes==1.1.1
debugpy @ file:///home/conda/feedstock_root/build_artifacts/debugpy_1660619049122/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
Deprecated==1.2.14
dill==0.3.1.1
distlib==0.3.6
dm-tree==0.1.8
docker==4.4.4
docopt==0.6.2
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
etils==0.9.0
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1678703645500/work
explainable-ai-sdk==1.3.3
fastapi==0.96.0
fastavro==1.7.4
fasteners==0.18
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1684761244589/work/dist
filelock==3.12.0
flatbuffers==2.0.7
flax==0.6.4
flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1684084314667/work/source/flit_core
fonttools==4.38.0
frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1659556953511/work
fsspec==2023.1.0
gast==0.4.0
gcsfs==2023.1.0
gitdb==4.0.10
GitPython==3.1.31
google-api-core==1.32.0
google-api-python-client==1.8.0
google-apitools==0.5.31
google-auth==1.35.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.4.6
google-cloud-aiplatform==1.17.1
google-cloud-artifact-registry==1.8.1
google-cloud-bigquery==2.34.4
google-cloud-bigquery-storage==2.16.2
google-cloud-bigtable==1.7.3
google-cloud-core @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-core_1658759191372/work
google-cloud-datastore @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-datastore_1661336072891/work
google-cloud-dlp==3.9.2
google-cloud-language==1.3.2
google-cloud-monitoring==2.15.0
google-cloud-pubsub==2.13.11
google-cloud-pubsublite==1.6.0
google-cloud-recommendations-ai==0.7.1
google-cloud-resource-manager==1.6.3
google-cloud-spanner==3.26.0
google-cloud-storage==2.9.0
google-cloud-videointelligence==1.16.3
google-cloud-vision==3.1.4
google-crc32c==1.5.0
google-pasta==0.2.0
google-resumable-media==2.5.0
googleapis-common-protos @ file:///home/conda/feedstock_root/build_artifacts/googleapis-common-protos-feedstock_1657690297642/work
gpustat==1.0.0
greenlet==2.0.2
grpc-google-iam-v1==0.12.6
grpcio @ file:///home/conda/feedstock_root/build_artifacts/grpc-split_1662408129152/work
grpcio-status @ file:///home/conda/feedstock_root/build_artifacts/grpcio-status_1662108958711/work
gviz-api==1.10.0
Gymnasium==0.26.3
gymnasium-notices==0.0.1
h11==0.14.0
h5py==3.8.0
hdfs==2.7.0
htmlmin==0.1.12
httplib2==0.21.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
ImageHash==4.3.1
imageio==2.31.0
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1653252814274/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1676919000169/work
iniconfig==2.0.0
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1666723258080/work
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1651240553635/work
ipython-genutils==0.2.0
ipython-sql==0.5.0
ipywidgets==7.7.5
jaraco.classes==3.2.3
jax==0.3.25
jaxlib==0.3.25
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
jeepney==0.8.0
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1654302431367/work
joblib==1.2.0
json5==0.9.14
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1669810440410/work
jupyter-http-over-ws==0.0.8
jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1671591499479/work
jupyter-server-mathjax==0.2.6
jupyter-server-proxy==3.2.2
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1673615989977/work
jupyter_core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1658332345782/work
jupyterlab==3.4.8
jupyterlab-git==0.41.0
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1649936611996/work
jupyterlab-widgets==1.1.4
jupyterlab_server==2.22.1
jupytext==1.14.6
keras==2.11.0
keras-tuner==1.3.5
keyring==23.13.1
keyrings.google-artifactregistry-auth==1.1.2
kiwisolver==1.4.4
kt-legacy==1.0.5
kubernetes==12.0.1
libclang==16.0.0
llvmlite==0.39.1
lz4==4.3.2
Markdown==3.4.3
markdown-it-py==2.2.0
MarkupSafe==2.0.1
matplotlib==3.5.3
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mdit-py-plugins==0.3.5
mdurl==0.1.2
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1675771498296/work
ml-metadata==1.12.0
ml-pipelines-sdk==1.12.0
more-itertools==9.1.0
msgpack==1.0.5
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1648882415996/work
multimethod==1.9.1
nb-conda @ file:///home/conda/feedstock_root/build_artifacts/nb_conda_1654442778977/work
nb-conda-kernels @ file:///home/conda/feedstock_root/build_artifacts/nb_conda_kernels_1636999991206/work
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1683202085119/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1665125402713/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1683636231832/work
nbdime==3.2.0
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1679336765223/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
networkx==2.6.3
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1680870634737/work
notebook-executor @ file:///home/kbuilder/miniconda3/conda-bld/dlenv-tf-2-11-gpu_1686189831411/work/packages/notebook_executor
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1682360583588/work
numba==0.56.4
numpy==1.21.6
nvidia-ml-py==11.495.46
oauth2client==4.1.3
oauthlib==3.2.2
objsize==0.6.1
onnx==1.12.0
opencensus==0.11.2
opencensus-context==0.1.3
opentelemetry-api==1.18.0
opentelemetry-exporter-otlp==1.18.0
opentelemetry-exporter-otlp-proto-common==1.18.0
opentelemetry-exporter-otlp-proto-grpc==1.18.0
opentelemetry-exporter-otlp-proto-http==1.18.0
opentelemetry-proto==1.18.0
opentelemetry-sdk==1.18.0
opentelemetry-semantic-conventions==0.39b0
opt-einsum==3.3.0
optax==0.1.4
orbax==0.1.0
orjson==3.9.0
overrides==6.5.0
packaging==20.9
pandas==1.3.5
pandas-profiling==3.6.6
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
papermill==2.4.0
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
patsy==0.5.3
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
phik==0.12.3
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow==9.5.0
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1633981968097/work
platformdirs==3.5.1
plotly==5.14.1
pluggy==1.0.0
portpicker==1.5.2
prettytable==3.7.0
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1684970902412/work
promise==2.3
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1677600924538/work
proto-plus==1.22.2
protobuf==3.19.6
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1666155398032/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
py-spy==0.3.14
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.7
pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1666656960991/work
pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work
pydantic==1.10.9
pydot==1.4.2
pyfarmhash==0.3.2
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1681904169130/work
PyJWT==2.7.0
pymongo==3.13.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1685514481738/work
pyparsing==3.0.9
pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1649013358450/work
PySocks @ file:///tmp/build/80754af9/pysocks_1594394576006/work
pytest==7.4.0
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
pytz==2023.3
pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work
PyWavelets==1.3.0
PyYAML==5.4.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1663830492333/work
ray==2.4.0
ray-cpp==2.4.0
regex==2023.6.3
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
requests-oauthlib==1.3.1
retrying==1.3.3
rich==13.4.1
rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016701961/work
scikit-image==0.19.3
scikit-learn==1.0.2
scipy==1.7.3
seaborn==0.12.2
SecretStorage==3.3.3
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1682601222253/work
Shapely==1.8.5.post1
simpervisor==0.4
six @ file:///tmp/build/80754af9/six_1644875935023/work
smart-open==6.3.0
smmap==5.0.0
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1662051266223/work
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1658207591808/work
SQLAlchemy==2.0.15
sqlparse==0.4.4
starlette==0.27.0
statsmodels==0.13.5
tabulate==0.9.0
tangled-up-in-unicode==0.2.0
tenacity==8.2.2
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-profile==2.13.0
tensorboard-plugin-wit==1.8.1
tensorboardX==2.6
tensorflow==2.11.0
tensorflow-cloud==0.1.16
tensorflow-data-validation==1.12.0
tensorflow-datasets==4.8.2
tensorflow-decision-forests==1.2.0
tensorflow-estimator==2.11.0
tensorflow-hub==0.12.0
tensorflow-io==0.29.0
tensorflow-io-gcs-filesystem==0.29.0
tensorflow-metadata==1.12.0
tensorflow-model-analysis==0.43.0
tensorflow-probability==0.19.0
tensorflow-serving-api==2.11.0
tensorflow-transform==1.12.0
tensorflowjs==4.4.0
tensorstore==0.1.28
termcolor==2.3.0
terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1670253674810/work
textwrap3==0.9.2
tf2onnx==1.14.0
tfx==1.12.0
tfx-bsl==1.12.0
threadpoolctl==3.1.0
tifffile==2021.11.2
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
toml==0.10.2
tomli==2.0.1
toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1656937818679/work
tqdm @ file:///opt/conda/conda-bld/tqdm_1647339053476/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
typeguard==2.13.3
typer==0.9.0
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1685704949284/work
uritemplate==3.0.1
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1678635778344/work
uvicorn==0.22.0
virtualenv==20.21.0
visions==0.7.5
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1673864653149/work
webencodings==0.5.1
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1684708148666/work
Werkzeug==2.1.2
widgetsnbextension==3.6.4
witwidget==1.8.1
wordcloud==1.9.2
wrapt==1.15.0
wurlitzer==3.0.3
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1648966511831/work
ydata-profiling==4.2.0
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1677313463193/work
zstandard @ file:///home/conda/feedstock_root/build_artifacts/zstandard_1655887611100/work
Describe the current behavior
python tfjs_next_page_prediction_e2e_test.py
get error:
FAIL: testTFJSPagePredictionPipeline (main.TFJSNextPagePredictionPipelineEndToEndTest)
TFJSNextPagePredictionPipelineEndToEndTest.testTFJSPagePredictionPipeline
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 813, in _convert_tf_saved_model
resource_ids_maps = _get_resource_ids_maps(model, concrete_func, resource_initializer_concrete_func)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 680, in _get_resource_ids_maps
captured_input_index = resource_id_to_captured_input_index[_id]
KeyError: 3207
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/Documents/tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_e2e_test.py", line 94, in testTFJSPagePredictionPipeline
LocalDagRunner().run(pipeline)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/tfx_runner.py", line 124, in run
return self.run_with_ir(pipeline_pb, run_options=run_options_pb, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/local/local_dag_runner.py", line 109, in run_with_ir
component_launcher.launch()
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/launcher.py", line 573, in launch
executor_output = self._run_executor(execution_info)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/launcher.py", line 448, in _run_executor
executor_output = self._executor_operator.run_executor(execution_info)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/python_executor_operator.py", line 135, in run_executor
return run_with_executor(execution_info, executor)
File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/python_executor_operator.py", line 59, in run_with_executor
execution_info.exec_properties)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/executor.py", line 178, in Do
run_fn(fn_args)
File "/home/Documents/tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_util.py", line 198, in run_fn
rewriter.ModelType.TFJS_MODEL)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/converters.py", line 131, in rewrite_saved_model
dst_model_type)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/converters.py", line 47, in _invoke_rewriter
rewriter_inst.perform_rewrite(original_model, rewritten_model)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/rewriter.py", line 106, in perform_rewrite
self._rewrite(original_model, rewritten_model)
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/tfjs_rewriter.py", line 91, in _rewrite
_ensure_str(rewritten_model.path))
File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/tfjs_rewriter.py", line 29, in _convert_tfjs_model
saved_model_path, destination_path
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 818, in convert
weight_shard_size_bytes, metadata_map)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 539, in _dispatch_converter
metadata=metadata_map)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 988, in convert_tf_saved_model
frozen_graph_dir=frozen_graph_dir)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 820, in _convert_tf_saved_model
output_node_names)
File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 434, in _freeze_saved_model_v1
sess, meta_graph_def, output_node_names)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func
return func(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 322, in convert_variables_to_constants
variable_names_denylist=variable_names_blacklist)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1288, in convert_variables_to_constants_from_session_graph
variable_names_denylist=variable_names_denylist))
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py", line 943, in init
graph_def = graph_util.extract_sub_graph(graph_def, output_node_names)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func
return func(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 251, in extract_sub_graph
_assert_nodes_are_present(name_to_node, dest_nodes)
File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 204, in _assert_nodes_are_present
assert d in name_to_node, "%s is not in graph" % d
AssertionError: Identity is not in graph
Describe the expected behavior
tfjs testcase pass
Standalone code to reproduce the issue
tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_e2e_test.py
Any reply is welcome, thanks
The text was updated successfully, but these errors were encountered: