Skip to content

tensorflow InvalidArgumentError: num_lower must be negative or less or equal to number of rows #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
arne-cl opened this issue Feb 24, 2021 · 2 comments
Open

tensorflow InvalidArgumentError: num_lower must be negative or less or equal to number of rows #2

arne-cl opened this issue Feb 24, 2021 · 2 comments

Comments

@arne-cl
Copy link
Member

arne-cl commented Feb 24, 2021

I discovered via issue NLPbox/stagedp-service#1 that NeuralEDUSeg crashes on some, mostly very short, input sentences:

$ cat /tmp/bad_input.txt                                                                                                                                                                        good food .

$ docker run -v /tmp:/tmp -it neural-edu-seg /tmp/bad_input.txt
Loading NeuralEDUSeg models...
[...]
Finished loading NeuralEDUSeg models.

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "neuralseg/splitter.py", line 334, in <module>
    main()
  File "neuralseg/splitter.py", line 323, in main
    result = segment_text(input_text, rst_data, model, spacy_nlp, output_format=args.output_format, debug=args.debug)
  File "neuralseg/splitter.py", line 279, in segment_text
    edus, spacy_edus, json_edus = segment_batches(model, data_batches, text, debug=debug)
  File "neuralseg/splitter.py", line 231, in segment_batches
    batch_edu_breaks = model.segment(batch)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 125, in segment
    feed_dict)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

Caused by op 'encoding/bilinear_attention/MatrixBandPart', defined at:
  File "neuralseg/splitter.py", line 334, in <module>
    main()
  File "neuralseg/splitter.py", line 321, in main
    rst_data, model, spacy_nlp = load_models(args)
  File "neuralseg/splitter.py", line 74, in load_models
    model = AttnSegModel(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 17, in __init__
    super().__init__(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/base_seg.py", line 37, in __init__
    self._build_graph()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/lstm_seg.py", line 20, in _build_graph
    self._encode()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/atten_seg.py", line 24, in _encode
    self.window_size)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/layers.py", line 51, in self_attention
    window_size, window_size)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2409, in matrix_band_part
    num_upper=num_upper, name=name)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

Expected behaviour would be sth. like:

$ cat /tmp/good_input.txt 
Judging from previous posts this used to be a good place, but not any longer.

$ docker run -v /tmp:/tmp -it neural-edu-seg /tmp/good_input.txt
Loading NeuralEDUSeg models...
[...]
Finished loading NeuralEDUSeg models.
{"paragraphs": [{"start": 0, "end": 78}], "sentences": [{"start": 0, "end": 77}], "edus": [{"start": 0, "end": 28}, {"start": 28, "end": 77}]}
@arne-cl
Copy link
Member Author

arne-cl commented Feb 24, 2021

Note: the example given in the description is using my wrapper around NeuralEDUSeg.
This is how it looks like using the original NeuralEDUSeg:

root@f697ebdcda53:/opt/neural-edu-seg/neuralseg# python run.py --segment --input_files /tmp/bad_input.txt --result_dir /tmp/results/
 2021-02-24 22:00:12,457 - SegEDU - INFO - Running with args : Namespace(batch_size=32, dev_files=None, dropout_keep_prob=0.9, ema_decay=0.9999, epochs=50, evaluate=False, gpu=None, hidden_size=200, input_files=['/tmp/bad_input.txt'], lea
rning_rate=0.001, log_path=None, max_grad_norm=5.0, model_dir='/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models', optim='adam', prepare=False, result_dir='/tmp/results/', rst_dir='../data/rst/', seed=123, seg
ment=True, test_files=None, train=False, train_files=None, weight_decay=0.0001, window_size=5, word_embed_path='../data/embeddings/glove.840B.300d.txt', word_embed_size=300, word_vocab_path='../data/vocab/word.vocab')
 2021-02-24 22:00:12,457 - SegEDU - INFO - Loading vocab...
 2021-02-24 22:00:12,522 - SegEDU - INFO - Word vocab size: 17243
 2021-02-24 22:00:12,522 - SegEDU - INFO - Loading the model...
2021-02-24 22:00:12.522775: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
 2021-02-24 22:00:17,873 - SegEDU - INFO - There are 4043611 parameters in the model
 2021-02-24 22:00:17,873 - SegEDU - INFO - Using Exp Moving Average to train the model with decay 0.9999.
 2021-02-24 22:00:19,063 - SegEDU - INFO - Time to build graph: 6.5401506423950195 s
/usr/local/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:22: FutureWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  FutureWarning)
 2021-02-24 22:00:32,036 - SegEDU - INFO - Model restored from /usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models/best
 2021-02-24 22:00:32,947 - SegEDU - INFO - Segmenting /tmp/bad_input.txt...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 50, in <module>
    segment(args)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/api.py", line 157, in segment
    batch_pred_segs = model.segment(batch)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 125, in segment
    feed_dict)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

Caused by op 'encoding/bilinear_attention/MatrixBandPart', defined at:
  File "run.py", line 50, in <module>
    segment(args)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/api.py", line 133, in segment
    model = AttnSegModel(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 17, in __init__
    super().__init__(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/base_seg.py", line 37, in __init__
    self._build_graph()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/lstm_seg.py", line 20, in _build_graph
    self._encode()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/atten_seg.py", line 24, in _encode
    self.window_size)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/layers.py", line 51, in self_attention
    window_size, window_size)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2409, in matrix_band_part
    num_upper=num_upper, name=name)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

It should look like this:

root@f697ebdcda53:/opt/neural-edu-seg/neuralseg# python run.py --segment --input_files /tmp/good_input.txt --result_dir /tmp/results/
 2021-02-24 21:58:50,750 - SegEDU - INFO - Running with args : Namespace(batch_size=32, dev_files=None, dropout_keep_prob=0.9, ema_decay=0.9999, epochs=50, evaluate=False, gpu=None, hidden_size=200, input_files=['/tmp/good_input.txt'], learning_rate=0.001, log_path=None, max_grad_norm=5.0, model_dir='/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models', optim='adam', prepare=False, result_dir='/tmp/results/', rst_dir='../data/rst/', seed=123, segment=True, test_files=None, train=False, train_files=None, weight_decay=0.0001, window_size=5, word_embed_path='../data/embeddings/glove.840B.300d.txt', word_embed_size=300, word_vocab_path='../data/vocab/word.vocab')
 2021-02-24 21:58:50,750 - SegEDU - INFO - Loading vocab...
 2021-02-24 21:58:50,822 - SegEDU - INFO - Word vocab size: 17243
 2021-02-24 21:58:50,823 - SegEDU - INFO - Loading the model...
2021-02-24 21:58:50.823501: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
 2021-02-24 21:58:55,833 - SegEDU - INFO - There are 4043611 parameters in the model
 2021-02-24 21:58:55,834 - SegEDU - INFO - Using Exp Moving Average to train the model with decay 0.9999.
 2021-02-24 21:58:56,994 - SegEDU - INFO - Time to build graph: 6.170761823654175 s
/usr/local/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:22: FutureWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  FutureWarning)
 2021-02-24 21:59:10,462 - SegEDU - INFO - Model restored from /usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models/best
 2021-02-24 21:59:11,021 - SegEDU - INFO - Segmenting /tmp/good_input.txt...
 2021-02-24 21:59:11,929 - SegEDU - INFO - Saving into /tmp/results/good_input.txt
root@f697ebdcda53:/opt/neural-edu-seg/neuralseg# cat /tmp/results/good_input.txt 
Judging from previous posts
this used to be a good place , but not any longer .

@arne-cl
Copy link
Member Author

arne-cl commented Feb 24, 2021 •
edited
Loading

Potential solution: PKU-TANGENT#1 (issue was closed, seemingly without fixing it).

I reported the issue again upstream: PKU-TANGENT#9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@arne-cl