forked from jayleicn/singularity
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtest_ret.err
97 lines (80 loc) · 7.84 KB
/
test_ret.err
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_train_1.json: 0%| | 0/9155 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_train_1.json: 100%|██████████| 9155/9155 [00:00<00:00, 455767.33it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_contact_swap.json: 0%| | 0/184 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_contact_swap.json: 100%|██████████| 184/184 [00:00<00:00, 272607.54it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_contact_swap_mani.json: 0%| | 0/184 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_contact_swap_mani.json: 100%|██████████| 184/184 [00:00<00:00, 531509.60it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_action_swap.json: 0%| | 0/62 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_action_swap.json: 100%|██████████| 62/62 [00:00<00:00, 440011.59it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_action_swap_mani.json: 0%| | 0/62 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_temporal_action_swap_mani.json: 100%|██████████| 62/62 [00:00<00:00, 391047.89it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_same_entity.json: 0%| | 0/102 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_same_entity.json: 100%|██████████| 102/102 [00:00<00:00, 447978.02it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_same_entity_mani.json: 0%| | 0/102 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_same_entity_mani.json: 100%|██████████| 102/102 [00:00<00:00, 490618.13it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_diff_entity.json: 0%| | 0/35 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_diff_entity.json: 100%|██████████| 35/35 [00:00<00:00, 367921.40it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_diff_entity_mani.json: 0%| | 0/35 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_neighborhood_diff_entity_mani.json: 100%|██████████| 35/35 [00:00<00:00, 369774.91it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_spatial.json: 0%| | 0/935 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_spatial.json: 100%|██████████| 935/935 [00:00<00:00, 316493.76it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_spatial_mani.json: 0%| | 0/935 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_spatial_mani.json: 100%|██████████| 935/935 [00:00<00:00, 539506.71it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_contact.json: 0%| | 0/1008 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_contact.json: 100%|██████████| 1008/1008 [00:00<00:00, 379622.74it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_contact_mani.json: 0%| | 0/1008 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_contact_mani.json: 100%|██████████| 1008/1008 [00:00<00:00, 535850.24it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_action.json: 0%| | 0/1168 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_action.json: 100%|██████████| 1168/1168 [00:00<00:00, 388959.67it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_action_mani.json: 0%| | 0/1168 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_action_mani.json: 100%|██████████| 1168/1168 [00:00<00:00, 535287.05it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_attribute.json: 0%| | 0/818 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_attribute.json: 100%|██████████| 818/818 [00:00<00:00, 541670.46it/s]
Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_attribute_mani.json: 0%| | 0/818 [00:00<?, ?it/s]Loading /home/wiss/zhang/Jinhe/singularity/Data/anetqa/anet_ret_counter_attribute_mani.json: 100%|██████████| 818/818 [00:00<00:00, 538524.67it/s]
Traceback (most recent call last):
File "tasks/retrieval.py", line 217, in <module>
main(cfg)
File "tasks/retrieval.py", line 145, in main
model_without_ddp, test_loader, tokenizer, device, config, prefix=test_name)
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/home/wiss/zhang/Jinhe/singularity/tasks/retrieval_utils.py", line 69, in evaluation_wrapper
i2t_x, t2i_x, i2t_emb, t2i_emb = eval_func(model, data_loader, tokenizer, device, config)
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
return func(*args, **kwargs)
File "/home/wiss/zhang/Jinhe/singularity/tasks/retrieval_utils.py", line 197, in evaluation
i2t_scores, t2i_scores = model.get_sim(_pooled_image_feats, text_feats[:, 0])
File "/home/wiss/zhang/Jinhe/singularity/models/model_retrieval_base.py", line 330, in get_sim
raise Exception('Unknown manipulation type')
Exception: Unknown manipulation type
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2057526) of binary: /home/wiss/zhang/anaconda3/envs/probe-sl/bin/python
Traceback (most recent call last):
File "/home/wiss/zhang/anaconda3/envs/probe-sl/bin/torchrun", line 33, in <module>
sys.exit(load_entry_point('torch==1.10.1', 'console_scripts', 'torchrun')())
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 345, in wrapper
return f(*args, **kwargs)
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/distributed/run.py", line 719, in main
run(args)
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/distributed/run.py", line 713, in run
)(*cmd_args)
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/wiss/zhang/anaconda3/envs/probe-sl/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 261, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
tasks/retrieval.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2023-10-19_12:40:27
host : worker-1
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 2057526)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================