Error trying to diff very large objects

[misaunde]

Hi! When trying to diff very large json objects, it reaches the maximum recursion depth and fails for me with this error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 862, in _diff_level
    return self.compare_list(level, drill)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 693, in compare_list
    return self.compare_list_with_order(level, drill)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 552, in compare_list_with_order
    score = self._compare_list_with_order(level, drill)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 420, in _compare_list_with_order
    lcs_pair_list = self.generate_lcs_pair_list(level)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 356, in generate_lcs_pair_list
    return self._generate_lcs_pair_list(level, left_size, right_size, dp_table)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 293, in _generate_lcs_pair_list
    return self._generate_lcs_pair_list(level, left_size - 1, right_size - 1, dp_table) + [
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 293, in _generate_lcs_pair_list
    return self._generate_lcs_pair_list(level, left_size - 1, right_size - 1, dp_table) + [
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 293, in _generate_lcs_pair_list
    return self._generate_lcs_pair_list(level, left_size - 1, right_size - 1, dp_table) + [
  [Previous line repeated 970 more times]
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 284, in _generate_lcs_pair_list
    if self.diff_level(TreeLevel(
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 887, in diff_level
    cache_key = f"[{level.get_key()}]@[{drill}]"
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 82, in get_key
    return f"{make_json_path_key(self.left_path)}/{make_json_path_key(self.right_path)}"
  File "/usr/local/lib/python3.9/site-packages/jycm/helper.py", line 24, in make_json_path_key
    return "->".join([f"[{v}]" if isinstance(v, int) else v for v in path_list])
  File "/usr/local/lib/python3.9/site-packages/jycm/helper.py", line 24, in <listcomp>
    return "->".join([f"[{v}]" if isinstance(v, int) else v for v in path_list])
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/[email protected]/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/site-packages/jycm/__main__.py", line 130, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/jycm/__main__.py", line 126, in main
    return run(left, right, rules, output, show)
  File "/usr/local/lib/python3.9/site-packages/jycm/__main__.py", line 59, in run
    same, result = diff_two_json_with_rules(left, right, rules)
  File "/usr/local/lib/python3.9/site-packages/jycm/__main__.py", line 39, in diff_two_json_with_rules
    same = ycm.diff()
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 914, in diff
    return self.diff_level(level=root_level, drill=False) == 1
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 890, in diff_level
    score = self._diff_level(level, drill)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 869, in _diff_level
    raise e
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 865, in _diff_level
    return self.compare_dict(level, drill)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 739, in compare_dict
    _score = self.diff_level(TreeLevel(
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 890, in diff_level
    score = self._diff_level(level, drill)
  File "/usr/local/lib/python3.9/site-packages/jycm/jycm.py", line 871, in _diff_level
    raise DiffLevelException(f"Error {e} [drill={drill}] when compare [{level}]")
jycm.jycm.DiffLevelException: Error maximum recursion depth exceeded while calling a Python object [drill=False] when compare 

I guess my json blob is big enough that even this didn't make a difference:
https://stackoverflow.com/a/3323013/3806701

Any ideas?

[eggachecat]

HI! Thanks for this performance issue and the reference!
Let me take a look soon..