Skip to content

[Bug] LiveCodeBench评测报错 #1930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
2 tasks done
AllenShow opened this issue Mar 10, 2025 · 0 comments
Open
2 tasks done

[Bug] LiveCodeBench评测报错 #1930

AllenShow opened this issue Mar 10, 2025 · 0 comments

Comments

@AllenShow
Copy link

先决条件

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

{'CUDA available': True,
'CUDA_HOME': '/usr/local/cuda',
'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0',
'GPU 0,1,2,3,4,5,6,7': 'NVIDIA A40',
'MMEngine': '0.10.4',
'MUSA available': False,
'NVCC': 'Cuda compilation tools, release 12.1, V12.1.66',
'OpenCV': '4.10.0',
'PyTorch': '2.5.1+cu124',
'PyTorch compiling details': 'PyTorch built with:\n'
' - GCC 9.3\n'
' - C++ Version: 201703\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2023.1-Product Build 20230303 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v3.5.3 (Git Hash '
'66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n'
' - OpenMP 201511 (a.k.a. OpenMP 4.5)\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - NNPACK is enabled\n'
' - CPU capability usage: AVX512\n'
' - CUDA Runtime 12.4\n'
' - NVCC architecture flags: '
'-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90\n'
' - CuDNN 90.1\n'
' - Magma 2.6.1\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, CUDA_VERSION=12.4, '
'CUDNN_VERSION=9.1.0, '
'CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, '
'CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 '
'-fabi-version=11 -fvisibility-inlines-hidden '
'-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
'-DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON '
'-DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK '
'-DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE '
'-O2 -fPIC -Wall -Wextra -Werror=return-type '
'-Werror=non-virtual-dtor -Werror=bool-operation '
'-Wnarrowing -Wno-missing-field-initializers '
'-Wno-type-limits -Wno-array-bounds '
'-Wno-unknown-pragmas -Wno-unused-parameter '
'-Wno-strict-overflow -Wno-strict-aliasing '
'-Wno-stringop-overflow -Wsuggest-override '
'-Wno-psabi -Wno-error=old-style-cast '
'-Wno-missing-braces -fdiagnostics-color=always '
'-faligned-new -Wno-unused-but-set-variable '
'-Wno-maybe-uninitialized -fno-math-errno '
'-fno-trapping-math -Werror=format '
'-Wno-stringop-overflow, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'TORCH_VERSION=2.5.1, USE_CUDA=ON, USE_CUDNN=ON, '
'USE_CUSPARSELT=1, USE_EXCEPTION_PTR=1, '
'USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, '
'USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, '
'USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, '
'USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n',
'Python': '3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0]',
'TorchVision': '0.20.1+cu124',
'lmdeploy': '0.7.1',
'numpy_random_seed': 2147483648,
'opencompass': '0.4.1+2176224',
'sys.platform': 'linux',
'transformers': '4.49.0'}

重现问题 - 代码/配置示例

python3 /nas/xz/opencompass/run.py --hf-path /nas/xz/model_para/Qwen2.5-7B --hf-type base --datasets livecodebench_gen --hf-num-gpus 8

重现问题 - 命令或脚本

评估过程报错，lcb_code_execution这一项只有1.46分

重现问题 - 错误信息

cat /nas/xz/LLaMA-Factory/saves/oc/Qwen2.5-7B-livecodebench_gen.log

INFO 03-10 20:14:29 [init.py:256] Automatically detected platform cuda.
03/10 20:14:31 - OpenCompass - INFO - Loading livecodebench_gen: /nas/xz/opencompass/opencompass/configs/./datasets/livecodebench/livecodebench_gen.py
03/10 20:14:31 - OpenCompass - INFO - Loading example: /nas/xz/opencompass/opencompass/configs/./summarizers/example.py
03/10 20:14:31 - OpenCompass - INFO - Current exp folder: outputs/default/20250310_201431
03/10 20:14:31 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/10 20:14:31 - OpenCompass - INFO - ./data/code_generation_lite does not exist!Start Download data automatically!If you have downloaded the data before,You can specific COMPASS_DATA_CACHE to avoid downloading~
Downloading http://opencompass.oss-cn-shanghai.aliyuncs.com/datasets/data/code_generation_lite.zip to /root/.cache/opencompass/data/code_generation_lite.zip
3.2/3.2 GB ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00
Extracting /root/.cache/opencompass/data/code_generation_lite.zip to /root/.cache/opencompass/data
lcb_code_generation test 400
lcb_code_generation train 400
03/10 20:20:18 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/execution-v2
lcb_code_execution test 479
lcb_code_execution train 479
03/10 20:20:18 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/test_generation
lcb_test_output test 442
lcb_test_output train 442
03/10 20:20:18 - OpenCompass - INFO - Partitioned into 1 tasks.
03/10 20:20:21 - OpenCompass - WARNING - Debug mode, log will be saved to tmp/3399630_debug.log
03/10 23:05:49 - OpenCompass - INFO - Partitioned into 3 tasks.
03/10 23:05:50 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/code_generation_lite
lcb_code_generation test 400
lcb_code_generation train 400
03/10 23:07:00 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/code_generation_lite
03/10 23:07:04 - OpenCompass - INFO - LCBCodeGeneration: Evaluating 400...
10%|▉ | 38/400 [00:08<01:12, 5.03it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'makeTheIntegerZero'"), <traceback object at 0x7f4b5be969c0>)
12%|█▏ | 49/400 [00:12<02:42, 2.16it/s]alarm went off
13%|█▎ | 53/400 [00:14<02:07, 2.71it/s]alarm went off
18%|█▊ | 70/400 [00:21<02:01, 2.72it/s]alarm went off
22%|██▎ | 90/400 [00:29<03:01, 1.71it/s]alarm went off
24%|██▎ | 94/400 [00:35<05:00, 1.02it/s]alarm went off
25%|██▌ | 101/400 [00:39<02:31, 1.98it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'minLengthAfterRemovals'"), <traceback object at 0x7f4b5be969c0>)
33%|███▎ | 132/400 [00:49<01:23, 3.21it/s]alarm went off
38%|███▊ | 154/400 [00:58<01:34, 2.59it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'minimumOperationsToMakeEqual'"), <traceback object at 0x7f499473e8c0>)
40%|███▉ | 158/400 [00:59<01:30, 2.67it/s]Fatal Python error: Segmentation fault

Current thread 0x00007f4b5c6bf740 (most recent call first):
File "", line 39 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
File "", line 44 in count_numbers
...

Extension modules: mkl._mklinit, mkl._py_mkl_service, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, yaml._yaml, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, _brotli, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.tslib, pandas._libs.lib, pandas._libs.hashing, pyarrow.lib, pandas._libs.ops, pyarrow._compute, pandas._libs.arrays, pandas._libs.index, pandas._libs.join, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.internals, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.tslibs.strptime, pandas._libs.groupby, pandas._libs.testing, pandas._libs.parsers, pandas._libs.json, regex._regex, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, sklearn.__check_build._check_build, psutil._psutil_linux, psutil._psutil_posix, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._highs.cython.src._highs_wrapper, scipy.optimize._highs._highs_wrapper, scipy.optimize._highs.cython.src._highs_constants, scipy.optimize._highs._highs_constants, scipy.linalg._interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._bspl, scipy.interpolate._ppoly, scipy.interpolate.interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._levy_stable.levyst, scipy.stats._stats_pythran, scipy._lib._uarray._uarray, scipy.stats._ansari_swilk_statistics, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.stats._unuran.unuran_wrapper, scipy.ndimage._nd_image, _ni_label, scipy.ndimage._ni_label, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, sklearn.utils._random, sklearn.utils._seq_dataset, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.utils._vector_sentinel, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.linear_model._sag_fast, pyarrow._parquet, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, xxhash._xxhash, pyarrow._json, markupsafe._speedups, PIL._imaging, gmpy2.gmpy2, PIL._imagingft, msgspec._core, zmq.backend.cython._zmq, msgpack._cmsgpack, google._upb._message, setproctitle, uvloop.loop, ray._raylet, sklearn.feature_extraction._hashing_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, websockets.speedups, tree_sitter._binding, tree_sitter_languages.core, rapidfuzz._feature_detector_cpp, rapidfuzz.distance._initialize_cpp, rapidfuzz.distance.metrics_cpp_avx2, rapidfuzz.fuzz_cpp_avx2, rapidfuzz.process_cpp_impl, rapidfuzz.utils_cpp, Levenshtein.levenshtein_cpp, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5o, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5l, h5py._selector (total: 256)
40%|████ | 161/400 [01:00<01:19, 3.00it/s]alarm went off
44%|████▍ | 177/400 [01:07<01:20, 2.77it/s]unable to get function error = (<class 'AttributeError'>, AttributeError("module 'tmp_sol' has no attribute 'maximumLength'"), <traceback object at 0x7f4b5be969c0>)
46%|████▌ | 182/400 [01:09<01:17, 2.83it/s]alarm went off
50%|█████ | 202/400 [01:14<00:54, 3.61it/s]alarm went off
100%|██████████| 400/400 [01:58<00:00, 3.37it/s]
03/10 23:09:03 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_code_generation]: {'extracted_predictions': {0: ['# YOUR CODE HERE'], 1: ['# YOUR CODE HERE'], 2: ['# YOUR CODE HERE'], 3: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n # Read the length of the string and the number of characters to be deleted\n n, k = map(int, input().split())\n \n # Read the string\n s = input()\n \n # Count the frequency of each character in the string\n freq = [0] * 26\n for char in s:\n freq[ord(char) - ord('a')] += 1\n \n # Count the number of characters with odd frequency\n odd_count = sum(1 for count in freq if count % 2 != 0)\n \n # Check if it is possible to remove exactly k characters to form a palindrome\n if odd_count <= k + 1:\n print("YES")\n else:\n print("NO")'], 4: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n # Read the size of the array and the number k\n n, k = map(int, input().split())\n \n # Read the array of integers\n a = list(map(int, input().split()))\n \n # Initialize the count of operations\n operations = 0\n \n # Iterate through the array\n for i in range(n):\n # If the current element is not divisible by k, increment the count of operations\n if a[i] % k != 0:\n operations += 1\n \n # Print the minimum number of operations needed\n print(operations)'], 5: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n # Read the integer n\n n = int(input())\n \n # Check if n is divisible by 3\n if n % 3 == 0:\n # If n is divisible by 3, Vanya wins\n print("First")\n else:\n # If n is not divisible by 3, Vova wins\n print("Second")'], 6: ['# Read the number of test cases\nt = int(input())\n\n# Iterate through each test case\nfor _ in range(t):\n # Read the number of boxes\n n = int(input())\n \n # Read the weights of the boxes\n a = list(map(int, input().split()))\n \n # Calculate the total weight of all boxes\n total_weight = sum(a)\n \n # Calculate the maximum absolute difference between the total weights of any two trucks\n max_diff = 0\n for k in range(1, n + 1):\n # Calculate the weight of each truck\n truck_weight = total_weight // k\n \n # Calculate the maximum and minimum weight of any two trucks\n max_weight = truck_weight * k\n min_weight = truck_weight * (k - 1)\n \n # Calculate the absolute difference between the maximum and minimum weight\n diff = abs(max_weight - min_weight)\n \n # Update the maximum absolute difference\n max_diff = max(max_diff, diff)\n \n # Print the maximum absolute difference\n print(max_diff)'], 7: ['# YOUR CODE HERE'], 8: ['# YOUR CODE HERE'], 9: ['class Solution:\n def countSeniors(self, details: List[str]) -> int:\n count = 0\n for detail in details:\n age = int(detail[11:13])\n if age > 60:\n count += 1\n return count'], 10: ['class Solution:\n def matrixSum(self, nums: List[List[int]]) -> int:\n for i in range(len(nums)):\n nums[i].sort(reverse=True)\n score = 0\n for j in range(len(nums[0])):\n max_num = 0\n for i in range(len(nums)):\n max_num = max(max_num, nums[i][j])\n score += max_num\n return score'], 11: ['class Solution:\n def maximumOr(self, nums: List[int], k: int) -> int:\n n = len(nums)\n left = [0] * n\n right = [0] * n\n left[0] = nums[0]\n right[n - 1] = nums[n - 1]\n for i in range(1, n):\n left[i] = left[i - 1] | nums[i]\n for i in range(n - 2, -1, -1):\n right[i] = right[i + 1] | nums[i]\n ans = 0\n for i in range(n):\n ans = max(ans, left[i - 1] | (nums[i] << k) | right[i + 1])\n return ans'], 12: ['class Solution:\n def maxStrength(self, nums: List[int]) -> int:\n if len(nums) == 1:\n return nums[0]\n nums.sort()\n if nums[-1] == 0:\n return 0\n if nums[0] == 0:\n return nums[-1]\n if nums[0] < 0 and nums[1] < 0:\n return nums[-1] * nums[-2] * nums[0]\n return nums[-1] * nums[-2]'], 13: ['class Solution:\n def minExtraChar(self, s: str, dictionary: List[str]) -> int:\n dp = [0] * (len(s) + 1)\n for i in range(len(s), -1, -1):\n dp[i] = dp[i + 1] + 1\n for word in dictionary:\n if s[i:i + len(word)] == word:\n dp[i] = min(dp[i], dp[i + len(word)])\n return dp[0]'], 14: ['class Solution:\n def buyChoco(self, prices: List[int], money: int) -> int:\n prices.sort()\n if prices[0] + prices[1] <= money:\n return money - (prices[0] + prices[1])\n else:\n return money'], 15: ['class Solution:\n def count(self, num1: str, num2: str, min_sum: int, max_sum: int) -> int:\n MOD = 10**9 + 7\n n1, n2 = len(num1), len(num2)\n dp = [[[0] * (max_sum + 1) for _ in range(2)] for _ in range(n1 + 1)]\n dp[0][0][0] = 1\n\n for i in range(n1):\n for j in range(2):\n for k in range(max_sum + 1):\n for d in range(10):\n if j == 0 and d > int(num1[i]):\n break\n if j == 1 and d > int(num2[i]):\n break\n dp[i + 1][j][k + d] += dp[i][j][k]\n dp[i + 1][j][k + d] %= MOD\n\n ans = 0\n for i in range(n1, n2 + 1):\n for j in range(2):\n for k in range(min_sum, max_sum + 1):\n ans += dp[i][j][k]\n ans %= MOD\n\n return ans'],
...
398: ['# YOUR CODE HERE'], 399: ['# YOUR CODE HERE']},

'eval_results': defaultdict(<class 'list'>, {0: [[-2]], 1: [[-2]], 2: [[-2]], 3: [[True, True, True, True]], 4: [[False]], 5: [[False]], 6: [[False]], 7: [[-2]], 8: [[-2]], 9: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 10: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 11: [[-1]], 12: [[False]], 13: [[-1]], 14: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 15: [[-1]], 16: [[True, -1]], 17: [[-1]], 18: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 19: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 20: [[False]], 21: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 22: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 23: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 24: [[False]], 25: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 26: [[False]], 27: [[False]], 28: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 29: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 30: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 31: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 32: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 33: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 34: [[True, True, True, False]], 35: [[True, True, True, True, True, True, True, True, True, True, True, True, True, -1]], 36: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 37: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 38: [[False]], 39: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 40: [[-2]], 41: [[-1]], 42: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 43: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 44: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 45: [[False]], 46: [[False]], 47: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 48: [[-1]], 49: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 50: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 51: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 52: [[True, False]], 53: [[False]], 54: [[False]], 55: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 56: [[False]], 57: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 58: [[True, True, True, True, True, True, True, -1]], 59: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 60: [[True, True, True, False]], 61: [[True, False]], 62: [[False]], 63: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 64: [[False]], 65: [[-2]], 66: [[True, True, True, True, True, True, True, True, True, True, True, True, False]], 67: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 68: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 69: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 70: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 71: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 72: [[False]], 73: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 74: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 75: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 76: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 77: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 78: [[True, True, True, True, False]], 79: [[False]], 80: [[False]], 81: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, -1]], 82: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 83: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 84: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 85: [[False]], 86: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 87: [[False]], 88: [[True, True, True, True, True, True, True, True, True, True, True, True, -1]], 89: [[-1]], 90: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 91: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 92: [[False]], 93: [[-1]], 94: [[True, False]], 95: [[False]], 96: [[-1]], 97: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 98: [[False]], 99: [[False]], 100: [[False]], 101: [[False]], 102: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 103: [[False]], 104: [[-2]], 105: [[False]], 106: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 107: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 108: [[True, True, True, True, True, False]], 109: [[False]], 110: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 111: [[False]], 112: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 113: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 114: [[False]], 115: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 116: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 117: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 118: [[False]], 119: [[False]], 120: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 121: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 122: [[True, True, True, True, True, False]], 123: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 124: [[True, True, True, True, True, True, True, True, True, True, True, -1]], 125: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 126: [[False]], 127: [[False]], 128: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 129: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 130: [[False]], 131: [[True, True, True, True, True, True, True, False]], 132: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 133: [[False]], 134: [[False]], 135: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 136: [[False]], 137: [[False]], 138: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 139: [[False]], 140: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 141: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 142: [[False]], 143: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 144: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 145: [[True, True, False]], 146: [[True, False]], 147: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 148: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 149: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 150: [[-1]], 151: [[True, False]], 152: [[False]], 153: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 154: [[False]], 155: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 156: [[-1]], 157: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 158: [[-2]], 159: [[True, False]], 160: [[-1]], 161: [[-2]], 162: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 163: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 164: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 165: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 166: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 167: [[True, False]], 168: [[-1]], 169: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 170: [[True, True, False]], 171: [[True, True, True, True, True, True, True, True, True, True, True, True, -1]], 172: [[False]], 173: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 174: [[False]], 175: [[False]], 176: [[False]], 177: [[True, True, True, True, True, True, True, False]], 178: [[False]], 179: [[-2]], 180: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 181: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 182: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 183: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 184: [[False]], 185: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 186: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 187: [[True, True, True, True, False]], 188: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 189: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 190: [[True, False]], 191: [[-2]], 192: [[-2]], 193: [[-2]], 194: [[-2]], 195: [[-2]], 196: [[-2]], 197: [[-2]], 198: [[-2]], 199: [[-2]], 200: [[-2]], 201: [[-2]], 202: [[False]], 203: [[-2]], 204: [[-2]], 205: [[False]], 206: [[-2]], 207: [[False]], 208: [[False]], 209: [[-2]], 210: [[-2]], 211: [[-2]], 212: [[-2]], 213: [[-2]], 214: [[-2]], 215: [[-2]], 216: [[-2]], 217: [[-2]], 218: [[-2]], 219: [[-2]], 220: [[-2]], 221: [[-2]], 222: [[True, True, True, True, True, True, True, True, True, True]], 223: [[-2]], 224: [[-2]], 225: [[False]], 226: [[-2]], 227: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 228: [[-1]], 229: [[False]], 230: [[-2]], 231: [[-2]], 232: [[-2]], 233: [[True, True, False]], 234: [[False]], 235: [[-2]], 236: [[-2]], 237: [[-2]], 238: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 239: [[-2]], 240: [[-2]], 241: [[-2]], 242: [[-2]], 243: [[-2]], 244: [[-2]], 245: [[-2]], 246: [[-2]], 247: [[-2]], 248: [[-2]], 249: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 250: [[-2]], 251: [[-2]], 252: [[-2]], 253: [[-2]], 254: [[-2]], 255: [[False]], 256: [[-2]], 257: [[False]], 258: [[-2]], 259: [[-2]], 260: [[-2]], 261: [[-2]], 262: [[-2]], 263: [[-2]], 264: [[-2]], 265: [[-2]], 266: [[True, True, True, True, True, True, True, -1]], 267: [[-2]], 268: [[-2]], 269: [[-2]], 270: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 271: [[-2]], 272: [[-2]], 273: [[-2]], 274: [[-2]], 275: [[-2]], 276: [[-2]], 277: [[-2]], 278: [[-2]], 279: [[-2]], 280: [[-2]], 281: [[-2]], 282: [[-2]], 283: [[-2]], 284: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 285: [[-2]], 286: [[-2]], 287: [[-2]], 288: [[-2]], 289: [[True, True, True, True, True, True, True, True, True, True, True]], 290: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 291: [[-1]], 292: [[-2]], 293: [[-2]], 294: [[-2]], 295: [[-2]], 296: [[-2]], 297: [[-2]], 298: [[-2]], 299: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 300: [[False]], 301: [[-2]], 302: [[-2]], 303: [[-2]], 304: [[-2]], 305: [[True, True, True, True, True, True, True, True, True, True, True]], 306: [[False]], 307: [[-2]], 308: [[-2]], 309: [[-2]], 310: [[-2]], 311: [[True, False]], 312: [[-2]], 313: [[False]], 314: [[-2]], 315: [[-2]], 316: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 317: [[-2]], 318: [[-2]], 319: [[-2]], 320: [[-2]], 321: [[True, True, True, True, True, True, True, True, True, True, True, True, True]], 322: [[-2]], 323: [[False]], 324: [[-2]], 325: [[-2]], 326: [[True, True, True, True, True, True, True, True, True]], 327: [[False]], 328: [[False]], 329: [[-2]], 330: [[-2]], 331: [[-2]], 332: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 333: [[True, True, False]], 334: [[-2]], 335: [[-2]], 336: [[-2]], 337: [[-2]], 338: [[True, False]], 339: [[-2]], 340: [[-2]], 341: [[-2]], 342: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 343: [[False]], 344: [[False]], 345: [[-2]], 346: [[-2]], 347: [[-2]], 348: [[-2]], 349: [[-2]], 350: [[-2]], 351: [[True, True, True, True, True, True, True, True, True, True, True, True]], 352: [[-2]], 353: [[-2]], 354: [[-2]], 355: [[-2]], 356: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 357: [[-2]], 358: [[-2]], 359: [[-2]], 360: [[-2]], 361: [[-2]], 362: [[-2]], 363: [[-2]], 364: [[True, True, True, True, True, True, True, True, True, True, True, True, True]], 365: [[-2]], 366: [[-2]], 367: [[-2]], 368: [[-2]], 369: [[True, True, True, False]], 370: [[True, True, True, True, True, True, True, True, True, True, True]], 371: [[-2]], 372: [[-2]], 373: [[-2]], 374: [[-2]], 375: [[True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]], 376: [[-2]], 377: [[False]], 378: [[-2]], 379: [[True, True, True, True, True, True, True, True, True, True, True, True]], 380: [[-2]], 381: [[-2]], 382: [[-2]], 383: [[-2]], 384: [[-2]], 385: [[-2]], 386: [[-2]], 387: [[-2]], 388: [[-2]], 389: [[-2]], 390: [[True, False]], 391: [[False]], 392: [[True, True, False]], 393: [[-2]], 394: [[-2]], 395: [[False]], 396: [[-2]], 397: [[-2]], 398: [[-2]], 399: [[-2]]}), 'pass@1': 27.75, 'detail': {'pass@1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 100.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0, 8: 0.0, 9: 100.0, 10: 100.0, 11: 0.0, 12: 0.0, 13: 0.0, 14: 100.0, 15: 0.0, 16: 0.0, 17: 0.0, 18: 100.0, 19: 100.0, 20: 0.0, 21: 100.0, 22: 100.0, 23: 100.0, 24: 0.0, 25: 100.0, 26: 0.0, 27: 0.0, 28: 100.0, 29: 100.0, 30: 100.0, 31: 100.0, 32: 100.0, 33: 100.0, 34: 0.0, 35: 0.0, 36: 100.0, 37: 100.0, 38: 0.0, 39: 100.0, 40: 0.0, 41: 0.0, 42: 100.0, 43: 100.0, 44: 100.0, 45: 0.0, 46: 0.0, 47: 100.0, 48: 0.0, 49: 100.0, 50: 100.0, 51: 100.0, 52: 0.0, 53: 0.0, 54: 0.0, 55: 100.0, 56: 0.0, 57: 100.0, 58: 0.0, 59: 100.0, 60: 0.0, 61: 0.0, 62: 0.0, 63: 100.0, 64: 0.0, 65: 0.0, 66: 0.0, 67: 100.0, 68: 100.0, 69: 100.0, 70: 100.0, 71: 100.0, 72: 0.0, 73: 100.0, 74: 100.0, 75: 100.0, 76: 100.0, 77: 100.0, 78: 0.0, 79: 0.0, 80: 0.0, 81: 0.0, 82: 100.0, 83: 100.0, 84: 100.0, 85: 0.0, 86: 100.0, 87: 0.0, 88: 0.0, 89: 0.0, 90: 100.0, 91: 100.0, 92: 0.0, 93: 0.0, 94: 0.0, 95: 0.0, 96: 0.0, 97: 100.0, 98: 0.0, 99: 0.0, 100: 0.0, 101: 0.0, 102: 100.0, 103: 0.0, 104: 0.0, 105: 0.0, 106: 100.0, 107: 100.0, 108: 0.0, 109: 0.0, 110: 100.0, 111: 0.0, 112: 100.0, 113: 100.0, 114: 0.0, 115: 100.0, 116: 100.0, 117: 100.0, 118: 0.0, 119: 0.0, 120: 100.0, 121: 100.0, 122: 0.0, 123: 100.0, 124: 0.0, 125: 100.0, 126: 0.0, 127: 0.0, 128: 100.0, 129: 100.0, 130: 0.0, 131: 0.0, 132: 100.0, 133: 0.0, 134: 0.0, 135: 100.0, 136: 0.0, 137: 0.0, 138: 100.0, 139: 0.0, 140: 100.0, 141: 100.0, 142: 0.0, 143: 100.0, 144: 100.0, 145: 0.0, 146: 0.0, 147: 100.0, 148: 100.0, 149: 100.0, 150: 0.0, 151: 0.0, 152: 0.0, 153: 100.0, 154: 0.0, 155: 100.0, 156: 0.0, 157: 100.0, 158: 0.0, 159: 0.0, 160: 0.0, 161: 0.0, 162: 100.0, 163: 100.0, 164: 100.0, 165: 100.0, 166: 100.0, 167: 0.0, 168: 0.0, 169: 100.0, 170: 0.0, 171: 0.0, 172: 0.0, 173: 100.0, 174: 0.0, 175: 0.0, 176: 0.0, 177: 0.0, 178: 0.0, 179: 0.0, 180: 100.0, 181: 100.0, 182: 100.0, 183: 100.0, 184: 0.0, 185: 100.0, 186: 100.0, 187: 0.0, 188: 100.0, 189: 100.0, 190: 0.0, 191: 0.0, 192: 0.0, 193: 0.0, 194: 0.0, 195: 0.0, 196: 0.0, 197: 0.0, 198: 0.0, 199: 0.0, 200: 0.0, 201: 0.0, 202: 0.0, 203: 0.0, 204: 0.0, 205: 0.0, 206: 0.0, 207: 0.0, 208: 0.0, 209: 0.0, 210: 0.0, 211: 0.0, 212: 0.0, 213: 0.0, 214: 0.0, 215: 0.0, 216: 0.0, 217: 0.0, 218: 0.0, 219: 0.0, 220: 0.0, 221: 0.0, 222: 100.0, 223: 0.0, 224: 0.0, 225: 0.0, 226: 0.0, 227: 100.0, 228: 0.0, 229: 0.0, 230: 0.0, 231: 0.0, 232: 0.0, 233: 0.0, 234: 0.0, 235: 0.0, 236: 0.0, 237: 0.0, 238: 100.0, 239: 0.0, 240: 0.0, 241: 0.0, 242: 0.0, 243: 0.0, 244: 0.0, 245: 0.0, 246: 0.0, 247: 0.0, 248: 0.0, 249: 100.0, 250: 0.0, 251: 0.0, 252: 0.0, 253: 0.0, 254: 0.0, 255: 0.0, 256: 0.0, 257: 0.0, 258: 0.0, 259: 0.0, 260: 0.0, 261: 0.0, 262: 0.0, 263: 0.0, 264: 0.0, 265: 0.0, 266: 0.0, 267: 0.0, 268: 0.0, 269: 0.0, 270: 100.0, 271: 0.0, 272: 0.0, 273: 0.0, 274: 0.0, 275: 0.0, 276: 0.0, 277: 0.0, 278: 0.0, 279: 0.0, 280: 0.0, 281: 0.0, 282: 0.0, 283: 0.0, 284: 100.0, 285: 0.0, 286: 0.0, 287: 0.0, 288: 0.0, 289: 100.0, 290: 100.0, 291: 0.0, 292: 0.0, 293: 0.0, 294: 0.0, 295: 0.0, 296: 0.0, 297: 0.0, 298: 0.0, 299: 100.0, 300: 0.0, 301: 0.0, 302: 0.0, 303: 0.0, 304: 0.0, 305: 100.0, 306: 0.0, 307: 0.0, 308: 0.0, 309: 0.0, 310: 0.0, 311: 0.0, 312: 0.0, 313: 0.0, 314: 0.0, 315: 0.0, 316: 100.0, 317: 0.0, 318: 0.0, 319: 0.0, 320: 0.0, 321: 100.0, 322: 0.0, 323: 0.0, 324: 0.0, 325: 0.0, 326: 100.0, 327: 0.0, 328: 0.0, 329: 0.0, 330: 0.0, 331: 0.0, 332: 100.0, 333: 0.0, 334: 0.0, 335: 0.0, 336: 0.0, 337: 0.0, 338: 0.0, 339: 0.0, 340: 0.0, 341: 0.0, 342: 100.0, 343: 0.0, 344: 0.0, 345: 0.0, 346: 0.0, 347: 0.0, 348: 0.0, 349: 0.0, 350: 0.0, 351: 100.0, 352: 0.0, 353: 0.0, 354: 0.0, 355: 0.0, 356: 100.0, 357: 0.0, 358: 0.0, 359: 0.0, 360: 0.0, 361: 0.0, 362: 0.0, 363: 0.0, 364: 100.0, 365: 0.0, 366: 0.0, 367: 0.0, 368: 0.0, 369: 0.0, 370: 100.0, 371: 0.0, 372: 0.0, 373: 0.0, 374: 0.0, 375: 100.0, 376: 0.0, 377: 0.0, 378: 0.0, 379: 100.0, 380: 0.0, 381: 0.0, 382: 0.0, 383: 0.0, 384: 0.0, 385: 0.0, 386: 0.0, 387: 0.0, 388: 0.0, 389: 0.0, 390: 0.0, 391: 0.0, 392: 0.0, 393: 0.0, 394: 0.0, 395: 0.0, 396: 0.0, 397: 0.0, 398: 0.0, 399: 0.0}}}
03/10 23:09:05 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/execution-v2
lcb_code_execution test 479
lcb_code_execution train 479
03/10 23:09:16 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_code_execution]: {'pass@1': 1.4613778705636742}
03/10 23:09:16 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/test_generation
lcb_test_output test 442
lcb_test_output train 442
100%|██████████| 442/442 [00:00<00:00, 38219.65it/s]
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 0)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 0)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 0)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
invalid syntax (, line 1)
03/10 23:09:16 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_test_output]: {'pass@1': 40.04524886877828, 'detail': {'pass@1': {0: 0.0, 1: 0.0, 2: 0.0, 3: 100.0, 4: 0.0, 5: 0.0, 6: 0.0, 7: 0.0, 8: 100.0, 9: 100.0, 10: 0.0, 11: 0.0, 12: 0.0, 13: 0.0, 14: 0.0, 15: 0.0, 16: 100.0, 17: 100.0, 18: 0.0, 19: 0.0, 20: 0.0, 21: 100.0, 22: 0.0, 23: 100.0, 24: 0.0, 25: 100.0, 26: 100.0, 27: 0.0, 28: 100.0, 29: 0.0, 30: 0.0, 31: 100.0, 32: 0.0, 33: 0.0, 34: 0.0, 35: 0.0, 36: 0.0, 37: 0.0, 38: 0.0, 39: 0.0, 40: 100.0, 41: 0.0, 42: 0.0, 43: 100.0, 44: 100.0, 45: 0.0, 46: 0.0, 47: 0.0, 48: 0.0, 49: 0.0, 50: 100.0, 51: 100.0, 52: 100.0, 53: 100.0, 54: 0.0, 55: 100.0, 56: 0.0, 57: 100.0, 58: 0.0, 59: 0.0, 60: 0.0, 61: 0.0, 62: 100.0, 63: 100.0, 64: 0.0, 65: 0.0, 66: 0.0, 67: 0.0, 68: 0.0, 69: 100.0, 70: 0.0, 71: 0.0, 72: 0.0, 73: 0.0, 74: 0.0, 75: 0.0, 76: 100.0, 77: 0.0, 78: 100.0, 79: 100.0, 80: 100.0, 81: 100.0, 82: 100.0, 83: 0.0, 84: 0.0, 85: 0.0, 86: 0.0, 87: 100.0, 88: 0.0, 89: 0.0, 90: 100.0, 91: 0.0, 92: 100.0, 93: 100.0, 94: 0.0, 95: 100.0, 96: 0.0, 97: 100.0, 98: 0.0, 99: 100.0, 100: 0.0, 101: 100.0, 102: 0.0, 103: 100.0, 104: 100.0, 105: 100.0, 106: 100.0, 107: 100.0, 108: 0.0, 109: 0.0, 110: 0.0, 111: 100.0, 112: 100.0, 113: 100.0, 114: 0.0, 115: 0.0, 116: 0.0, 117: 100.0, 118: 0.0, 119: 0.0, 120: 0.0, 121: 0.0, 122: 0.0, 123: 0.0, 124: 0.0, 125: 100.0, 126: 0.0, 127: 0.0, 128: 0.0, 129: 0.0, 130: 0.0, 131: 100.0, 132: 0.0, 133: 0.0, 134: 100.0, 135: 100.0, 136: 100.0, 137: 0.0, 138: 0.0, 139: 100.0, 140: 100.0, 141: 0.0, 142: 0.0, 143: 0.0, 144: 100.0, 145: 100.0, 146: 0.0, 147: 0.0, 148: 100.0, 149: 0.0, 150: 100.0, 151: 100.0, 152: 0.0, 153: 0.0, 154: 0.0, 155: 0.0, 156: 100.0, 157: 100.0, 158: 100.0, 159: 100.0, 160: 100.0, 161: 100.0, 162: 100.0, 163: 100.0, 164: 0.0, 165: 0.0, 166: 100.0, 167: 100.0, 168: 100.0, 169: 0.0, 170: 0.0, 171: 0.0, 172: 0.0, 173: 100.0, 174: 100.0, 175: 100.0, 176: 100.0, 177: 0.0, 178: 100.0, 179: 0.0, 180: 100.0, 181: 0.0, 182: 0.0, 183: 100.0, 184: 0.0, 185: 100.0, 186: 100.0, 187: 100.0, 188: 0.0, 189: 100.0, 190: 100.0, 191: 0.0, 192: 0.0, 193: 0.0, 194: 0.0, 195: 0.0, 196: 0.0, 197: 100.0, 198: 0.0, 199: 0.0, 200: 0.0, 201: 0.0, 202: 100.0, 203: 0.0, 204: 0.0, 205: 100.0, 206: 0.0, 207: 0.0, 208: 100.0, 209: 100.0, 210: 100.0, 211: 100.0, 212: 100.0, 213: 0.0, 214: 100.0, 215: 0.0, 216: 0.0, 217: 100.0, 218: 100.0, 219: 0.0, 220: 0.0, 221: 100.0, 222: 0.0, 223: 100.0, 224: 0.0, 225: 0.0, 226: 0.0, 227: 100.0, 228: 100.0, 229: 0.0, 230: 0.0, 231: 0.0, 232: 0.0, 233: 100.0, 234: 0.0, 235: 100.0, 236: 0.0, 237: 100.0, 238: 0.0, 239: 0.0, 240: 0.0, 241: 0.0, 242: 0.0, 243: 100.0, 244: 100.0, 245: 0.0, 246: 0.0, 247: 100.0, 248: 100.0, 249: 0.0, 250: 0.0, 251: 0.0, 252: 100.0, 253: 0.0, 254: 0.0, 255: 0.0, 256: 0.0, 257: 0.0, 258: 100.0, 259: 100.0, 260: 100.0, 261: 0.0, 262: 100.0, 263: 0.0, 264: 0.0, 265: 100.0, 266: 0.0, 267: 100.0, 268: 0.0, 269: 0.0, 270: 0.0, 271: 100.0, 272: 0.0, 273: 0.0, 274: 0.0, 275: 0.0, 276: 0.0, 277: 0.0, 278: 100.0, 279: 100.0, 280: 0.0, 281: 100.0, 282: 100.0, 283: 0.0, 284: 0.0, 285: 0.0, 286: 0.0, 287: 100.0, 288: 0.0, 289: 0.0, 290: 100.0, 291: 0.0, 292: 100.0, 293: 100.0, 294: 0.0, 295: 0.0, 296: 0.0, 297: 100.0, 298: 0.0, 299: 100.0, 300: 0.0, 301: 0.0, 302: 0.0, 303: 0.0, 304: 0.0, 305: 0.0, 306: 0.0, 307: 100.0, 308: 0.0, 309: 100.0, 310: 100.0, 311: 100.0, 312: 0.0, 313: 100.0, 314: 0.0, 315: 100.0, 316: 100.0, 317: 0.0, 318: 100.0, 319: 100.0, 320: 100.0, 321: 0.0, 322: 100.0, 323: 0.0, 324: 0.0, 325: 100.0, 326: 100.0, 327: 100.0, 328: 100.0, 329: 100.0, 330: 0.0, 331: 0.0, 332: 0.0, 333: 100.0, 334: 100.0, 335: 100.0, 336: 0.0, 337: 100.0, 338: 100.0, 339: 100.0, 340: 0.0, 341: 100.0, 342: 100.0, 343: 0.0, 344: 0.0, 345: 0.0, 346: 100.0, 347: 100.0, 348: 100.0, 349: 100.0, 350: 0.0, 351: 0.0, 352: 100.0, 353: 0.0, 354: 0.0, 355: 0.0, 356: 0.0, 357: 0.0, 358: 0.0, 359: 0.0, 360: 100.0, 361: 100.0, 362: 0.0, 363: 0.0, 364: 100.0, 365: 0.0, 366: 0.0, 367: 100.0, 368: 100.0, 369: 0.0, 370: 0.0, 371: 0.0, 372: 0.0, 373: 100.0, 374: 0.0, 375: 0.0, 376: 0.0, 377: 0.0, 378: 100.0, 379: 100.0, 380: 0.0, 381: 0.0, 382: 100.0, 383: 100.0, 384: 100.0, 385: 100.0, 386: 0.0, 387: 0.0, 388: 100.0, 389: 0.0, 390: 100.0, 391: 0.0, 392: 0.0, 393: 100.0, 394: 0.0, 395: 0.0, 396: 100.0, 397: 0.0, 398: 0.0, 399: 0.0, 400: 0.0, 401: 0.0, 402: 0.0, 403: 100.0, 404: 0.0, 405: 0.0, 406: 100.0, 407: 100.0, 408: 0.0, 409: 0.0, 410: 0.0, 411: 0.0, 412: 100.0, 413: 0.0, 414: 0.0, 415: 0.0, 416: 0.0, 417: 100.0, 418: 0.0, 419: 0.0, 420: 0.0, 421: 0.0, 422: 100.0, 423: 0.0, 424: 100.0, 425: 0.0, 426: 0.0, 427: 0.0, 428: 0.0, 429: 0.0, 430: 100.0, 431: 0.0, 432: 0.0, 433: 0.0, 434: 100.0, 435: 0.0, 436: 100.0, 437: 0.0, 438: 0.0, 439: 100.0, 440: 0.0, 441: 0.0}}}
dataset version metric mode Qwen2.5-7B_hf

lcb_code_generation f0ed6c pass@1 gen 27.75
lcb_code_execution 24f99f pass@1 gen 1.46
lcb_test_output 1fe37c pass@1 gen 40.05
03/10 23:09:16 - OpenCompass - INFO - write summary to /nas/xz/LLaMA-Factory/saves/oc/outputs/default/20250310_201431/summary/summary_20250310_201431.txt
03/10 23:09:16 - OpenCompass - INFO - write csv to /nas/xz/LLaMA-Factory/saves/oc/outputs/default/20250310_201431/summary/summary_20250310_201431.csv

The markdown format results is as below:

dataset version metric mode Qwen2.5-7B_hf
lcb_code_generation f0ed6c pass@1 gen 27.75
lcb_code_execution 24f99f pass@1 gen 1.46
lcb_test_output 1fe37c pass@1 gen 40.05

03/10 23:09:16 - OpenCompass - INFO - write markdown summary to /nas/xz/LLaMA-Factory/saves/oc/outputs/default/20250310_201431/summary/summary_20250310_201431.md

其他信息

cat tmp/3399630_debug.log

03/10 20:20:24 - OpenCompass - INFO - Task [Qwen2.5-7B_hf/lcb_code_generation,Qwen2.5-7B_hf/lcb_code_execution,Qwen2.5-7B_hf/lcb_test_output]
INFO 03-10 20:20:27 [init.py:256] Automatically detected platform cuda.
Sliding Window Attention is enabled but not implemented for sdpa; unexpected results may be encountered.
Loading checkpoint shards: 100%|██████████| 4/4 [00:05<00:00, 1.27s/it]
03/10 20:20:37 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/code_generation_lite
lcb_code_generation test 400
lcb_code_generation train 400
03/10 20:21:49 - OpenCompass - INFO - Start inferencing [Qwen2.5-7B_hf/lcb_code_generation]
[2025-03-10 20:21:50,415] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-10 20:21:50,416] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|██████████| 50/50 [15:04<00:00, 18.10s/it]
03/10 20:36:55 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/execution-v2
lcb_code_execution test 479
lcb_code_execution train 479
03/10 20:36:55 - OpenCompass - INFO - Start inferencing [Qwen2.5-7B_hf/lcb_code_execution]
[2025-03-10 20:36:55,583] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-10 20:36:55,583] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|██████████| 60/60 [1:32:48<00:00, 92.81s/it]
03/10 22:09:44 - OpenCompass - INFO - Try to load the data from /root/.cache/opencompass/./data/test_generation
lcb_test_output test 442
lcb_test_output train 442
03/10 22:09:44 - OpenCompass - INFO - Start inferencing [Qwen2.5-7B_hf/lcb_test_output]
[2025-03-10 22:09:44,358] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-10 22:09:44,358] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|██████████| 56/56 [56:02<00:00, 60.04s/it]
03/10 23:05:46 - OpenCompass - INFO - time elapsed: 9922.02s
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@AllenShow