Error during the final stage of the DepthMap node

[yerlyj]

I've been using Meshroom quite successfully to generate 3D models from various objects. However, I've encountered an issue specifically with the DepthMap node when processing a certain set of images. To provide some context, I have captured two different sets of images of the same object using the same camera. Interestingly, I can successfully create a 3D model using one set of images, but I encounter problems with the other set.

The issue arises during the DepthMap node's execution. It appears to progress without any apparent issues until it reaches the final stage, where it ultimately fails. I receive the following error message in the terminal at this point. I'm hoping to understand what might be causing this error and how I can address it to successfully generate a 3D model from this particular set of images.

ERROR:root:Error on node computation: Error on node "DepthMap_1(17)":
Log:
[2023-09-28 22:08:40.329346] [0x00007f4aeb047000] [trace] Embedded OCIO configuration file: '/home/tesla/Documents/Meshroom/Meshroom-2023.2.0-av3.1.0-centos7-cuda11.3.1/aliceVision/share/aliceVision/config.ocio' found.
Program called with the following parameters:

• autoAdjustSmallImage = 1
• chooseTCamsPerTile = 1
• colorOptimizationEnabled = 1
• colorOptimizationNbIterations = 100
• customPatchPatternGroupSubpartsPerLevel = 0 (default)
• customPatchPatternSubparts = Unknown Type "St6vectorIN11aliceVision8depthMap24CustomPatchPatternParams13SubpartParamsESaIS3_EE" (default)
• downscale = 2
• exportIntermediateCrossVolumes = 0
• exportIntermediateDepthSimMaps = 0
• exportIntermediateNormalMaps = 0
• exportIntermediateTopographicCutVolumes = 0
• exportIntermediateVolume9pCsv = 0
• exportIntermediateVolumes = 0
• exportTilePattern = 0
• imagesFolder = "/mnt/MountFilerch/RAD/CIBM_CVMR/JeromeYBackup/Transfer/Meshroom/Projects/Tree/Series5/MeshroomCache/PrepareDenseScene/74105d6015557c3c23c9d51da9639611a4673991"
• input = "/mnt/MountFilerch/RAD/CIBM_CVMR/JeromeYBackup/Transfer/Meshroom/Projects/Tree/Series5/MeshroomCache/StructureFromMotion/d434aefeb671d01aa4a676cd3869ab91af853fce/sfm.abc"
• maxCoresAvailable = Unknown Type "j"
• maxMemoryAvailable = 9223372036854771712
• maxTCams = 10
• maxViewAngle = 70
• minViewAngle = 2
• nbGPUs = 0
• output = "/mnt/MountFilerch/RAD/CIBM_CVMR/JeromeYBackup/Transfer/Meshroom/Projects/Tree/Series5/MeshroomCache/DepthMap/41c2f0de203fc753df7013c9f86297ded9fbc3e6"
• rangeSize = 3
• rangeStart = 51
• refineEnabled = 1
• refineGammaC = 15.5
• refineGammaP = 8
• refineHalfNbDepths = 15
• refineInterpolateMiddleDepth = 0
• refineMaxTCamsPerTile = 4
• refineScale = 1
• refineSigma = 15
• refineStepXY = 1
• refineSubsampling = 10
• refineUseConsistentScale = 0
• refineUseCustomPatchPattern = 0
• refineWSH = 3
• sgmDepthListPerTile = 1
• sgmDepthThicknessInflate = 0
• sgmFilteringAxes = "YX"
• sgmGammaC = 5.5
• sgmGammaP = 8
• sgmMaxDepths = 1500
• sgmMaxSimilarity = 1
• sgmMaxTCamsPerTile = 4
• sgmP1 = 10
• sgmP2Weighting = 100
• sgmScale = 2
• sgmSeedsRangeInflate = 0.2
• sgmStepXY = 2
• sgmStepZ = -1
• sgmUseConsistentScale = 0
• sgmUseCustomPatchPattern = 0
• sgmUseSfmSeeds = 1
• sgmWSH = 4
• tileBufferHeight = 1024
• tileBufferWidth = 1024
• tilePadding = 64
• verboseLevel = "info"

Hardware :
Detected core count : 96
User upper limit on core count : 96
OpenMP will use 96 cores
Detected available memory : 492036 Mo

[22:08:40.334644][info] CUDA-Enabled GPU.
Device information:
- id: 0
- name: NVIDIA RTX A6000
- compute capability: 8.6
- clock frequency (kHz): 1800000
- total device memory: 48676 MB
- device memory available: 44318 MB
- per-block shared memory: 49152
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 1536
- max block sizes: {1024,1024,64}
- max grid sizes: {2147483647,65535,65535}
- max 2D array texture: {131072,65536}
- max 3D array texture: {16384,16384,16384}
- max 2D linear texture: {131072,65000,2097120}
- max 2D layered texture: {32768,32768,2048}
- number of SM(x)s: 84
- registers per SM(x): 65536
- registers per block: 65536
- concurrent kernels: yes
- mapping host memory: yes
- unified addressing: yes
- texture alignment: 512 byte
- pitch alignment: 32 byte
Device information:
- id: 1
- name: NVIDIA RTX A6000
- compute capability: 8.6
- clock frequency (kHz): 1800000
- total device memory: 48669 MB
- device memory available: 48352 MB
- per-block shared memory: 49152
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 1536
- max block sizes: {1024,1024,64}
- max grid sizes: {2147483647,65535,65535}
- max 2D array texture: {131072,65536}
- max 3D array texture: {16384,16384,16384}
- max 2D linear texture: {131072,65000,2097120}
- max 2D layered texture: {32768,32768,2048}
- number of SM(x)s: 84
- registers per SM(x): 65536
- registers per block: 65536
- concurrent kernels: yes
- mapping host memory: yes
- unified addressing: yes
- texture alignment: 512 byte
- pitch alignment: 32 byte
CUDA build version: 11.3
[22:08:40.591272][info] Supported CUDA-Enabled GPU detected.
[22:08:41.092089][info] Found 1 image dimension(s):
[22:08:41.092180][info] - [1800x1200]
[22:08:41.632947][info] Overall maximum dimension: [1800x1200]
[22:08:41.632980][warning] Tile buffer size (width: 1024, height: 1024) is larger than the maximum image size (width: 900, height: 600).
[22:08:41.633002][warning] Single tile computation, override SGM maximum number of T cameras per tile (before: 4, now: 10).
[22:08:41.633005][warning] Single tile computation, override Refine maximum number of T cameras per tile (before: 4, now: 10).
[22:08:41.633009][warning] Single tile computation, override SGM step XY (before: 2, now: 1).
[22:08:41.633079][info] Tiling information:
- parameters:
- buffer width: 1024 px
- buffer height: 1024 px
- padding: 64 px
- maximum downscale: 2
- maximum image width: 900 px
- maximum image height: 600 px
[22:08:41.633096][info] Maximum image size is smaller than one tile, use only one tile.
[22:08:41.633115][info] SGM parameters:
- scale: 2
- stepXY: 1
[22:08:41.633143][info] Refine parameters:
- scale: 1
- stepXY: 1
[22:08:41.633399][info] Number of GPU devices: 2, number of CPU threads: 96
[22:08:41.633646][info] CPU thread 0 (of 2) uses CUDA device: 0
[22:08:41.633732][info] CPU thread 1 (of 2) uses CUDA device: 1
[22:08:41.660633][info] Device memory:
- available: 35453 MB
- requirement for the first tile: 919.845 MB
- # computation buffers per tile: 851.867 MB (Sgm: 757.867 MB, Refine: 94 MB)
- # input images (R + 10 Ts): 67.9779 MB (single mipmap image size: 6.17981 MB)
[22:08:41.660706][info] Parallelization:
- # tiles per image: 1
- # simultaneous depth maps computation: 38
- # simultaneous tiles computation: 38
aliceVision_depthMapEstimation: /opt/AliceVision_git/src/aliceVision/depthMap/cuda/host/DeviceStreamManager.cpp:17: aliceVision::depthMap::DeviceStreamManager::DeviceStreamManager(int): Assertion `nbStreams > 0' failed.

WARNING:root:Downgrade status on node "DepthMapFilter_1(0)" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "DepthMapFilter_1(1)" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "DepthMapFilter_1(2)" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "DepthMapFilter_1(3)" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "DepthMapFilter_1(4)" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "DepthMapFilter_1(5)" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "Meshing_1" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "MeshFiltering_1" from Status.SUBMITTED to Status.NONE
WARNING:root:Downgrade status on node "Texturing_1" from Status.SUBMITTED to Status.NONE
WARNING:root:QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-tesla'

[natowi]

I think the issue can be found in this section:

[22:08:41.092089][info] Found 1 image dimension(s):
[22:08:41.092180][info] - [1800x1200]
[22:08:41.632947][info] Overall maximum dimension: [1800x1200]
[22:08:41.632980][warning] Tile buffer size (width: 1024, height: 1024) is larger than the maximum image size (width: 900, height: 600).
[22:08:41.633002][warning] Single tile computation, override SGM maximum number of T cameras per tile (before: 4, now: 10).
[22:08:41.633005][warning] Single tile computation, override Refine maximum number of T cameras per tile (before: 4, now: 10).
[22:08:41.633009][warning] Single tile computation, override SGM step XY (before: 2, now: 1).
[22:08:41.633079][info] Tiling information:

• parameters:
• buffer width: 1024 px
• buffer height: 1024 px
• padding: 64 px
• maximum downscale: 2
• maximum image width: 900 px
• maximum image height: 600 px
  [22:08:41.633096][info] Maximum image size is smaller than one tile, use only one tile.

What are your tiling settings? Do they differ for both datasets?

[yerlyj]

My tiling settings are identical for both datasets, one that was successful and the other that encountered issues (see the attached screenshots). I've also ensured that my settings match the ones you provided. However, I noticed that in my version of Meshroom (2023.2.0), there's no 'Auto Adjust Small Image' checkbox. I've attempted to adjust the tiling settings, suspecting that the issue might be related to them, but I haven't found any settings that resolve the problem. Any insights or suggestions would be greatly appreciated.

Successful DepthMap node computation

Failed DepthMap node computation

[natowi]

Auto Adjust Small Image is an "advanced" option (three dots icon in the upper right corner of ode settings). This setting Automatically adjusts depth map parameters if images are smaller than one tile.

I don´t have any ideas on the topic at the moment. Maybe try another downscale factor?