Is it possible to use a better resolution than QQVGA?

[senceryucel]

Hey, @meenchen

Like you have written in the person_detection_demo script,

sensor.set_framesize(sensor.LCD)  # Set frame size to QVGA 160x128

we made our inference on QQVGA (160x128):

sensor.set_framesize(sensor.QQVGA)

(since we use the frame buffer instead of an LCD). It worked very nice. However, I tried to test it with higher resolutions but there did not occur any detection. I was wondering if it is possible to have a higher resolution with using this engine. Thanks a lot in advance.

[meenchen]

Hi @senceryucel,

Glad to hear that works out. It is possible to use a higher resolution, but it may require some change in the memory allocation of openmv (i.e., the new patch https://github.com/mit-han-lab/tinyengine/pull/74/files) for the framebuffer and increases the memory footprint of TinyEngine.

I would recommend changing the following resolution configurations, regenerating the code using gen_code.sh, and recompiling openmv, to see if your setting works out:

tinyengine/examples/detection_fpn.py

Line 28 in d412e11

h = 128

tinyengine/examples/openmv_person_detection/openmv_person_detection.patch

Line 423 in d412e11

+#define IMAGE_H 160

If you encounter some insufficient memory issue during compiling openmv, reducing the framebuffer in omv_boardconfig .h and try again.

[senceryucel]

Hi @meenchen

I've tried to change some stuff but in the end, I always get the memory overflowed error:

/usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/bin/ld: /home/senceryucel/Desktop/tinyengine/examples/openmv_person_detection/openmv/src/build/bin/firmware.elf section `.bss' will not fit in region `SRAM1'
/usr/lib/gcc/arm-none-eabi/6.3.1/../../../arm-none-eabi/bin/ld: region `SRAM1' overflowed by 595344 bytes
collect2: error: ld returned 1 exit status
omv/ports/stm32/omv_portconfig.mk:649: recipe for target 'firmware' failed
make: *** [firmware] Error 1
make: *** Waiting for unfinished jobs....

I am not sure which line do I need to change in omv_boardconfig.h

As far as I understood, I am supposed to change the corresponding board config parts in the new patch file which you have written for OpenMV H7 Plus. So, I need to change one of these below lines:

diff --git a/src/omv/boards/OPENMV4P/omv_boardconfig.h b/src/omv/boards/OPENMV4P/omv_boardconfig.h
index 178c8ea6..6ec5761d 100644
--- a/src/omv/boards/OPENMV4P/omv_boardconfig.h
+++ b/src/omv/boards/OPENMV4P/omv_boardconfig.h
@@ -146,6 +146,7 @@
 // Note: fb_alloc is a stack-based, dynamically allocated memory on FB.
 // The maximum available fb_alloc memory = FB_ALLOC_SIZE + FB_SIZE - (w*h*bpp).
 #define OMV_MAIN_MEMORY         SRAM1       // data, bss and heap
+#define OMV_MAIN_MEMORY2        SRAM5       // my memory
 #define OMV_STACK_MEMORY        ITCM        // stack memory
 #define OMV_DMA_MEMORY          SRAM3       // Misc DMA buffers memory.
 #define OMV_FB_MEMORY           DRAM        // Framebuffer, fb_alloc
@@ -153,12 +154,15 @@
 #define OMV_JPEG_MEMORY_OFFSET  (31M)       // JPEG buffer is placed after FB/fballoc memory.
 #define OMV_VOSPI_MEMORY        SRAM4       // VoSPI buffer memory.
 #define OMV_FB_OVERLAY_MEMORY   AXI_SRAM    // Fast fb_alloc memory.
-#define OMV_FB_OVERLAY_MEMORY_OFFSET    (496*1024) // Fast fb_alloc memory size.
+// #define OMV_FB_OVERLAY_MEMORY_OFFSET    (496*1024) // Fast fb_alloc memory size.
+#define OMV_FB_OVERLAY_MEMORY_OFFSET    (240*1024) // Fast fb_alloc memory size.
+
 
 #define OMV_FB_SIZE             (20M)       // FB memory: header + VGA/GS image
 #define OMV_FB_ALLOC_SIZE       (11M)       // minimum fb alloc size
 #define OMV_STACK_SIZE          (64K)
-#define OMV_HEAP_SIZE           (240K)
+// #define OMV_HEAP_SIZE           (240K)
+#define OMV_HEAP_SIZE           (140K)
 #define OMV_SDRAM_SIZE          (32 * 1024 * 1024) // This needs to be here for UVC firmware.
 
 #define OMV_LINE_BUF_SIZE       (11 * 1024) // Image line buffer round(2592 * 2BPP * 2 buffers).
@@ -175,14 +179,18 @@
 #define OMV_DTCM_LENGTH         128K
 #define OMV_ITCM_ORIGIN         0x00000000
 #define OMV_ITCM_LENGTH         64K
-#define OMV_SRAM1_ORIGIN        0x30000000
-#define OMV_SRAM1_LENGTH        256K        // SRAM1 + SRAM2 + 1/2 SRAM3
+// #define OMV_SRAM1_ORIGIN        0x30000000
+// #define OMV_SRAM1_LENGTH        256K        // SRAM1 + SRAM2 + 1/2 SRAM3
+#define OMV_SRAM1_ORIGIN        0x24000000
+#define OMV_SRAM1_LENGTH        512K
 #define OMV_SRAM3_ORIGIN        0x30040000
 #define OMV_SRAM3_LENGTH        32K
 #define OMV_SRAM4_ORIGIN        0x38000000
 #define OMV_SRAM4_LENGTH        64K
-#define OMV_AXI_SRAM_ORIGIN     0x24000000
-#define OMV_AXI_SRAM_LENGTH     512K
+// #define OMV_AXI_SRAM_ORIGIN     0x24000000
+// #define OMV_AXI_SRAM_LENGTH     512K
+#define OMV_AXI_SRAM_ORIGIN     0x30000000
+#define OMV_AXI_SRAM_LENGTH     256K
 #define OMV_DRAM_ORIGIN         0xC0000000
 #define OMV_DRAM_LENGTH         32M

But none of the changes I have made positively effected the result. I am having the same memory overflowing. Can you provide some further help if possible?

Thanks a lot.

[meenchen]

Hi @senceryucel,

Look like the memory overflowed by a lot (~600KB). I am not sure we can fit it into the board (only 1MB SRAM in total). Can you share the resolution you use?

[senceryucel]

Hi @meenchen

I have tried it for 2 different resolutions. First, it was VGA (640x480). It threw the overflow with more than 600kBs, then I changed the resolution to QVGA (480x320). The outputs belong to the QVGA configs.

[meenchen]

Hi @senceryucel,

I don't think the board is able to fit these 2 resolutions, at least not with the current memory scheduler of TinyEngine. Since our memory scheduler is completely decoupled with the framebuffer of the openmv firmware, it means we need to maintain two input images at the same time. This can easily cause memory overflow for such high resolution.

[senceryucel]

Hi, @meenchen

Thanks for your reply. I could not understand why do we need to maintain two input images at the same time.

Memory overflow for 640x480 resolution was approximately 610kB, where it was 595kB for 480x320. There is a huge difference between these 2 resolutions, but the memory overflow difference is too small. This made me think that if we can handle maintaining 2 inputs problem and achieve to handle only one at the same time, the memory usage will probably cut to the half of what it is right now. Hence we won't have any overflows (since the overflow is 600kB and the memory is 1MB, if we can make it to maintain 1 input instead of 2, the memory usage will be something like 800kB which will not produce any overflow).

Am I correct? Your engine is like a treasure for me but I have to make it runnable on higher resolutions, at least for VGA. I hope there exists a way to do so.

Thank you so much for all you have done so far and in advance.