Question about the Program

[JianDan0212]

Hello,

I am using your project to process data from my light field camera, but I have encountered an issue. I have reviewed previous issues and did not find any similar problems, so I have submitted a new issue request and hope you can provide some assistance.

While using your open-source project to process our light field data, I found that the results vary significantly. One concern I have is that my light field camera's microlens array is arranged in a rectangular pattern rather than the commonly used hexagonal pattern. However, I have already modified the camera parameter file in the XML file during calibration, but the results still exhibit significant problems, so I would like to ask for your advice on how to solve this issue.

It is worth noting that our light field camera has a single focal length lens, meaning all of our microlenses have the same focal length. Additionally, it should be noted that our MLA microlens array is arranged in a rectangular pattern.

I have attached my raw data, the camera calibration XML file, a white image, and one of the results I processed.
0811.zip

[JianDan0212]

This image is one of the results I obtained after processing.

[JianDan0212]

Looking forward to your reply。

[freerafiki]

Hello @JianDan0212 ,

I need some time to think about it. The first thing that crosses my mind is that you are actually using a Plenoptic Camera 1.0 (the first type). This toolbox was specifically designed to handle Plenoptic Camera 2.0 (the second type). The differences are mainly the different micro-lens types and the arrangement (hexagonal) which you do not have in your (custom, I suppose?) model. However, by modifying the XML file (I see you changed the basis to handle a rectangular grid) it should be possible.

Are you sure about diameters and offsets? Because it looks like something is off (you should not see circular microimages in the rendered image).

So, on one hand, you could use the LightFieldToolbox - although in Matlab, this toolbox is designed for type 1 plenoptic cameras (rectangular grid, only one lens type) and may give best results out of the box. It is widely used in the literature. On the other hand, if for some reason you need to use this toolbox, I will try to take a look at your data and the code and see what should be changed to accomodate your camera. I will update you soon, but since I am not anymore actively working on this, I might need some time to figure it out.

[JianDan0212]

Thank you for your response. I am currently using a 2.0 light field camera. In the attached zip file, you will find my raw images, which contain the light field data corresponding to the 2.0 type. However, I am using a rectangular MLA arrangement. The diameter and offset values are generally correct, though there may be some minor discrepancies.

I need to use your toolbox and would be immensely grateful if you could help me identify the issue. Your assistance would be greatly appreciated.

[freerafiki]

Ok got it. I am experiencing some problem with the zip file extraction (would it be possible to upload it somewhere else?).
Plus, I think realistically that I will manage only next week to check because I need some time to look at it. I will do my best and update you as soon as I have something.

[JianDan0212]

Thank you very much, I will re-upload the files in ZIP to googl cloud disk later, so you should be able to download.

[JianDan0212]

https://drive.google.com/drive/folders/1jy5oeU3VDyqsEfRvNqzKdPqL21ZdBKja?usp=sharing

Here is the address of my file, I am using your render_view_3g.py and disparity_sample.py files。

[freerafiki]

Ok now I see the data (new_white.bmp, new_device.xml, new_device.png).

A couple of things I see:

• the white image we used to capture to calibrate was completely white. We used a white filter and a light diffuser for best results, but anything that comes close to getting complete white light is okay. In your white image you still see the same scene. The white image is used to calculate the light propagation and distribution (vignetting mostly), so the whiter it is, the better. You can capture something more white. Anyway, this does not create direct problem.
• I do not see the disparity image. These types of cameras need the disparity to "know" where the object is, because the size of the micropatches used to render (let's say the resolution at which an object is captured in each microlens) is depth-dependent. Can you also include that? Also because the disparity_sample.py script should use the grid (so we can also understand if the problem is on the conversion from hexagonal to rectangular grid!) and should output a json file with the parameter (also useful to check that all went fine).
• I think the diameter could be increased a bit (70/72, border pix could go up to 3), but these will not change too much. I need to check with more time how the code uses the sub_grid_base value and how the now rectangular grid is added (if the coordinates are correct) because looking at your picture it seems the grid is off, there is vertical black space and also the circular micro-images are reported, meaning that the patch size could be too large!

I will continue next week, if you can also upload disparity image and json output it could help. Thanks!

[JianDan0212]

https://drive.google.com/drive/folders/1gikizb4bwpOaIxrJ8A0AGRh97SW41EXv?usp=sharing

This is the updated link, which contains the disparity map and JSON file. However, these files were generated using your disparity_sample.py script with my own image data. You can take a look or rerun the script to obtain the same results. Regarding the issue you mentioned:
Indeed, theoretically, the white image should be as white as possible, but due to the material we are using, the image turns green, which is unavoidable.
I have uploaded the disparity map to Google Cloud.
As for the third issue, there really shouldn't be any black gaps, but I haven't yet found an effective solution for this.
Thank you very much for your help!

[freerafiki]

Ok, this clarifies already something. As you can (I hope) see, the disparity is wrong. The disparity should be consistent and contain the information about the scene geometry.

You already see the problem that the disparity is organized in hexagonal grid, while your input image is captured with rectangular grid! The coordinates are not working the way you want them to. I need more time to run the program and I do not manage soon, but I can give you a few hints on where to look in the code!

1. You are using the real_lut method (my fault) which is hardcoded as a choice: at line 74 of disparity_sample.py you should replace params.method = 'real_lut' with something else (like params.method = args.method and then adding also parser.add_argument('-m', dest='method', default='plain') ). If you wonder what this is, it is the method to control the choice of the lenses. real_lut means using a Look Up Table which I built (full paper here) for the best quick selection. However, you do not need it (also you have rectangular grid), so you should change the method.

2. The disparity_sample.py calls estimate_disp which is here , which does the job to get the lenses and estimate the disparity. If you change the method to plain, you can use the calc_costs_plain().

3. Inside calc_costs_plain you get the lenses and microimages. You could set a breakpoint here and check from there how the coordinates and the microimage look like!

[JianDan0212]

  Thank you for your reply. I will continue debugging based on the approach you mentioned to observe the corresponding results. Additionally, regarding the lookup table you mentioned, I briefly went through your article and noticed that you are using a microlens array with three different focal lengths. There’s one thing I would like to clarify: in my camera, I am using a microlens array where all the lenses have the same focal length, meaning that the focal lengths of all my microlenses are identical, with no distinction. In this case, would the method described in your article still be applicable? 
  Additionally, I’d like to ask if you know the approximate ratio by which the focal lengths of the 0, 1, and 2 types of lenses in your microlens array increase or decrease. I’m considering whether I can customize an MLA with corresponding specifications based on the parameters used in your program. I believe this could be a relatively quick validation approach.

[freerafiki]

Ok, let me split the answer:

Look Up Table

The look up table contains the combinations to use for hexagonal grid and 3 micro lens types (the "standard" at least back then when I used it). If you have rectangular grid with only one lens type, you can just use all of the neighbouring lenses, there is no need to use any sophisticated strategy for the lens selection.
For example, you can use the 4 or 8 neighbouring microlenses to compute the depth. You can also define a new strategy which uses only these lenses and stick to that. The strategies I used (figures from the paper) are defined starting here. You can create new rectangular strategies and use those!

How the strategies were used

The code works by first making a rough estimate of the disparity with the "first ring" strategy (the ring around, in your case this could be the 4-neighbours!). Then when you have an idea, you can choose the best strategy (for example, the disparity is very small and the object is repeated across many microlenses, you can use even 20 lenses, if disparity is large, you can stick to the 4 neighbours only). You can also just use one single strategy everywhere, but you will lose some information.

Microlenses focal range ratio

I do not have precise information on these, there was one for close, one for medium and one for long. You can find more info about using 3 focal types on the 2013 Perwass paper, but you will not have the exact values.

PS: You can try to change your microlenses, but if you do not fix the coordinate system and the lens selection, having one or three (or any) number of microlens type will not make the code work, the error is on the extraction of the microlens image given the coordinates!

[JianDan0212]

Thank you for your explanation. I think I now have a better understanding. My main issue seems to be fixing the coordinate system and lens selection. In other words, the micro-lens images I am currently extracting are incorrect, which is the key reason why the results I get are so poor, correct?

Additionally, regarding "fix the coordinate system and the lens selection," by coordinate system, do you mean the rectangular arrangement of my MLA? And by lens selection, do you mean that I need to choose 4, 8, or some other number of micro-lenses for disparity estimation? Is my understanding correct?

[freerafiki]

Thank you for your explanation. I think I now have a better understanding. My main issue seems to be fixing the coordinate system and lens selection.In other words, the micro-lens images I am currently extracting are incorrect, which is the key reason why the results I get are so poor, correct?

Yes, exactly!

Additionally, regarding "fix the coordinate system and the lens selection," by coordinate system, do you mean the rectangular arrangement of my MLA? And by lens selection, do you mean that I need to choose 4, 8, or some other number of micro-lenses for disparity estimation? Is my understanding correct?

Correct!

[JianDan0212]

Okay, got it, I'm just trying to modify your program so that it can use rectangular MLA arrays, thank you for your guidance and explanation, thanks

[freerafiki]

Great! Please do let me know in the end how it worked (and it would be great if you would be willing also to release the code (you can make a pull request or anyway you want) so that we have in the repo a working version)

[JianDan0212]

I am very happy to publish my program. I am now trying to modify your processing logic, but I may encounter some troubles. This may take some time to debug and modify. After my work is completed, I will submit a merge request. , integrated into your github, but this may take some time.

[JianDan0212]

I am very sorry to bother you again, but I have another question I'd like to ask. Regarding the preprocessing part, I have completed the extraction of MLA images by using an all-white image to replace the XML file. Below is one of the result images I extracted. However, I found that the disparity map I calculated may still have significant issues. I wanted to ask you what might be the reason for this. The attached image is the extraction result I obtained by using an all-white image to replace your XML file, using the disparity_sample.py program.

[freerafiki]

Hi - sorry, can you caption the images so that I know which is what (disp, processed, ecc.).

Let's try to divide step by step.

1. First the microlens image extraction. Did you arrange the rectangular grid (I see again hexagonal grid above). Is this part working? Because without this, the rest will probably be almost only noise.
   If so, can we see one image with the rectangular grid extracted and the corresponding .XML file? That would be great. Even we can merge it.

2. Once that is done, we can compute the disparity. You need (as discussed above) to change the lens selection "strategy" meaning you select neighbouring lenses (which are arranged in the rectangular grid). Which lenses did you select?

I think these questions should be answered (and solved) from the above pictures I cannot understand, plus I see an hexagonal grid and you talked about rectangular grid, so I am a bit confused. Can you try to elaborate a bit more about the details of what you did?

[JianDan0212]

I apologize for any confusion caused, I have changed the previous strategy. The images you are seeing now are actually produced by replacing the hexagonal MLA arrangement; the last of the three images is the png image generated during imaging runtime, which should be the processed image. The first image is the _conf_plain_1.0_8.5_sad.bmp image generated after the program runs, and the second image is the _disp_col_plain_1.0_8.5_sad.bmp image generated after the program runs; At this point, I can confirm that the microlens image extraction part is working correctly, which has been my main focus recently. I have already been able to obtain the extracted grid images and corresponding XML files, as shown in the three images.