Understanding the XML file

[freerafiki]

As one master student correctly pointed out via mail, there seems to be some lack of information about using/creating/understanding the .XML files which are used in the code.

What are the XML files for?

They contains information about the micro-lens array (MLA). They were generated from Raytrix Software during the calibration. But they are relatively easy to re-use and modify for our needs.
An example can be found here

How can I calibrate my camera?

There are several ways of calibrating your camera - another one we used was from Nousias et al. (ICCV 2017) - the corresponding code is here https://github.com/RViMLab/ICCV2017-plenoptic-camera-calibration. It is old, but you might try it if needed.

Additional Material on this

• the Raytrix SDK slides
• the 2014 Fleischmann paper Lens-Based Depth Estimation for Multi-focus Plenoptic Cameras (unfortunately behind paywall).
• the 2013 Perwass paper Single Lens 3D-Camera with Extended Depth-of-Field (available for example on ResearchGate)

Going through the XML file

An example of the file (you can find in the Dataset) is the following

<RayCalibData version="1.0">
  <offset units="pix">
    <x>15.707222938538</x>
    <y>12.639554023743</y>
  </offset>
  <diameter units="pix">31.609000000000</diameter>
  <rotation units="rad">0.000885307090</rotation>
  <lens_border units="pix">0.500000000000</lens_border>
  <tcp units="virtual_depth">2.000000000000</tcp>
  <lens_base_x units="lens">
      <x>-1.000000000000</x>
      <y>0.000000000000</y>
  </lens_base_x>
  <lens_base_y units="lens">
      <x>-0.500000000000</x>
      <y>0.866025269032</y>
  </lens_base_y>
  <sub_grid_base units="lens">
      <x>3.000000238419</x>
      <y>1.732050538063</y>
  </sub_grid_base> 

We can break it down (check Raytrix Slides):

MLA grid

We build a microlens array (MLA) grid. This means we have a set of points which are the centers of each microlens image. To do this, we need a reference (for the grid).

<offset units="pix">
    <x>15.707222938538</x>
    <y>12.639554023743</y>
</offset>

Slide 7 from Raytrix. Imagine we have an image of NxM pixels. The center is at (N/2, M/2). To create the MLA grid we want to set the center of "our" grid (the reference) in the middle of a lens (in raytrix type, they choose type 1 lens). So the center of (any) type 1 lens which is closest to (N/2, M/2) will be chosen. The offset is the difference between the center of the image and the center of our microlens grid! The image on Slide 7 will help (black arrow for the offset).

Here I add also the rotation:

<rotation units="rad">0.000885307090</rotation>

This is also relative to the microlens grid we create (it is the rotation of the grid). If you set offset and rotation to 0, the grid will be created on the center of the image. Usually you have a small x,y offset to center the grid and a very small angle if the image is tilted. This depends on your camera (self-built camera sometimes had large angle, one we had in Spain had it, Raytrix camera usually have almost 0 angle)

Single MLA image

<diameter units="pix">31.609000000000</diameter>
<lens_border units="pix">0.500000000000</lens_border>

Slide 8 helps here. Once we have the centers of the MLA images, we want to know how big each image is. So lens diameter tells us from each center we have to go (diameter/2) in each direction to extract the (circular) image! And lens border (usually 0-2 pixels) are pixels which are at the border between microlenses and are discarded. Some old cameras had large problems with these, newer had almost zero!

Virtual Depth

<tcp units="virtual_depth">2.000000000000</tcp>

The concept of virtual depth was introduced in 2013 Perwass paper, section 3.3. We can describe it as the amount of times something is repeated in adjacent microlenses. So if you have very large virtual depth, your scene is repeated many times on many microlenses! This is good for depth reconstruction (multiple correspondences) but bad for resolution (super small). A compromise should be found. They also explain that you need virtual depth > 2/3 otherwise you have no correspondences to actually compute depth, and best is VD > 5 and VD < 10/12 as far as I remember. I don't think this is used in the Plenoptic Toolbox, you can just leave it there as 2.

Grid Basis

<lens_base_x units="lens">
    <x>-1.000000000000</x>
    <y>0.000000000000</y>
</lens_base_x>
<lens_base_y units="lens">
    <x>-0.500000000000</x>
    <y>0.866025269032</y>
</lens_base_y>

This is a bit more complex, but well explained. In Section 2.1 of Fleischmann 2014 paper, you have a description of the lens grid in mathematical terms. These values are the shifts in the x and y axis to go from one to the other lens (it's an hexagonal grid, I forgot to say!).

If you check the values, they are (almost) the same (there is a minus one and they are inverted). You can leave these values as they are, and if you look at the drawings of the paper and check it, these allow you to shift between lens. You can see the four values of the matrix B of the paper in the .XML ($\sqrt{3/2} = 0.866$ you see in the .XML). Read the paper for more details, I think this should be okay then.

Sub-Grid

<sub_grid_base units="lens">
    <x>3.000000238419</x>
    <y>1.732050538063</y>
</sub_grid_base>

This is again related to the distance between one microlens of type 1 and the next one of type 1. Specific to the 3-microlens type of Raytrix. You can find a visual explanation on Slide 10 of Raytrix SDK. Probably also not used in the code!