Introduction to Photogrammetry with VisualSFM

0
1515

Photogrammetry is the discipline that deals with the study of the geometric properties of objects or scenes as well as their spatial characteristics from photographs. The main attribute of these techniques is that while photography only represents two-dimensional attributes, photogrammetry works with three-dimensional information obtained from various overlapping two-dimensional images. Through these overlapping areas, common points can be determined with which 3D views are recreated.

Throughout this article, and surely in upcoming articles, I will talk about photogrammetry, but in reality, I will obtain these photogrammetric models through Structure from Motion techniques. Although both strategies result in a 3D model, they are significantly different technologies. Photogrammetry is a rather old technique based on establishing common points between images taken in a controlled manner and with calibrated cameras, with the essential purpose of taking measurements between these points, although in modern practice it is possible to extract more or less effective 3D models. However, Structure from Motion is a technique that, although inspired by photogrammetry, comes from the field of computer vision and allows creating 3D models from collections of unstructured images, without calibrated cameras, and even using different cameras, which makes it a more versatile and robust strategy. Currently, photogrammetry and SfM are intertwined in various programs that support this type of task, reducing everything to simple photogrammetry processes, when we should rather talk about SfM, although ultimately our results will be approximately similar.

Works related to photogrammetry are quite booming in the fields of archaeology, rock art, conservation of cultural assets, and generally all disciplines related to the protection, documentation, digitization, or virtualization of heritage, since it is a fairly accessible tool, not only in terms of costs but also in the simplicity of its use and that it can constitute a quite relevant source of documentation.

There is not a large market offer of this type of tools, the most notable being the veteran Photomodeler from the company Eos Systems with more than two decades of experience in this field and which is properly a photogrammetry tool, which in its latest versions incorporates automatic processes clearly inspired by SfM. Another widely used tool in this field is Photoscan from the Russian company Agisoft, whose operation, although its developers do not make it clear, is exclusively SfM.

Among free or open-source alternatives, we do not have many options: on one hand, we have the online application developed by Microsoft called Photosynth, which to download the work done online requires an intermediate tool called SynthExport available only for Microsoft platforms. Online, we have another tool called Hypr3D that allows us to create 3D models from photographs and export them to other environments. Another alternative is INSIGHT3D available for Windows and Linux, but its development is obsolete, making installation very difficult as it is not compatible with the latest OpenCV libraries, also presenting some instability. As a last option, we also have ARC3D, a tool oriented towards uses related to 3D digitization of heritage, which is available through a client that accesses an online server via web services where all processes are carried out; like other web solutions, prior registration on the platform is required.

Another free alternative is tools like Bundler, which I will address in later articles, based on similar SfM processes that I will describe below, but their use is usually command-line based, which makes it more tedious.

Fortunately, from the University of Seattle, led by the young engineer Changchang Wu, we have VisualSFM, a tool that allows us to create 3D models from 2D images.

VisualSFM, similarly to other tools, aims to create a 3D point cloud with the help of various tools. These point clouds are formed by simple points described by spatial XYZ coordinates and RGB values that will be responsible for restoring the original texture through what is known as dense point clouds. The working concept is close to Photomodeler; however, with VisualSFM it is not necessary to establish reference points between images nor calibrate our camera, which speeds up the process and allows us to work with large quantities of images without much difficulty. VisualSFM also does not incorporate measurement tools, a process we must leave to other tools such as MeshLab, although in the latest versions we do have tools for georeferencing point clouds by GCP (ground control points) and GPS.

The operating process of VisualSFM, like many SfM tools, is based on the following steps:

  1. SIFT: detection of key or matching points between multiple images
  2. Bundle Adjustment: Creates the sparse point cloud from the key points detected in the previous step. In this step, the shape of our model can be intuited and the positions of the cameras obtained.
  3. Multi-View Stereo: This will be the process responsible for generating the dense point cloud from all the previous data. This process is handled by the CMVS and PMVS2 tools from Yasutaka Furukawa, a Google Maps engineer; in fact, PMVS2 is a tool used in that technology.

VisualSFM is available for WIN, Mac, and Linux. It is this last system on which I have decided to install, not without some difficulty.

In this first part of a series of articles on the topic of photogrammetry, we will proceed to install VisualSFM for Ubuntu 12. The following tutorial is largely taken from the work of Scott Sawyer, to which I have added certain aspects I encountered that were not described in this previous work, which may be of interest if you encounter any difficulties I have not considered.

Anyway, on the VisualSFM website, there is a more or less effective tutorial for installation on various systems, which it is advisable to briefly review.

First, we need the following files:

Dependencies:

sudo apt-get install libgtk2.0-dev libglew-dev libdevil-dev libboost-all-dev  libatlas-cpp-0.6-dev libatlas-dev 
imagemagick libatlas3gf-base libcminpack-dev libgfortran3 libmetis-edf-dev libparmetis-dev freeglut3-dev 
libgsl0-dev libboost-all-dev liblapack-dev

estructuraBefore starting, it is advisable to create a single directory called, for example, vsfm and unzip all downloaded packages inside it with unzip or the usual tar xf archivo.tar.gz

The initial idea of the installation is to compile VisualSFM and install certain libraries and binaries in its bin directory. In the screenshot on the left, you can see the necessary directories and the installation result in the vsfm/bin path.

We can start by compiling VisualSFM:

unzip VisualSFM_linux_64bit.zip
cd vsfm
make

Next, we proceed to compile SiftGPU and copy the library to the root of vsfm

unzip SiftGPU-V382.zip 
cd SiftGPU
make 
cp bin/libsiftgpu.so ../vsfm/bin

If we want to use CUDA, something feasible as long as we have nVidia graphics cards, besides downloading the framework, we must take some precautions: The SiftGPU makefile does a locate libcudart.so to check if CUDA is installed or not; if the locate database is not updated, it may not recognize that CUDA is installed, so either we update our database or skip the condition that checks if libcudart.so is installed by forcing siftgpu_enable_cuda = 1 afterwards. On the other hand, we must check the CUDA path; by default, it is located in /usr/local/cuda but in recent versions it is in /usr/local/cuda-5.0, so it is necessary to update the CUDA_INSTALL_PATH parameter.

If we compile with CUDA, it is likely that when starting VisualSFM we encounter the error “unable to load libsiftgpu.so”; this happens because SiftGPU does not find libcudart.so.4. To fix this, we copy this file from /usr/local/cuda/lib64/libcudart.so.4 to the directory where libsiftgpu.so is located.

In the next step, we can proceed to work with PBA; on one hand, we need to modify the file pba/src/pba/SparseBundleCU.h and add the following include:

#include <stdlib.h>

Next, we proceed to compile PBA. If we have previously installed CUDA Toolkit, we can simply run make, but if we do not have CUDA support, we must compile with:

make -f makefile_no_gpu

Then we will copy the generated file libpba_no_gpu.so to vsfm/bin/. Attention must be paid that if we are using CUDA, the generated library will be libpba.so; otherwise, this library will be libpba_no_gpu.so, which must later be renamed to libpba.so.

The next step will be to compile Graclus 1.2, for which we must previously edit Makefile.in and adjust -DNUMBITS=64 if necessary to suit our system. With a make, we can compile without problems.

Next, we proceed to compile PMVS-2. During the compilation of this binary, a couple of problems may arise; on one hand, certain errors in the mylapack library, which we can solve with the following sequence:

tar xf pmvs-2.tar.gz
cd pmvs-2/program/main/
make
cp mylapack.o mylapack.o.backup
make clean
cp mylapack.o.backup mylapack.o
make depend
make

On the other hand, it is likely that the clapack libraries (clapack.h and f2c.h) are missing. These libraries can be downloaded from http://www.netlib.org/clapack/. We can install them on our system, for example by placing the directory in “/usr/include” or simply modify the mylapack.cc file in the lines:

#include <clapack/f2c.h>
#include <clapack/clapack.h>

To define the correct path, replacing the < > with quotes, for example, we can place the clapack directory in the path “program/base/numeric/”, in case we need to specify the exact path of these files; however, the correct way is to indicate the directory path in the YOUR_INCLUDE_PATH parameter of the makefile.

Finally, we must configure and compile CMVS; for this, after decompressing it with the usual:

tar xf cmvs-fix2.tar.gz

We will copy the mylapack.o file from PMVS-2 to CMVS with:

cp pmvs-2/program/main/mylapack.o cmvs/program/main/

Now we must edit the file cmvs/program/base/cmvs/bundle.cc and add the following includes:

#include <vector>
#include <numeric>

We do something similar with cmvs/program/main/genOption.cc inserting at the beginning:

#include <stdlib.h>

Now we go to the Makefile file in the cmvs/program/main path and edit it so that we specify certain paths of our system as follows:

#Your INCLUDE path (e.g., -I/usr/include)
YOUR_INCLUDE_PATH =
#Your metis directory (contains header files under graclus1.2/metisLib/)
YOUR_INCLUDE_METIS_PATH = -I/opt/vsfm/graclus1.2/metisLib
#Your LDLIBRARY path (e.g., -L/usr/lib)
YOUR_LDLIB_PATH = -L/opt/vsfm/graclus1.2

In the version I downloaded, the “Your…” comments were not declared with the corresponding hash; if the same happens to you, pay attention to declare these lines correctly. The YOUR_INCLUDE_PATH parameter can be left empty or specify the path to clapack.

With these steps, the installation is complete; now we must add certain environment variables to our system to locate the VisualSFM executable:

export PATH=$PATH:/opt/vsfm/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vsfm/bin

If we are using Ubuntu, we must add these lines at the end of the file:

vim /etc/bash.bashrc

However, this option would only allow us to launch VisualSFM from the terminal. If we are using Ubuntu 12, the most appropriate is to create a launcher of the .desktop type in the “/usr/share/applications” path; for this, we create a file in this path called “visualsfm.desktop”

#!/usr/bin/env xdg-open
[Desktop Entry]
Encoding=UTF-8
Name=VisualSFM
Icon=/usr/share/icons/VisualSFM.png
Comment=VisualSFM
Exec=env LD_LIBRARY_PATH=/opt/vsfm/bin /opt/vsfm/bin/VisualSFM
Type=Application
Categories=Application

In this way, we introduce our environment variable in the “Exec” parameter. The icon is optional; I created a small icon for the application in the “/usr/share/icons/” path.

Brief introduction to VisualSFM

The use of VisualSFM is quite simple, and although I will elaborate more in detail in later articles, to create a first model we simply need:

  1. A series of images with various viewpoints of the same object or scene.
  2. They are loaded into the application with the folder icon accompanied by the + symbol.
  3. Next, matches are searched for between the images; this is done with the button symbolized by the four opposite arrows.
  4. When this process finishes, we use the adjacent button formed by two arrowheads (red and blue).
  5. Finally, with the button symbolized by the CMSVS acronym, we reconstruct the texture. At this point, the path to save the project will be indicated.
  6. When the previous process finishes, which may take a few minutes, to see the final result we go to the View menu and activate Dense 3D Points.

fotogrametriaAs we see in the result, the 3D recreation is somewhat rough, as there is considerable noise around the initial object due to the backgrounds of the images, so it will be necessary to clean the image to obtain a good recreation of it. Similarly, various “holes” caused by the lack of information or density of common points can be observed, which hinders the exact reconstruction of the texture. It is interesting to observe how we can see the different viewpoints.

In subsequent articles, I will delve a little more into the treatment of the resulting project with free tools such as MeshLab as well as the colorimetric performance aspects of these point clouds, which is somewhat where I wanted to arrive with this approach to photogrammetry oriented towards the digitization and documentation of cultural heritage.