Welcome

These webpages contain the full documentation of the VisualHull Project by Miro Rashid.

Table of Contents


Idea and Motivation

The general idea for this project is to reconstruct 3D objects solely by its silhouettes (i.e. filled outline from a picture). But of course, it is not only restricted to recreation of existing objects, it can also be used for the construction of 3D models, by using fictional silhouettes.

I got the motivation from a specific type of anamorphic sculptures: Wire models that show a picture in one specific perspective but show a different picture in another perspective, so basically this would make the sculpture “bi-anamorphic”, e.g.:

Anamorphic Wire Sculpture

(“The Revolution of Giraffes” © Matthieu Robert-Ortis https://laughingsquid.com/wp-content/uploads/2017/04/Anamorphic-Wire-Sculptures.jpg)

This wire sculpture is showing an elephant from the side but the frontal view shows two giraffes. It is related to the Visual Hull project because we could interpret the wire form pictures as silhouettes and therefore create a maximal 3D object that shows both pictures.


Procedure

The concept of a visual hull is to create “generalized cones”, i.e. cones that use a silhouette as its base, for each silhouette and then intersecting these. Note that for orthographic projection, we would use “generalized cylinders” instead, because here we would have no perception of depth.

There are many procedures, but the one I focused on, was to create a cuboid, consisting of voxels (“volume-pixels”, simply atomic cubes). Then we use this cuboid as our canvas from which we then carve out those pixels which would generate a bigger silhouette than the given silhouettes.

The exact procedure is described in the documentation of the System Functions.

Here is a demonstration of the process, using the example Mini Tree, where the two silhouettes are fictional and created in GIMP in a orthographic projection: Mini Tree Procedure


Usage

The scripts will work on any system that supports Python 3 (tested on Python 3.9) Also following modules have to be preinstalled (e.g. via pip install):

This one-liner to install the prerequisites might work for you:

pip install numpy pypng pygame

For the parameters and arguments to use as input for the program, see here: Main Parameters.


Examples

There are some example results that I created for testing and demonstrating the visual hull creation. These are fully described in About the Examples.


Problems and Future

There are obviously some (more or less major) problems that the visual hull method has.

Problem: Concavities

This is an issue for all visual hull algorithms. It is not possible to reconstruct a concave area (e.g. hole, dent) correctly, because the silhouettes do not encode information about depth and the sides of a concavity are blocking light/vision.

One way to fix this, is to include some depth information in the silhouette pictures, e.g. using gradient colors for the silhouettes instead of binary pixels (z-buffering). But here it might be difficult to extract depth information from the color/texture of pictures of an object. It might be possible to use motion footage to get a greater perception of the form of an object.

Problem: Inaccurate

If using too little silhouettes, round objects cannot be reconstructed properly. For example, the example Minerva has a round/circular pedestal, but because the reconstruction used 8 silhouettes, the resulting object has a hexagonal pedestal.

An obvious fix would be to use many silhouettes. But to avoid this, the textural properties could also be used to get a better grasp of forms. It would be also possible to smooth and round the surfaces afterwards to get better results.

Problem: Voxels

This problem is specific to the method used in this project. Voxels are not very popular among the mainstream for 3D-Objects and also the efficiency is bounded for this application.

One way to fix this, is to convert the voxel raster to a polyhedral object consisting of triangles (e.g. via Marching Cube Algorithm). If one might like to keep the cube form, it could also be helpful to only safe the visible vertices and faces.

Future Problem: Texturing

In this project we created uncolored 3D-objects. It is also possible to give the objects color and texture from the picture it was reconstructed from.

Here we run into another problem: the lighting in the pictures might distort the actual colors, as some surfaces are reflective enough to have “white” spots, even though the color is just not visible there because of overexposure.

Future Problem: Position and Rotation Reconstruction

In this project we have to input all parameters of the silhouettes manually. It would be also possible to use machine learning methods to estimate the original position and rotation in which a silhouette is captured.

The problem here would be, that we would have to give up reconstruction of forms that might be not that common. For example, the algorithm could recreate a cube from a trapezoid.

Future: Higher Dimensions

The visual hull method is used in lower dimensions: the creation of 1D-silhouettes to reconstruct the area of an 2D-object. In this project we looked at 2D-silhouettes to create 3D-objects.

But it might be also interesting to “see” the construction of 4D-objects from 3D-silhouettes.