Face Replacement Results
 
Team Members
 
  • Shahzad Malik
  •  
    Report and Presentation Slides
     
    This page shows the results from the Digital Face Replacement project. For a more thorough discussion of the system, please download the full report which is available in PDF format.

    Download Full Report [PDF]

    Download Presentation [PPT]
     
    Summary
     

    The ability to automatically replace a face in a photograph with that of another person has huge implications in the entertainment and special effects industries. For example, consider a Hollywood stunt double performing a dangerous routine while remaining in full-view of the camera, and then having a post-processing step that automatically replaces each instance of the stunt double's face with that of the desired actor. Or imagine watching your favorite movie where the lead actor's role is seamlessly replaced with your own face.

    While there are many different components that are needed in order to fully realize such a system, the focus of this project is on a system to capture the illumination condition across a human face in a single 2D image, and then applying this lighting onto some replacement face. This newly lit replacement face can then be seamlessly merged into the original photograph.

     
    Results
     
    Example #1
    This image shows the face replacement algorithm being applied to an arbitrary portrait of an individual's face. The image at the left is the original, untouched image. The middle image represents a front-facing, fully-lit target image that we wish to place over top of the original face. The image to the right is the final result of the face replacement. Notice that the lighting has been recreated fairly well, and the skin tone around the new photograph is fairly consistent. The algorithm failed to convert some of the extremely bright areas of the original face (such as the forehead and neck) during the replacement process, largely due to the use of a simple flesh detection algorithm. One other problem is the new "manly" facial features that the replacement face has taken on. This is because the original face had a relatively long and sharp chin, whereas the replacement woman has a soft and rounded chin. The algorithm currently does not take this into consideration.

    Example #2
    This image shows the face replacement algorithm being applied to the same original photograph in the previous example, but with a new replacement face. Notice that the final output image exhibits the same problems as the previous result, namely the bright spots in the top right area of the forehead and the bottom right area of the neck. Nevertheless, the remaining parts of the conversion are convincing.

    Example #3
    This image shows how the algorithm handles head poses that are not necessarily front-facing. The image at the left is the original picture, and the image in the middle is the new image after replacing Steven Seagal's face with the person on the right. While the result is fairly good, there are a few noticeable problems. The first is a lack of expression on the new face, which is clearly evident in the original. Another problem is the lack of shadows in the jaw area of the replaced face. Additionally, the replaced face exhibits somewhat of a "plastic" look, largely due to our use of a use of a Lambertian reflectance model.

    Example #4
    This example shows what happens if our flesh detection algorithm fails. The photograph on the left is the original image. The middle image shows the detected flesh pixels in white. Notice how some of the background pixels have been classified incorrectly as flesh pixels. Since these pixels are connected to the facial flesh area, our flood fill algorithm also considers the connected background pixels as flesh as well. As a result, the face replacement image to the right also has some background and hair pixels being incorrectly converted to the new replacement face flesh tones.

    Example #5
    This example image shows the result of augmenting some virtual object into the original scene to see how it becomes illuminated. In this case, we augment a virtual checkered hat onto Steven Seagal's head. Notice how the light estimation is fairly accurate, causing the hat to be realistically illuminated from the left of the scene.

    Copyright © 2003 Shahzad Malik
    Last updated: January 27, 2003