Software transforms digital photos into old-fashioned paintings

Brooks is a computer scientist at Dalhousie University, and also has a background in traditional drawing. The method he recently developed can transform digital photos taken with an ordinary camera into mixed-media paintings and portraits, making him a pioneer in a subject area that is largely unexplored. While the system requires very little input from the user, the results can be tweaked if desired, and produce realistic compositions of “nonphotorealistic renderings” (NPR).

“Mixed media compositions incorporate two or more traditional visual media such as oil, watercolor, tempera, ink, or pencil,” Brooks explained. “By mixing materials, the artist is able to explore new channels of artistic expression or rejuvenate existing ones.”

As Brooks explains, mixed media is often used by artists when a single medium is insufficient to fully develop an artistic idea. From ancient Egyptians to Rembrandt and Picasso, and even contemporary cartoonists such as Dave McKean, mixed media has been used to give variety to work and emphasize certain details.

To give the appearance of mixed media to photos, Brooks’ system analyzes the input image and computes a measure of detail at each pixel. The areas of the image with different levels of detail are separated into distinct regions, and then each region is processed independently with a different NPR filter chosen by the user. Examples of filters can be “paint daubs,” “soft glow,” “crystallize,” “ink outlines,” “cartoon,” etc. Choosing the filters is the only step that requires user input.

Finally, the different areas of the image are smoothly fused. Here, the user has the option of determining the gradient level of blended areas, but Brooks notes that all the samples he presents in his research are created automatically by the system. The system performs the blending process in separate stages to allow the user to see the intermediate processing. Brooks explains that this step-by-step process resembles how an artist produces a mixed media work in stages.

One of the most challenging types of images to transform is facial portraits, as people are highly sensitive to the treatment of faces. Using Brooks’ standard technique, the facial areas around the cheeks and forehead are perceived as low detail areas, while the eyes and mouth have more detail, which produces undesirable results. Brooks approached this special case by developing systems for skin filtering, face point detection, and face point clustering to create a more satisfactory mixed media portrait.

“The most advanced component of my system is the module that handles portraits,” Brooks said. “Automatic face detection and processing is quite tricky if one wants to make the system robust. I use a combination of skin detection, cascaded weak detectors, and Gaussian mixture model clustering.”

Brooks is currently exploring more advanced features and improvements, including integrating 2.5-dimensional physical materials such as felts and cardboards, as well as the possibility of adapting the technique to video.

“We have recently begun collaborating with the film school at the Nova Scotia College of Art and Design,” he said. “We are looking at developing a new kind of digital film production system which could involve extending mixed media processing to video. So, it may be released as part of a larger film processing package.”

Citation: Brooks, Stephen. “Mixed-Media Painting and Portraiture.” IEEE Transactions on Visualization and Computer Graphics, vol. 13, no. 5, September/October 2007.