A second look at non-photorealistic rendering

Non-photorealistic rendering(NPR) is a thread of research I've been following for a while, without ever being completely satisfied with publications in the field. NPR is often overlooked by computer graphics researchers as  it is difficult to evaluate the quality of the work numerically. The combination of perception methods with NPR can address this issue, at least for some applications (It is certainly the case for visualization applications). Perception theories are verified using psychophysics experiments which consist in showing images to a group of subjects, asking them questions about those images and computing some statistics on their answers.

NPR hasn't drawn any attention from the artistic crowd either because the results shown in scientific publications often lack aesthetic qualities.

This issue is partly addressed by the newly proposed programming paradigms I discussed in a previous post. I'm confident that programming these effects will be accessible to even the most novice programmers and most probably to the crowd of artists who can already script in Maya or AfterEffects. If it is the case, we might start to see some interesting NPR-generate images.

Another problem is that traditional 3D modelling tools only bring indirect control to animators over the final look of their images. The 2.5D cartoon-type of approach I discussed in a previous post might help giving more direct control.

One other problem I find is that every object in a scene has to be physically coherent in 3D, which is a serious limitation for 2D artists. A paper called "Modeling and Rendering impossible figures" presented an approach that might be the beginning of a solution to that problem. Here's a simple video explaining the approach:



Finally, the most important problem for me is that you can always tell NPR images have been generated from a 3D model. Rendered objects usually look like 3D models on which paint or hatching patterns have been applied rather than true 2D depictions. Some publications are starting to address this issue this year by using a combination of 3D and 2D information in the rendering process. It was the case for the 2.5D cartoon paper I discussed earlier and it is the case again in a talk I attended thursday morning called "NPR Gabor Noise for Coherent Stylization".

When it comes to animation as an art form, my preferred technique is still frame-by-frame 2D animation. But what I've seen this year at Siggraph forces me to take a second look at NPR, especially since I think it could be very useful for my visualization work.

The conference is now over. This post is being edited early friday morning on a boarding dock in the Pearson airport. I'll write a last post later on this weekend about my predictions for future trends in graphics. I think it will be funny to read that in 10 years from now.

Enhanced display resolution

A quick post to mention a nice little paper presented this morning in the "Perceptual rendering methods" session. In "Apparent Display Resolution Enhancement for Moving Images", authors present a method to display a high-resolution image on a lower resolution display but still maintaining the high-quality of the image. In short, the idea is to slightly jitter the image over time to give the impression of a higher resolution image. TV manufacturers just found yet another reason to make you buy a brand new 240Hz HD TV: the possibility to show your 15 Megapixels photographs in a much nicer way.

Another interesting paper in the same session, "Subtle gaze direction" shows how to attract the attention of viewers on a certain part of an image using imperceptible color modulations. The work is supported by a simple psychophysics experiment.

And now, some entertainment

In the evening, the conference schedules some more entertaining event.

Yesterday evening I attended "Siggraph dailies", a novelty this year. The goal is to "celebrates excellence in computer graphics by showcasing images and short animations of extraordinary power and beauty". Presenters have 1-2 minutes to talk about their work while a short sequence of animation they produced runs in loop. This could be the perfect venue to attract some attention on brain images visualization. Most productions this year came from the film and game industries, but a couple also came from the world of scientific visualization.

The second show I attended is the real-time simulations session. The presenters there are, again, mostly from game studios but also from the world of visualization. The participants show their work on a huge screen. They have to be able to interact with the material and the thing needs to run at interactive rates. Many of the presentation where really impressive. A man from NVidia was showing a film quality water simulation that uses a pretty cleaver screen-space technique to compute the water surface from the position of a few hundred thousand small sphere who's position is computed in real-time on the GPU. Another impressive presentation was given by a team from INRIA and CNRS in France. They built a "realistic virtual Earth model and browser with physically based rendering and animation of terrain, ocean, atmosphere, and clouds, and seamless transitions from ground to space views". It is difficult to believe that the rendering is done in real-time.

Tonight I was at the screening of the films that are competing in the computer animation festival. One of my favorites (which happen to win the jury prize) is "Loom" a short by director James Cunningham from New Zealand. The film takes place in the trenches during the first world war. It is one of the most interesting looking 3D CGI film I've seen in a long time.

Pixar's "Day and Night" is by far the best short I've seen from this studio in a long time.

"Visualizing empire decline" (see embeded video) by Pedro Cruz and Penousal Machado from University of Coimbra in Portugal managed to keep me interested during all its 3:30 min of watching nothing more than 2D color bubbles representing the rise and fall of 4 of the great maritime empires of europe. Its yet another proof that the meaning behind an animated film is more important than the fancy graphics.



I finally got a chance to see "Logorama", which I liked a lot.

The interesting films I just listed accounted for about 20 minutes out of the 2 hours screening. The rest of the projection was filled with technically advanced, but extremely boring sequences of explosions, monsters, pseudo medieval crap a lot of CGI animators dream of. I overdosed...

Your brain on the web

Doctors at hospitals like the Neurological Institute already have access to some tools to visualize brain scans of their patients on the web before surgery. But these tools are currently limited to 2D visualization of slices of the scans. The Web3D consortium promises to change that with the release of the X3D standard for medical imaging. This standard includes all the features needed to display and interact with DICOM images in 3D in a web browser. The functionality even including volume rendering. I attended the talk given by the consortium this morning, and I think it looks really promising. I'm under the impression that implementations of the standard are not there yet. In the afternoon, I chatted with a Google employee at the company's booth in the exhibition hall. He was showing some demos of WebGL running on Chrome (Google's web browser). He told me that the company was fully dedicated to support the format.

Perception...again

I've already mentioned that the use of human visual perception concepts is a growing trend this year at Siggraph. Today's technical papers session entitled "Perception, presence and animation" just confirmed to me the strength of that trend.

Two of the papers presented dealt with identifying what details of a visual scene can be discarded without affecting the quality and the readability of the rendered images. This is exactly what visual artists have always been doing, that is, avoiding to render what is not needed to tell the story. And it is especially true for animators who have had to find ways to maintain the amount of drawing required to a reasonable level. In "Fool me twice: Exploring and exploiting error tolerance in physics-based animation", authors investigate our tolerance to different kinds of errors in the accuracy of physical simulations. The goal is to reduce computation times of such simulations. In "Seeing is Believing: Body Motion Dominates in Multisensory Conversations", authors explore visual and auditory elements of a conversation between a group of people that make it believable for an external observer. Using this information, they hope to be able to automatically animate a crowd with a sound using a very small amount of example data.

In "Using Blur to Affect Perceived Distance and Size", depth perception principles are used to enhance a rendered image by exploring the relationship between depth perception and image blurring. The results from this paper reminded me of a question that I've always asked myself: Is there a relation between the perceptual correctness of an image and its aesthetic qualities. I don't have an answer, but I feel it is the case. At the end of his talk, the presenter added a comment about how his work could be extended to stereo images. He shows how stereo disparity is directly correlated to blurring, which means that stereoscopic filmakers must be very careful to vary these 2 parameters in a coherent way. It immediately reminded me of some of the barely watchable live action shots in "Facing Champlain", a stereoscopic movie I worked on 2 years ago at the National Film Board. This must have been the problem.

It would be interesting to put together vision scientists and renown animators to analyze the techniques used by those animators from the point of view of the science of human visual perception. I think both fields would benefit from that collaboration.

Tomorrow, more papers related to human visual perception in a session entitled "Perceptual rendering methods".

Programmer is not going away - part 2

Can artists become programmers? Can programmers become artists? So does seem to think Don Marinelli, one of the founders of Carnegie Mellon's Entertainment Technology Centre and yesterday's keynote speaker here at Siggraph. "There is no such thing as left-brained and right-brained people" he told the audience. "If I open up your skulls, we are going to find two hemispheres in everybody's brain". Don insisted a lot on the importance of inter-diciplinarity in the field of computer graphics. And it seems to be a trend this year at the conference (probably not unrelated to the fact that Siggraph  seems to be going through a crisis and is trying to broaden its scope - more on that in a later post).

In any case, if programmer and artist can't live in a single person, they will have to keep working together for a while. Today's technical paper session on "Expressive rendering & Illustrations" brings another proof of that fact.

The first of these, entitled Programmable motion effects, is inspired by the work of James E. Cutting on the depiction of motion in static images in art, science and popular culture. The framework presented allows one to program the trace created on a rendered image by the motion of a 3D object. The following video shows some results optained:



It is interesting to note that once again, a technical paper has its roots in previous work in the field of human visual perception. This is a trend.

The second paper, called "Programmable rendering of Line Drawing from 3D scenes", presents a framework that allows to programatically describe the style and placement of lines used to render the contours of a 3D model. Even though I'm not a big fan of line drawing depiction of 3D scenes, I must admit that images from this paper are more promising than what I have seen so far. If no artistic value, this kind of technique could at least be used to improve the readability of technical illustrations.

Real-time ray-tracing

When I came back from Siggraph in 2001, I told my colleagues at Ubisoft where I was working back then that games and real-time 3D applications would render exclusively using Ray-tracing within 5 years. I was wrong about only 1 thing: it was going to take 10 years or so. Siggraph 2001 had been all about the new capabilities of graphics card that allowed them to be programmed. I thought it was only a question of time before the programming languages become flexible enough to do ray-tracing.

For the less technical readers, ray-tracing is a 3D rendering technique that computes the path of the light that hits each pixel of the image to render. Because it is based on the physical laws governing the propagation of light, it produces images of a far better quality than the polygonal rendering techniques traditionally used for real-time rendering. Ray-tracing has been used for years to render animation movies, but it has always been too computationally intensive for real-time applications.

For the last 2-3 years, the idea of doing real-time ray-tracing on the graphics card has gained some interest and a couple of simple hardware-assisted ray-tracing projects have been successful. It is the case with some volume rendering features in medical imaging applications. But today, the technique is going mainstream with the release of NVidia's OptiX. I've attended NVidia's technical presentation this afternoon and it was very convincing. With only a few lines of codes, it is possible to write your own custom ray-tracer.

For the world of 3D computer animation, it means that what they see in Maya or Blender when they animate will get a step closer to the look of the final rendering.

In the world of medical imaging, this kind of tool will be very interesting to create volume rendering applications that consider principles of human perception (how about some depth of field effect to help surgeons perceive depth?).

At the end of the afternoon, I passed by NVidia's booth in the exhibition hall and they are showing real-time stereoscopic volume rendering of a beating heart (scanned with an MRI) with some overlaid blood flow simulation. Everything is computed on the new 6800 Quadro card (including the flow simulation). The women at the boot said the dataset is 4Gb and it's all residing on the card.

Stereo cartoons for free

I always follow closely what is coming out of Fredo Durand's lab at MIT. They have proved to have a good artistic intuition and they usually come up with very simple ideas. The "2.5D cartoon models" paper they presented this morning is no exception. The video clip that comes with the paper explains the whole thing better than I could do it:


MIT 2.5D Cartoon Models from lester banks on Vimeo.

Their system requires no 3D modelling. Instead, you draw your character from different points of views and the system extrapolates the 3D information.

The program is available for free on the project web site (without the source code unfortunately). They don't mention the possibility to create stereoscopic animation, but it can easily be done without modifying the software. The only device required to operate the software is a simple mouse or a tablet and it runs on computers with very modest hardware.

The programmer is not going away

There is an ongoing race in the world of computer graphics between those who want to democratize the tools and those who are making them more complicated. And unfortunately, the former is never catching up.

For those who hoped they would no longer need a programmer, guys behind the paper entitled "The Frankencamera: An Experimental Platform for

Computational Photography" come with bad news. Andrew Adams started his presentation with the following statement: "we think you should be able to program your camera". Fortunately, he's providing a simple C API that allows to decide when and for how long the shutter opens, how wide the aperture and to set a bunch of other shooting parameters. The acquired images can then be processed within the camera.

None of the applications presented (HDR, panoramas, denoising, sharpening) are new, but it just got a lot simpler for researchers in the field to experiment with new computational photography algorithms especially since the presented API also runs on a standard Nokia N900 smartphone.

Other than this Frankencamera paper, the computational photography research presented this year is not very interesting.

The big picture

One of the first things that struck me when I started to work on a demo for a film about the brain at the NFB was how little I knew about the brain, even though I had worked at the Montreal Neurological Institute for 3 years before. I wish I had taken more time to get the big picture. That's why I was very interested by what Stephen Larson from UC San Diego had to say.

Larson was here to introduce the Siggraph crowd to the world of brain image visualization during a session called "Biomedical Applications: what you need to know". He's interested in using visualization tools to help study the brain across all scales, something that is really difficult to do now. He is involved with the development of a program called "The Whole Brain Catalog" who's goal is to do exactly that. He's shown a particularly interesting video that they produced at UC San Diego to demonstrate the possibilities of the program.

Larson mentioned an interesting web resource called molecularmovies.org. The web site presents "an organized directory of cell and molecular animations, as well as a collection of original tutorials for life science professionals learning 3D visualization".

Some of my NFB colleagues might be interested in knowing that Larson used a map of the London subways system as a metaphor to illustrate interconnections in the brain. They might also be interested to know that Steve Larson's talk generated a huge interest here at the conference and that they might not have the only film about the brain in preparation.

Human perception and graphics (Stereo)

A very enlightening course this afternoon about the way knowledge of the human visual system can be applied to several aspects of computer graphics. For example, the course went though different notions that might help in directing the viewer's attention to certain parts of an image.

Part of the course (talk by Marty Banks from UC Berkeley) focused on stereoscopic displays. Among other things, he highlighted a problem with stereoscopic displays that I wasn't aware of: the decoupling of convergence and accomodation. When watching a real-world scene, the eyes converge and focus on the same point. With a stereoscopic display, the eyes always focus on the screen but converge at different distances depending on the content of the images. That's what makes it difficult and tiring to watch objects that are too far off the screen. Banks has built many psychophysics experiments to highlight weaknesses of stereo displays and he suggests techniques to compensate for them.

Human visual perception is a topic to which the computer graphics community should pay more attention.

The future of computer graphics research

One of the reasons you attend Siggraph is to hear what the future of computer graphics will be made of. So when a group of legend of the field join for a panel called “Future Directions in Graphics Research”, you attend. That’s what I did this afternoon.

The panel participants each went through a thread of potential future research. The topics covered included accurate physical modelling of the human body or working with extremely large data sets to the new opportunities for using computer graphics in education and digital literature. My favorite part concerned new ways to interact with graphics on the computer. The recent multiplication of cheap hardware devices that can be used to produce or interact with graphics (cell-phone cameras by the millions, multi-touch screens, 3D cameras, etc.) forces us to reconsider user interface paradigms. We need to draw from psychology to create interfaces that are more engaging. Furthermore, the ubiquity of broadband network opens up new possibilities for programs that let users collaborate. This is good news for one of my recent projects (Lines), a frame-by-frame animation program that lets many artists work on the same animation in real-time.

Perhaps the most interesting comment came from the audience. The man said that until recently, a lot of the research in computer graphics focused on producing synthetic images that are indistinguishable from a real photograph. We are not there yet, but close enough to have many people wonder: “what’s next”. He suggested the new holy grail should be something like creating a physical device that is indistinguishable from a real window.
I heard many interesting ideas during this panel, but overall, nothing groundbreaking. I'm a little disappointed.

Mecca

Sunday morning in the LA convention center. The impressive crowd of the Siggraph conference (roughly 30 000 people) is rushing through registration. I feel like a kid in a candy store. I have to remember to eat at least once every day.

5 days before flying to LA

Only 5 days before I fly to LA for my first Siggraph in a while. I'll be using this blog to share my thoughts on the cool graphics I see at the conference and the millions of software ideas it will generate in my head.