Now, with lots of help from Brent Cowan (our Graphics TA), I've developed a proper cel shader which looks totally awesome! Cel shading tutorials online (like lighthouse3D for GLSL) also helped, and provided me with information and know-how to understand how a cel shader does what it does.
Above is an image contrasting my shader vs OpenGL default lighting. There are actually 2 separate shaders working to achieve this effect.
The first shader is a shader that does the outlines. The shader file itself is actually very, very simple. All it's doing is moving a vertex along the normal to 'fatten' the model. I'm telling OpenGL to disable lighting (so that it's one solid colour), and to cull faces that face the camera, as opposed to away from the camera before the shader is actually used. The back-face colour is what makes the colour of the outline, the shader can take that as a parameter (colour) and also line width. That way, we can have different line thicknesses and colours for each object, or even a different state of the same object.
This would be something like a 'selected' effect |
The second shader takes care of the lighting, and actually draws another teapot inside the 'fattened' version. Depending on the angle between the normal and the light, the shader just clamps diffuse/specular (etc.) values in a certain range. This can be done for a variety of light types, not just diffuse.
Diffuse + Specular cel shading |
'Glassy' + Diffuse + Specular |
The darker areas look a bit gray due to the ambient value, I'm not sure if either looks better.
Without the ambience, the black outlines blend into the teapot, so we're going to stick with ambience for now.