GLSL Fruit Shaders

Fig 1.1: Screen captures of orange and apple shaders

For an OpenGL project I am helping fellow NYU-ITP graduate student Matt Parker with; I developed two real-time shaders written in GLSL to be applied to poly-spheres representing two types of fruit: apples and oranges. Due to the sheer number of spheres being displayed, the object on my end was to get a high-amount of surface detail using a limited amount of polygons. To accomplish this, I used a combination of displacement and normal mapping calculation in the shader, meaning that all surface detail is accomplished through the GPU.

I have already written about how displacement is done through shaders - the very same technique I used in the past was also used in this set of shaders:
Geometry Displacement through GLSL

Color variance along the geometric surface in regard to light has changed apart from my past shader work, where I have written in ambient and specular calculations through accessing the light (L) and given surface material (M), where:

Diffuse:


Ambient:


Specular (where H is the given half-vector):


To simulate the surface shape variance needed to pull off the appearance of an orange, a popular technique called "bump-mapping" was used. Bump-mapping is generally accomplished by having a normal-map which is sampled as XYZ vectors which are supplemented across the geo-normal matrix that the shader is applied to. The Lambert term for the normal is calcuated, which is simply the dot product of the normal and the light direction. If the lambert term is above 0.0, calcuate specular and add to the final color for the fragment. Here is some sample code to illustrate:

vec3 N = texture2D(normalMap, gl_TexCoord[0].st).rgb *
               2.0 - 1.0;
float lambert_term = max(dot(N,L),0.0);

if (lambert_term > 0.0)    {
// Compute specular
vec3 E = normalize(eyeVec);
vec3 R = reflect(-L, N);
float specular = pow(max(dot(R, E), 0.0),
                     gl_FrontMaterial.shininess);
}

The finished shader encompasses texturing, displacement, and normal mapping, resulting in a nice aesthetic simulating a complex surface with light calculations matching the surface variance. All of this occurring on a low-poly (16 x 16) quadric sphere object. Rotation animation simulated in real-time of the orange shader below:


Fig 1.2: Orange shader rotation animation


Sources for general information/reference and equation images:
ozone3D.net
Lighthouse3D.com
OpenGL.org

Maya: Fluid Gradient Programming (Plug-In)

Fig 1.1: Still of fluid-based fire simulation, with colors generated through plug-in gradient mapping system

I am currently developing a plug-in for Maya, a popular 3D modeling/animation package. The goal for the plug-in as a whole is to reinvent how fluid dynamic systems/effects are developed in Maya by constructing a node-based GUI which can be used within Maya, with each node encapsulating a current feature Maya fluids currently have - essentially adding on a layer of abstraction for the user to more easily generate simulations. The plug-in also aims to extend the current feature set of fluids within Maya.

I decided to start the development of the plug-in with a quick experiment, that being the integration of the generic gradient object class I developed earlier this semester with the Maya native gradient data driving color and incandescence. The aim being that images could be sampled and have their most dominate colors applied to the color/incandescence of the fluid voxel-system within Maya - thus automating the construction of the color palette for a given effect. The results and rendered video are posted below:


Fig 1.2: Default Maya Fluid Effects values



Fig 1.3: Early development of Maya plug-in as a drop-menu



Fig 1.4: Image chosen to have dominate colors sampled



Fig 1.5: Image sampled gradient constructed and mapped into Maya's attribute data



Fig 1.6: Fluid fire simulation animation, colors derived from gradient sampling without modification

Farewell ITP

So, I have decided to leave ITP. I will be transferring at the end of the semester.

There are some decisions that deserve explanation - this probably qualifies as one. I do my best to write down important decisions I make throughout life so that I may reflect on my thought-process later on... for better or for worse, many times. I have thought about leaving ITP for a couple of months now - and I have done my very best to give myself a chance to make sure that leaving is the right choice, so this is by no means a rash decision on my part. I am writing at length about my decision to leave not so much as to somehow glorify my personal decision on this matter, but rather; to explain why I am leaving in a way that will hopefully be helpful.

ITP is a great place... for the right type of person. Having a department which is open-ended allows for a very nurturing environment for both those still searching for their career path and for those simply interested in thinking outside the normal realm of art or tech-related education. The faculty is outstanding and generally eager to pass their knowledge along. There is plenty of motivation to experiment and little pressure to fully implement - it is less about low-level execution and more about conceptualizing ideas. ITP is a playground; you can essentially do anything - and that is just fine... if that is what you are looking for. However, if you are looking for a very specific education and research support dedicated to that topic - ITP is not going to give you those kinds of resources. You are given a sampling of skills to absorb during your first semester - then you pick your own classes, which with the ITP curriculum, is nearly impossible to align along a certain path. This is clearly by design. In short, while you may earn a Masters degree, you'll be hard pressed to 'master' a certain skill-set at ITP, and rather finish being decent in many areas. This may be a positive for some, a negative for others such as myself - but it is certainly the nature of the program.

A large part of why I am leaving ITP is due to lack of project consequence. Projects are designed, developed to a prototypical-level, then for the most part, are forgotten about. An area ITP would certainly benefit from would be in interfacing with the professional world more aggressively. This would achieve several things: it would make projects matter and have more tangible consequence, which they currently almost certainly do not achieve outside of personal achievement. Such professional connections would establish ITP more firmly as a presence in various industries, and it would allow students to get realistic perspective on design and execution moving forward in their studies. This will potentially enable students to work with materials normally out of their price range with funding from corporate sponsors. Now, it is my understanding that many at ITP do not want to interface with the industry and want to develop their ideas in their own way - and that is just fine, such collaboration would ideally be purely optional. However, to the many others suggesting that being funded by corporations is "selling out" - I believe that it is concise enough to say: it is time to grow up.

Explaining what ITP is, as many know - is a rather tricky thing. As I recall - in an interview I had recently with a company, I was asked to explain what my graduate department was and what was researched there. This task ended up taking me quite awhile. I did try to sneak a quick definition by first in hopes they wouldn't persist... "Well, it is essentially a study in varying topics relating to technology." No sale. They wanted specifics - and rightfully so, despite my best efforts at deflection. So, with a deep breath - I did my best to explain. "It is consortium of ideas relating to technology - disciplines relating to art, design, technical understanding, these skills are tied together to approach technology. Topics relating to social networking, sustainable energy, physical computing, graphics programming, web design - to name a few, are evaluated from a theory basis, and to a degree, are iterated on through implementation." I can assure you, that quote is not exactly a perfect representation on how I phrased it, as there were surely several "uh"'s and pauses on my part. The reaction? As was interviewer concisely put it: "So, it isn't really a concentration so much as a general think tank on technology, correct?" Yikes. As someone applying for a very technical and implementation-centric position, having your graduate degree related to a "general think tank" is like being a chef and having your food specialty being related to fast food. This relation could be due to what may have been a poor definition of ITP on my part - but looking back on it, it was really the best definition I could provide, and still is. I must reiterate, this definition may be just fine for a lot of people - but for me, it defines a place clearly not for me. The reason I cited this encounter is because it was what made me realize that I needed a change, and I feel it really encapsulates my motivation for leaving.

And so, that is my long explanation as to why I am leaving. I'll be leaving on April 28th to return back to the Bay Area for the summer - then I will begin my new graduate studies in the Fall. I have enjoyed my time in New York and will look back on it fondly, there is no city quite like NYC. It has been wonderful meeting a lot of the great people involved at ITP - I sincerely wish you all well!

Thanks,
William McDonald
wmcdonald@cmu.edu

Animation: Luma-Displacement / Glow (OpenGL / GLSL)

Sphere with an animated displacement GLSL shader that calculates a glow / bloom effect along defined distance from midpoint.

Luma-Based Displacement

Fig 1.1: In-progress glacier shader with geometric displacement

Python vs. Java vs. Python/Weave

In case you haven't noticed, I am a rather big proponent of using Python whenever possible. Between Python's flexibility and just the enjoyment I get out of writing in the language, I generally look to use Python before turning to a static language such as C/C++ or Java. The times I have had to turn to a compiled language are usually linked to when I need computational speed.

There are several ways to get Python moving faster - some a bit more ugly than others; code speedups with Python are usually about trading in readability for performance, but that doesn't mean it can't be straight-forward in implementation. In the past, I have used the usual bag of Python tricks: list comprehensions, method/variable assignment outside loops, generators, and countless more pure-Python solutions.

Recently, I stumbled upon Weave, a module existing inside the popular SciPy library which allows for C code to be mixed with Python code. Using Weave allows for the coder to stay inside the bounds of Python, yet use to power of C to compute the heavier algorithms. Not only does it provide a speed increase... it beats Java. Consider this (nonsensical) algorithm comparison between pure Python, Java, and Weave + Python:


Python Example:

Time: 1.681 sec.


Java Example:

Time: 0.037 sec.


Python + Weave Example:

Time: 0.017 sec.


Using identical algorithms in all three tests; using Weave with Python was nearly 100 times faster (98.9x) than using pure Python alone. In comparison to Java, Weave + Python was a shade over 2 times faster (2.1x) than Java. Although the Python/Weave code itself is bigger than the pure Python and Java examples - the speed it provides is absolutely phenomenal, without ever leaving your .py file.

Hopefully, I'll be able to use the power of mixing C with Python for more elaborate purposes - but this test alone is enough to get me excited about future projects with Python.

DayBRIGHT: Gradient / Render Integration

I have now managed to get DayBRIGHT to render out a full animation using a gradient as the light source. While the animation scene itself is rather basic, with three spheres inside a box, it demonstrates some of the primary goals of the project.

Now that the gradient data structure integrates with the rendering process, it now becomes a matter of writing the translators for each type of media I'd like to use - as well as developing a more interesting scene to light.




Fig 1.1 + 1.2: Animation using displayed gradient as lighting source over time


Indirect surface shader code:

DayBRIGHT: Post-Processing

Changing gears a bit, I decided to begin writing in some post-image processing functionality to DayBRIGHT to handle the final image output. The first post-process was writing a color noise filter that blended a 1:1 ratio of random colored pixels with each rendered image . This is done to counter the fact that the illusion of computer graphics being reality is often blown due to a discrepancy in noise artifacts across the image, most commonly noticed when graphics are composited with film or video footage. Even without footage, the human mind does not normally process perfectly smooth color and shape with reality. Reality rarely has instances of complete visual fidelity, at least from an everyday human perspective.


Fig 1.1: Simple color noise composite process

Another post-process that is necessary was gamma correction. The algorithm resulting from the equation is straight-forward, where a gamma corrected set of values from 0-255 is created, then each RGB value from the given image is changed to the gamma corrected equivalent. Below is the simple math equation for gamma correction and corresponding algorithm in (non-optimized) Python code:


Fig 1.2: Gamma correction equation

DayBRIGHT: Renderman Rendering

Below are some test renders coded through Renderman and woven into the DayBRIGHT graphics and data handling subsystems. Using the classic Cornell box, I was able to successfully simulate global illumination in combination with image based lighting. The photon calculation isn't as smooth (or as fast) as I'd like it to be - but optimization will be taken care of later once I can generate reliable image sequences.


Fig 1.1: Global illumination / color bleeding


Fig 1.2: Image-based lighting with accurate shadows. Shadows no longer gradient of grays, but rather mixed with color pertaining to casting objects.


Fig 1.3: Classic Cornell box setup with pronounced color bleed

DayBRIGHT: Image to Gradient Translation

The first translator I decided to fully flesh out for DayBRIGHT is the static image translator. The underlying system I have developed only has one principle rule - anything can be translated and used, as long as that translation returns a gradient object. So, how can a gradient be created from an image?

I took the route of writing an algorithm which finds and stores the most dominate colors in an image - then using each sample as a control point within a varying length linearly-smoothed gradient, where the most dominate colors are in order from left to right. Below are some examples of the results, where the top is the source that was sampled and gradient is the output visualized:


Fig 1.1: Sky converted to a gradient based on dominate colors


Fig 1.2: Composition VI, W. Kandinsky - image to gradient conversion


Fig 1.3: Composition VII, W. Kandinsky - image to gradient conversion

Now that a method existed to translate static images into gradients, it becomes a trivial task to write a translator for a video feed, as video can be easily treated as a series of images. Simple alterations to a subsystem can allow for a calculated sample of each frame in a gradient. So, if a video feed were parsed once every hour for twenty four hours, a gradient could be constructed of n-length, linearly being smoothed evenly across each respective control point based on n+1 distance.