Will at ITP

GLSL Fruit Shaders

2008-04-23T12:12:00.014-04:00

Fig 1.1: Screen captures of orange and apple shaders

For an OpenGL project I am helping fellow NYU-ITP graduate student Matt Parker with; I developed two real-time shaders written in GLSL to be applied to poly-spheres representing two types of fruit: apples and oranges. Due to the sheer number of spheres being displayed, the object on my end was to get a high-amount of surface detail using a limited amount of polygons. To accomplish this, I used a combination of displacement and normal mapping calculation in the shader, meaning that all surface detail is accomplished through the GPU.

I have already written about how displacement is done through shaders - the very same technique I used in the past was also used in this set of shaders:
Geometry Displacement through GLSL

Color variance along the geometric surface in regard to light has changed apart from my past shader work, where I have written in ambient and specular calculations through accessing the light (L) and given surface material (M), where:

Diffuse:

Ambient:

Specular (where H is the given half-vector):

To simulate the surface shape variance needed to pull off the appearance of an orange, a popular technique called "bump-mapping" was used. Bump-mapping is generally accomplished by having a normal-map which is sampled as XYZ vectors which are supplemented across the geo-normal matrix that the shader is applied to. The Lambert term for the normal is calcuated, which is simply the dot product of the normal and the light direction. If the lambert term is above 0.0, calcuate specular and add to the final color for the fragment. Here is some sample code to illustrate:

vec3 N = texture2D(normalMap, gl_TexCoord[0].st).rgb *
               2.0 - 1.0;
float lambert_term = max(dot(N,L),0.0);

if (lambert_term > 0.0)    {
// Compute specular
vec3 E = normalize(eyeVec);
vec3 R = reflect(-L, N);
float specular = pow(max(dot(R, E), 0.0),
                     gl_FrontMaterial.shininess);
}

The finished shader encompasses texturing, displacement, and normal mapping, resulting in a nice aesthetic simulating a complex surface with light calculations matching the surface variance. All of this occurring on a low-poly (16 x 16) quadric sphere object. Rotation animation simulated in real-time of the orange shader below:

Fig 1.2: Orange shader rotation animation

Sources for general information/reference and equation images:
ozone3D.net
Lighthouse3D.com
OpenGL.org

Maya: Fluid Gradient Programming (Plug-In)

2008-04-16T15:15:00.010-04:00

Fig 1.1: Still of fluid-based fire simulation, with colors generated through plug-in gradient mapping system

I am currently developing a plug-in for Maya, a popular 3D modeling/animation package. The goal for the plug-in as a whole is to reinvent how fluid dynamic systems/effects are developed in Maya by constructing a node-based GUI which can be used within Maya, with each node encapsulating a current feature Maya fluids currently have - essentially adding on a layer of abstraction for the user to more easily generate simulations. The plug-in also aims to extend the current feature set of fluids within Maya.

I decided to start the development of the plug-in with a quick experiment, that being the integration of the generic gradient object class I developed earlier this semester with the Maya native gradient data driving color and incandescence. The aim being that images could be sampled and have their most dominate colors applied to the color/incandescence of the fluid voxel-system within Maya - thus automating the construction of the color palette for a given effect. The results and rendered video are posted below:

Fig 1.2: Default Maya Fluid Effects values

Fig 1.3: Early development of Maya plug-in as a drop-menu

Fig 1.4: Image chosen to have dominate colors sampled

Fig 1.5: Image sampled gradient constructed and mapped into Maya's attribute data

Fig 1.6: Fluid fire simulation animation, colors derived from gradient sampling without modification

Animation: Luma-Displacement / Glow (OpenGL / GLSL)

2008-04-02T22:49:00.005-04:00

Sphere with an animated displacement GLSL shader that calculates a glow / bloom effect along defined distance from midpoint.

Luma-Based Displacement

2008-04-01T21:07:00.003-04:00

Fig 1.1: In-progress glacier shader with geometric displacement

Python vs. Java vs. Python/Weave

2008-03-24T13:44:00.011-04:00

In case you haven't noticed, I am a rather big proponent of using Python whenever possible. Between Python's flexibility and just the enjoyment I get out of writing in the language, I generally look to use Python before turning to a static language such as C/C++ or Java. The times I have had to turn to a compiled language are usually linked to when I need computational speed.

There are several ways to get Python moving faster - some a bit more ugly than others; code speedups with Python are usually about trading in readability for performance, but that doesn't mean it can't be straight-forward in implementation. In the past, I have used the usual bag of Python tricks: list comprehensions, method/variable assignment outside loops, generators, and countless more pure-Python solutions.

Recently, I stumbled upon Weave, a module existing inside the popular SciPy library which allows for C code to be mixed with Python code. Using Weave allows for the coder to stay inside the bounds of Python, yet use to power of C to compute the heavier algorithms. Not only does it provide a speed increase... it beats Java. Consider this (nonsensical) algorithm comparison between pure Python, Java, and Weave + Python:

Python Example:

Time: 1.681 sec.

Java Example:

Time: 0.037 sec.

Python + Weave Example:

Time: 0.017 sec.

Using identical algorithms in all three tests; using Weave with Python was nearly 100 times faster (98.9x) than using pure Python alone. In comparison to Java, Weave + Python was a shade over 2 times faster (2.1x) than Java. Although the Python/Weave code itself is bigger than the pure Python and Java examples - the speed it provides is absolutely phenomenal, without ever leaving your .py file.

Hopefully, I'll be able to use the power of mixing C with Python for more elaborate purposes - but this test alone is enough to get me excited about future projects with Python.

DayBRIGHT: Gradient / Render Integration

2008-03-12T13:09:00.014-04:00

I have now managed to get DayBRIGHT to render out a full animation using a gradient as the light source. While the animation scene itself is rather basic, with three spheres inside a box, it demonstrates some of the primary goals of the project.

Now that the gradient data structure integrates with the rendering process, it now becomes a matter of writing the translators for each type of media I'd like to use - as well as developing a more interesting scene to light.

Fig 1.1 + 1.2: Animation using displayed gradient as lighting source over time

Indirect surface shader code:

DayBRIGHT: Post-Processing

2008-03-03T19:24:00.008-05:00

Changing gears a bit, I decided to begin writing in some post-image processing functionality to DayBRIGHT to handle the final image output. The first post-process was writing a color noise filter that blended a 1:1 ratio of random colored pixels with each rendered image . This is done to counter the fact that the illusion of computer graphics being reality is often blown due to a discrepancy in noise artifacts across the image, most commonly noticed when graphics are composited with film or video footage. Even without footage, the human mind does not normally process perfectly smooth color and shape with reality. Reality rarely has instances of complete visual fidelity, at least from an everyday human perspective.

Fig 1.1: Simple color noise composite process

Another post-process that is necessary was gamma correction. The algorithm resulting from the equation is straight-forward, where a gamma corrected set of values from 0-255 is created, then each RGB value from the given image is changed to the gamma corrected equivalent. Below is the simple math equation for gamma correction and corresponding algorithm in (non-optimized) Python code:

Fig 1.2: Gamma correction equation

DayBRIGHT: Renderman Rendering

2008-03-01T14:02:00.007-05:00

Below are some test renders coded through Renderman and woven into the DayBRIGHT graphics and data handling subsystems. Using the classic Cornell box, I was able to successfully simulate global illumination in combination with image based lighting. The photon calculation isn't as smooth (or as fast) as I'd like it to be - but optimization will be taken care of later once I can generate reliable image sequences.

Fig 1.1: Global illumination / color bleeding

Fig 1.2: Image-based lighting with accurate shadows. Shadows no longer gradient of grays, but rather mixed with color pertaining to casting objects.

Fig 1.3: Classic Cornell box setup with pronounced color bleed

DayBRIGHT: Image to Gradient Translation

2008-02-26T16:43:00.007-05:00

The first translator I decided to fully flesh out for DayBRIGHT is the static image translator. The underlying system I have developed only has one principle rule - anything can be translated and used, as long as that translation returns a gradient object. So, how can a gradient be created from an image?

I took the route of writing an algorithm which finds and stores the most dominate colors in an image - then using each sample as a control point within a varying length linearly-smoothed gradient, where the most dominate colors are in order from left to right. Below are some examples of the results, where the top is the source that was sampled and gradient is the output visualized:

Fig 1.1: Sky converted to a gradient based on dominate colors

Fig 1.2: Composition VI, W. Kandinsky - image to gradient conversion

Fig 1.3: Composition VII, W. Kandinsky - image to gradient conversion

Now that a method existed to translate static images into gradients, it becomes a trivial task to write a translator for a video feed, as video can be easily treated as a series of images. Simple alterations to a subsystem can allow for a calculated sample of each frame in a gradient. So, if a video feed were parsed once every hour for twenty four hours, a gradient could be constructed of n-length, linearly being smoothed evenly across each respective control point based on n+1 distance.

DayBRIGHT: Generic Structures

2008-02-19T21:07:00.003-05:00

The underlying system DayBRIGHT uses to translate various data structures has become less about constraining data to a set format, and more about developing a generic structure (a gradient) to fully represent the data in an elegant way. I find this an interesting topic to explore because it has really opened a lot of doors as to how far I can take his project; in particular, in the sheer variance of data I can now use.

RSS feeds, video streams, static images, audio tracks, physical sensor data - all of which can be harbored by a single generic data structure. Each piece of data simply needs its own translator towards being converted to the structure; once that is done - there is no further special-casing needed.

My goal for this project is to have, by the end of semester, a demonstration of how the various types of data listed above can be used to achieve results for an identical operation. In the case of DayBRIGHT, I will be using the gradient structures to light 3D geometry. How this will be done merits its own detailed entry - which is forthcoming.

I am really excited about this idea, it is a blend of engineering/mathematics and visual aesthetics that I have always been drawn to. I am dedicating all of my classes to it in hope that my time commitment will really push this idea as far as it can go.

DayBRIGHT: Proposal + Test Video

2008-02-15T17:59:00.008-05:00

The idea behind the DayBRIGHT project is to be able to visualize data through HDR light probes being influenced by various types of datasets. Each dataset, be it an RSS weather forecast, webcam frame sequence, arbitrary color selection, or any other value - will be translated into a common format to be applied to a core HDR image.

The DayBRIGHT project will be encapsulated within a core software application which will communicate with various data feeds. As mentioned, each type of data will be converted to a common format: a color gradient. A color gradient will allow for analysis of RGBA, as well as time by sampling across its length. Depth can also be calculated by converting RGBA to luminance values.

To render the results, the initial solution will be to use 3Delight, an implementation of the Renderman standard, first introduced by Pixar. Using Renderman will allow for programming flexibility and high-quality renders at HD resolutions, if needed. An example of the early results is shown below, where an orange -> cyan color gradient has been applied:

DayBRIGHT: Gradient Sampling

2008-02-13T21:45:00.004-05:00

One of the core concepts of the DayBRIGHT project that I am developing over this semester is to enable the ability to drive HDR lighting over time using a variety of methods - the first of which is using a simple color gradient.

For every frame to be rendered, a copy of the core HDR image is created, then a color overlay operation is applied to that copy - with the color applied being the current sample value of the gradient. This relationship is graphically represented below:

Fig 1.1: Graphical example of gradient sampling applied to image

Mind.Scribble.Form

2008-02-10T22:04:00.000-05:00

My mind blurred, sketched, articulated, and scribbled on paper - most of which makes little to no sense to me currently. I normally never write things down to remember for the future - I generally only do so to calculate or brainstorm on the spot. Apparently, in parallel with my messy drawings, I also take photos without much care either. I think I'll write that off as part of my "creative process" and hope that you believe me (which you shouldn't).

Blast Radius

2008-02-06T13:47:00.000-05:00

Several layers of particles revolving around a central sphere - each layer being rotated based on assigned attraction forces. Particles are symbolic of populous movement combined with the effect of unseen influence over collective entities - physically, socially, and otherwise.

Fig. 1.1: Straight-on shot

Fig 1.2: Corner angle

Source:
BlastRadiusMain.java
GLSL.java

Interactive and Time-Based HDR

2008-02-01T22:29:00.000-05:00

I have been thinking a lot about pixels lately - most likely due to having a class with the given root word in it... twice. So, we have all of this information nicely packed within a frame - now what to do with it?

What I find interesting is how an image can be mined for data that is unseen upon first glance - and one of my favorite ideas in regard to this way of thinking is the creation and use of HDR images. High-Dynamic Range images are useful for an assortment of reasons - in the field of computer graphics, they are many times used as a light source. Paul Debevec is known as one of the groundbreaking researchers on the topic - and what initially began as technique for expensive high-end rendering has now been implemented in real-time in the past couple of years. Check out his site to see some of the amazing things HDR images can be used for.

Fig 1.1: Example of a light-probe HDR image
Credit: http://www.debevec.org

So, what else can been done with HDR - and how can it be used to create something interesting? For me, I want to explore moving away from leaving HDR lighting as a static image - and instead have it evolve over time while lighting a static scene. A few ideas are: time-lapse HDR lighting, real-time interaction with the HDR image itself, tying HDR colorization with weather forecasting, integration with existing data sets, and so forth.

Programming, Ubiquity, Devices, and You

2008-01-17T17:23:00.000-05:00

After telling fellow students at ITP about the classes I decided to sign up for the coming Spring semester, I have often been asked the following two questions:

1. "Why are you taking programming centric classes if that is already one of your strengths?"

2. "Why aren't you taking any physical computer related classes?"

The answer to the first one is rather concise; I feel that staying constantly involved in software and mathematics allows me to build myself as a better overall problem-solver on both an analytical and creative basis. Writing code to solve a problem is concurrently a step-by-step breakdown of the problem itself; patterns emerge and reveal themselves, woven with systems of varying elegance and complexity. Understanding concepts which reveal such harmony will endure far past simply designing the next iPhone - leading me to the answer to the second question:

My position on physical computing at it pertains to projects at ITP (and in general) is that the development of devices which we physically interact with is inevitably obsolete. This is the century of biology - and consequently, biological micro-interfacing will be the future of device technology. Increased ubiquity leading to result/reward has nearly always been the trend in tool advancements - computing is to exist behind a curtain whenever possible. While it is certainly a valid venture to develop large-scale projects involving common electronics and so forth - I personally have little interest in pursuing similar implementation methods.

Software for Physical Display Project

2007-12-10T22:12:00.000-05:00

Group: William McDonald, Sunghun Kim, Matt Parker

Further documentation:
Sunghun with videos / images
Matt on construction / mechanics

For our group project, we wanted to develop a physical system to represent graphical depth data. We decided to use a collection of sine and noise waves to start off with - as using depth data really only makes (practical) sense with a 3D grid.

Using Processing, a straight-forward GUI was created featuring four buttons, with a graphical representation of the waves being dynamically created to the right of the buttons. The fourth button allows the user to paint his/her own wave (or any shape, for that matter) and the software will generate a visualization and physical output for the image painted.

Fig 1.1: Drawing canvas allows user to draw images to physically visualize

Fig. 1.2: Visualization of user input wave drawing

To sample each wave - an algorithm was written which samples across the width of a given image proportional to the resolution of the physical system - in our case, 20 samples. At each sample - the program checks to see how far down (Y) the image it takes in pixels before a color other than white is found. Upon finding a color, the value of that sample is received by the simple equation of :

sample_value = y_coord / (image_height / max_value)

Fig 1.3: Graphical representation of sampling method on wave images

In regard to feeding this data into the servo motor array - we decided to use four arduino microprocessors connected via USB to the computer running the software application; with each arduino effectively controlling five servos:

Fig 1.4: Computer / Arduino / Servo array configuration

Shiver - Final

2007-12-08T15:35:00.000-05:00

Fig 1.1: Splash screen for Shiver

The last week of work on Shiver has primarily been a focus on adding features on the GUI side and tying up some loose ends. The largest new feature is the ability to 'simulate' - which allows a user to simulate a full rotation (or however many wanted) of the globe, outputting the frames as an image sequence. This process allows the user to see the rotation as a movie clip running at full speed. The video in this blog entry is derived from that process.

Fig 1.2: Simulate options window

Shiver is still very much a work in progress - however, I am pleased with the result of my work over the past 6-7 weeks, particularly with knowing that I was able to achieve high-quality imagery in real-time using Python and OpenGL together.

shiver_simulation.mov - 10.6mb

Images from Shiver:

Shiver Development: Event Density + Time Charting

2007-12-04T21:19:00.000-05:00

For the past few days, I decided to develop a type of visualization which attempts to better communicate the areas that have a high density of earthquakes over time. I decided to do this by having the program create a texture on the fly which is fed into the primary globe fragment shader, where areas of high seismic event activity would get higher amounts of red. Since the texture being created is simply a collection red blobs on a black background, it is simple process to add the red areas on top of the globe through the shader. Additionally, the seismic events themselves are visualized through simple lines aligned to the normals of the sphere - with the length of the line determined by its Richter scale value.

Fig 1.1: Globe with event density visualization

After wrapping up event density mapping, I wanted to start trying to visualize events as they occurred over time. The visualization itself doesn't communicate time as much as it displays the erratic nature of seismic activity happening around the world over an extended period (currently, over seven days). To create lines that seemingly wrap themselves around the curvature of the globe - I decided to write a small bezier curve generator which had end points at the 3D coordinates of selected events. The distance of those events is calculated using the Euclidean distance formula of:

Where the distance value is used to apply weight (w) to midpoints in rational bezier curves, such that:

Fig 1.2: Bezier curves mapped across globe based on time of occurrence

Shiver Development: Displacement

2007-11-28T01:22:00.000-05:00

As I continue to polish the overall look of the globe - I decided to explore how to display the topology of the earth, which will become more important upon the incorporation of zooming in and out. Another plus for having a working shader pipeline is the ability to write vertex shaders in a way which changes the geometry of the given shader is attached to - better known as displacement.

Fig 1.1: Altitude map provided by the NASA Blue Marble project

Having an altitude map which precisely matches the texture maps I am already using is quite convenient - so there was no need to alter the image or the coordinate mappings in the my shaders. The concept of displacing geometry per vertex is a rather simple one:

Fig 1.2: Formula for displacement per vertex, where:

P0 = original vertex position
p1 = new vertex position
N = vertex normal position
df = normalized displacement factor
uf = user-defined scaling value

Fig 1.3: Diagram of per vertex displacement (Image credit: oZone3d.net)

Fig 1.4: No displacement / displacement comparison in Shiver (wireframe)

My next goal for Shiver is to implement a sunlight calculation system which will auto-generate a texture map accurately representing where how the sun is lighting the earth based on the time of day. Additionally, I plan on (finally) starting the preliminary visualization of the actual seismic events being geomapped.

Woven

2007-11-23T17:04:00.000-05:00

I developed a small piece of software for a group presentation I had in Applications class that has to do with visualizing collaboration among 1st year students at ITP. Someone during the Q+A asked if the software was open-source and if so, where could the source be downloaded? The answers were "yes" and "soon, on my blog". Unfortunately, I don't really have the time as it stands to package the software up nicely for download - so I am just going to post the raw source code for now, with the hope that I can package it later on...

Source:
ProcessingMain.java
StudentDataParser.java
Student.java
Button.java
student_data.txt

Shiver Development: Event Mapping

2007-11-23T11:22:00.000-05:00

A couple of new updates; one being that I have a new fog shader working on the globe - which adds to the effect of the atmospheric layering. The ability to program shaders and apply them to geometry is really what separates imagery that is noticeably fake and that which is nearing photorealistic. Real-time photorealism is the next big step in computer graphics - although you could make a very valid claim that photorealism hasn't been reached in pre-rendered form either.

Fig 1.1: Comparison between shader enhanced and non-shaded geometry

The next update in progress is actually a rather large one - that being that I can now plot events on the earth sphere, needing only an event's latitude and longitude values. This was actually a larger challenge than originally anticipated, as I didn't take into consideration polar coordinates in regard to texture mapping in OpenGL as it relates to sphere mapping. I also ran into an issue pushing and popping the transformation matrix - but that turned out to be an instance of not having certain draw events happening between the correct combination of push/pop commands.

Fig 1.2: Seismic events mapped on globe

Shiver Development: Shaders in PyOpenGL

2007-11-11T21:43:00.000-05:00

After a turbulent weekend trying to get shaders working with PyOpenGL - I finally got my first vertex and fragment files compiled and working inside Shiver. Without any documentation to bail me out of my usual jams with new code - I hacked away through the weekend until I stumbled upon a way of changing C-type function arguments, then converted variables into C-compliant data to be fed into the altered function in question. In English: I needed to convert data so it could be read and processed correctly.

Fig 1.1: Globe with atmospheric shader applied

So - in the lab I am often asked: "why are shaders important"? Shaders can provide surface appearance variance in regard to color, opacity, reflection, refraction, etc... even physical alterations to the geometry itself. In relation to what I am attempting to achieve - it is vital to be able to replicate light and physical phenomena that exists when viewing the earth from space.

My first goal was to get a representation of the atmospheric layer that wraps around the earth - a thin haze of a light blue hue. The trick is - the haze should not rotate along with the earth as the camera moves around the globe. To achieve this; a shader was created that calculates how light is hitting the surface at each given frame - then simply finds where the light drop-off occurs on the surface (the edges) and colors those areas a varying hue of blue based on how much light exists on a given fragment of the geometry. Since I applied the shader on a sphere and the lighting is both hitting the sphere straight on and is static in all attributes over time - we can assume perfect symmetry in the light distribution across the sphere at any given time, which allows for the Lambertian Reflection calulation of:

Where:
I0 = reflected intensity
Ld = light diffuse color,
Md = material diffuse coefficient

Once we have the reflected intensity of each fragment, simple logic can be installed within the shader that only allows for color to be applied a certain areas of intensity - and, since we are dealing with a sphere with light shooting right at it (symmetrical light reflection!) - it becomes simple to create a halo light effect.

Fig 1.2: Closer look of the glow-like effect of the atmospheric shader

The next steps with using shaders are to create a fog shader for the globe, as well as the potential for a very subtle bump-map, as well as other effects as time permits.

Shiver - Final Proposal

2007-10-31T17:16:00.000-04:00

A few posts back - I wrote about developing a seismic data visualizer for my ICM final. This is still the case - however, the details involving the execution of the project are now very different. The most notable change involves adding several physical components to the project, meaning; this project will now be both my ICM and my Physical Computing project for this semester. Excited? Splendid - now on to the details...

Technical
On the ICM front - I made a decision a week ago that I would in fact use Python as my primary programming language opposed to my original choice of Java. Why? Well, as I mentioned in the past - Python is slow in regard to intensive computation in comparison to compiled languages such as C/C++ or Java. However, where Python shines is in its fast development time - and through its flexibility to glue together various components effectively. Without a doubt, hitting the visual high-bar I am aiming for in regard to the graphics in this project will put an enormous strain on Python - however, much of that strain can ideally be handled through C code, if necessary. I will be using an OpenGL binding called pyOpenGL - which is a mature library with strong documentation, so I should have information/learning support when I need it. For the GUI components, I will be using the wxPython toolkit - which allows for native OS widget use, so the program will automatically reflect the standard appearance of whatever OS the program is being run on. Very slick.

Fig 1.1: Shiver in early development, click to enlarge.

Representation
As for how the data will be represented, it will be done so on a 3D globe - much like applications such as Google Earth and NASA's World Wide Wind. When the applications gathers seismic data activity from the USGS - each event will be mapped to the globe according to its latitude and longitude values. Based on an event's Richter Scale value - the event will be visualized along the nearest normal to in correlation to its lat/long. mapping. Events can be selected through an easily organized directory structure tree, or on the globe itself. When selected - various attributes are displayed, as well as a simulation option - which leads us to the physical computing side of the project...

Physical
After sitting down with fellow ITP'er Sunghun Kim, we decided that we would like to team up and pursue a physical representation of seismic data that can be partnered with software I am developing for my ICM final. This idea will involve creating a 2D representation of the seismic events - dictated by the Richter scale value of each event. This will ideally result in a mesh being created by rods pushing and pulling at a rubber-like surface. Each rod will be driven of an average value of pixel color values dependent on the ratio between the rod count and the image resolution. Once a 2D simulation is established, it will ideally branch out to a 3D grid - but that is likely for another semester.

Elevator Previsualizer

2007-10-25T01:39:00.000-04:00

This is a midterm project I worked on with a talented group of people in Physical Computing. Follow the link for more:

Elevator Previsualizer Main Site

Phys. Comp Midterm: Implementation / User Observations

2007-10-17T16:22:00.000-04:00

For our midterm in Phys. Comp - my group decided to take on the task of breaking up the boredom which happens when waiting for an elevator. Petra, Sunghun, and myself set up what we had completed for the project this far the first time -- with mixed results. It is clear that we have a lot of implementation issues to resolve before getting the type of user interaction data we need for the later stages of polishing usability.

Fig 1.1: Will (me) interacting with camera.

Implementation Issues/Observations

1. The projector, if angled from the ceiling, will display a distorted shape due to the light being shot at an angle.

2. The image being shot through the projector does not immediately fit in the space of the door. The video can be clipped, but the light (although black) still shows up.

3. Even if the projector is raised up on the ceiling; the light from the projector will be blocked by people standing directly in front of the door.

4. There is no (reasonable) way of preventing significant light pollution at any given time because we are unable to strictly control the pubic environment, and we are bound by the collective lumen value the given projector can output.

5. We do not yet have a good solution for binding the light-blocking module to the projector – although this likely has a simple solution.

6. Due to the central and elevated nature of both the camera and the projector, light/motion recursion through the camera is very likely to occur unless steps are taken to filter an area – which has its consequences that will result in trade-offs likely too severe to compromise over.

7. The location of the crank has to be outside the field of the camera and projector. It is possible to have the crank located right in the middle, within the camera range but not obstructing the projector light - but this will mean that the user of the crank will be the focus of the image, which may create static results.

8. Due to the environment we are contained in, the projector will have to be turned sideways to establish the amount of vertical space needed to cover the door.

User Issues/Observations

1. Users are drawn to the computer if it is out in the open. They seem to glance at the projected image, then want to look “behind the curtain” at the computer screen. The computer should be hidden from view.

2. Users seem to want to view the projected image by being in front of it, just as people watch television.

3. People stepping out of the elevator (or stepping in) do not particularly enjoy being blasted in the face by the projector light. (this issue has been solved, however).

4. The moment seems fleeting when people interact, people interact and a result is presented, then instantly replaced by a new result. This does promote a sense of real-time, but doesn’t reward

Fig 1.2: Software cutting out background, leaving only the person displayed.

Fig 1.3: Petra using a crank which drives the projected image up and down.

ICM Final Project Proposal: Global Data Visualizer

2007-10-12T13:23:00.000-04:00

Proposal:
When developing software, I feel it is important to develop around a system that is flexible, yet powerful when focus is needed on a particular task. Many times, a generic solution to a problem requiring precision results in software that has the ability to handle many tasks, yet does so in a mediocre way. Software should be quiet in interface, elegant in result - and predictable in terms of being intuitive to use.

My commentary on this aspect of software leads me to my proposal -- that being; I want to develop a system which can visualize various types of data as it pertains to our planet, and I'd like to do so in a way which promotes ease of use and flexibility in how it handles different data. Ideally, the data will be presented in a way that will be understood, yet a good deal more 'artsy' than most data visualizers I have seen. Whether 'artsy' means abstraction or simply a clean/slick way of visualizing the data I cannot really say, I very rarely ever know what I will create on the art-side until I am up to my neck in code architecture. That isn't to say that the art is an after-thought, because the fact negating such an idea lies in much of my past code, where I practically destroy my nicely planned out system for the sake of making the output look 'cool'. Of course, I'd prefer to maintain both... but the engineer inside me brings a knife to a gunfight if I have to choose only one due to time restraints.

Technical:
Initially, my task will be to visualize seismic activity happening around the world. This will ideally expand to harbor other data as time permits. I have actually already done this to an extent - but I'd like to have the events be projected on a 3D globe with correct coordinates an so forth. I have made the decision that I will be using OpenGL through Java for this project. I toyed with the idea of using C++ or Python in combination with OpenGL - but C++ coding is still a very slow process for me and Python simply isn't going to provide the optimal speed I need (which is too bad, coding in Python is really enjoyable). I was planning on using JOGL - but the ever helpful Daniel Shiffman pointed me in the direction of LWJGL (Lightweight Java Game Library) which looks really promising, so I will be venturing in that direction.

Here are a few links that are helping me think things through on this project:
Lightweight Java Game Library
NASA Blue Marble
USGS.gov

Elevator Project: Motion Tracking

2007-10-09T21:44:00.000-04:00

For my group project in Physical Computing, we have decided to address the issue of people waiting outside of elevators and the boredom that inevitably occurs during the wait. Petra, Sunghun, and myself all agreed that a good way to break the static nature of waiting is to allow people to interact with something that has immediate output in correlation with their movements. This allows those waiting to interact quickly with limited effort, thus making the amount of time one actually has to wait for the elevator to arrive non-consequential, which matters a great deal since upon arrival, the elevator could be very close to arriving, or not at all.

My part in the project has been developing the software - using Java/Processing for the graphics and motion tracking system, as well as C coding for the Arduino microprocessor we are using. The motion tracking system does a combination of background subtraction and motion detection, which basically filters out all imagery which not both moving and not within a certain brightness range. The pixels that meet these prerequisites have their coordinates tracked and drawn on using a combination of small rectangles of lines being drawn between each tracked pixel in sequential order, which results in an abstraction of imagery that is vastly different from real-life, yet familiar enough to be predictably interacted with, at least in the regard towards positioning oneself towards making an intended impact on a defined space.

Fig 1.1: Although heavily abstracted, a face emerges through motion.

Fig 1.2: Rapid motion completely destroys all recognizable form, as intended.

Fig 1.3: Slight motion will in turn bring about only slight abstraction, to
the point of nearing a keyed cutout of the form.

Processing:
ProcessingMain
MotionCapture

Arduino:
cb_software

Shiver - Early Development

2007-09-26T01:26:00.000-04:00

I have decided to go ahead and start dedicating a good amount of time toward one of the two major projects I hope to accomplish this semester at ITP. The project name is tentatively called "Shiver" and it involves parsing seismic event data and using that content as a way of provoking both visual and physical simulations. Right now, I am concentrating on the visual side - as it is a nice starting place to begin prototyping the "how" as it relates to progress I will need to make later in development. However, one of the major tasks I wanted to immediately address was the ability to interface hardware I create with the software I am programming. I was able to get an analog signal to influence variables inside Processing using its serial library - with the pictures and code below:

Fig 1.1: Simple circuit hooked up to a Arduino and variable resister
to test analog/serial communication.

Fig 1.2: Still image of Shiver at current development point

Link:
Main Demonstration Page

Processing - Source code:
parse_blocks
xml_parser

Arduino - Source code:
serial_communication

Motion Tracking Glove Prototype

2007-09-22T01:04:00.000-04:00

In my first attempt in pairing concepts from my Physical Computing and Computational Media classes - I decided that I would like to try my hand (pun fully intended) at developing a type of glove-based motion tracking system. Since I have a larger project in mind for Computational Media which involves tracking movement through a camera - this was a perfect opportunity to start roughing out a few of the pixel tracking / blob detection algorithms that would be easily reusable when I move on to developing the larger project. So, after an afternoon of traversing through pixels, I had the color tracker working on high densities of near-pure red:

Fig 1.1: Ellipse following over pixels being tracked

So, with the tracking algorithms working well enough to run some practical tests - it was then time to move into the physical computing side of my endeavor. The wiring was a rather simple affair, as I really only intended to have two LEDs, one which was to be consistently on and the other to require a button press to light up. After a successful test with my Arduino code - I cut up some cardboard and wrapped two pieces of equal length in eletrical tape, then sandwiched my board into an enclosure, as seen below:

Fig 1.2: An ugly yet effective circuit board enclosure

Fig 1.3: The enclosure opened - revealing the breadboard and
microprocessor.

I had bought some gloves from K-Mart that were of the variety where if you flip down a flap the gloves could become mittens. The flap turned out the be the (near) perfect holder for my enclosure. The fit isn't exact - but just fine for the prototypical nature of this project. A few holes through fabric and some snaking of the wire through the palm and fingers of the glove and it was ready for action.

Fig 1.4: Motion tracking glove - ready for your local fashion show.

Fig 1.5: Glove plugged into a USB port and working

After putting on the glove and using it in combination with the Processing program I wrote, the test results indicate that better lit rooms gives the color tracking algorithms a bit too many high values than it can currently handle - which results in flickering of the tracking point. However, in moderate to low lit rooms - the glove works rather well. Below is the screen shot of the glove in action with the Processing program:

Fig 1.6: Motion/Color tracking algorithms paired with Processing graphics, pointer
being directed by glove use.

Processing - Source code:
motion_tracker
blinds
gui_components

Arduino - Source code:
motion_glove

Physical Computer - Lab #2

2007-09-22T00:17:00.000-04:00

Moving forward in the adventures of the electrical nature - it was already time to dive into the programming side of physical computing. Lab #2 consisted of wiring up a simple circuit with a switch and a couple of LEDs and being able to program a Arduino board to process simple logic through authored code. After working through the previous lab - I found the circuit-boarding side of this lab to be far easier, even with the addition of the Arduino microprocessor, which as the time was completely foreign to me. Below are a couple of pictures of the process, plus a link to the source code for my first Arduino program.

Fig 1.1: First success using Arduino

Fig 1.2: Using a simple switch logic gate

Fig 1.3: Six LEDs blinking one after another

Source Code: six_led_lights

Note: I have noticed that one of the (many) limitations of the programming language initially available to use with the Ardunio microprocessor is the inability to directly get the size of an array through a method or any sort of standalone function. This becomes problematic when needing to accurately loop through an array of LED pins, or anything else stored in an array for that matter. A way around this limitation without resorting to counting the elements manually and hard-coding that number to a variable is to instead use the following code, as additionally outlined in my source code.

int led_pins[] = {3,4,5,6,7,8};
int led_pin_count = sizeof(led_pins) / 2;

The global function 'sizeof' returns a value which represents the number of bytes currently stored in a given variable. Since we can assume that an integer is 2 bytes; dividing the value sizeof returns by 2 will result in the elemental size of our array.

Physical Interface: Cash Register

2007-09-19T22:11:00.000-04:00

In all likelihood - I have walked more in these past couple of weeks than I had in any given week during my time as an undergrad. I am fortunate enough to live within walking distance (roughly 1.8 miles) to the Interactive Telecommunications Department where I am undertaking my graduate research at NYU. More often than not - I drop into a local deli on my way to ITP to grab a vitamin water or perhaps one of those energy bars of varying brand. Upon selecting what is likely going to turn out to be lunch - I walk over to the cashier to pay my usual $4.00 in cash, a much more 'pleasant' affair now after being glared at several times for sticking out a credit card. Besides the usual stiff attitude of the deli cashiers, I have always been taken by the speed of which the cashiers operate. Usually, the cash register itself is of the older variety - a 'do it yourself' style machine where numbers of prices are punched in rather than scanned. What is most interesting is that in my experience, the old-style cash registers are substantially faster and the users seem far less likely to have a "someone get the manager over here" type moment compared to a scan-based register with touch screens. So, is this an instance of technology getting in the way rather than helping? Here is my completely research-free and high-level look:

First, there needs to be an acceptance to the fact that technology isn't always built for speed. Be it for all the right or wrong reasons - technology many times can substitute what was traditionally a simple and fast process with more functionality; resulting in a steeper learning curve for the end-user(s). I have chosen cash registers as an example for a user-interface to analyze because its advancement in features are an interesting mix of positives and negatives, as well as a device that has taken on a new relationship with its users. The cash register has 'evolved' from a interface requiring one employee to use the device, to needing an employee and a customer to interact with, to most recently - only needing a customer to interact with, now coming full-circle. I won't focus on the third advancement, but I want to write a quick comparison of the first two user-interfaces - the singular and dual user interface cash registers.

Fig 1.1: Older model cash register: simple and fast
(Stoic NYC cashier attendant not pictured)

In my example of the dual-user register - I recently went to a Best Buy in lower Manhattan to buy a DVD (The Office: Season 3... a must have) and approached the cashier in front of a rather new looking cash register system, complete with a customer input pad and stylus pen for a signature. In the end, this transaction took several times longer than an average transaction at the deli - with the very same amount of items using the exact same type of payment: a credit card (to the chagrin of the deli cashier). Generally with technology, if a user is being asked to allocate more time, then the end result is expected to be enhanced to return. My end result as a customer yielded no instant enhancements - I still received a receipt, my item, and a debit from my banking account in both transactions. As a user-interface, the technologically enabled cash register at Best Buy required effort on my part by signing with the stylus (awkwardly, as always) and paying attention to which buttons to press to proceed. The cashier seemingly did less work in relation to number of button-presses and analysis than that of the deli cashier - but the overall workload wasn't necessarily shortened, only abstracted across multiple users (such as myself) and varying interfaces. Sure, there are a few hidden benefits of the high-end register such as receipt caching for faster returns and cash back options using a debit card - but what is the ratio between transactions and the collective number of returns and debit card cash back requested on a given day? I'd say it is very likely a substantially lopsided ratio leaning toward the quantity of transactions - yet the technological advances cater to the lower frequency of occurrences in the case of the cash register.

Fig 1.2: "Sign here and press button to continue, now
press this combination of buttons to confirm...hey,
pay attention or this transaction isn't happening."

My concluding thought is that you could make the case that technology advancement for the cash register earns its keep because it lessens the work load of the cashier - but then again, the cashier is getting paid to work less at the expense of my time, so as a customer I wouldn't exactly call that a positive step unless I was on the other side of the register.

ICM - Assignment 2: Mouse Tracker

2007-09-17T23:19:00.001-04:00

This is just a quick demonstration of how shapes can change states based on mouse interaction - and continue animation after the interaction has taken place. This example (which I cannot embed in this blog because of its large size) is just the simplistic animation mouse tracking part of a larger project I am undertaking which involves using a connected camera to track movements of a certain color, then using that concentration of color as a pointer. More on this project later...

Mouse Tracker using Processing / Java

Physical Computing - Lab #1

2007-09-11T22:41:00.000-04:00

So, this is lab was my first venture into the depths of physical computing. I feel that overall, things went rather smoothly - escaping without any serious injuries or damage to private property. For my efforts - I was able to make multiple LEDs of various colors light up, as well as garnering the ability to fade them in and out and using a switch to turn them on and off. Perhaps not too exciting to read about - but certainly exciting to accomplish on my end.

My first circuit.

Switch Activated LED

Fading LED using a potentiometer.

This lab served as a great introduction to some of the really core basics/principles of electrical engineering. I am looking forward to start programming microprocessors - especially when it comes pairing code logic with sensors. More to come.

ICM - Assignment 1: Circle Creator

2007-09-07T14:36:00.001-04:00

I wrote up a little applet in Processing which simply creates circles of varying sizes at pseudo-random coordinates within a bounding box.

For this assignment, I wanted to use the time I had to write up a couple of class objects I knew that I could reuse later on in this class (and just in general) - in particular a general use button and a border frame. Clicking the button signals a callback of sorts to the main circle drawing function - pretty straight forward, although not being able to pass a function as an argument to another function makes it less flexible currently than I'd like.

This browser does not have a Java Plug-in.

Get the latest Java Plug-in here.

Handle Events:
Mouse click (within viewport box): erase viewport
Key 'r': Red scale
Key 'g': Green scale
Key 'b': Blue scale

Source code: circle_creator

Built with Processing