Programming, Ubiquity, Devices, and You

After telling fellow students at ITP about the classes I decided to sign up for the coming Spring semester, I have often been asked the following two questions:

1. "Why are you taking programming centric classes if that is already one of your strengths?"

2. "Why aren't you taking any physical computer related classes?"

The answer to the first one is rather concise; I feel that staying constantly involved in software and mathematics allows me to build myself as a better overall problem-solver on both an analytical and creative basis. Writing code to solve a problem is concurrently a step-by-step breakdown of the problem itself; patterns emerge and reveal themselves, woven with systems of varying elegance and complexity. Understanding concepts which reveal such harmony will endure far past simply designing the next iPhone - leading me to the answer to the second question:

My position on physical computing at it pertains to projects at ITP (and in general) is that the development of devices which we physically interact with is inevitably obsolete. This is the century of biology - and consequently, biological micro-interfacing will be the future of device technology. Increased ubiquity leading to result/reward has nearly always been the trend in tool advancements - computing is to exist behind a curtain whenever possible. While it is certainly a valid venture to develop large-scale projects involving common electronics and so forth - I personally have little interest in pursuing similar implementation methods.

Software for Physical Display Project

Group: William McDonald, Sunghun Kim, Matt Parker

Further documentation:
Sunghun with videos / images
Matt on construction / mechanics

For our group project, we wanted to develop a physical system to represent graphical depth data. We decided to use a collection of sine and noise waves to start off with - as using depth data really only makes (practical) sense with a 3D grid.

Using Processing, a straight-forward GUI was created featuring four buttons, with a graphical representation of the waves being dynamically created to the right of the buttons. The fourth button allows the user to paint his/her own wave (or any shape, for that matter) and the software will generate a visualization and physical output for the image painted.



Fig 1.1: Drawing canvas allows user to draw images to physically visualize



Fig. 1.2: Visualization of user input wave drawing


To sample each wave - an algorithm was written which samples across the width of a given image proportional to the resolution of the physical system - in our case, 20 samples. At each sample - the program checks to see how far down (Y) the image it takes in pixels before a color other than white is found. Upon finding a color, the value of that sample is received by the simple equation of :

sample_value = y_coord / (image_height / max_value)




Fig 1.3: Graphical representation of sampling method on wave images


In regard to feeding this data into the servo motor array - we decided to use four arduino microprocessors connected via USB to the computer running the software application; with each arduino effectively controlling five servos:



Fig 1.4: Computer / Arduino / Servo array configuration

Shiver - Final

Fig 1.1: Splash screen for Shiver

The last week of work on Shiver has primarily been a focus on adding features on the GUI side and tying up some loose ends. The largest new feature is the ability to 'simulate' - which allows a user to simulate a full rotation (or however many wanted) of the globe, outputting the frames as an image sequence. This process allows the user to see the rotation as a movie clip running at full speed. The video in this blog entry is derived from that process.



Fig 1.2: Simulate options window


Shiver is still very much a work in progress - however, I am pleased with the result of my work over the past 6-7 weeks, particularly with knowing that I was able to achieve high-quality imagery in real-time using Python and OpenGL together.



shiver_simulation.mov - 10.6mb


Images from Shiver:

Shiver Development: Event Density + Time Charting

For the past few days, I decided to develop a type of visualization which attempts to better communicate the areas that have a high density of earthquakes over time. I decided to do this by having the program create a texture on the fly which is fed into the primary globe fragment shader, where areas of high seismic event activity would get higher amounts of red. Since the texture being created is simply a collection red blobs on a black background, it is simple process to add the red areas on top of the globe through the shader. Additionally, the seismic events themselves are visualized through simple lines aligned to the normals of the sphere - with the length of the line determined by its Richter scale value.



Fig 1.1: Globe with event density visualization

After wrapping up event density mapping, I wanted to start trying to visualize events as they occurred over time. The visualization itself doesn't communicate time as much as it displays the erratic nature of seismic activity happening around the world over an extended period (currently, over seven days). To create lines that seemingly wrap themselves around the curvature of the globe - I decided to write a small bezier curve generator which had end points at the 3D coordinates of selected events. The distance of those events is calculated using the Euclidean distance formula of:

Where the distance value is used to apply weight (w) to midpoints in rational bezier curves, such that:

Fig 1.2: Bezier curves mapped across globe based on time of occurrence

Shiver Development: Displacement

As I continue to polish the overall look of the globe - I decided to explore how to display the topology of the earth, which will become more important upon the incorporation of zooming in and out. Another plus for having a working shader pipeline is the ability to write vertex shaders in a way which changes the geometry of the given shader is attached to - better known as displacement.


Fig 1.1: Altitude map provided by the NASA Blue Marble project

Having an altitude map which precisely matches the texture maps I am already using is quite convenient - so there was no need to alter the image or the coordinate mappings in the my shaders. The concept of displacing geometry per vertex is a rather simple one:


Fig 1.2: Formula for displacement per vertex, where:

P0 = original vertex position
p1 = new vertex position
N = vertex normal position
df = normalized displacement factor
uf = user-defined scaling value


Fig 1.3: Diagram of per vertex displacement (Image credit: oZone3d.net)



Fig 1.4: No displacement / displacement comparison in Shiver (wireframe)

My next goal for Shiver is to implement a sunlight calculation system which will auto-generate a texture map accurately representing where how the sun is lighting the earth based on the time of day. Additionally, I plan on (finally) starting the preliminary visualization of the actual seismic events being geomapped.

Woven

I developed a small piece of software for a group presentation I had in Applications class that has to do with visualizing collaboration among 1st year students at ITP. Someone during the Q+A asked if the software was open-source and if so, where could the source be downloaded? The answers were "yes" and "soon, on my blog". Unfortunately, I don't really have the time as it stands to package the software up nicely for download - so I am just going to post the raw source code for now, with the hope that I can package it later on...



Source:
ProcessingMain.java
StudentDataParser.java
Student.java
Button.java
student_data.txt

Shiver Development: Event Mapping

A couple of new updates; one being that I have a new fog shader working on the globe - which adds to the effect of the atmospheric layering. The ability to program shaders and apply them to geometry is really what separates imagery that is noticeably fake and that which is nearing photorealistic. Real-time photorealism is the next big step in computer graphics - although you could make a very valid claim that photorealism hasn't been reached in pre-rendered form either.


Fig 1.1: Comparison between shader enhanced and non-shaded geometry

The next update in progress is actually a rather large one - that being that I can now plot events on the earth sphere, needing only an event's latitude and longitude values. This was actually a larger challenge than originally anticipated, as I didn't take into consideration polar coordinates in regard to texture mapping in OpenGL as it relates to sphere mapping. I also ran into an issue pushing and popping the transformation matrix - but that turned out to be an instance of not having certain draw events happening between the correct combination of push/pop commands.


Fig 1.2: Seismic events mapped on globe

Shiver Development: Shaders in PyOpenGL

After a turbulent weekend trying to get shaders working with PyOpenGL - I finally got my first vertex and fragment files compiled and working inside Shiver. Without any documentation to bail me out of my usual jams with new code - I hacked away through the weekend until I stumbled upon a way of changing C-type function arguments, then converted variables into C-compliant data to be fed into the altered function in question. In English: I needed to convert data so it could be read and processed correctly.


Fig 1.1: Globe with atmospheric shader applied

So - in the lab I am often asked: "why are shaders important"? Shaders can provide surface appearance variance in regard to color, opacity, reflection, refraction, etc... even physical alterations to the geometry itself. In relation to what I am attempting to achieve - it is vital to be able to replicate light and physical phenomena that exists when viewing the earth from space.

My first goal was to get a representation of the atmospheric layer that wraps around the earth - a thin haze of a light blue hue. The trick is - the haze should not rotate along with the earth as the camera moves around the globe. To achieve this; a shader was created that calculates how light is hitting the surface at each given frame - then simply finds where the light drop-off occurs on the surface (the edges) and colors those areas a varying hue of blue based on how much light exists on a given fragment of the geometry. Since I applied the shader on a sphere and the lighting is both hitting the sphere straight on and is static in all attributes over time - we can assume perfect symmetry in the light distribution across the sphere at any given time, which allows for the Lambertian Reflection calulation of:



Where:
I0 = reflected intensity
Ld = light diffuse color,
Md = material diffuse coefficient

Once we have the reflected intensity of each fragment, simple logic can be installed within the shader that only allows for color to be applied a certain areas of intensity - and, since we are dealing with a sphere with light shooting right at it (symmetrical light reflection!) - it becomes simple to create a halo light effect.


Fig 1.2: Closer look of the glow-like effect of the atmospheric shader

The next steps with using shaders are to create a fog shader for the globe, as well as the potential for a very subtle bump-map, as well as other effects as time permits.

Shiver - Final Proposal

A few posts back - I wrote about developing a seismic data visualizer for my ICM final. This is still the case - however, the details involving the execution of the project are now very different. The most notable change involves adding several physical components to the project, meaning; this project will now be both my ICM and my Physical Computing project for this semester. Excited? Splendid - now on to the details...


Technical
On the ICM front - I made a decision a week ago that I would in fact use Python as my primary programming language opposed to my original choice of Java. Why? Well, as I mentioned in the past - Python is slow in regard to intensive computation in comparison to compiled languages such as C/C++ or Java. However, where Python shines is in its fast development time - and through its flexibility to glue together various components effectively. Without a doubt, hitting the visual high-bar I am aiming for in regard to the graphics in this project will put an enormous strain on Python - however, much of that strain can ideally be handled through C code, if necessary. I will be using an OpenGL binding called pyOpenGL - which is a mature library with strong documentation, so I should have information/learning support when I need it. For the GUI components, I will be using the wxPython toolkit - which allows for native OS widget use, so the program will automatically reflect the standard appearance of whatever OS the program is being run on. Very slick.



Fig 1.1: Shiver in early development, click to enlarge.


Representation
As for how the data will be represented, it will be done so on a 3D globe - much like applications such as Google Earth and NASA's World Wide Wind. When the applications gathers seismic data activity from the USGS - each event will be mapped to the globe according to its latitude and longitude values. Based on an event's Richter Scale value - the event will be visualized along the nearest normal to in correlation to its lat/long. mapping. Events can be selected through an easily organized directory structure tree, or on the globe itself. When selected - various attributes are displayed, as well as a simulation option - which leads us to the physical computing side of the project...


Physical
After sitting down with fellow ITP'er Sunghun Kim, we decided that we would like to team up and pursue a physical representation of seismic data that can be partnered with software I am developing for my ICM final. This idea will involve creating a 2D representation of the seismic events - dictated by the Richter scale value of each event. This will ideally result in a mesh being created by rods pushing and pulling at a rubber-like surface. Each rod will be driven of an average value of pixel color values dependent on the ratio between the rod count and the image resolution. Once a 2D simulation is established, it will ideally branch out to a 3D grid - but that is likely for another semester.

Elevator Previsualizer

This is a midterm project I worked on with a talented group of people in Physical Computing. Follow the link for more:

Elevator Previsualizer Main Site