A Brief History of Graphics and Rendering
Alex Darby our technical lead initially sent this to all of the programmers on the current Gamer Camp: Pro course.
It was so good that we thought those following our blog would be interested in what he has to say.
I need to make sure that, as a group, we have a more level understanding of the graphics side of things.
Let's start at the beginning (note this may not be 100% historically accurate, but I've explained it all incrementally).
Models in a "standard" real time 3D graphics engine are all built in Maya or 3DS Max or some other similar 3D modelling tool.
These models are fundamentally composed of vertices positioned in 3D, each with a position of the form [x, y, z].
Vertices can be rendered, but they don't look very fancy :)
We covered the maths involved in rendering vertices (i.e. transforming them from the model's space in 3D onto the 2D screen) in overview in module 3. Each vertex is transformed as follows:
from "model space" (where it is defined) into "world space"
from world space into "camera space" (the frame of reference of the camera)
from camera space into "projection space" (also often called "clip space")
from projection space into "screen space" (this converts the 3D [x, y, z] point into a 2D [x, y] point)
To make 3D graphics with high visual fidelity using just vertices is not simple. If you render them as single pixels, it requires a number of vertices large enough that there is at least one vertex in the model for each pixel on the screen when it is rendered at its maximum size.
The Death Star plans in the original star wars film were done as just vertices, as were computer graphics in Alien (they were done using vector graphics displays, which is a whole different thing to bitmap graphics though... google it if you're interested)
Each vertex requires 3 values [x, y, z] to describe its 3D position (and a colour value if we wanted to do coloured graphics) and must be transformed from model space into screen space; so this is both memory AND processor intensive. This is basically the same as using "voxels" ( a voxel is a "volumetric pixel" - i.e. a 3D pixel look it up on google for more information).
In order to reduce the memory and computational expense, less 3D points must be used. People started research into using 3D surfaces to describe models since they can use only a few vertices to describe large areas of model surface. There are lots of different 3D surfaces that can be used, and lots of research has been done (and no doubt still is somewhere) into the various options.
The simplest 3D surface that can be described is a planar polygon, and the simplest planar polygon is a triangle, it is also one of the simplest to render in 2D so it was an obvious target for hardware acceleration and so eventually became the standard way to do 3D models.
When a model is made out of triangles, the vertices are grouped together into 3s.
The maths for rendering triangles is not that much more complex than for rendering vertices. The three vertices are transformed into screen space, and then all the pixels contained within the 2D triangle described by these points are rendered to render the triangle (determining the specific pixels that are within this 2D triangle is a process called "rasterisation").
Assuming we were doing coloured graphics, the data for each triangle we wish to render must now include the 3 vertices and its colour.
Flat coloured triangles by themselves don't look that great, and so someone invented the idea of "lighting" them.
The most simple way of lighting a triangle is to adjust the colour of the whole triangle based on the angle between some light source, the triangle, and the camera.
This calculation is done in 3D before converting the triangle into 2D (usually in camera space, or world space) and the colour stored in the triangle data is modified by the lighting clculation and stored so it can be used to fill the pixels of the 2D triangle.
It is often called "faceted lighting" because it makes 3D models look like cut gemstones. This produces graphics that look like Starfox on the SNES.
Obviously, a surface that was supposed to be smoothly curved (like a cylinder) doesn't look very good when it's made of faceted triangles. The next advance was in smoothing the lighting of surfaces to remove the "faceting" effect. To accomplish this, rather than lighting the whole triangle with the same light value as a flat surface, the lighting value must change across the surface of a triangle so that it appears not to be flat.
The simple "light the whole triangle" technique can use a single "surface normal" vector to describe the direction the face is pointing (or it can be calculated in real time using the cross product from 2 adjacent edges of the triangle).
To achieve smooth lighting across adjacent triangles, each vertex that is part of a "smooth" surface of adjacent triangles must store its own surface normal which lies between the actual face angles of the triangles it is part of. This averaged (or "smoothed") normal gives the contacting edges of adjacent triangles the same lighting value amd so the boundaries between them become invisible.
Additionally, the 3D lighting calculation must now be done once per vertex rather than once per triangle, and the lighting value applied to each 2D pixel that lies within the triangle must be adjusted (interpolated) based on where in the triangle it is relative to each of the 3 vertices.
See below image "FacetedVsSmoothNormals.jpg" for a visual representation of how this works.
So, we're looking a lot better, for example this technique could be used to render a car pretty convincingly.
However, the limit of a single colour per triangle means that this doesn't look great for anything organic (i.e. with high surface detail) unless each model is made up of so many triangles that each one is approx. the size of an onscreen pixel when it's rendered.
This lead people to research ways to get better visual fidelity with less triangles, and the next advance was texture mapping - to put images (or "textures") onto the triangles so that large triangles can have lots of visual detail.
With texture mapping, the data required to render a triangle includes a texture, and each vertex rendered requires an additional 2D coordinates that defines its position in the texture (rather than [x, y] texture coordinates are typically referred to as [u, v]).
When the triangle is rendered in 2D, the base colour of each pixel in it is now derived by calculating the [u, v] coordinate of that screen pixel within the texture based on the [u, v] coordinates of the triangle 3 vertices (modern "correct" texture mapping includes other factors too - check out http://en.wikipedia.org/wiki/Texture_mapping).
See Below image "UvMapping.jpg" for a visual representation of how this works.
Ok, so now we're more or less at the level of graphics as they were on the original PlayStation (but the PSX didn't have perspective correct texturing...), so let's review where we're at.
A 3D mesh for real time rendering at PSX standards contains the following data:
the texture (or texture map) is used to render all triangles in the mesh
Each vertex has a position and a normal
Let's assume they're in an array and we can index them.
Each triangle has a colour, It also refers to three vertices by index and stores a corresponding U and V value for each vertex, specifying its position in the texture.
If you wanted to have more than one texture you'd end up with more than one mesh in your 3D model.
See Below image "PsxStyleSingleTextureMesh.jpg" for a visual representation of a 3D mesh with a single texture.
Since the PSX the changes to the data needed for real-time rendering, and the pipeline iteself have largely been incremental improvments rather than fundamental changes.
Let's leap ahead to modern graphics pipelines such as those in the PS3 or X360 GPUs (i.e. ignoring the relatively new concept of the "geometry shader").
The first and most major difference relative to our simple, single texture model is that the vast majority of the computation work involved in rendering is now done on the graphics card either by "fixed function" hardware or by "programmable" parts of the Graphics Processing Unit (GPU) called "shaders".
The model data is exported from the 3D package and then the exported mesh is loaded by the game into memory accessible by the GPU.
This mesh data contains references to all the data needed to render it - typically at least the same amount of data as would have been required for a PSX mesh - vertices, triangles, and a texture.
When the game side code wants to render the model it tells the GPU which mesh data to render, and supplies:
a "model matrix" (model space to world space)
a "view matrix" (world space to view space (view space == camera space))
The GPU then steps through each vertex in the model data and passes it to a program that runs on the GPU called a "vertex shader".
Vertex shaders are entirely customiseable and can be used to do per vertex lighting calculations, generate extra data needed for rendering etc. In its simplest form, a vertex shader program processes the vertex data passed to it from model space into view space by multiplying its position by the model and view matrices.
Once the vertex has been processed by the vertex shader it is multiplied by a "projection matrix" (also supplied to the GPU by the code) which projects it from 3D into Projection (or "clip") space. This is where perspective is applied to vertices, so vertices further from the screen (in their Z coordinate) have their X and Y coordinates scaled down so they look smaller.
In projection space is where vertices outside the viewable area are discarded. Any triangle with one or more vertices outside the viewable area must be processed so that the part of it that is viewable is still rendered. This processing is done on the GPU, and involes creating extra vertices to replace the ones removed.
See below image "ProjectionSpaceClipping.jpg" for a visual representation of clipping a triangle (n.b. red dots are vertices, the clipped triangle has more vertices).
Once the vertex has been clipped it is converted to 2D "screen coordinates" (via "device coordinates" on PC where the view may be windowed). Once the vertices are in the correct 2D space the triangles can be rasterised to produce the final pixels output to the screen.
The GPU steps through each pixel within each 2D screen space triangle and passes it to a program called a "pixel shader" (also known as "fragment shader" on the PS3). The pixel shader's job is to calculate the output colour of the pixel that is passed to it.
Pixel shaders have special operations they can do to look up their colour in a texture and so forth. The real power of pixel shaders is that the textures they can use do not have to be images, they can also be clever things like normal maps (http://en.wikipedia.org/wiki/Normal_mapping), ambient occlusion maps (http://en.wikipedia.org/wiki/Ambient_occlusion), or specular maps(http://en.wikipedia.org/wiki/Specularity).
The colour value output from each pixel shader is then written into the screen buffer by the GPU.
So how does this change the data we need to render something from our previous simple PSX model earlier? The answer is, again, incrementally rather than fundamentally.
The concept of a material is that it represents a physical material like concrete, rusty metal, or brushed aluminium.
A material may have multiple texture maps, but it definitely has exactly one vertex shader, and one pixel shader.
The vertex shader is run once by the GPU for each vertex in the mesh the material applies to.
The pixel shader is run once by the GPU for each pixel in each triangle the material applies to.
Each vertex has a position.
Let's assume they're in an array and we can index them.
Each vertex may have other data associated with it as required by the vertex shader e.g.:
a tangent for normal mapping
a specular power for specular lighting
Each triangle refers to three vertices by index.
Each triangle may also have other data associated with each vertex in it as required by the pixel shader; typically one or more pairs of [u, v] values, specifying the vertex's position(s) in the various texture(s) used by the material.
Whew! That's (broadly) how real time graphics works on PS3 and X360.
So, in general you should at least have a fairly good idea of what data is pushed about in the average real-time graphics pipeline, if not precisely how it's pushed.
If you have any questions or want to know more, ask each other, google it, or ask me :)
An excellent place to get a more full understanding of the modern graphics pipeline is chapter 2 of Real-Time Rendering 3rd edition (with Sack-boy on the cover). They should have this in the library.
Whilst I cannot condone finding .pdf versions of this book on the internet you will probably find it a very cost effective way to read just the 2nd chapter (the rest of it is pretty maths heavy and full of squiggles and won't be of much use to anyone not interested in GPU programming).
The best book is (I now I go on about it!) 3D math primer for graphics and game development. This explains all of it in a bit more of a beginner friendly way than the 1st book, and has more diagrams. Best. Book. Ever.
Alex Darby
Technical Course Lead @ Gamer Camp
prguitarman
theartofmadeline
kaledo-art
turnnoffyourmind