1) A little bit of matrices

First of all it's essential that you understand what a normal 3x3 matrix is
made up of. Let's consider the unit matrix, i.e. the one that's implicitly used
for a non-transformed space:

|1 0 0|
|0 1 0|
|0 0 1|

What this matrix actually tells us is the X,Y,Z coordinates for the X,Y,Z
unit world axises.

|1 0 0|
|0 1 0|
|0 0 1|
 ^ ^ ^
 | | +- Zw = [0,0,1]
 | +--- Yw = [0,1,0]
 +----- Xw = [1,0,0]

As you can see, for a normal untransformed spaces it's quite obvious that
the world axises have coords [1,0,0] (the x-axis), [0,1,0] (the y-axis),
[0,0,1] (the z-axis).

Important: Conslusion, each column in the rotation matrix gives us the
coordinates for each of the unit axises in our transformed space, described
in the untransformed space's coordinates.

More generally the rotation matrix:

     |Rxx, Ryx, Rzx|
Mr = |Rxy, Ryy, Rzy|
     |Rxz, Ryz, Rzz|
       ^    ^    ^
       |    |    +- Zw = [Rzx, Rzy, Rzz]
       |    +------ Yw = [Ryx, Ryy, Ryz]
       +----------- Xw = [Rxx, Rxy, Rxz]

I.e. the world axises for our rotated space expressed in our current
untransformed space coordinates are:

Xw = [Rxx, Rxy, Rxz]
Yw = [Ryx, Ryy, Ryz]
Zw = [Rzx, Rzy, Rzz]

Ok, enough about rotation matrices and world axises. If you don't get it,
re-read and only move forward until you really get this.



2) Deriving the vertices for the unit cube using the rotation matrix

The coordinates for a unit cube are:

p1 = [ 1,  1,  1 ]
p2 = [ 1,  1, -1 ]
p3 = [ 1, -1,  1 ]
p4 = [ 1, -1, -1 ]
p5 = [-1,  1,  1 ]
p6 = [-1,  1, -1 ]
p7 = [-1, -1,  1 ]
p8 = [-1, -1, -1 ]

By symmetry we quickly se that it can be reduced to:
p1 = [ 1,  1,  1 ]
p2 = [ 1,  1, -1 ]
p3 = [ 1, -1,  1 ]
p4 = [ 1, -1, -1 ]
p5 = -p4
p6 = -p3
p7 = -p2
p8 = -p1

So, we really only need to calculate p1-p4. Let's consider p1 first:

p1 = [1,1,1]

Remeber the unit matrix:

|1 0 0|
|0 1 0|
|0 0 1|
 ^ ^ ^
 | | +- Zw = [0,0,1]
 | +--- Yw = [0,1,0]
 +----- Xw = [1,0,0]


Xw+Yw+Zw = [1,0,0] + [0,1,0] + [0,0,1] = [1,1,1] = p1

As you see, by simply adding the world axises you get a corner of the unit
cube. It's quite obvious; to get from origo to a corner of a cube you need
to take one step in each dimension.

For p1-p4 we get:

Xw+Yw+Zw = [1,0,0] + [0,1,0] + [0,0,1] = [ 1,  1,  1 ] = p1
Xw+Yw-Zw = [1,0,0] + [0,1,0] - [0,0,1] = [ 1,  1, -1 ] = p2
Xw-Yw+Zw = [1,0,0] - [0,1,0] + [0,0,1] = [ 1, -1,  1 ] = p3
Xw-Yw-Zw = [1,0,0] - [0,1,0] - [0,0,1] = [ 1, -1, -1 ] = p4

Now, in general rotation matrix form:

     |Rxx, Ryx, Rzx|
Mr = |Rxy, Ryy, Rzy|
     |Rxz, Ryz, Rzz|
       ^    ^    ^
       |    |    +- Zw = [Rzx, Rzy, Rzz]
       |    +------ Yw = [Ryx, Ryy, Ryz]
       +----------- Xw = [Rxx, Rxy, Rxz]

Hence we get:

Xw+Yw+Zw = [Rxx, Rxy, Rxz] + [Ryx, Ryy, Ryz] + [Rzx, Rzy, Rzz] = [Rxx+Ryx+Rzx, Rxy+Ryy+Rzy, Rxz+Ryz+Rzz] = p1
Xw+Yw-Zw = [Rxx, Rxy, Rxz] + [Ryx, Ryy, Ryz] - [Rzx, Rzy, Rzz] = [Rxx+Ryx-Rzx, Rxy+Ryy-Rzy, Rxz+Ryz-Rzz] = p2
Xw-Yw+Zw = [Rxx, Rxy, Rxz] - [Ryx, Ryy, Ryz] + [Rzx, Rzy, Rzz] = [Rxx-Ryx+Rzx, Rxy-Ryy+Rzy, Rxz-Ryz+Rzz] = p3
Xw-Yw-Zw = [Rxx, Rxy, Rxz] - [Ryx, Ryy, Ryz] - [Rzx, Rzy, Rzz] = [Rxx-Ryx-Rzx, Rxy-Ryy-Rzy, Rxz-Ryz-Rzz] = p4

So, we have now stated that given a general rotation matrix you can derive all eight vertices of a cube by simple additions. (Good for a C64!!)



3) Deriving the normals of the faces of the unit cube using the rotatio matrix:

For filled vectors we must do backface culling, and the approach I'll give will use the face normals hence we need to derive them.

The coordinates for the 6 face normals are:

n1 = [  1,  0,  0 ]
n2 = [ -1,  0,  0 ]
n3 = [  0,  1,  0 ]
n4 = [  0, -1,  0 ]
n5 = [  0,  0,  1 ]
n6 = [  0,  0, -1 ]

By symmetry:

n1 = [  1,  0,  0 ]
n2 = -n1
n3 = [  0,  1,  0 ]
n4 = -n3
n5 = [  0,  0,  1 ]
n6 = -n5

Remeber the unit matrix:

|1 0 0|
|0 1 0|
|0 0 1|
 ^ ^ ^
 | | +- Zw = [0,0,1]
 | +--- Yw = [0,1,0]
 +----- Xw = [1,0,0]

As you can see:

Xw = [1,0,0] = n1
Yw = [0,1,0] = n3
Zw = [0,0,1] = n5

Now, in general rotation matrix form:

     |Rxx, Ryx, Rzx|
Mr = |Rxy, Ryy, Rzy|
     |Rxz, Ryz, Rzz|
       ^    ^    ^
       |    |    +- Zw = [Rzx, Rzy, Rzz]
       |    +------ Yw = [Ryx, Ryy, Ryz]
       +----------- Xw = [Rxx, Rxy, Rxz]

Hence we get:

Xw = [Rxx, Rxy, Rxz] = n1
Yw = [Ryx, Ryy, Ryz] = n3
Zw = [Rzx, Rzy, Rzz] = n5



To sum 2) and 3) up, the coordinates for all eight vertices and all six normals expressed in rotational matrix terms are:

Xw+Yw+Zw = [Rxx, Rxy, Rxz] + [Ryx, Ryy, Ryz] + [Rzx, Rzy, Rzz] = [Rxx+Ryx+Rzx, Rxy+Ryy+Rzy, Rxz+Ryz+Rzz] = p1
Xw+Yw-Zw = [Rxx, Rxy, Rxz] + [Ryx, Ryy, Ryz] - [Rzx, Rzy, Rzz] = [Rxx+Ryx-Rzx, Rxy+Ryy-Rzy, Rxz+Ryz-Rzz] = p2
Xw-Yw+Zw = [Rxx, Rxy, Rxz] - [Ryx, Ryy, Ryz] + [Rzx, Rzy, Rzz] = [Rxx-Ryx+Rzx, Rxy-Ryy+Rzy, Rxz-Ryz+Rzz] = p3
Xw-Yw-Zw = [Rxx, Rxy, Rxz] - [Ryx, Ryy, Ryz] - [Rzx, Rzy, Rzz] = [Rxx-Ryx-Rzx, Rxy-Ryy-Rzy, Rxz-Ryz-Rzz] = p4
p5=-p4
p6=-p3
p7=-p2
p8=-p1

Xw = [Rxx, Rxy, Rxz] = n1
n2 = -n1
Yw = [Ryx, Ryy, Ryz] = n3
n4 = -n3
Zw = [Rzx, Rzy, Rzz] = n5
n6 = -n5



4) Back face culling by checking which side the eye is of a plane

First consider the plane equation:

A*x+B*y+C*z+D = 0

[A,B,C] = plane normal
[x,y,z] = reference point to test
D = distance from origo to the plane expressed in negative normal lengths.

If [x,y,z] is chosen to be exactly on the plane the equation will be =0.
If [x,y,z] is chosen to be below the plane, i.e. on the anti-normal side, the equation will be <0.
If [x,y,z] is chosen to be above the plane, i.e. on the normal side, the equation will be >0.

Consider one of the faces of the unit cube, for example the face with a normal n1.

n1 = [1,0,0] = [A,B,C]
D = -1 since the distance negative distance from origo to the unit cube's faces are exactly one normal in length.

1*x + 0*y + 0*z - 1 = 0

If we chose an [x,y,z] that should lie in the plane, for example [1,0,0] we get:

1*1 + 0*0 + 0*0 - 1 = 1-1 = 0

If we chose an [x,y,z] that lies below the plane, for example [0,0,0] we get:

1*0 + 0*0 + 0*0 - 1 = 0-1 = -1

If we chose an [x,y,z] that lies above the plane, for example [10,0,0] we get:

1*10 + 0*0 + 0*0 - 1 = 10-1 = 9

As you can see, the experimental values coincide with what we stated above.


Now let's consider how to use this for backface culling:

If you eye is on the normal side of a plane we consider the plane visible, i.e. if the result of Ax+By+Cz+D > 0 the plane is considered visible.

More over, let's assume the object rotated around origo and that the eye is fixed at a distance say [0,0,-3] (typical for c64 vectors) we get:

A*0 + B*0 + C*-3 + D > 0


I.e. to determine if a plane A,B,C,D is visible, simply calculate A*0+B*0+C*-3+D and check if it's positive. Or to simplify: C*-3+D > 0.


Now, as I stated before, for the unit cube the negative distance to all faces from origo is -1. I.e. D=-1 for all planes of the unit cube in origo. This reduced our check to:

C*-3-1 > 0

Shuffle around:

C > 1/-3


Let's call 1/-3 for e. e = 1 / -distance to object (still assuming unit cube where D=-1 for all planes).



5) Combine 3) and 4) to get useful results for a c64 vector:

We've stated that D=-1 for all 6 planes of the unit cube.

We've stated that the normal vectors for all 6 planes of the unit cube expressed in general rotation matrix form are:

Xw = [Rxx, Rxy, Rxz] = n1
n2 = -n1
Yw = [Ryx, Ryy, Ryz] = n3
n4 = -n3
Zw = [Rzx, Rzy, Rzz] = n5
n6 = -n5

We've fixed the eye position at [0,0,-3] giving us an e-value of e=1/-3.

We've stated that if C>e then the plane is visible.

We've stated that [A,B,C] is the coordinats of a plane normal.

I.e. to determine if n1 is visible:

Check if: Rxz > e

For all faces:

n1: Rxz > e
n2: -n1 > e = -Rxz > e = Rxz <= e
n3: Ryz > e
n4: -n1 > e = -Ryz > e = Ryz <= e
n5: Rzz > e
n6: -n1 > e = -Rzz > e = Rzz <= e



6) Conclusion

We've shown that giving the rotational matrix we can derive all 8 vertices of the unit cube by simple additions.
We've further shown that we can also derive all 6 face normals by simply extract the world axises.
We've shown that assuming a fixed eye position at [0,0,-z] we can simply compare the z-component of a face normal with a constant (e).
We've shown how to calculate e properly.

If you wonder how to calculate a rotational matrix quickly on the C64 I recommend you read Krill's 3d-math tutorials.

	ldx frameLo
!:	cpx visibilityTable0
	bne !+
	lda faceVisible+0
	eor #1
	sta faceVisible+0
	inc !- +1
!:	cpx visibilityTable1
	bne !+
	lda faceVisible+1
	eor #1
	sta faceVisible+1
	inc !- +1
!:	cpx visibilityTable2
	bne !+
	...