Help with filling stencil buffer

[arpaagent]

ok, I am trying to draw to only a certain portion of the screen by using stencilling, but I think i am having trouble filling in the stencil data. From snooping around the forums, it seems like the framebuffer alpha channel and the stencil buffer are one in the same. Is this correct? So, for example, if i want to fill the stencil buffer with 0xff and then draw a polygon with stencil value 0x00 in the stencil buffer, could I do it like this?

Code: Select all

sceGuClearStencil( 0xff );
sceGuClear( GU_STENCIL_BUFFER_BIT );
// this lets us draw to the alpha channel without affecting color channels
sceGuPixelMask( 0x00ffffff );

// this will give us 0 in the alpha channel
sceGuColor( 0x00000000 );
// draw polygons in the shape of desired stencil
sceGumDrawArray( ... );

and then later, I could use the stencil values i created with blending...like this?? So that anything with a stencil value of 0xff would pass but anything with stencil value of 0x00 would not get drawn.

Code: Select all

sceGuEnable( GU_BLEND );
sceGuBlendFunc( GU_ADD, GU_DST_ALPHA, GU_ONE_MINUS_DST_ALPHA, 0, 0 );
// draw polygons that will get stencilled
sceGumDrawArray( ... );

I've been messing around with this for a while now and I just can't get it to work like I want it to.

Any insight would be greatly appreciated! thanks!!

[Smong]

I can't seem to find where sceGuStencilClear is defined, I have r2309 of the psptoolchain from about 2 weeks ago.

[arpaagent]

sorry, i meant sceGuClearStencil( );
edited my post

[Smong]

Edit: post completely rewritten with (partial?) solution.

After looking at the gu/reflection and playing around with some code I have come to these conclusions:
- You need to enclose the sceGumDrawArray that you want to be affected by stencilling with sceGuEnable(GU_STENCIL_TEST).
- You need to call sceGuStencilFunc and sceGuStencilOp twice, once before drawing your stencil, then again before you draw the things you want stencilled.
- sceGuPixelMask isn't necessary, but you must set it back to 0 if you do call it.
- The blend function should be done on SRC, not DST.
- The relevant code in the reflection sample is between lines 188 and 205

Here's my test function:

Code: Select all

struct Vertex
{
	float x, y, z;
};

#define DO_VERT(vert, xx, yy, zz) \
	(vert)->x = xx; \
	(vert)->y = yy; \
	(vert)->z = zz; \
	vert++;

/*
when setting up gu:
sceGuClearStencil(0x00);
sceGuEnable(GU_STENCIL_TEST); // Stencil test (might want to move this to only enclose the things you want stencilled)
sceGuEnable(GU_BLEND);
sceGuBlendFunc(GU_ADD, GU_SRC_ALPHA, GU_ONE_MINUS_SRC_ALPHA, 0, 0);

at the start of your rendering loop (note last flag):
sceGuClear(GU_COLOR_BUFFER_BIT | GU_DEPTH_BUFFER_BIT | GU_STENCIL_BUFFER_BIT);

before drawing stuff you want stencilled:
Gfx_StencilRectangle(...)

some parts copied from reflection sample:
Copyright (c) 2005 Jesper Svennevid
Copyright &#40;c&#41; 2005 McZonk <[email protected]>
*/

void Gfx_StencilRectangle&#40;int x, int y, int w, int h&#41;
&#123;
	struct Vertex *vdata = sceGuGetMemory&#40;sizeof&#40;struct Vertex&#41; * 4&#41;;
	struct Vertex *v = vdata;

	sceGuStencilFunc&#40;GU_ALWAYS, 1, 1&#41;; // always set 1 bit in 1 bit mask
	sceGuStencilOp&#40;GU_KEEP, GU_KEEP, GU_REPLACE&#41;; // keep value on failed test &#40;fail and zfail&#41; and replace
	sceGuDepthMask&#40;GU_TRUE&#41;;

	// turn off alpha &#40;0&#41; so drawing doesn't actually effect the color buffer only the stencil buffer
	// if you turn alpha on &#40;>0&#41; then it will be give the stencilled area a background color
	sceGuColor&#40;0x00000000&#41;;

	// draw polygons in the shape of desired stencil
	DO_VERT&#40;v, x, y, 0&#41;;
	DO_VERT&#40;v, x + w, y, 0&#41;;
	DO_VERT&#40;v, x, y + h, 0&#41;;
	DO_VERT&#40;v, x + w, y + h, 0&#41;;

	// disable GU_TEXTURE_2D since we aren't using GU_TEXTURE_* flag
	sceGuDisable&#40;GU_TEXTURE_2D&#41;;
	sceGumDrawArray&#40;GU_TRIANGLE_STRIP,
		GU_VERTEX_32BITF | GU_TRANSFORM_3D,
		4, 0, vdata&#41;;
	sceGuEnable&#40;GU_TEXTURE_2D&#41;;

	sceGuStencilFunc&#40;GU_EQUAL, 1, 1&#41;; // allow drawing where stencil is 1
	sceGuStencilOp&#40;GU_KEEP, GU_KEEP, GU_KEEP&#41;; // keep the stencil buffer unchanged
	sceGuDepthMask&#40;GU_FALSE&#41;;
&#125;

It doesn't work 100% I can see some artifacts (text from psplink showing through), probably because I'm also drawing textures with alpha which mess up the stencil buffer.

[arpaagent]

Smong, thanks for the insight

I think it makes a little more sense to me now. The stencilFunc is not only used for testing against the stencil to mask where objects are drawn, but it is also used to put values into the stencil buffer, using the GU_ALWAYS flag. Also, would it be correct to say that in the first StencilOp, the the fail conditions will never occur since the previous call to the StencilFunc is GU_ALWAYS, implying that any drawing done will result in a pass value for stencil purposes? Just trying to get a better grasp on what exactly is going on.

I also did notice that some text from pspDebugScreenPrint was showing up when I was messing with it earlier, although I thought that would be resolved by making a call to sceGuClear( ...GU_STENCIL_BUFFER_BIT... ). I guess I will have to play with it some more.

Later this evening I will try to recode up what i was doing and see if that solves my problems. I will let you know how it goes.

Thanks again!

[Smong]

Also, would it be correct to say that in the first StencilOp, the the fail conditions will never occur since the previous call to the StencilFunc is GU_ALWAYS, implying that any drawing done will result in a pass value for stencil purposes?

After some quick testing it seems like that is the case. I tried changing the GU_KEEP's to GU_REPLACE, GU_INVERT and GU_ZERO in the first sceGuStencilOp call and it didn't seem to make any difference.