Q3 symbol list, call for participation

[holger]

Hi,

if anybody of you missed the news, John Carmack/id-software released the (mostly) GPL'd Q3 source code a few hours ago: ftp://ftp.idsoftware.com/idstuff/source ... source.zip

Didn't had the time yet to investigate the source in detail, but it may look quite nice on the PSP. Q3 Has already been reported to run on the Dreamcast (I believe officially released by id-software), so there could be a real chance to get it running on the PSP, too. A quick compile test on OS-X using the default settings lead to a binary slightly larger than 11MB, with the following unresolved symbols:


$ size ./code/macosx/build/Quake3.app/Contents/MacOS/Quake3
__TEXT __DATA __OBJC others dec hex
1576960 7979008 4096 2342912 11902976 b5a000

$ nm ./code/macosx/build/Quake3.app/Contents/MacOS/Quake3 | grep -w U
U .objc_class_name_NSAutoreleasePool
U .objc_class_name_NSBundle
U .objc_class_name_NSConstantString
U .objc_class_name_NSCursor
U .objc_class_name_NSDate
U .objc_class_name_NSDictionary
U .objc_class_name_NSException
U .objc_class_name_NSFileManager
U .objc_class_name_NSImage
U .objc_class_name_NSImageView
U .objc_class_name_NSNumber
U .objc_class_name_NSObject
U .objc_class_name_NSOpenGLContext
U .objc_class_name_NSOpenGLPixelFormat
U .objc_class_name_NSOpenPanel
U .objc_class_name_NSPanel
U .objc_class_name_NSPasteboard
U .objc_class_name_NSProcessInfo
U .objc_class_name_NSString
U .objc_class_name_NSThread
U .objc_class_name_NSUserDefaults
U .objc_class_name_NSWindow
U _AudioDeviceAddIOProc
U _AudioDeviceGetProperty
U _AudioDeviceRemoveIOProc
U _AudioDeviceSetProperty
U _AudioDeviceStart
U _AudioDeviceStop
U _AudioHardwareGetProperty
U _CFDictionaryGetValue
U _CFRelease
U _CFStringCompare
U _CGAssociateMouseAndMouseCursorPosition
U _CGDisplayAvailableModes
U _CGDisplayBounds
U _CGDisplayCapture
U _CGDisplayCurrentMode
U _CGDisplayIDToOpenGLDisplayMask
U _CGDisplayRelease
U _CGDisplayRestoreColorSyncSettings
U _CGDisplaySwitchToMode
U _CGGetActiveDisplayList
U _CGGetDisplayTransferByTable
U _CGGetLastMouseDelta
U _CGLClearDrawable
U _CGLDescribeRenderer
U _CGLErrorString
U _CGLQueryRendererInfo
U _CGLSetFullScreen
U _CGSetDisplayTransferByTable
U _CGWarpMouseCursorPosition
U _IOBSDNameMatching
U _IOIteratorNext
U _IOMasterPort
U _IOObjectRelease
U _IORegistryEntryCreateCFProperties
U _IORegistryEntryGetParentIterator
U _IOServiceGetMatchingServices
U _NSAddressOfSymbol
U _NSApp
U _NSApplicationMain
U _NSDefaultRunLoopMode
U _NSFilePosixPermissions
U _NSFileType
U _NSFileTypeDirectory
U _NSGenericException
U _NSIsSymbolNameDefined
U _NSIsSymbolNameDefinedWithHint
U _NSLog
U _NSLookupAndBindSymbol
U _NSLookupAndBindSymbolWithHint
U _NSRealMemoryAvailable
U _NSRunAlertPanel
U _NSSearchPathForDirectoriesInDomains
U _NSStringPboardType
U _NSZoneFree
U _NSZoneMalloc
U _NSZoneRealloc
U _NXCloseEventStatus
U _NXGetMouseScaling
U _NXOpenEventStatus
U _NXSetMouseScaling
U __DefaultRuneLocale
U __NSAddHandler2
U __NSConstantStringClassReference
U __NSExceptionObjectFromHandler2
U __NSRemoveHandler2
U __Unwind_Resume
U __ZTVN10__cxxabiv117__class_type_infoE
U __ZTVN10__cxxabiv120__si_class_type_infoE
U __ZdaPv
U __ZdlPv
U __Znam
U __Znwm
U ___CFStringMakeConstantString
U ___cxa_guard_acquire
U ___cxa_guard_release
U ___eprintf
U ___error
U ___gcc_qadd
U ___gcc_qdiv
U ___gcc_qmul
U ___gcc_qsub
U ___gxx_personality_v0
U ___keymgr_dwarf2_register_sections
U ___keymgr_global
U ___sF
U ___tolower
U ___toupper
U __cthread_init_routine
U __dyld_register_func_for_add_image
U __dyld_register_func_for_remove_image
U __init_keymgr
U __keymgr_get_and_lock_processwide_ptr
U __keymgr_set_and_unlock_processwide_ptr
U __setjmp
U _abort
U _acos
U _asctime
U _asin
U _atan2
U _atexit
U _atof
U _atoi
U _atol
U _bind
U _bootstrap_port
U _calloc
U _ceil
U _clock
U _close
U _closedir
U _cos
U _ctime
U _dlclose
U _dlerror
U _dlopen
U _dlsym
U _errno
U _exit
U _fclose
U _fflush
U _fileno
U _floor
U _fopen
U _fputs
U _fread
U _free
U _fseek
U _ftell
U _fwrite
U _getcwd
U _getenv
U _gethostbyname
U _getmntinfo
U _getpwuid
U _gettimeofday
U _getuid
U _glAlphaFunc
U _glArrayElement
U _glBegin
U _glBindTexture
U _glBlendFunc
U _glCallList
U _glClear
U _glClearColor
U _glClearDepth
U _glClearStencil
U _glClipPlane
U _glColor3f
U _glColor4f
U _glColor4ubv
U _glColorMask
U _glColorPointer
U _glCullFace
U _glDeleteTextures
U _glDepthFunc
U _glDepthMask
U _glDepthRange
U _glDisable
U _glDisableClientState
U _glDrawBuffer
U _glDrawElements
U _glEnable
U _glEnableClientState
U _glEnd
U _glFinish
U _glGetError
U _glGetIntegerv
U _glGetString
U _glHint
U _glLineWidth
U _glLoadIdentity
U _glLoadMatrixf
U _glMatrixMode
U _glOrtho
U _glPolygonMode
U _glPolygonOffset
U _glPopMatrix
U _glPushMatrix
U _glReadPixels
U _glScissor
U _glShadeModel
U _glStencilFunc
U _glStencilMask
U _glStencilOp
U _glTexCoord2f
U _glTexCoord2fv
U _glTexCoordPointer
U _glTexEnvf
U _glTexImage2D
U _glTexParameterf
U _glTexParameterfv
U _glTexSubImage2D
U _glTranslatef
U _glVertex2f
U _glVertex3f
U _glVertex3fv
U _glVertexPointer
U _glViewport
U _gluErrorString
U _inet_addr
U _ioctl
U _kCFAllocatorDefault
U _localtime
U _longjmp
U _mach_error_string
U _mach_init_routine
U _malloc
U _memcmp
U _memcpy
U _memmove
U _memset
U _mkdir
U _objc_msgSend
U _objc_msgSend_stret
U _opendir
U _perror
U _pow
U _pthread_cond_init
U _pthread_cond_signal
U _pthread_cond_wait
U _pthread_create
U _pthread_detach
U _pthread_mutex_init
U _pthread_mutex_lock
U _pthread_mutex_unlock
U _putenv
U _qsort
U _rand
U _read
U _readdir
U _realloc
U _recvfrom
U _remove
U _rename
U _rint
U _select
U _sendto
U _setjmp
U _setsockopt
U _setvbuf
U _sin
U _socket
U _sqrt
U _srand
U _stat
U _strcasecmp
U _strcat
U _strchr
U _strcmp
U _strcpy
U _strdup
U _strerror
U _strlen
U _strncat
U _strncmp
U _strncpy
U _strrchr
U _strstr
U _sysctl
U _tan
U _time
U _tmpfile

We can skip for now the Objective-C, the NextStep- and CoreAudio stuff, also the CocoaGL binding -- they are in the backend code and would get replaced by a EGL or glut backend, or a wgl- or glX-fake-implementation.

Most interesting are the GL, networking and pthread functions. Quite a lot of the required GL functions are implemented, but most of them not yet well-tested. We definitely need to get the texture object, general vertex array and display list support working, Jeremy and me are discussing this.

Is somebody aware of the file-i/o and networking code in the libc? What has to be done there? What about a pthread wrapper, is anybody working on this? It may be useful for other projects, too...

Holger

[jsgf]

It isn't really using display lists for anything.

It doesn't look like the threads are being used for anything. Q3 supports running the renderer in a separate thread, but that's only intended for SMP systems; it doesn't help much anyway. There's also a streaming file I/O thread, but the Linux port doesn't seem to use it.

I think the qvm stuff is actually an interpreted bytecode. They modified lcc to generate q3asm assembler, which are assembled by q3asm. So I think this is architecture-neutral.

[inomine]

It will be interesting to see how far you guys manage to get, but after you manage to get the engine to run you are going to hit the problem of limited RAM.

The PSP only has 32Mb of it, 24Mb of which is only really useable, Quake3 needs a lot more than that. The DC port relied on heavily modified map and texture files to get it to run, using the standard Q3 datasets will not work out. Your best bet is to get hold of the map-pack that was released for PC users to allow them to play against DC players, it will at least hold the reduced geometry, but then your problem becomes the textures.

By all means port the engine, it will be superb as a full feautured starting point for other devs, but I wouldn't hold me breath about people playing DM17 on the way to work any time soon.

[AnonymousTipster]

Well, good luck, I know everyone is rooting for you to bring this superb game to the PSP.
It really depends on what you can stream from MS - RAM during the game, I don't think it's possible to stream the BSP maps, but it might be possible to stream some of the textures, depending on where they are in realation to the player. As inomine said, port the engine, and get it working with small maps and textures, then work up from there.

Good Luck.

[inomine]

I just had a poke around the DC map pack and it looks like they have nicely provided just about all the reduced data set. All textures, all models, and all sounds, and importantly the BSP files are tiny (~2mb on average). Most likely the only thing they left out are the ambient acoustics and the stuff for the console and hud.

[jsgf]

I don't expect Q3A to be at all playable as a game; the PSP's controls are just too wrong for a twitch FPS. But if I can get a botmatch (or even just walking around) working nicely on the first simple map, I'll be happy. My main interest in Q3 is as a good exercise for the GL implementation; Q3 would also be a useful engine for homebrew games.

Where can you get the DC map pack from?

J

[McZonk]

I'm working on a Quake 3 Port too. But I think it should be playable. Even it will be a lot of extra work.
But you're right. The memory will be the critical part not the speed. I think it is a better idea to port the opengl calls to pspgu calls manually.

[jboldiga]

Dreamcast map-pack: Linky

edit: warning 50 meg download.[/url]

[holger]

can you please elaborate, why you think it's better to port the GL calls to pspgu manually?

I've prepared a build system for psp-gcc and all stubs for a generic-Posix/gcc-glut target, this should be easily testable on the desktop for acceptable test- and development turnaround cycles, and it will be easily portable to other consoles and platforms. Right now Quake3 is booting and crashes in the first glGet() call, when it's asking for the screen resolution (this crash is natural, I ripped out all OS initialisation stuff, so the GL context is not yet initialized at this point).

Most empty stub functions are quite easy to port, quite a lot can even be taken directly from the linux- and BSD-target. Since they are many small functions, we can well work on them in parallel.

Is there interest that I commit this stuff to SVN even in this early stage?

[holger]

update: check out http://svn.ps2dev.org/listing.php?repna ... v=912&sc=1

In the current state Q3 is booting, setting up the GL, until it tries to load it's Shader Files (then it crashes, I don't any installed. I will care about this tomorrow). Compile it by typing

$ make ARCH=""

for a native build testable and debuggable on the host system (you'll most probably want to step through it using gdb). A simple "make" will start the cross build for your PSP, but this will fail soonish unless you provide some fake implementations of the missing functions.

The build is tested only on OS-X, hope I configured the linux/cygwin flags correctly.

All stubs are located in the pspgl/test-q3/generic/ directory. If you want to apply further patches, you can simply place them in the build directory, the wildcard rule in the Makefile will automatically catch them.

[Produkt]

there is a more optomized version of Q3A based off of Quake 2...

its called Qfusion which is just the engine. then there is Qbism which supports all aspects of Q3.

they are better optomized than Q3 which would ease the psp CPU...

Im working on a port for it myself...

[McZonk]

I think the pspgu and opengl are to different for an effective wrapper. Even if both architectures are based on a state machine. The calls sceGuEnable vs glEnable are not critcal. But drawing polygons is. I looked at the pspgl port and I don't like it, even if I like OpenGL very much.
But I also think OpenGL will die in the next years. Not because of DirectX (never used, mac user), but if I look at my 3d programs at the pc: a lot of OpenGL stuff I used some years ago is dropped out (lighting, fog and all this I do with Shaders in Cg now). OpenGL is only used for pushing modeldata, state/buffer activation and some extension for speed up.

I can be happy with pspgl, that's no problem, but why we don't work together on a Q3A port? My current state is: Change all file/dir operations to psp stuff; port most of the Sys_.... calls; map controller input to joystick and keyboard. I would like to work in a team. Better if we are all doing the same work.
This is my current idea of movement:
Analog Nub = Looking; The right 4 keys = movement; Start = Menu; Select = Console; L = Jump; R = Attack; Left = Prev Weapon; Down = Next Weapon; Right = Use Item; Up = Screenshot
It's not really usable. But you need both: looking and movement.

I would enjoy to work with you, holger.

[holger]

You are not talking about the API that is going to die, but about the hardware generation this API is accessing: fixed-function GPUs don't have much future.

For all fixed-function graphics processors we have now and from the past, OpenGL is a quite good match, and there are some other consoles out there that could benefit from a generic-Q3 port that has no window-system dependencies...

For sure we'll get much sooner to a playabkle state if we're working together. Could you post your work somewhere, or, even better, merge it into the pspgl/test-q3 SVN directory?

[McZonk]

Sorry, I cannot merge my data before Thursday or Friday when I'm back home. I'll work in another city during the week.
What's the state of your version and where I can help? Still crashing at shader loading? I'll check out your version. I'm workingon mac too. So it should not produce problems with compiling :)

Edit:
I read your sources. Wow, we make two really different things :) I have make a lot of changes in the original quake 3 source code, most of them in common.c. Mapping File access to sceIo... and such stuff. Are the normal calls socket(), fopen() and such things from the standard c libs workling on the psp? I tought I had to use sceIo... for all this.

[holger]

quite a lot are working, some are missing, some are buggy. I believe the time is much better spent to fix them and add the missing bits, this is reusable by other projects, too...

[AnonymousTipster]

With regard to the controls, I am working on an FPS of my own (not quite Q3 standard), and i'm using the same control system. It seems to work quite well, but I usually use the D-Pad instead, because it is easier. I have changed my analog code so that it is ramped, rather than linear; this means that the controls are much finer, and work suprisingly well in an FPS.

I doubt it helps much, but here is the code I am using:

Code: Select all

if&#40;pad.Lx-127 > analogXCutoff || pad.Lx-127 < -analogXCutoff&#41;&#123;
			//analog stick tripped
			int multiplyer;
			//we need to process the analog value to give a value between -127 and 127
			float LxF = -&#40;pad.Lx-127&#41;;

			//now we need to make the amount ramp up, to give finer control &#58;&#41;
			if&#40;LxF < 0&#41;&#123;multiplyer = -1;&#125;else&#123;multiplyer = 1;&#125;
			LxF = &#40;&#40;&#40;pad.Lx-127&#41;*&#40;pad.Lx-127&#41;&#41;/127&#41;*multiplyer;
			playerLookLeftRight&#40;LxF, time&#41;;
&#125;

Hopefully, when I release my game, I hope to have infinitely customizable controls, so that you can choose exactly what works best for you, this may prove useful for playability testing if the Q3 port isn't at that point by release.

[holger]

status update: the generic/glut backend works so far that you can do some basic navigation and watch the demo on your desktop in 480x272 resolution. The networking stuff is not yet done, so real games fail when Q3 is searching for a server. Anyways, now it's time to work on the missing pieces in the psp libraries... check out the SVN repository.

[McZonk]

Did you ever test to compile your version for the psp? I think there are a lot of issues. Perhaps I can help at this. My version has mapped all file stuff (FILE* and DIR*) to PSP functions.
But there are two problems I haven't deal with yet. If I build Q3 for the PC I'll get 2libraries. One of them is the SIMP Lib which will be ignored if it is not there. The PSP has no SIMP instructions so there is no problem. But what is the second library?
The other problem is the network interface. To play quake I need sockets. Even if I play single player (using localhost connection). But I don't found sockets in my psp libs and I found nothing like sceSocket too.
I think holger's idea to use pspgl is fine, I also use it. But I think to use Glut and belive you get a version thats completly platform independent is far away from reality. I wish it were not. There is so much stuff inside you have to touch: fileio, mutexes, networking.

[holger]

Attempting to build a static binary for the psp by typing just "make" leads to:

$ grep "undefined reference" /tmp/build.log | sort +5 +1 | cut -d ' ' -f 5- | uniq
`__assert'
`asctime'
`closedir'
`ctime'
`getcwd'
`glArrayElement'
`glBindTexture'
`glCallList'
`glClearDepth'
`glClearStencil'
`glClipPlane'
`glColorMask'
`glDeleteTextures'
`glDepthMask'
`glDrawBuffer'
`glLineWidth'
`glPolygonMode'
`glReadPixels'
`glStencilMask'
`glTexSubImage2D'
`localtime'
`opendir'
`putenv'
`readdir'
`setvbuf'
`sscanf'
`stat'

not too far from reality, isn't it?

The network interface is not used right now (you can only watch the demos in this version), I want to care for the GL stuff first. File-I/O maps quite well on the existing PSP functions, most of them are already part of the psp-libc.

Sockets are a virtual abstraction, you can put them over every networking API, we can care about this later, when the basic demo is running on the PSP. See any libc implementation for an example, e.g. the newlibc or the dietlibc.

[crazyc]

readdir, opendir, and closedir are in newlib but there is no header. sys/dirent.h should look something like this.

Code: Select all

#include <pspiofilemgr_dirent.h>

#ifdef __cplusplus
extern "C" &#123;
#endif

#define dirent SceIoDirent
#define readdir _readdir
#define opendir _opendir
#define closedir _closedir
	
DIR *opendir&#40;const char *&#41;;
struct dirent *readdir&#40;DIR *&#41;;
int closedir&#40;DIR *&#41;;
#ifdef __cplusplus
&#125;
#endif

The defines are wrong but they work around sdk holes. Also here is an untested implementation of stat.

Code: Select all

#include <pspiofilemgr.h>
#include <sys/stat.h>
#include <string.h>

static time_t psp_to_epoch_time&#40;ScePspDateTime psp_time&#41;
&#123;
	struct tm conv_time;
	conv_time.tm_year = psp_time.year;
	conv_time.tm_mon = psp_time.month;
	conv_time.tm_mday = psp_time.day;
	conv_time.tm_hour = psp_time.hour;
	conv_time.tm_min = psp_time.minute;
	conv_time.tm_sec = psp_time.second;
	conv_time.tm_isdst = -1;
	return mktime&#40;&conv_time&#41;;
&#125;

int _stat&#40;const char *filename, struct stat *buf&#41;
&#123;
	SceIoStat psp_stat;
	memset&#40;buf, '\0', sizeof&#40;struct stat&#41;&#41;;
	if&#40;sceIoGetstat&#40;filename, &psp_stat&#41; < 0&#41;
		return -1;
	
	buf->st_ctime = psp_to_epoch_time&#40;psp_stat.st_ctime&#41;;
	buf->st_atime = psp_to_epoch_time&#40;psp_stat.st_atime&#41;;
	buf->st_mtime = psp_to_epoch_time&#40;psp_stat.st_mtime&#41;;

	buf->st_mode = &#40;psp_stat.st_mode & 0xfff&#41; |
		       &#40;&#40;FIO_S_ISLNK&#40;psp_stat.st_mode&#41;&#41;?&#40;S_IFLNK&#41;&#58;&#40;0&#41;&#41; |
		       &#40;&#40;FIO_S_ISREG&#40;psp_stat.st_mode&#41;&#41;?&#40;S_IFREG&#41;&#58;&#40;0&#41;&#41; |
		       &#40;&#40;FIO_S_ISDIR&#40;psp_stat.st_mode&#41;&#41;?&#40;S_IFDIR&#41;&#58;&#40;0&#41;&#41;;
	buf->st_size = psp_stat.st_size;
	return 0;
&#125;

[McZonk]

I see there are just a few functions make problems on the psp. Sorry, I won't criticise your work. I think I was wrong. But what is with the libraries?

[mrbrown]

Sockets are a virtual abstraction, you can put them over every networking API, we can care about this later, when the basic demo is running on the PSP. See any libc implementation for an example, e.g. the newlibc or the dietlibc.

Your toolchain shipped with newlib, which defines all of the libc functions you listed. Replace -lpsplibc with -lc.

For the life of me I don't know why people insist on using psplibc when trying to do ports :).

[holger]

*g* because they're cheating and stealing their LFLAGS from the samples ;-)

thanks for the hint!!

[holger]

Marcus, in which order one needs to link the libraries? Don't get this right, I always have some unresolved references left. I'll update my toolchain now, just for the case you changed things in the newlibc...

Could you please take a look in the test-q3/Makefile? How would you write the LFLAGS?

[mrbrown]

-lc needs to go before -lpspuser and -lpspkernel. There may be some platform-dependent calls that are still missing like the _stat() code crazyc posted. crazyc's _stat() will probably go in later today.

[holger]

Now we have some more undefined references, and as before most of them are trivial. Isn't gettimeofday() somewhere implemented? I believed open/close/seek/read/write/opendir/readdir/closedir too.... if not, they're easy.

`_close'
`_fstat'
`_getpid'
`_gettimeofday'
`_kill'
`_link'
`_lseek'
`_open'
`_read'
`_stat'
`_unlink'
`_write'
`closedir'
`getcwd'
`isatty'
`opendir'
`readdir'

What about kill, link and isatty()?

../../../../../newlib/libc/stdio/makebuf.c:96: undefined reference to `isatty'
../../../../../newlib/libc/reent/signalr.c:61: undefined reference to `_kill'
../../../../../newlib/libc/reent/signalr.c:96: undefined reference to `_getpid'

Need to find out why the signal code is touched... May the code become more compact when using the psplibc? The number of undefined references and the amount of required work seems to be about the same...

[crazyc]

`_close'
`_gettimeofday'
`_lseek'
`_open'
`_read'
`getcwd'
`_write'
`_unlink'

These are in and work, there must be something wrong with your toolchain. If you have an older toolchain, you have to link with libpspglue.

`_fstat'

This is in but is a stub. It will require some work to implement.

`isatty'
`_link'

These are in as stubs.

`_getpid'
`_kill'

These are not in.

`_stat'
`closedir'
`opendir'
`readdir'

Look up.

[holger]

thanks for the hint, now it works! we're getting closer:

`_stat'
`closedir'
`glArrayElement'
`glBindTexture'
`glCallList'
`glClipPlane'
`glDeleteTextures'
`glDrawBuffer'
`glReadPixels'
`glStencilMask'
`glTexSubImage2D'
`opendir'
`readdir'

[mrbrown]

You need to simplify your LFLAGS. libpspglue.a no longer exists and you don't want to link both libc.a and psplibc.a at the same time. You want something like this:

Code: Select all

LFLAGS += -lpspdebug -lpspge -lpspdisplay -lpspctrl -lpspsdk -lc -lm -lpspuser

If it's been awhile since you last updated your toolchain you should completely wipe /usr/local/pspdev before rerunning toolchain.sh (that would've caught libpspglue.a).

[McZonk]

Okey, I looked again at pspgl and I like it. Perhaps it is really possible to write platform indepentend stuff.
I'm at home this evening. So what can I do? Writing the missing pspgl functions? I think I can help with:
glReadPixels (used for screenshots, I have written this my self in my quake version, but I think your port is better, holger)
glStencilMask (not sure if this call can be mapped 1:1 but if I call sceGuStencilFunc() with the old parameters (have to be kept than, when I call glStencilFunc) and the new mask it should have the same behavior)