PS3 Ubuntu/Yellowdog file handling issues

SamuraiX · Post by **SamuraiX** » Mon Apr 14, 2008 5:56 am

Ok, Now i've ported my application to work on a x86 Ubuntu machine a while back and it works great. Now I've just installed Ubuntu on my ps3 and tried to compile it using the same steps I've used in my x86 Ubuntu machine and for the strangest reason I can't get it working.

It seems to have problems with reading files in binary mode as the values are all wrong. Is there something I'm missing or should perform differently when using PS3 Ubuntu versus a x86 Ubuntu machine?

Here is an example of what I am seeing:

Code: Select all

#define	PACKMAGIC 0x4B434150
int magic;
// Read magic dword &#40;"PACK" identifier&#41;
    if&#40;read&#40;handle,&magic,4&#41;!=4 || magic!=PACKMAGIC&#41;
	&#123;
		close&#40;handle&#41;;
		return -1;
    &#125;

Now on Windows, PSP, x86 Ubuntu and every other platform, the variable magic is always populated with the following value: 1262698832 (0x4B434150) which is correct.

However, on PS3 Ubuntu or Yellowdog the value is 1346454347 (0x5051434b)

Now this tells me that the order is incorrect. On PS3 its populating the magic value from right to left 0x5051434b but on all other platforms it reads the value left to right which is the correct way.

Any advice? Thanks in advance!

ldesnogu · Post by **ldesnogu** » Mon Apr 14, 2008 6:09 am

Your program works on little-endian machines, but the PS3 is big-endian.

ouasse · Post by **ouasse** » Mon Apr 14, 2008 10:01 pm

some code like :

Code: Select all

char magic&#91;4&#93;;
if &#40;read&#40;handle, magic, 4&#41;!=4 || memcmp&#40;magic, "PACK", 4&#41;&#41;
&#123;
   .....
&#125;

should work on all platforms. You shouldn't speculate on the machine's endianness when performing I/O.

SamuraiX · Post by **SamuraiX** » Mon Apr 14, 2008 11:43 pm

ldesnogu wrote:Your program works on little-endian machines, but the PS3 is big-endian.

That did it! Once I swapped the order to little-endian, the data is no longer reversed and everything is running fine.

Thank You very much!

Now I just need to figure out why my app reports I'm out of memory. Perhaps I should just have it run as an otheros instead of trying to share resources within Ubuntu. Funny thing is, I figured it would be a straight port to PS3/UBuntu since I'm using SDL framework and the port from x86/Windows to x86/Ubuntu only took minutes to do.

ldesnogu · Post by **ldesnogu** » Tue Apr 15, 2008 12:44 am

SamuraiX wrote:Now I just need to figure out why my app reports I'm out of memory. Perhaps I should just have it run as an otheros instead of trying to share resources within Ubuntu. Funny thing is, I figured it would be a straight port to PS3/UBuntu since I'm using SDL framework and the port from x86/Windows to x86/Ubuntu only took minutes to do.

I think it will probably be much easier to develop/debug on PS3 Linux. My understanding is that bare-metal dev on the PS3 is not ready for prime time :)

Other things that might kill you, apart from endianism, are some non portable assumptions about long, and pointer sizes, especially if your ubuntu is 64 bits. You could try to compile your program with -m32 to see if it changes its behavior.

jimparis · Post by **jimparis** » Tue Apr 15, 2008 3:24 pm

SamuraiX wrote:Now I just need to figure out why my app reports I'm out of memory. Perhaps I should just have it run as an otheros instead of trying to share resources within Ubuntu. Funny thing is, I figured it would be a straight port to PS3/UBuntu since I'm using SDL framework and the port from x86/Windows to x86/Ubuntu only took minutes to do.

As ldesnogu says, debugging under Linux should be a lot easier than trying to do anything lower-level. Try using massif:

Code: Select all

valgrind --tool=massif ./myprogram

(either on your PC or on the PS3) to figure out if maybe you're just really allocating too much memory.

Jim · Post by **Jim** » Tue Apr 15, 2008 10:49 pm

Now I just need to figure out why my app reports I'm out of memory

I suspect another endian problem, just a little further in.
Normally
malloc (0x00 00 00 01)
becomes
malloc (0x01 00 00 00)
and you run out of ram.

Jim

SamuraiX · Post by **SamuraiX** » Tue Apr 15, 2008 11:57 pm

I'm sorry for this noobish question but I have only worked on little endian systems, are there any preprocessor commands that will have a Big Endian system revert to Little Endian? Or must I be byte swapping all my data from right->left to left->right?

But this doesn't make sense, in the fact that malloc returns NULL (I will double check). What do you recommend for malloc to perform like it does in a little endian system. Otherwise, I would have to assume allot of rework/byte swapping is going to be need for my application to run under PS3/Ubuntu.

{Edit}

I found this great article on porting apps from 32 bit to 64 bit OS(s) here: link

Seems its going to be a painful road ahead..... ;p

ldesnogu · Post by **ldesnogu** » Wed Apr 16, 2008 1:08 am

SamuraiX wrote:I'm sorry for this noobish question but I have only worked on little endian systems, are there any preprocessor commands that will have a Big Endian system revert to Little Endian? Or must I be byte swapping all my data from right->left to left->right?

It's a property of the processor not of the language, so you can't change it magically. You will have to examine your code carefully especially if you are reading some binary file as you seem to do.

But this doesn't make sense, in the fact that malloc returns NULL (I will double check). What do you recommend for malloc to perform like it does in a little endian system. Otherwise, I would have to assume allot of rework/byte swapping is going to be need for my application to run under PS3/Ubuntu.

I think Jim hinted what's happening. If you read your malloc size from a binary file assuming a 32 bit entry, then it's wrongly ordered and you get some crazy value which will make malloc fail...

Rewrap malloc and make it print its parameter to see what's going on. Or use valgrind as Jimparis suggested.

SamuraiX · Post by **SamuraiX** » Wed Apr 16, 2008 3:50 am

This makes complete sense why I would potentially run out of memory! Certain mallocs within my application are based off the image size (width*height) that is read from the file's header, now the pieces are starting to line up! So the lesson learned here is check all binary file operations and be sure to add checks or wrappers for byte swapping, depending on endian format.

Thank You all very much!

SamuraiX · Post by **SamuraiX** » Wed Apr 16, 2008 1:56 pm

Ok.... I'm finished! I've finished porting OpenBOR to the PS3/Ubuntu setup. Incase you guys don't know what OpenBOR is... Its very custimizable engine that is used for recreating old side-scrolling bashers. Streets of Rage, Final FIght and so on... The original creators of the BOR engine can be found here: http://www.senileteam.com and the official OpenBOR site is http://www.lavalit.com (my site).

So all you guys need to do if you want to check it out is the following:

Grab the source here: svn co https://openbor.svn.sourceforge.net/svnroot/openbor
Required Libs prior to compilation: libSDL, libSDL_gfx, LibSDL_mixer, LibSDL_image, zlib and libpng.
Automated Building type: ./build.sh
Manual Building type:
source environ.sh 4
make BUILD_LINUX=1

Automated
If you have used the automated process then under releases directory you will have a LINUX folder, Go to my site (www.lavalit.com - registration required) and download a few mods aka pak files and place them in the Paks folder.

Manual
If you did not use the automated build system then create the following directories along side of the executable (case sensitive):
Paks, Logs, Saves, ScreenShots

Place the xbox/data/menu.pak into the Paks directory and the mods you downloaded from my site (www.lavalit.com - registration required) also into the Paks directory.

Note: Make sure all the directories have 777 for permissions.

Now you are ready for some good old school bashing! With Golden Axe - Remake I get 166 fps @ 480x272. Now obviously its not PS3 optimized (Waiting for PS3SDK from this great site here!) but compared to my PSP optimized port (120 fps @ the same resolution). Not too shaby since Ubuntu does run rather slow on PS3.

Thank you to all that helped me with the Big Endian suggestions and that great tool for watching memory.