[QUESTION] What is the max heap size of a user prx?

theHobbit · Post by **theHobbit** » Wed Jul 23, 2008 3:39 am

Hi there, just wondering, what is the max heap size of a user mode prx?
Cuz i'm trying to set PSP_HEAP_SIZE_KB to 6Mb but i'm getting some errors with malloc and fopen, like i cant open a file larger than 6Kb and cant allocate around 2Mb of memory.

Also i checked the free memory with sceKernelTotalFreeMemSize after loading the prx and it just takes around 2Mb to load it.

Thanks in advance.

Hellcat · Post by **Hellcat** » Wed Jul 23, 2008 4:55 am

The max. value I have tested myself so far is 22.5 MB - working on Fat and Slim.

I think I even did 23MB once, but I'm not sure yet....

J.F. · Post by **J.F.** » Wed Jul 23, 2008 5:00 am

If you want the max on both a Phat and a Slim with some space left over for extra threads, use a negative heapsize. That's why we added that. Just do something like

Code: Select all

    PSP_HEAP_SIZE_KB&#40;-128&#41;;

And it will use everything but 128K on both the Phat and the Slim. Don't forget to put

Code: Select all

PSP_LARGE_MEMORY = 1

in the makefile to activate the extra memory on the Slim. It is safe for the Phat as well where it does nothing.

Another big advantage of using the negative heap size is that you don't have to ever worry about changing the size of the program. Those maximums you and the other guy were talking about will vary depending on how big your program is. As your program gets larger as you add to it, that value will have to be decreased. The negative size never has to be changed.

hlide · Post by **hlide** » Wed Jul 23, 2008 6:50 am

well with psplink v3 oe, it doesn't work very well :/

J.F. · Post by **J.F.** » Wed Jul 23, 2008 7:09 am

hlide wrote:well with psplink v3 oe, it doesn't work very well :/

Is there a problem with getting the max memory in newlib under psplink? Seems that would affect quite a few things. Best to keep a small heap for debugging with psplink, then use negative heapsize for the release, or don't use psplink to debug (I never do).

theHobbit · Post by **theHobbit** » Wed Jul 23, 2008 8:35 am

Hi, thanks for the answers. But i guess i need to be more specific about my problem. I have a main app prx wich loads several modules (plugins) in the user memory partition, and everytime it loads a plugin i check how much memory is left in the main app with sceKernelTotalFreeMemSize but even if i set a heap size of 6MB in a plugin the memory used to load it is just around 2MB. So i was wondering if the plugin is getting the 6MB or just the 2MB that takes to load it.

Because first i tryied the plugin code as a standalone app with only 6MB of heap and it worked fine. But when i tryied to load it as a plugin in the main app there were several memory allocation errors, and it couldn't even open a 6KB file with fopen :(.

Oh, and i'm using psplink and a psp slim to debug the app, maybe it has something to do with psplink?

hlide · Post by **hlide** » Wed Jul 23, 2008 6:44 pm

J.F. wrote:
hlide wrote:well with psplink v3 oe, it doesn't work very well :/
Is there a problem with getting the max memory in newlib under psplink? Seems that would affect quite a few things. Best to keep a small heap for debugging with psplink, then use negative heapsize for the release, or don't use psplink to debug (I never do).

There is several main advantages to use psplink even without gdb :

- (re)build your project then just type the filename of your PRX binary on pspsh prompt to run it WITHOUT any first transfer on MS. Development is faster, easier and less restricting.

- using host0: instead of ms0: to handle files is INSANELY FASTER. My dynarec was able to generate a disassembly output in less than 1 seconds in a log file instead of more than 30 seconds if I'm doing it through ms0: (the log file is about 700Kb of text). Not speaking about sparing my MS this way. I want to see my log file ? just open it from the PC directory where host0: is. No need to transfer it from MS since it was directly output on my PC.

- stdout printf can be used to output useful messages.

- using TTY command on PSPSH, you can also read stdin on PSP to interact with it. To leave TTY mode, you must type "~." at the begining of a new line on PSPSH console.

hlide · Post by **hlide** » Wed Jul 23, 2008 6:52 pm

theHobbit wrote:Oh, and i'm using psplink and a psp slim to debug the app, maybe it has something to do with psplink?

I can only tell you what I experienced :

without PSPLINK, I was able to allocate a block to the end of 64 MB (slim).

with PSPLINK, I was able to have the same result JUST ONCE after testing it without psplink, then I was always stuck with a block allocated at the end of lower 32 MB.

EDIT: i'm using PSPLINK V3 OE from Heimdall's PSPSDK

moonlight · Post by **moonlight** » Wed Jul 23, 2008 8:18 pm

If you have various user prx, each one will reserve in the system the specified amount of heap size, wether it is actually using all of it or not, leaving less space for the rest of modules heaps. I think, however, the heap is not inmediatelly allocated, but the first time a function (yours or from a lib) calls malloc or similar.

You have two options:
- Create the modules in a way that the addition of all them heap is not bigger than that maximum of ~22.5 MB
- Find a way of sharing the memory handling between all of them, this is probably the most proper solution.

An example of how an inter-user modules memory sharing works is the own vsh: they have a module which has the full libc including malloc, and which creates the heap (paf.prx), and the rest of modules use that heap through paf functions, you'll have to find a way of doing something like that, or just do the first solution if you are lazy ;)

hlide · Post by **hlide** » Wed Jul 23, 2008 9:58 pm

moonlight wrote:I think, however, the heap is not inmediatelly allocated, but the first time a function (yours or from a lib) calls malloc or similar.

I'm using "memalloc" to allocate several 1-byte long blocs aligned to 1MB-boundary :

Code: Select all

#define KB * 1024
#define MB * 1024 KB

s32 i = 64;
u8 *block;
u8 *blocks&#91;64&#93;;

// allocate several 1-byte long blocks aligned to 1MB-boundary up to the end of main ram
while &#40;&#40;block = &#40;u8 *&#41;memalloc&#40;1 MB, 1&#41;&#41;&#41;
&#123;
	if &#40;i-- < 0&#41;
		break;
	blocks&#91;i&#93; = block;
&#125;

// try to allocate my 3MB block at the end of main ram
while &#40;!&#40;block = &#40;u8 *&#41;memalloc&#40;1 MB, 3 MB&#41;&#41;&#41;
&#123;
	if &#40;i < 64&#41;
		break;
	free&#40;blocks&#91;i++&#93;&#41;;
&#125;

// free the other blocks
while &#40;i < 64&#41;
	free&#40;blocks&#91;i++&#93;&#41;;

I guess "memalloc" uses "malloc".

Smong · Post by **Smong** » Wed Jul 23, 2008 11:26 pm

hlide wrote:

Code: Select all

s32 i = 64;
u8 *blocks&#91;64&#93;;

...

// try to allocate my 3MB block at the end of main ram
while &#40;!&#40;block = &#40;u8 *&#41;memalloc&#40;1 MB, 3 MB&#41;&#41;&#41;
&#123;
	if &#40;i < 64&#41;
		break;
	free&#40;blocks&#91;i++&#93;&#41;;
&#125;

I'm not entirely sure what you're trying to do but it looks like this would free blocks[64], which is off the end of the array.

hlide · Post by **hlide** » Thu Jul 24, 2008 12:08 am

Smong wrote:
hlide wrote:
Code: Select all
s32 i = 64;
u8 *blocks&#91;64&#93;;

...

// try to allocate my 3MB block at the end of main ram
while &#40;!&#40;block = &#40;u8 *&#41;memalloc&#40;1 MB, 3 MB&#41;&#41;&#41;
&#123;
	if &#40;i < 64&#41;
		break;
	free&#40;blocks&#91;i++&#93;&#41;;
&#125;
I'm not entirely sure what you're trying to do but it looks like this would free blocks[64], which is off the end of the array.

I don't think so. Those "free" calls occur only if i < 64 everywhere I use it.

What I want is to be sure to have my 3 MByte block to be allocated at the end of the main ram. "malloc" and "memalloc" try to allocate from the begining to the end of the heap, so I need to allocate several blocks until we reach the end of the heap (obviously the end of main ram), then try to free the last blocks at the end until I can allocate my 3 Mbyte block, then I free the left blocks.

Post by **TyRaNiD** » Thu Jul 24, 2008 4:12 am

Not strictly on topic but:

hlide: The version of PSPLINK you used did you rebuild the eboot to actually implement large memory support? PSPLINK should no longer load anything in low memory except for its initial bootstrap (which goes away once the shell is up and running) and it shouldn't try allocating anything. Try typing memfree (or was it meminfo ? :p) into the shell with no app running and see how big partition 2 is.

hlide · Post by **hlide** » Thu Jul 24, 2008 5:10 am

TyRaNiD wrote:Not strictly on topic but:

hlide: The version of PSPLINK you used did you rebuild the eboot to actually implement large memory support? PSPLINK should no longer load anything in low memory except for its initial bootstrap (which goes away once the shell is up and running) and it shouldn't try allocating anything. Try typing memfree (or was it meminfo ? :p) into the shell with no app running and see how big partition 2 is.

In the Makefile, I added "PSP_LARGE_MEMORY = 1".

In main.cpp, I have : "PSP_HEAP_SIZE_KB(-64);"

Code: Select all

host0&#58;/> ./myuserprx.prx
My allocated Block starts at 0x09C00000
meminfo
Memory Partitions&#58;
N  |    BASE    |   SIZE   | TOTALFREE |  MAXFREE  | ATTR |
---|------------|----------|-----------|-----------|------|
1  | 0x88000000 |  3145728 |    294912 |    274688 | 000C |
2  | 0x08800000 | 25165824 |    327680 |    262144 | 000F |
3  | 0x88000000 |  3145728 |    294912 |    274688 | 000C |
4  | 0x88300000 |  1048576 |   1048576 |   1048576 | 000C |
5  | 0x08400000 |  4194304 |   4194304 |   4194304 | 000F |
6  | 0x08800000 | 25165824 |    327680 |    262144 | 000F |
8  | 0x8A000000 | 29360128 |  29360128 |  29360128 | 000C |
10 | 0x8BC00000 |  4194304 |   4194304 |   4194304 | 000C |
11 | 0x8A000000 | 29360128 |  29360128 |  29360128 | 000C |
host0&#58;/>reset
Resetting psplink
host0&#58;/> meminfo
Memory Partitions&#58;
N  |    BASE    |   SIZE   | TOTALFREE |  MAXFREE  | ATTR |
---|------------|----------|-----------|-----------|------|
1  | 0x88000000 |  3145728 |    309248 |    284928 | 000C |
2  | 0x08800000 | 25165824 |  25149440 |  25149440 | 000F |
3  | 0x88000000 |  3145728 |    309248 |    284928 | 000C |
4  | 0x88300000 |  1048576 |   1048576 |   1048576 | 000C |
5  | 0x08400000 |  4194304 |   4194304 |   4194304 | 000F |
6  | 0x08800000 | 25165824 |  25149440 |  25149440 | 000F |
8  | 0x8A000000 | 29360128 |  29360128 |  29360128 | 000C |
11 | 0x8A000000 | 29360128 |  29360128 |  29360128 | 000C |
host0&#58;/>

It looks as if addresses beyond 0A000000 are kernel-only ram !?

Post by **TyRaNiD** » Thu Jul 24, 2008 6:07 am

Ah yes well there is a small issue with the boot271 makefile of course :P It doesn't use the built in makefile at all :P

Edit it and change the line

mksfo 'PSPLink v3.0 OE' PARAM.SFO

to

mksfoex -d MEMSIZE=1 'PSPLink v3.0 OE' PARAM.SFO

then make clean, make and see if it helps :) You should see a MEMSIZE string in EBOOT.PBP if it is correct.

I guess I really have been out of the game a long time :)

hlide · Post by **hlide** » Thu Jul 24, 2008 7:35 am

TyRaNiD wrote:Ah yes well there is a small issue with the boot271 makefile of course :P It doesn't use the built in makefile at all :P

Edit it and change the line

mksfo 'PSPLink v3.0 OE' PARAM.SFO

to

mksfoex -d MEMSIZE=1 'PSPLink v3.0 OE' PARAM.SFO

then make clean, make and see if it helps :) You should see a MEMSIZE string in EBOOT.PBP if it is correct.

I guess I really have been out of the game a long time :)

I'm lost, which project ? I tried to find "mksfo 'PSPLink v3.0 OE' PARAM.SFO" as a key in the "trunk\psplink" I checked out and got no result.

Post by **TyRaNiD** » Thu Jul 24, 2008 7:49 am

Probably cause you need to be looking in trunk/psplinkusb/boot271 perhaps

hlide · Post by **hlide** » Thu Jul 24, 2008 8:28 am

TyRaNiD wrote:Edit it and change the line

mksfo 'PSPLink v3.0 OE' PARAM.SFO

to

mksfoex -d MEMSIZE=1 'PSPLink v3.0 OE' PARAM.SFO

then make clean, make

it rocks !!! :) thanx !

Smong · Post by **Smong** » Thu Jul 24, 2008 8:30 am

hlide wrote:I don't think so. Those "free" calls occur only if i < 64 everywhere I use it.

What I want is to be sure to have my 3 MByte block to be allocated at the end of the main ram. "malloc" and "memalloc" try to allocate from the begining to the end of the heap, so I need to allocate several blocks until we reach the end of the heap (obviously the end of main ram), then try to free the last blocks at the end until I can allocate my 3 Mbyte block, then I free the left blocks.

I see what you're trying to do now. The following patch contains 2 changes, one to prevent a memory leak (which probably won't occur), the other is a re-explanation of what I was getting at originally.

Code: Select all

--- orig.c      2008-07-23 23&#58;25&#58;52.312500000 +0100
+++ fixed.c     2008-07-23 23&#58;25&#58;28.250000000 +0100
@@ -9,14 +9,17 @@
 while &#40;&#40;block = &#40;u8 *&#41;memalloc&#40;1 MB, 1&#41;&#41;&#41;
 &#123;
    if &#40;i-- < 0&#41;
+   &#123;
+      free&#40;block&#41;;
       break;
+   &#125;
    blocks&#91;i&#93; = block;
 &#125;
 
 // try to allocate my 3MB block at the end of main ram
 while &#40;!&#40;block = &#40;u8 *&#41;memalloc&#40;1 MB, 3 MB&#41;&#41;&#41;
 &#123;
-   if &#40;i < 64&#41;
+   if &#40;i == 64&#41;
       break;
    free&#40;blocks&#91;i++&#93;&#41;;
 &#125;

Your original loop would break immediately since "i" will almost certainly be less than 64, having the knock on effect of the 3mb block never getting allocated successfully. I could also have put ">=" as the new condition but with "==", "i" will never go over 64 so felt it unnecessary.

hlide · Post by **hlide** » Thu Jul 24, 2008 8:46 am

as I told you, it was just from memory and I was at work so I wrote it too quickly. Oh well, forget.

Smong · Post by **Smong** » Thu Jul 24, 2008 9:13 am

Sorry I didn't know that, I was just trying to help.

hlide · Post by **hlide** » Thu Jul 24, 2008 9:38 am

Smong wrote:Sorry I didn't know that, I was just trying to help.

no problem.

theHobbit · Post by **theHobbit** » Fri Jul 25, 2008 3:32 am

Hi there, thanks all for the answers. I finally managed to fix the problem. As suggested i found that the plugin prxs were using all the memory left in the app, so first i rebuilt psplink with the MEMSIZE=1 patch and the app worked fine with the extra memory in the slim. But since i want to make the app compatible with psp fat, i had to make some changes in the plugin to use less memory.

And just to let you know my app is a music player called GameMusicGear
wich plays several music formats and i'm just about to finish the next release so if anyone is interested just keep an eye at pspupdates.

So thanks again to everyone for the fast answers. Thats why i really like the forum.

Post by **ooPo** » Wed Aug 27, 2008 1:16 am

$ svn diff
Index: boot271/Makefile
===================================================================
--- boot271/Makefile (revision 2426)
+++ boot271/Makefile (working copy)
@@ -1,11 +1,11 @@
release: all
- mksfo 'PSPLink v3.0 OE' PARAM.SFO
+ mksfoex -d MEMSIZE=1 'PSPLink v3.0 OE' PARAM.SFO
pack-pbp EBOOT.PBP PARAM.SFO psplink.png NULL NULL NULL NULL psplink_boot.prx NULL

TARGET = psplink_boot
OBJS = main.o

-INCDIR =
+INCDIR =
CFLAGS = -O2 -G0 -Wall
CXXFLAGS = $(CFLAGS) -fno-exceptions -fno-rtti
ASFLAGS = $(CFLAGS)
@@ -13,6 +13,7 @@
LIBDIR =
LDFLAGS =

+PSP_LARGE_MEMORY = 1
BUILD_PRX = 1

PSPSDK=$(shell psp-config --pspsdk-path)

$ svn commit
Sending boot271/Makefile
Transmitting file data .
Committed revision 2427.

Committed.

Post by **TyRaNiD** » Wed Aug 27, 2008 2:30 am

Of course you might not actually want to use large memory support depends on what you are trying to target ;)

Post by **ooPo** » Wed Aug 27, 2008 5:47 am

Around here, we do everything BIG!

BIG!

Post by **ooPo** » Wed Aug 27, 2008 5:48 am

Unfortunately, that's as big as it gets.

Raphael · Post by **Raphael** » Thu Aug 28, 2008 9:20 pm

Then, why isn't it called PSP_BIG_MEMORY? ;)