Malloc and Realloc on psp

[sauron_le_noir]

Is there a function available that give you the remaining space available for a malloc or realloc function ? the new sumatra pdf load a load of resource in main memory and
for some pdfs (pdf with large image) the realloc function get a out of mem

[moonlight]

Try mallinfo, maybe it works.

[cory1492]

I think the malloc/free or realloc still has some leakage, if you are getting tight on memory you may want to keep that in mind.

[hlide]

you mean fragmentation ? because fragmentation and leakage aren't the same thing. A leakage would be unacceptable and I cannot imagine libc to have a defective malloc and realloc. On the other hand, malloc and realloc might use an algorithm which can lead to a lot of fragmented free blocks of small size under some conditions, preventing you from allocating a big one even if total free space is enough.

[sauron_le_noir]

it's the second one there are many little block available but not one big enought
that is the function creating this.it increment by 4k until it find a buffer large enought
to read the PDF stream resource.
Pdf containing a lot of streams ( essentialy pdfs with a lot a images) got a out of memory error. AM i right the psp doesn't have virtual memory or DAT (dynamic address translator) ?

Here is the function causing the trouble

Code: Select all

*
 * Utility function to consume all the contents of an input stream into
 * a freshly allocated buffer; realloced and trimmed to size.
 */

enum { CHUNKSIZE = 1024 * 4 };

fz_error *
fz_readall(fz_buffer **bufp, fz_stream *stm)
{
	fz_error *error;
	fz_buffer *real;
	unsigned char *newbuf;
	unsigned char *buf;
	int len;
	int pos;
	int n;

	len = 0;
	pos = 0;
	buf = nil;

	while (1)
	{
		if (len - pos == 0)
		{
			if (len == 0)
				len = CHUNKSIZE;
			else
				len *= 2;
			newbuf = fz_realloc(buf, len);
			if (!newbuf)
			{
				fz_free(buf);
				return fz_throw("outofmem: scratch buffer");
			}
			buf = newbuf;
		}

		error = fz_read(&n, stm, buf + pos, len - pos);
		if (error)
		{
			fz_free(buf);
			return fz_rethrow(error, "cannot read data");
		}

		pos += n;

		if &#40;n < CHUNKSIZE&#41;
		&#123;
			if &#40;pos > 0&#41;
			&#123;
				newbuf = fz_realloc&#40;buf, pos&#41;;
				if &#40;!newbuf&#41;
				&#123;
					fz_free&#40;buf&#41;;
					return fz_throw&#40;"outofmem&#58; scratch buffer"&#41;;
				&#125;
			&#125;
			else newbuf = buf;

			real = fz_malloc&#40;sizeof&#40;fz_buffer&#41;&#41;;
			if &#40;!real&#41;
			&#123;
				fz_free&#40;newbuf&#41;;
				return fz_throw&#40;"outofmem&#58; buffer struct"&#41;;
			&#125;

			real->refs = 1;
			real->ownsdata = 1;
			real->bp = newbuf;
			real->rp = newbuf;
			real->wp = newbuf + pos;
			real->ep = newbuf + pos;
			real->eof = 1;

			*bufp = real;
			return fz_okay;
		&#125;
	&#125;
&#125;

[/code]

[sauron_le_noir]

[Can somebody explain me or point me a tutorial that explain the organisation of the memory partition on a psp (can be technical i'm a old os/2 device driver develloper)
also a explanation of the meminfo whould be great

Code: Select all

host0:/> meminfo
Memory Partitions:
N  |    BASE    |   SIZE   | TOTALFREE |  MAXFREE  | ATTR |
---|------------|----------|-----------|-----------|------|
1  | 0x88000000 |  3145728 |    289792 |    268544 | 000C |
2  | 0x08800000 | 54525952 |    782336 |    524288 | 000F |
3  | 0x88000000 |  3145728 |    289792 |    268544 | 000C |
4  | 0x88300000 |  1048576 |   1048576 |   1048576 | 000C |
5  | 0x08400000 |  4194304 |   4194304 |   4194304 | 000F |
6  | 0x08800000 | 54525952 |    782336 |    524288 | 000F |
8  | 0x8BC00000 |        0 |         0 |         0 | 000C |
10 | 0x8BC00000 |  4194304 |   4194304 |   4194304 | 000C |
11 | 0x8BC00000 |        0 |         0 |         0 | 000C |
host0:/> warning: cannot realloc 286720 bytes
+ stream/stm_misc.c:91: fz_readall(): outofmem: scratch buffer
| mupdf/pdf_stream.c:510: pdf_loadstream(): cannot load stream into buffer (366)
| mupdf/pdf_resources.c:165: preloadxobject(): cannot load image resource 366
| mupdf/pdf_resources.c:430: pdf_loadresources(): cannot load xobject resource
| mupdf/pdf_page.c:233: pdf_loadpage(): cannot load page resources

i guess ATTR give me the type of memory kernel or is partition number 2 the user partition ?
in the meminfo i see that i have 286720 byte free but the realloc failed

Initializing triMemory
allocating 557056 bytes, in 2176 blocks
allocated at address 0x4000000
allocating 557056 bytes, in 2176 blocks
allocated at address 0x4088000
Initializing triInput
allocating 3072 bytes, in 12 blocks
allocated at address 0x4110000
allocating 5120 bytes, in 20 blocks
allocated at address 0x4110c00
allocating 475136 bytes, in 1856 blocks
allocated at address 0x4112000

why have i 0x400000 ,4112000 etc as adrs it is display with a printf (" %p"

[sauron_le_noir]

moonlight you are a genious
look at : http://www.gnu.org/software/libtool/man ... for-Malloc very interesting
and
http://www.gnu.org/software/libtool/man ... -of-Malloc

i try the tuto to see if working on the psp

[sauron_le_noir]

grrr malloc_hook are gone in 4.3.1

----> not use malloc hooks. They cannot work reliably and will disappear
in the next malloc implementation. Ignore them.

--
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖

http://www.linuxjournal.com/article/6390
malloc_stats() works on psp

[hlide]

sauron_le_noir, again, please use tag

Code: Select all

[code]your code here[/code]

to put your code therein. Any code you post here is unreadable because the indentation is lost. Nobody will read your code if your persist this way.

[sauron_le_noir]

it's done

[cory1492]

you mean fragmentation ? because fragmentation and leakage aren't the same thing. A leakage would be unacceptable and I cannot imagine libc to have a defective malloc and realloc. On the other hand, malloc and realloc might use an algorithm which can lead to a lot of fragmented free blocks of small size under some conditions, preventing you from allocating a big one even if total free space is enough.

All I know for sure at this point is, if I malloc+realloc until all available memory is used (in small chunks), then free it, my next malloc cannot be the same amount I initially allocated. Whether this is an "on purpose" function of malloc I couldn't say, but it sure looks like some kind of leakage (or error) is occurring to me and is something I figured would be worth mentioning to sauron_le_noir.

Due to this behavior, I for one try to avoid malloc.

[Raphael]

Initializing triMemory
allocating 557056 bytes, in 2176 blocks
allocated at address 0x4000000
allocating 557056 bytes, in 2176 blocks
allocated at address 0x4088000
Initializing triInput
allocating 3072 bytes, in 12 blocks
allocated at address 0x4110000
allocating 5120 bytes, in 20 blocks
allocated at address 0x4110c00
allocating 475136 bytes, in 1856 blocks
allocated at address 0x4112000

why have i 0x400000 ,4112000 etc as adrs it is display with a printf (" %p"

Those allocs are the ones from triVAlloc for VRAM (which allocate in 0x4000000-0x4200000). For the normal triMalloc log see triMemoryLog.txt

[sauron_le_noir]

I've found why sumatra pdf fails on some pdfs it is not a question of numbers of pages i have a redbook from IBM with 631 rendering wihout any problems but i have a pdf with 200 pages every page is a bitmap ARGB 480x272 and it breaks. Sumatra pdf library store all the resource in a pdf store in memory so the last pdf get a out of mem when reading the page 134. I'm rewritting the pdf_store code of the library to accept
to use memory or Memory stick swap space for his pdf store.
So i hope to be able to render every pdf on the psp that sumatra can render on a pc.
I've also investigated alternative malloc,free,realoc than standard libgc
It is a work in progress...... keep in touch :p:p:p
And raphael your library is really great !!!! i have solved the bitmap problems H > 1024 and W > 1024
I have a ARGB with a bitmap greater than 1024x1024
i just created a bitmap of 480x272 and extract the portion form the big buffer so problem solve and it is very fast.