Hypervisor API

StrontiumDog · Post by **StrontiumDog** » Sat Mar 10, 2007 11:17 am

Hello,

I am in the process of documenting, for my own purposes, the Hypervisor API. I am mainly collecting the info held in the Linux Kernel sources, but eventually I will also be probing the Hypervisor to try and learn more about the operations provided by it, that the kernel doesn't expose.

Instead of keeping this info to myself, I think it may be a useful addition to the wiki. Is that OK? Does anyone mind if I start adding PS3 Hypervisor API documentation to the wiki. Obviously if anyone wants to help in the process that would be great, because there are around 114 Hypervisor calls that have documented names, and by my reckoning another potential 141 that don't. None of this info comes from proprietary sources, it will all come from either published open source code, or from reverse engineering.

Anyway, let me know if this is anything this site is interested in having in its wiki or not.

StrontiumDog

Post by **Oobles** » Sat Mar 10, 2007 4:10 pm

Go for it. :)

David.

ralferoo · Post by **ralferoo** » Sat Mar 10, 2007 7:38 pm

StrontiumDog wrote:Does anyone mind if I start adding PS3 Hypervisor API documentation to the wiki.

Wiki? What wiki? I was looking for a wiki to use for PS3 related things the other day and concluded there weren't any...

But yeah, sounds like a fantastic idea. I was also wondering about the possibility of tricking the hypervisor controlled SPU into giving up some of its code or the hypervisor, e.g. whether we could trick it to writing SPU local store memory to disk, etc.

At the moment, I'm most interested in learning about SBE assembler and seeing what I can get out of mutliple processors, but ultimately I'll want much more than just framebuffer access. It'd be cool if we did manage to gain some kind of access to the RSX though. Even if we only got access to the PS2 emulation hardware we'd be able to achieve some very cool stuff when coupled with SPU for data calculations.

I'm all for documenting as much of everything as we can!

StrontiumDog · Post by **StrontiumDog** » Sat Mar 10, 2007 9:49 pm

ralferoo wrote:Wiki? What wiki?

http://wiki.ps2dev.org/

and more specifically, for the beginning of the hypervisor API docs, see:

http://wiki.ps2dev.org/ps3:hypervisor

comments appreciated.

ralferoo · Post by **ralferoo** » Sat Mar 10, 2007 10:35 pm

Cool. Thank's. I might also start a development section when I have time!

One thing - you mention that all inputs and outputs are 64 bit. Is this just because they could hold ints (32 bits) or PPU EAs (64 bits)? Is there never a chance that they get used as 128 bit vectors?

StrontiumDog · Post by **StrontiumDog** » Sat Mar 10, 2007 11:00 pm

ralferoo wrote:Cool. Thank's. I might also start a development section when I have time!

One thing - you mention that all inputs and outputs are 64 bit. Is this just because they could hold ints (32 bits) or PPU EAs (64 bits)? Is there never a chance that they get used as 128 bit vectors?

The short answer is that they use the 64 bit PowerPC architecture registers. The operations in the kernel explicitly create 64 bit values in the registers.

The longer answer is, in the limited number of functions I have looked into deeply, there have only been values up to 64 bits. It would certainly be possible that there are 128 bit vectors, but they would have to be split across 2 64 bit PowerPC Registers. R3-R10 are only 64 bits wide.

I have been focusing on the GPU Hypervisor calls. In those cases, what appears to be happening, is that the call returns an address that the kernel can map to its own memory space. Having done this, memory on the other side of the Hypervisor can be accessed. If there is a mechanism to pass 128 bit vectors, then I would expect the operation will be something like getting a pointer to memory mapped registers and then writing directly into the GPU, rather than passing everything through the Hypervisor, which I expect would be very slow.

The SC operand is a "context synchronising" operation, which is a BAD thing for performance, so we do not really want that to be the way to do anything that might want to be high performance. Which is what I expect you are looking for here, some way to pass 128 bit vectors to the GPU.

I am sure some of this will become a lot clearer as we document what is known about the Hypervisor, and what is learned through experimentation.

ralferoo · Post by **ralferoo** » Sun Mar 11, 2007 1:30 am

StrontiumDog wrote:The short answer is that they use the 64 bit PowerPC architecture registers. ... R3-R10 are only 64 bits wide.

Ah! I see. I've never done any PPC development before and I haven't looked into that side at all as I'm only playing with SPE assembler at the moment!

StrontiumDog wrote:The SC operand is a "context synchronising" operation, which is a BAD thing for performance, so we do not really want that to be the way to do anything that might want to be high performance.

Isn't the disk IO implemented this way or does the kernel use mailbox channels to communicate directly with the SPU? (Well, I guess if I bothered to read the kernel source tree, I'd discover that for myself!)

StrontiumDog wrote:I am sure some of this will become a lot clearer as we document what is known about the Hypervisor, and what is learned through experimentation.

Indeed. I'm quite excited about this avenue too; I'm sure by testing edge cases in hypervisor calls we'll find some way of opening up the GPU address space (although the call called lv1_gpu_context_iomap sounds very exciting!)

StrontiumDog · Post by **StrontiumDog** » Sun Mar 11, 2007 8:48 am

ralferoo wrote:
StrontiumDog wrote:The short answer is that they use the 64 bit PowerPC architecture registers. ... R3-R10 are only 64 bits wide.
Ah! I see. I've never done any PPC development before and I haven't looked into that side at all as I'm only playing with SPE assembler at the moment!

I have ~10 years PPC systems development, although i've never done any vector programming, and as you say the SPE are a different beast.

ralferoo wrote:
StrontiumDog wrote:The SC operand is a "context synchronising" operation, which is a BAD thing for performance, so we do not really want that to be the way to do anything that might want to be high performance.
Isn't the disk IO implemented this way or does the kernel use mailbox channels to communicate directly with the SPU?

They say its implemented by calls through he Hypervisor. Its probably correct, because Sony don't want people doing "privileged" operations on the drives, and if there was a direct mechanism to talk to the drive hardware through memory mapping, they wouldn't be able to control it. So I am guessing that accessing drives is going to hit performance, but given the relative speed of drives anyway, it may not be measurable.

ralferoo wrote:(Well, I guess if I bothered to read the kernel source tree, I'd discover that for myself!)

It is well worth doing. I recommend checking out the latest kernel tree from GIT. use:

Code: Select all

git-clone git&#58;//git.kernel.org/pub/scm/linux/kernel/git/geoff/ps3-linux.git ps3-linux

ralferoo wrote:
StrontiumDog wrote:I am sure some of this will become a lot clearer as we document what is known about the Hypervisor, and what is learned through experimentation.
Indeed. I'm quite excited about this avenue too; I'm sure by testing edge cases in hypervisor calls we'll find some way of opening up the GPU address space (although the call called lv1_gpu_context_iomap sounds very exciting!)

I don't have a PS3 I can do development on yet, I will get that when they are released in Australia in 2 weeks time. If I started hacking on my US one, my kids would kill me. I believe the hypervisor can be called from userland. If anyone has a PS3 set up, I would be interested if the hypervisor system call can be made successfully from userland or not.

If it can, it will make poking around easier, if not I (or someone) will have to write a kernel driver to do it, which is a lot less convenient.

I already believe there is a lot more that can be done. For example, there is at least a blitter available. The Kernel does this:

Code: Select all

xres = ps3fb_res&#91;i&#93;.xres;
	yres = ps3fb_res&#91;i&#93;.yres;
	...
	offset = xres * yres * BPP * frame;

	fb_ioif = GPU_IOIF + FB_OFF&#40;i&#41; + offset;
	status = lv1_gpu_context_attribute&#40;ps3fb.context_handle,
					   L1GPU_CONTEXT_ATTRIBUTE_FB_BLIT,
					   offset, fb_ioif,
					   L1GPU_FB_BLIT_WAIT_FOR_COMPLETION |
					   &#40;xres << 16&#41; | yres,
					   xres * BPP&#41;;	/* line_length */

If that is not a blit from one area to another I don't know what is. It appears what FB Driver is doing is allocating a 18MB (aligned to 1MB boundary) buffer in main system memory and that is what is used for the 2 frame buffers. It the uses this "blit" operation to transfer the information from the "virtual" frame buffer in main system memory, to the GPU physical frame buffer using the above call.

In that case, it should be possible to bypass the virtual frame buffer and write directly to GPU memory using this operation. I know its not a 3d operation, but it's a start. If this is correct, the speed of this blitter needs to be benchmarked to see if it is useful as a general blit operation. Also, the speed of the hypervisor system call needs to be benchmarked to see what sort of performance hit it really has, as that will depend heavily on the system architecture and I can only comment on the general principal that "context synchronizing" operations hit performance badly.

Strontium Dog

StrontiumDog · Post by **StrontiumDog** » Sun Mar 11, 2007 10:15 pm

OK,

I have documented as many calls as I can for the next few days. If anyone can have a look at what I have done so far. Especially things like layout, format, etc. If there are any constructive criticisms I would love to hear them, before I add more stuff. Changing format now would not be a big problem, but once I add lots more stuff it would be a drag.

StrontiumDog

jimparis · Post by **jimparis** » Mon Mar 12, 2007 2:47 am

StrontiumDog wrote:Especially things like layout, format, etc. If there are any constructive criticisms I would love to hear them, before I add more stuff. Changing format now would not be a big problem, but once I add lots more stuff it would be a drag.

What about using something more standard like Doxygen for the structures and lv1 calls? It would then be easier to use your declarations, and keeping it in under revision control in the PS3 SVN repository would make it easier to manage.

ralferoo · Post by **ralferoo** » Mon Mar 12, 2007 4:21 am

It might be more obvious to spot undocumented vs. missing calls if the number was in hex, although some of the groups seem to start at odd boundaries anyway.

We can always work that out when we're looking at an area of interest anyway.

StrontiumDog · Post by **StrontiumDog** » Mon Mar 12, 2007 7:29 am

jimparis wrote:What about using something more standard like Doxygen for the structures and lv1 calls? It would then be easier to use your declarations, and keeping it in under revision control in the PS3 SVN repository would make it easier to manage.

What I am trying to do here is collect information that is known or learned about the Hypervisor API. All of the code I am using at the moment to do this is in the PS3 Linux Kernel tree. I am not writing a module for accessing the Hypervisor, as the Kernel already has that in it.

If access to the Hypervisor is useful from userland, the Kernel code can easily be adapted for this work. So what I am saying is there is already a code repository for all of this stuff, and I think at this stage replicating that would not be a good idea, as it would mean continual merges as the Kernel tree is updated.

I prefer the wiki approach to Doxygen comments in SVN, because the SVN approach requires management. If someone works out what 1 parameter does, I think it would be easier for them to just drop onto the wiki, and make an annotation. Rather than "checkout svn tree, make changes, submit patches perform merge." Which I think will be a huge impediment to the free exchange of what is learned. I do not want to make this my life's work either, I am encouraging anyone with an interest to grab a copy of the linux tree, pick a function and describe what Linux does with it.

Strontium Dog

StrontiumDog · Post by **StrontiumDog** » Mon Mar 12, 2007 7:39 am

ralferoo wrote:It might be more obvious to spot undocumented vs. missing calls if the number was in hex, although some of the groups seem to start at odd boundaries anyway.

We can always work that out when we're looking at an area of interest anyway.

I have updated the index of the Hypervisor calls.

1. Hex Numbers. I adopted the numbering scheme in the Kernel, so that 1:1 matches were easier to make. If I put them in Hex, that would require continual mental conversion when comparing the Kernel tree with the Wiki documentation.

2. Numbered Boundaries. I have put little table pads between all of the discontinuities in the table to try and highlight where "unrevealed" codes may lay.

3. Undocumented functions. There are 2, the unrevealed codes one I have dealt with above. In the case of a function where we know its name and number of parameters/returns but nothing more, I have made functions that "something more" is known about in bold. The idea being if you see a non-bold green function, you know straight away that the entry is a place holder, and that nothing more than its name and number of parameters is known. I picked this approach over not documenting the call, and therefore leaving it red, because red calls are ones no one has looked into yet. That way we know if a call is not documented because no one has bothered yet, or because no info is known about it from the Kernel.

Does this show the information you were after better?

Strontium Dog

Post by **Warren** » Wed Apr 18, 2007 3:28 pm

Your Hypervisor documentation efforts are greatly appreciated StrontiumDog! When we become accustomed to coding for the SPUs and decide we want to tackle getting rid of linux your efforts will be invaluable.

Keep up the great work!

unsolo · Post by **unsolo** » Wed Apr 18, 2007 6:14 pm

Why not make a /dev/hypervisor module.. (i asked someone in #gentoo-amd64 to take a look at this yesterday i dont know if its feasable tho)
blocking out those things we wouldn't want to execute..

urchin · Post by **urchin** » Tue Apr 24, 2007 6:09 am

I've created a hypervisor module. I've tidied it up and if it's up to scratch then it should be in SVN soon.

It's not totally useful (many things can only be performed from kernel space) but it's fun for poking around.

subdub · Post by **subdub** » Wed May 02, 2007 10:10 am

Hi,

Thanks for contributing to the documentation of the hypervisor.
I was trying silly stuff with it and came accross something.

Calling hypervisor from userland. It's actually possible even if the first 256 function returns -3 (LV1_NO_PRIVILEGE). I was even able to crash the kernel from userland, either by calling function #0 a fair amount of time or while trying incremental calls to the hypervisor. I used this code:

Code: Select all

#include <stdio.h>
#include <limits.h>

int main&#40;&#41;
&#123;
  long ret;
  unsigned long i;

  ret = 0;
  for &#40;i = 0; i < ULONG_MAX; i++&#41;
    &#123;
      printf&#40;"Trying hypervisor function %#08x ...\n", i&#41;;
      __asm__ volatile&#40;"mr %%r11,%0;\n" \
                       "sc 1;\n" \
                       "mr %1,%%r3;\n"
                       &#58; "=r" &#40;ret&#41;
                       &#58; "r" &#40;i&#41;&#41;;
      printf&#40;"\t-> r3 = %ld\n\n", ret&#41;;
    &#125;
  return 0;
&#125;

I'm using Ubuntu 7.04 / Firmware 1.70.

Also, I was wondering how have you build this list ? Only by looking in the Linux kernel sources ? Great job anyway !

Thanks,
Subdub.

laichung · Post by **laichung** » Wed May 02, 2007 5:36 pm

All the stuff about lv1 call is here / include / asm-powerpc / lv1call.h

And you need to read other file to find how to use those call.
Happy hacking~

Anyway, does the lv1 call limited to 256?

urchin · Post by **urchin** » Wed May 02, 2007 6:25 pm

It's not explicitly limited to 256 as the parameter is a u64, though there are no calls above 255 publicly defined as yet.

boxbuilder · Post by **boxbuilder** » Mon Dec 17, 2007 11:01 am

Forgive me if this is n00bish, but I believe the hypervisor runs on the illusive 7th spe. But I remember reading that the spe's can't access memory. Doesn't that mean that they can't access the devices without asking the ppe? If so, then the hypervisor must either:
1. Be two pieces, a kernel which runs on the main processor (over Linux), and the part on the 7th spe.
2. The hypervisor is not actually relaying messages between Linux, and devices, but rather converting hypervisor calls into a form which the devices understand, then piping them back to the ppe (Linux) to then be sent to the devices (Hackers Christmas).
Either situation would mean the hypervisor's security is more lax than I had thought. Anyway, thought I'd just throw that out.

BTW: excellent work mapping function calls, I wish I could do more to help.

mc · Post by mc » Wed Dec 19, 2007 8:09 am

Hypervisor calls run on the PPE. It's just like a regular syscall, it only elevates the CPU privilege level and jumps to a set address.

Compound · Post by **Compound** » Wed Dec 19, 2007 8:33 am

how are people determining functions from the hypervisor? is it possible to dump it in some way? or is it literally guess work as to what different memory addresses do?

jimparis · Post by **jimparis** » Wed Dec 19, 2007 8:52 am

The Linux kernel source has a ton of information on how the hypervisor works. That's the source of basically everything we know, with a little trial-and-error too. It's not really hidden, there's a lot of info available in the kernel.

ralferoo · Post by **ralferoo** » Wed Dec 19, 2007 8:53 am

Compound wrote:how are people determining functions from the hypervisor? is it possible to dump it in some way? or is it literally guess work as to what different memory addresses do?

Calling the functions and seeing what happens for the most part. We also have examples of some of them being used in the publically available linux drivers Sony release.

forums.ps2dev.org

Hypervisor API

Hypervisor API

Re: Hypervisor API

Re: Hypervisor API