For remote play over the Internet, the PSP opens five HTTP connections to the PS3 on port 9293. I've posted dumps of a short session here: http://ps3.jim.sh/remoteplay/
The five connections are session, ctrl, video, audio, and pad. For example, the first is:
The audio, video, and pad data appears encrypted. If there is more interest in figuring out these protocols (so that we can create our own remote play clients and servers), I think the next step would be to reverse the remote play functions in the PSP firmware to figure out what's up with the authentication, encryption, and data format. Anyone bored enough to give it a look? :)
For remote play auto-start, the PS3 wakes whenever it sees an ARP request for the IP address that it was using when it turned off. The PSN attempts to connect to it via TCP and that triggers the ARP from your firewall or whatever.
The PS3 still sends a DHCP release before powering off, so it's possible for some other device to grab the address instead. In that case the PS3 would wake on seeing an ARP to that other device, but then it just fetches a new DHCP lease, and it turns back off automatically when the remote play connection doesn't start for a minute or two.
As for turning off the PS3, that seems to be controlled by the PS3 itself, because it just displays a "Turn off the system?" prompt over the normal remote play graphical connection.
Still waiting for some bored PSP hacker to actually reverse the PSN sign-in and the remote play protocol... :P
Sorry to bump this, but I'm trying to spark some interest in hacking the remote play as well.
It would be great to create a java or flash app that can handle remote play the same as the psp can, only on a computer or other handhelds. I recently did a dump of the HTTP transactions as well, and have the same findings as jimparis does above.
I'm in no way very technical at all about any of this, but what I can tell is that the ps3 is sending video and audio separately (obviously). The audio's format is AT3, which correct me if I'm wrong, but only the PSP has the capability/drivers to playback.