Porting to PSP of Speech recognition Pocket Sphinx

Discuss the development of new homebrew software, tools and libraries.

Moderators: cheriff, TyRaNiD

Post Reply
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Porting to PSP of Speech recognition Pocket Sphinx

Post by mypspdev »

Hi! Happy 2008 to all!
Here is source code and documentation: http://www.speech.cs.cmu.edu/pocketsphinx/.

I've completed the porting to PSP, it picks up audio input and propose some sentences according to best fit of utterance.
It's fast and nice.
I'm experiencing some crashes at the following procedure where is something that I do not understand.

I've a pointer to unsigned char:

Code: Select all

unsigned char *pid_cw0 ...
Then it's firstly used:

Code: Select all

pid_cw0 = (s->OPDF_8B[j][s->f[j][0].codeword]);
But when used here:

Code: Select all

tmp1 = pid_cw0[n] + w0;
the program on PSP crashes. Something is wrong with [n] position reading.

I've substituted pid_cw0[n] with forced values: the program flows over to end.
Something is wrong reading pid_cw0[n]. and I'am not able to understand what's the meaning of reading this pointed value at position [n].

Is there missing any alloc of array? or something similar? Any initialization of [n] positions is required?
The original source code is fully and succesfully compiled both for PSP and for PC under Windows, but when running on PSP it crashes at that instruction.

Has anybody any suggestion on investigating or by-pass or complete this original instruction set?
Thanks a lot, here is source code of the procedure, then the structure of s2_Semi_mgau_t.

Code: Select all

static int32
get_scores4_8b(s2_semi_mgau_t * s)
{
    register int32 j, n, k;
    int32 tmp1, tmp2;
    unsigned char *pid_cw0, *pid_cw1, *pid_cw2, *pid_cw3;
    int32 w0, w1, w2, w3;       /* weights */

    memset(senone_scores, 0, s->CdWdPDFMod * sizeof(*senone_scores));
    for &#40;j = 0; j < s->n_feat; j++&#41; &#123;

        /* ptrs to senone prob ids */
        pid_cw0 = &#40;s->OPDF_8B&#91;j&#93;&#91;s->f&#91;j&#93;&#91;0&#93;.codeword&#93;&#41;;
        pid_cw1 = &#40;s->OPDF_8B&#91;j&#93;&#91;s->f&#91;j&#93;&#91;1&#93;.codeword&#93;&#41;;
        pid_cw2 = &#40;s->OPDF_8B&#91;j&#93;&#91;s->f&#91;j&#93;&#91;2&#93;.codeword&#93;&#41;;
        pid_cw3 = &#40;s->OPDF_8B&#91;j&#93;&#91;s->f&#91;j&#93;&#91;3&#93;.codeword&#93;&#41;;

        w0 = s->f&#91;j&#93;&#91;0&#93;.val.score;
        w1 = s->f&#91;j&#93;&#91;1&#93;.val.score;
        w2 = s->f&#91;j&#93;&#91;2&#93;.val.score;
        w3 = s->f&#91;j&#93;&#91;3&#93;.val.score;

        /* Floor w0..w3 to 256<<10 - 162k */
        if &#40;w3 < -99000&#41;
            w3 = -99000;
        if &#40;w2 < -99000&#41;
            w2 = -99000;
        if &#40;w1 < -99000&#41;
            w1 = -99000;
        if &#40;w0 < -99000&#41;
            w0 = -99000;        /* Condition should never be TRUE */

        /* Quantize */
        w3 = &#40;511 - w3&#41; >> 10;
        w2 = &#40;511 - w2&#41; >> 10;
        w1 = &#40;511 - w1&#41; >> 10;
        w0 = &#40;511 - w0&#41; >> 10;

        for &#40;k = 0; k < n_senone_active; k++&#41; &#123;
	    n = senone_active&#91;k&#93;;

//#### HERE IS CRASHING!
            tmp1 = pid_cw0&#91;n&#93; + w0;
//#### HERE IS CRASHING!

            tmp2 = pid_cw1&#91;n&#93; + w1;
            tmp1 = LOG_ADD&#40;tmp1, tmp2&#41;;
            tmp2 = pid_cw2&#91;n&#93; + w2;
            tmp1 = LOG_ADD&#40;tmp1, tmp2&#41;;
            tmp2 = pid_cw3&#91;n&#93; + w3;
            tmp1 = LOG_ADD&#40;tmp1, tmp2&#41;;

            senone_scores&#91;n&#93; -= tmp1 << 10;
        &#125;
    &#125;
    return 0;
&#125;

Code: Select all

typedef struct &#123;
    union &#123;
        int32	score;
        int32	dist;	/* distance to next closest vector */
    &#125; val;
    int32 codeword;		/* codeword &#40;vector index&#41; */
&#125; vqFeature_t;
typedef vqFeature_t *vqFrame_t;

typedef struct s2_semi_mgau_s s2_semi_mgau_t;
struct s2_semi_mgau_s &#123;
    int32   **dets;	/* det values foreach feature */
    mean_t  **means;	/* mean vectors foreach feature */
    var_t   **vars;	/* var vectors foreach feature */

    unsigned char ***OPDF_8B; /* mixture weights */

    int32 n_feat;	/* Number of feature streams */
    int32 *veclen;	/* Length of feature streams */
    int32 n_density;	/* Number of mixtures per codebook */

    int32 topN;		/* Number of top densities to compute &#40;<S2_MAX_TOPN&#41; */
    int32 CdWdPDFMod;	/* Legacy thing, actually means number of mixw */

    kd_tree_t **kdtrees;
    uint32 n_kdtrees;
    uint32 kd_maxdepth;
    int32 kd_maxbbi;

    int32 num_frames;
    int32 ds_ratio;

    int32 use_mmap; /**< Are mixture weights mmap&#40;&#41;ed? */

    /* Top-N scores and codewords from current, last frame. */
    vqFeature_t **f, **lastf;
    int32 *score_tmp;
&#125;;
Art
Posts: 642
Joined: Wed Nov 09, 2005 8:01 am

Post by Art »

If I read correctly you've declared pid_cw0 as a single character variable,
but then you treat it as an array with the line you say is crashing it.
As I see, unless the value of n is zero, the line you say is crashing it
would change the value of some other byte in memory.
If not actually, then potentially.
jimparis
Posts: 1145
Joined: Fri Jun 10, 2005 4:21 am
Location: Boston

Post by jimparis »

There could be lots of reasons, that code is complicated.
For example, if OPDF_8B is full of junk data than the error makes sense. Printing out the values of all relevant variables (pid_cw0, OPDF_8B, n, etc) might help pinpoint the error.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

Thanks for replies and helps.
I've checked that OPDF_8B is ***, a 3d array 4x256x5220.
pid_cwX is pointing to the array 4x256 and should have 5220 elements.
The crash happens also with n=170, so <5220.
The memory space for OPDF_8B is allocated correctely in a previous function without errors: I am excluding memory space saturation on PSP, but I'm still investigating.
Now I'll dump the full 3d array values of OPDF_8B.
For who interested I'll explain how to port Pocket Sphinx to PSP: 4 libraries must be compiled (from SphinxBase), then the running programs of Pocketsphinx in the main.c.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

{SOLVED}Porting to PSP of Speech recognition Pocket Sphinx

Post by mypspdev »

mistake... sorry
Last edited by mypspdev on Thu Jan 03, 2008 7:31 pm, edited 1 time in total.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Re: {SOLVED}Porting to PSP of Speech recognition Pocket Sphi

Post by mypspdev »

[SOLVED]
Now PSP has also Speech Recognition by both audio input: GoCam or Socom Microphone!!

I've eliminated the pid_cwX pointers and I've put in place of them directly the array OPDF_8B[j][k] element required.

It works nice!
A new Module is added to MyPSP Robotics: MyTalk, Voice command recognition!
Art
Posts: 642
Joined: Wed Nov 09, 2005 8:01 am

Post by Art »

Have you posted a demo anywhere we can try?

Did you user the Genuine Socom headset or DIY mic?

I have never found a DIY mic as clear, or loud as an original PSP one.
I wonder if the difference will cause any issues.
If not actually, then potentially.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

sorry... mistake, I apologize for double posts.
Last edited by mypspdev on Sun Jan 06, 2008 1:34 am, edited 1 time in total.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

mypspdev wrote:
Art wrote:Have you posted a demo anywhere we can try?

Did you user the Genuine Socom headset or DIY mic?

I have never found a DIY mic as clear, or loud as an original PSP one.
I wonder if the difference will cause any issues.
here is MyPSP v5.0.0 with the last module for voice recognition (english) porting Pocketsphinx :
http://rapidshare.com/files/81029828/My ... 0.rar.html
Sorry, some bugs are rised now on old modules because RAM is heavely occupied.
I'm changing dimensions of thread spaces and buffers and MyPSPv5.1.0 is going to be delivered fully working over all modules again.

MyPSPv5.0.0 is only for Slim, doesn't work on Fat (RAM shortage).

Both microphones of coyrse for MyTAlk: Socom headset and GoCam/Talkman USB mic.

Now MyPSP has integrated:
- MyVoice: Text to speech or Text to Wave (Flite)
- MyTalk: speech recognition (Pocketsphinx)
- MyEye: OpenCV, SIFT, GOCR
- MyVVoIP: Video and Voice over IP
- MyAI: OpenMind Knowledge Data base

Hi Art!
J.F.
Posts: 2906
Joined: Sun Feb 22, 2004 11:41 am

Post by J.F. »

MyTalk with a SOCOM input blows up at the main utterance loop. The MyAudio recorder works fine, so it's probably not the SOCOM input routines.

This is on a Slim with 3.71 M33-4.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

J.F. wrote:MyTalk with a SOCOM input blows up at the main utterance loop. The MyAudio recorder works fine, so it's probably not the SOCOM input routines.

This is on a Slim with 3.71 M33-4.
you are right I'm sorry! I'll prepare the socom update soon.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

J.F. wrote:MyTalk with a SOCOM input blows up at the main utterance loop. The MyAudio recorder works fine, so it's probably not the SOCOM input routines.

This is on a Slim with 3.71 M33-4.
@J.F.:
tomorrow I'll correct MyTalk for Socom, I apologize for GoCam activation even if you choose Socom Audio Input.
At present, may be it's better to use MyPSPv5.1.0
http://rapidshare.com/files/81479325/My ... 0.rar.html
or here
http://www.psp-ita.com/?module=news&id= ... ew_reply=1
RAM shortage needs some reduced spaces for threads and buffers, v5.1.0 now should work well also with MyVVoIP module for Video and Voice TRX from PSP-Server to PSP-Client even using Socom or GoCam audio input.
Tomorrow I'll deliver also a MyVVoIP module with full duplex Video and Voice trx-rx between two PSPs.
(in the package some prxs by Sony are not included, please get them)

J.F., do you have the remote-controller for slim? is it the same than RC for Fat, of course with the little plug?

Thanks for interest.
J.F.
Posts: 2906
Joined: Sun Feb 22, 2004 11:41 am

Post by J.F. »

mypspdev wrote:
J.F. wrote:MyTalk with a SOCOM input blows up at the main utterance loop. The MyAudio recorder works fine, so it's probably not the SOCOM input routines.

This is on a Slim with 3.71 M33-4.
@J.F.:
tomorrow I'll correct MyTalk for Socom, I apologize for GoCam activation even if you choose Socom Audio Input.
At present, may be it's better to use MyPSPv5.1.0
http://rapidshare.com/files/81479325/My ... 0.rar.html
or here
http://www.psp-ita.com/?module=news&id= ... ew_reply=1
RAM shortage needs some reduced spaces for threads and buffers, v5.1.0 now should work well also with MyVVoIP module for Video and Voice TRX from PSP-Server to PSP-Client even using Socom or GoCam audio input.
Tomorrow I'll deliver also a MyVVoIP module with full duplex Video and Voice trx-rx between two PSPs.
(in the package some prxs by Sony are not included, please get them)

J.F., do you have the remote-controller for slim? is it the same than RC for Fat, of course with the little plug?

Thanks for interest.
Of course I have the remote - you can't use the SOCOM headset mike without it. :D The slim remote is a smidgen different from the phat - it has a different end connector (since the connector on the slim is different because of the video out), and the remote itself is a smidgen different. Like I said, your voice recording routine worked fine, so the SOCOM is working, and the code for that part is working fine.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

J.F wrote:Like I said, your voice recording routine worked fine, so the SOCOM is working, and the code for that part is working fine.
For using Socom or GoCam, it's simple:

Code: Select all

int pspAudioInput&#40;s32 length, s32 sample_rate, void *buf&#41; &#123;
	if &#40;!audio_input_ready&#41;
		return -1;
		
	if &#40;MyMic==PSP_GOCAM_MIC&#41;
	&#123;	
		sceUsbCamReadMic&#40;buf, length&#41;;
	&#125;
	if &#40;MyMic==PSP_SOCOM_MIC&#41;
	&#123;
            sceAudioInput&#40;length, sample_rate, buf&#41;;
	&#125;
	return 0;
&#125;

Code: Select all

int pspAudioInputBlocking&#40;s32 length, s32 sample_rate, void *buf&#41; &#123;
	if &#40;!audio_input_ready&#41;
		return -1;
	if &#40;MyMic==PSP_GOCAM_MIC&#41;
	&#123;	
		sceUsbCamReadMicBlocking&#40;buf, length&#41;;
	&#125;
	if &#40;MyMic==PSP_SOCOM_MIC&#41;
	&#123;
	      sceAudioInputBlocking&#40;length, sample_rate, buf&#41;;
	&#125;
	return 0;
&#125;
I've not tested well Socom on Slim because I have remote control only for fat... now I'm going to separate MyTalk and the reduced stand alone module should work also on fat.
Tomorrow I'll complete the input from Socom, too.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

sorry mistake...
Last edited by mypspdev on Sun Jan 06, 2008 9:36 am, edited 1 time in total.
mypspdev
Posts: 178
Joined: Wed Jul 11, 2007 10:30 pm

Post by mypspdev »

[quote="mypspdev"]@J.F.:
for Socom Audio Input to MyTalk, please would you try on Slim to use this eboot.pbp in place of previous one on the same folder MyPSPv5.1.0 ?
I've added the if (MyMic==PSP_SOCOM_MIC)...
let me know, thanks

EDIT:
Here new eboot for Socom Audio Input for MyTalk:
http://rapidshare.com/files/81579280/EBOOT.rar.html
Art
Posts: 642
Joined: Wed Nov 09, 2005 8:01 am

Post by Art »

I was wondering the difference for a DIY mic compared to a genuine one
(both are Socom mics). I guess I'll try, but I can't compare to a genuine
Sony Socom mic anymore.
If not actually, then potentially.
User avatar
jean
Posts: 489
Joined: Sat Jan 05, 2008 2:44 am

Post by jean »

Just to reply @Art... i've sectioned PSP's remote :) and even if it's all thight and integrate SMD technology, i can guess that the fourth ring of the audio connector (that SOCOM mic uses) doesn't go directly to the "serial" connector, but is preamplified (we could do as well in a DIY mic with a simple single-transistor circuit powered by the SIO +/- pins). Moreover, this is strictly necessary if your microphonic capsule is a capacitive (and not resistive) one, as those you can find in mobile GSM phones'. By now, i'm using a nokia mic soldered directly into the PSP remote, and it's working fine, but if you want to be independant from remote controller, then you should try different mics and (if you want it loud) some simple preamplifiers. PM if you want some schematics.

jean
Art
Posts: 642
Joined: Wed Nov 09, 2005 8:01 am

Post by Art »

Sorry, I missed that one at the time you posted it jean.

Even when I used a DIY headset with 3.5mm 4 conductor plug via the headphone remote,
it still
sounded like crap, and furthermore, I destroyed my first Socom headset
to try to get to the bottom of it, and the mic works just as well connected straight to the headphone port on the PSP.

The mic itself is just a little electret mic with one cap surface mounted on a tiny PCB.
It didn't look special.

Now I'm on my second copy of Socom Fire Team Bravo :rolleyes: :D

Looks like the answer to get the Talkman mic going with Audio Mechanica might be a few posts above,
if the Talkman mic works as easy as the GoCam one.
I don't know if anyone can use it yet. Is it any different?
If not actually, then potentially.
Post Reply