The paper addresses a problem on speaker identification system using Hidden Markov Model (HMM) caused by the training data selected far from its distribution centre. Four scenarios for unguided data have been conducted to partition the data into training data and testing data. The data were recorded from ten speakers. Each speaker uttered 80 times with the same physical (health) condition. The data collected then pre-processed using Mel-Frequence Cepstrum Coefficients (MFCC) feature extraction method. The four scenarios are based on the distance of each speech to its distribution centre, which is computed using Self Organizing Map (SOM) algorithm. HMM with many number of states (from 3 up to 7) showed that speaker with multi-modals distribution will drop the system accuracy up to 9% from its highest recognition rate, i.e. 100%.