MSc/MEng Data Mining and Machine Learning (2024)
Lab 3 – Speech Recognition using HTK
Introduction
The purpose of this laboratory is to familiarise you with automatic speech recognition. You will
use the Hidden Markov Model Toolkit (HTK) to build a connected digit recognition system which
takes an acoustic speech signal as input, performs training of the HMM for each digit and evaluate
the performance of the system on a provided dataset. The entire HTK consists of several tools
(exe-files), each performing a specific operation, e.g., feature extraction, HMM training, etc. Each
tool is executed in the Command Prompt window by typing its name together with passing all the
required input parameters. The exe-files of the individual HTK tools are included in the
LabASR.zip file to be downloaded from Canvas. The zip-file also includes the manual for the
HTK software – the manual is big but you are going to need it only occasionally and only as a
reference in order to find out the meaning of (some of) the input/output parameters which are
passed when using a specific HTK tool.
Getting started
Download the zip-file LabASR.zip from Canvas to your drive. Open the zip-file and copy the
entire directory structure to your drive. Run the Command Prompt Window by going to the
Windows Start menu and typing ‘cmd’ (no quotes). Use the ‘cd’ command to set your directory
to the place you copied the unzipped file. You are now set to start running some HTK tools.
Dataset
The dataset used in the laboratory contains recording of spoken digit sequences, where a digit is
one of the following: one, two, three, four, five, six, seven, eight, nine, zero, oh. The recordings
are stored in .wav format. The first letter in the filename of each .wav file indicates whether the
recording is from a male (M) or a female (F) speaker. The data is split into training part (folder
TRAIN) and testing part (folder TEST). In each (train/test) part, there is a set of clean (noise-free)
recordings (folder CLEAN1) and a set of recordings corrupted by an additive noise (i.e., noise
signal added to the clean signal) at the signal-to-noise ratio (SNR) of 15 dB and 10 dB (folder
N1_SNR15, N1_SNR10, respectively). The additive noise illustrates the effect of a background
ambient noise in practice.
Viewing the signal
In this initial exercise you will practice the use of the HList tool. This tool allows you to view
wav-files or files containing features extracted from wav-files (the feature extraction can be
performed using the HCopy tool which will be the subject of the next section). Typing the below
gives the values of samples in the wav-file and these are stored in the file logHList_wav:
HTK3.2bin\\HList -h -C config/config_HList_wav
dataAurora2/wavLabDMML/TRAIN/CLEAN1/FAC_13A.wav > logHList_wav
You can examine the file containing the MFCC features (after you have created them as described
in the next section) by typing:
HTK3.2bin\\HList -h -C config/config_HList_mfcc
dataAurora2/specLabDMML/TRAIN/CLEAN1/FAC_13A.mfcc > logHList_mfcc
1
2
Feature extraction
The HCopy tool enables to extract a sequence of feature vectors from a given wav-file. It is
capable of extracting several different types of features, e.g., logarithm filter-bank energies,
MFCCs, etc. By typing the below, you can convert the MAE_12A.wav file into a file with the same
name but extension .mfcc which contains the MFCC features (note that the feature file will be
located in a different directory):
HTK3.2bin\\HCopy -C config/config_HCopy_MFCC_E
dataAurora2/wavLabDMML/TRAIN/CLEAN1/FAC_13A.wav
dataAurora2/specLabDMML/TRAIN/CLEAN1/FAC_13A.mfcc
The HCopy tool can be used to extract features for a set of files listed in a given text-file. This can
be performed by using the HCopy as below, where the
listTrainHCopy_LabDMML_CLEAN1.scp is a text-file containing the list of files (with a full
path) to be processed. This file is located in the list directory. Open and view this file and you
can see that each line contains name of two files (with a full path) – the first is the file to be used
as the input and the second is the file to be used as the output. You will need to modify the path
here to be the path where your data are located. After you have done the path modifications,
type:
HTK3.2bin\\HCopy -C config/config_HCopy_MFCC_E –S
list/listTrainHCopy_LabDMML_CLEAN1.scp
The option -S is used to specify a script file name (listTrainHCopy_LabDMML_CLEAN1.scp)
that contains the list of files to be converted.
Building the digit recognition system – parameter set-up
In the previous section, we have converted a set of wav-files into files containing the features.
Now, you start to build your digit recognition system. You will need the following:
- Vocabulary list – file wordList_noSP located under the lib directory – this contains the
list of words the recogniser is going to be able to recognise. A model will be built for each
vocabulary word.
- Dictionary (or pronunciation model) – file wordDict located under the lib directory –
this defines the mapping of words to acoustic units, i.e., how model of each vocabulary
word is built using a single (or a sequence of concatenated) HMMs. Since we are using in
this example HMMs of whole words, the dictionary contains a repetition of each
vocabulary word. Note that this would be different in a case of building HMMs of each
phoneme.
- Language model (or grammar) – file wordNetwork located under the lib directory – this
defines (in a specific format) the set of possible sentences that can be recognised, as well
as their relative prior probabilities. If needed, it can be written by hand or more
conveniently using the tool HParse.
- Features extracted for the training / testing data – are located under dataAurora2
directory.
- Label files for the training / testing data – file label_LabDMML_noSP.mlf located under
the label directory is to be used in the first instance. You can open this text file and see
that it contains the labels (i.e., transcription of what have been spoken in terms of the
digits) for all the training data.
- Prototype HMM – file proto_s1d13_st8m1_LabDMML_MFCC_E located under the lib
directory. You can open this text file and see that it contains a definition of the type of
HMM to be used – it defines the dimension of the features, the number of states in the
HMM, initial values for means, variances and weights for each state (these values are
indicative only – they inform about the structure of the HMM), and the transition
probability matrix which determines the possible transitions between states (the
transitions assigned to zero will not be possible).
- Configuration file for the individual tools – each tool may have different configuration file
(containing the parameters of the processing to be performed).
Building the digit recognition system – training the HMMs
1. Create the directory hmm0 under hmmsTrained. The initial parameters of HMMs are going to
be estimated using the tool HCompV. By executing the following, the initially trained HMM
parameters will be located in the file hmmdef (and vFloors) under the directory
hmmsTrained/hmm0. Note that you will need to modify the path in the
listTrainFullPath_LabDMML_CLEAN1.scp file.
HTK3.2bin\\HCompV -C config/config_train_MFCC_E -o hmmdef -f 0.01 -m -S
list/listTrainFullPath_LabDMML_CLEAN1.scp -M hmmsTrained/hmm0
lib/proto_s1d13_st8m1_LabDMML_MFCC_E
2. Now you will create 2 files (could be done manually but you are provided exe-files which do
the work automatically for you).
Type the below – it will create file with name models containing the HMM definition of all the
11 digits and the silence model. The models file could be created manually by simply copying
the content of hmmdef several times (for each vocabulary unit) and replacing the name
according to the vocabulary.
HTK3.2bin\\models_1mixsil hmmsTrained/hmm0/hmmdef hmmsTrained/hmm0/models
Type the below, which creates the so-called macro-file having basically the same content as the
file vFloors but slightly modified structure. The value 13 indicates the dimension and MFCC_E
the type of features – you will need to modify these when using different features/dimension.
HTK3.2bin\\macro 13 MFCC_E hmmsTrained/hmm0/vFloors hmmsTrained/hmm0/macros
3. The next step is to run several iterations of the Baum-Welch training procedure. This can be
done using the tool HERest. Among the input parameters for this tool is the input directory
containing the current HMM parameters (which is now hmmsTrained/hmm0) and the output
directory containing the new re-estimated HMM parameters (which is now
hmmsTrained/hmm1). Thus, you need to create the new directory hmm1 and then run:
HTK3.2bin\\HERest -C config/config_train_MFCC_E -I
label/label_LabDMML_noSP.mlf -t 250.0 150.0 1000.0 -S
list/listTrainFullPath_LabDMML_CLEAN1.scp -H hmmsTrained/hmm0/macros -H
hmmsTrained/hmm0/models -M hmmsTrained/hmm1 lib/wordList_noSP
3
4
Altogether, perform three iterations of the HERest. Before each iteration, make a new
directory (hmm1, hmm2, and hmm3) where the newly trained HMMs are going to be stored. At
each iteration, you should not forget to change the corresponding input and output directory
names in the above HERest command – use the output directory from the current iteration
as the input directory in the next iteration.
4. Now create two new directories hmm4 and hmm5. Then copy the content of the directory hmm3
into the hmm4 directory.
5. Create the model for a short-pause sp by performing the two commands as below:
HTK3.2bin\\spmodel_gen hmmsTrained/hmm3/models hmmsTrained/hmm4/models
HTK3.2bin\\HHEd -H hmmsTrained/hmm4/macros -H hmmsTrained/hmm4/models -M
hmmsTrained/hmm5 lib/tieSILandSP_LabDMML.hed lib/wordList_withSP
6. Perform another three iterations of the HERest (with sp this time) – before each iteration,
make a new directory where the newly trained HMMs will be stored.
HTK3.2bin\\HERest -C config/config_train_MFCC_E -I
label/label_LabDMML_withSP.mlf -t 250.0 150.0 1000.0 -S
list/listTrainFullPath_LabDMML_CLEAN1.scp -H hmmsTrained/hmm5/macros -H
hmmsTrained/hmm5/models -M hmmsTrained/hmm6 lib/wordList_withSP
Training finished! – you have now obtained trained models of digits in the folder hmm8, each
modelled by 10 state HMM with a single Gaussian PDF with diagonal covariance matrices. Let’s
go to do testing (recognition).
Building the digit recognition system – recognition
1. The tool HVite is to be used for testing of the recognition system. This performs the Viterbi
decoding and gives the sequence of models which are most likely to produce the given
unknown utterance. Among the input parameters to the HVite tool are the trained HMMs
and the list of testing utterances (from the testing data directory). First, you need to extract
features from the testing wav-files using the HCopy tool as described at the beginning of the
lab (when you created features for the training utterances). Then, you can run the Viterbi
decoding using:
HTK3.2bin\\HVite -H hmmsTrained/hmm8/macros -H hmmsTrained/hmm8/models -S
list/listTestFullPath_LabDMML_CLEAN1.scp -C config/config_test_MFCC_E -w
lib/wordNetwork -i result/result.mlf -p 0 -s 0.0 lib/wordDict
lib/wordList_withSP
2. Tool HResults is to be used for analysing the results of the HVite and providing the final
recognition accuracy of the system. The -e option will cause that sil and sp models will be
omitted from counts for the overall recognition performance.
HTK3.2bin\\HResults -e "???" sil -e "???" sp -I label/labelTest_LabDMML.mlf
lib/wordList_withSP result/result.mlf >> result/recognitionFinalResult.res
HResults provides results on sentence (SENT) level and Word (WORD) level – these indicate
how well the entire sentences or words were recognised. In the results, the ‘H’, ‘D’, ‘S’, ‘I’, and
‘N’ denote the number of hits, deletions, substitutions, insertions and total number of
words/sentences, respectively. If there is a large difference between the number of deletions
(‘D’) and insertions (‘I’), this indicates that the recognition system is not well balanced. To
improve this balance, there is a parameter referred to as -p flag in the HVite command – this
is word insertion penalty (WIP), a penalty on transiting from one model to other model. The
WIP can be used to balance the number of deletions and insertions. If needed, change the
value from 0 to some other positive or negative value (e.g., in steps of 10).
Perl scripts
In the Lab directory in Canvas you can find the file perlScripts_LabASR.zip – this contains
several Perl scripts which in a neat way incorporate all the above commands. The
ASR_LabDMML_MFCC_E.pl script does all the above (feature extraction, training and testing)
and the ASR_LabDMML_onlyTest_MFCC_E.pl performs testing only (assuming the training has
been performed). You will need to change paths inside the Perl scripts. Then you can run the
first Perl script by typing perl ASR_LabDMML_MFCC_E.pl in the Command Prompt window –
it should perform the feature extraction, the entire training and testing. For a reference, an
introduction to Perl is located in the Lab directory in Canvas.
Lab Report Tasks:
For all the tasks below, if needed, modify the –p flag (in HVite) to achieve reasonable balance of
the number of deletions and insertions.
1. Explore the effect of delta and delta-delta features. Using the provided Perl script, modify the
recognition system developed above such that it uses not only the static MFCC features (i.e.,
MFCC_E) but also the delta and delta-delta features (i.e., MFCC_E_D_A). You will need to
perform modifications at several places. In the HCopy config modify the TARGETKIND to
MFCC_E_D_A and set the DELTAWINDOW=3 and ACCWINDOW=2. The MFCC_E_D_A features
will not be 13 dimensional (as were the MFCC_E features) but 39 dimensional – so, you will
need to make modifications at places where the feature dimension information appears. You
will also need to modify the TARGETKIND in config_train and config_test and will need
to use the proto_s1d39_st8m1_LabDMML_MFCC_E_D_A. Train the system using the clean
training data. Perform experimental evaluations on clean test data. Report and discuss your
results. [20 marks]
2. Investigate the effect of using Gaussian mixture state PDF modelling. Modify the provided Perl
scripts (and configuration files) to develop a recognition system that uses the MFCC_E_D_A
features and employs 3 Gaussian mixture components per state. Train the system using the
clean training data. Perform experimental evaluations on clean testing data and compare the
results with those obtained using a single Gaussian per state as obtained from Task 1. Report
and discuss your results. [20 marks]
3. Explore the effect of noise. [40 marks]
a. Perform experimental evaluations of the recognition system developed under Task 2
separately on each provided noisy test data (N1_SNR10, N1_SNR15).
b. Then develop a new system – this should be as the system in Task 2 (i.e., using
MFCC_E_D_A features and 3 Gaussian mixture components) but trained using a
combined set of all the clean and noisy training data together – to do this, you will
need to create a new list file containing all the filenames of all the clean and noisy
5
training data. Perform evaluations of this system separately on clean and on each
noisy test data (N1_SNR10, N1_SNR15).
Report, compare and discuss your results.
4. Consider that you have available the trained system from Task 3b (in a case you did not do this
task you may consider the system from Task 2). Suggest how you could (in a similar concept
as used in Task 3b) try to improve the performance of the system for ‘female’ speakers.
Develop the modified system and perform suitable experiments on noisy test data N1_SNR10.
Report, compare and discuss your results. [20 marks]
Lab Report Submission
You should report concisely on each of the above tasks. Describe clearly what changes you
needed to make to perform the task and discuss the obtained results. Your report from this lab
is expected to be no longer than 7 pages and the submission is through Canvas. Standard penalty
of 5% per day applies for late submissions.
請加QQ：99515681 郵箱：99515681@qq.com WX：codinghelp
