Theodoros Emmanouilidis

Notes & Thoughts

Running LDA Algorithm With Mahout

September15

The following article explains the usage of Apache Mahout’ s implementation of the Latent Dirichlet Allocation learning algorithm. The idea behind the example is to extract given number of topics from a collection of text files located in a given directory. For simplicity we will use Mahout’ s ability to run locally, but there is no real difference  in the procedure if it’ s applied to a Mahout build configured to use a Hadoop cluster.

The first step is to download and install Mahout. Download from here the latest stable release and extract the archive to your preferred working directory. The only thing that needs to be configured for a minimal run is the JAVA_HOME environment variable.

For ubuntu just type:

1
export JAVA_HOME=/usr/lib/jvm/java-6-sun

to the console or change the declared path with the appropriate path for your release / set-up. Since the export command works only for the current session, you can edit the {mahout directory}/bin/mahout file and paste the export command there.

The second step is to create the directory that will contain the text files we want to extract topics from and populate it.

1) Converting text documents to SequenceFile format

Mahout has a built in utility to convert text files contained to a given directory to SequenceFile format:

1
seqdirectory

The options of the command are as follows:

  --input (-i) input                             Path to job input directory.
  --output (-o) output                           The directory pathname for
                                                 output.
  --overwrite (-ow)                              If present, overwrite the
                                                 output directory before
                                                 running job
  --chunkSize (-chunk) chunkSize                 The chunkSize in MegaBytes.
                                                 Defaults to 64
  --fileFilterClass (-filter) fileFilterClass    The name of the class to use
                                                 for file parsing. Default:
                                                 org.apache.mahout.text.PrefixAd
                                                 ditionFilter
  --keyPrefix (-prefix) keyPrefix                The prefix to be prepended to
                                                 the key
  --charset (-c) charset                         The name of the character
                                                 encoding of the input files.
                                                 Default to UTF-8
  --help (-h)                                    Print out help
  --tempDir tempDir                              Intermediate output directory
  --startPhase startPhase                        First phase to run
  --endPhase endPhase                            Last phase to run

but the absolutely minimum to run the utility are (every command is executed from {mahout directory}/bin directory):

1
./mahout seqdirectory --input {full path to the folder containing text files} --output {full path to folder that we want to save sequence files} -c UTF-8

or for me

1
./mahout seqdirectory --input /home/tgi/Desktop/ruby --output /home/tgi/Desktop/1 -c UTF-8

the document id generated is {prefix}{relative path from parent}/document.txt

2) Creating vectors from SequenceFile

Again, Mahout has a built in utility for this:

1
seq2sparse

All options of the command are:

  --minSupport (-s) minSupport        (Optional) Minimum Support. Default
                                      Value: 2
  --analyzerName (-a) analyzerName    The class name of the analyzer
  --chunkSize (-chunk) chunkSize      The chunkSize in MegaBytes. 100-10000 MB
  --output (-o) output                The output directory
  --input (-i) input                  input dir containing the documents in
                                      sequence file format
  --minDF (-md) minDF                 The minimum document frequency.  Default
                                      is 1
  --maxDFPercent (-x) maxDFPercent    The max percentage of docs for the DF.
                                      Can be used to remove really high
                                      frequency terms. Expressed as an integer
                                      between 0 and 100. Default is 99.
  --weight (-wt) weight               The kind of weight to use. Currently TF
                                      or TFIDF
  --norm (-n) norm                    The norm to use, expressed as either a
                                      float or "INF" if you want to use the
                                      Infinite norm.  Must be greater or equal
                                      to 0.  The default is not to normalize
  --minLLR (-ml) minLLR               (Optional)The minimum Log Likelihood
                                      Ratio(Float)  Default is 1.0
  --numReducers (-nr) numReducers     (Optional) Number of reduce tasks.
                                      Default Value: 1
  --maxNGramSize (-ng) ngramSize      (Optional) The maximum size of ngrams to
                                      create (2 = bigrams, 3 = trigrams, etc)
                                      Default Value:1
  --overwrite (-ow)                   If set, overwrite the output directory
  --help (-h)                         Print out help
  --sequentialAccessVector (-seq)     (Optional) Whether output vectors should
                                      be SequentialAccessVectors. If set true
                                      else false
  --namedVector (-nv)                 (Optional) Whether output vectors should
                                      be NamedVectors. If set true else false
  --logNormalize (-lnorm)             (Optional) Whether output vectors should
                                      be logNormalize. If set true else false

The minimum command to issue, continuing the example, is:

1
./mahout seq2sparse -i /home/tgi/Desktop/1 -o /home/tgi/Desktop/2 -wt tf

3) Invoking LDA algorithm

With the documents prepared we can now invoke the LDA algorithm. options to run the algorithm are:

  --input (-i) input                      Path to job input directory.
  --output (-o) output                    The directory pathname for output.
  --overwrite (-ow)                       If present, overwrite the output
                                          directory before running job
  --numTopics (-k) numTopics              The total number of topics in the
                                          corpus
  --numWords (-v) numWords                The total number of words in the
                                          corpus (can be approximate, needs to
                                          exceed the actual value)
  --topicSmoothing (-a) topicSmoothing    Topic smoothing parameter. Default is
                                          50/numTopics.
  --maxIter (-x) maxIter                  The maximum number of iterations.
  --help (-h)                             Print out help
  --tempDir tempDir                       Intermediate output directory
  --startPhase startPhase                 First phase to run
  --endPhase endPhase                     Last phase to run

Continuing the example we issue the command:

1
./mahout lda -i /home/tgi/Desktop/2/tf-vectors -o /home/tgi/Desktop/3 -k 50 -v 200000

choosing to compute 50 topics in our corpus. The numWords parameter must exceed the total number of words in the previously computed dictionary. The easiest way to do this is to run it with a fictional number the first time, find this in the job initialization log outputted in the console

1
INFO: record buffer = 262144/327680

and re run LDA with -v > 327680 in the example.

The input directory must point to the output directory of the previous face /tf-vectors or /tfidf-vectors depending on our previous choice.

4) Output the computed topics

After running LDA you can obtain an output of the computed topics using another Mahout utility:

1
ldatopics

All options of the command are:

  --dict (-d) dict                         Dictionary to read in, in the same
                                           format as one created by
                                           org.apache.mahout.utils.vectors.lucen
                                           e.Driver
  --output (-o) output                     Output directory to write top words
  --words (-w) words                       Number of words to print
  --input (-i) input                       Path to an LDA output (a state)
  --dictionaryType (-dt) dictionaryType    The dictionary file type
                                           (text|sequencefile)
<span class="Apple-style-span" style="font-family: Georgia, 'Times New Roman', 'Bitstream Charter', Times, serif; font-size: 13px; line-height: 19px; white-space: normal;">and to print out the topics of the example corpus we type:  </span>
1
./mahout ldatopics -i /home/tgi/Desktop/3/state-9/ -d /home/tgi/Desktop/2/dictionary.* -o /home/tgi/Desktop/4 --dictionaryType sequencefile

Be careful here to use as the input directory the last state before convergence of the algorithm (sate-9 for the example). The output should be a set of files that each represent a computed topic and contain the words of that topic. Something like this:

end [p(end|topic_47) = 0.02054285450740693
class [p(class|topic_47) = 0.019173433558138983
you [p(you|topic_47) = 0.011081809083779152
ruby [p(ruby|topic_47) = 0.010939372894063723
code [p(code|topic_47) = 0.009912782417548813
pm [p(pm|topic_47) = 0.009565600712295945
07 [p(07|topic_47) = 0.008512880092545057
x [p(x|topic_47) = 0.007979504372069835
3 [p(3|topic_47) = 0.007552053207748147
from [p(from|topic_47) = 0.007501607551193776
your [p(your|topic_47) = 0.0074405804416695824
use [p(use|topic_47) = 0.007428773869438666
page [p(page|topic_47) = 0.007026473801340893
def [p(def|topic_47) = 0.006772484477458351
chapter [p(chapter|topic_47) = 0.006635243376794015
10 [p(10|topic_47) = 0.00600798063831032
have [p(have|topic_47) = 0.005999418317965101
2 [p(2|topic_47) = 0.005959277207972052
need [p(need|topic_47) = 0.005849514834292144
method [p(method|topic_47) = 0.005676919181434945

If you run LDA using a Hadoop cluster, you should substitute all paths used in the example with hdfs paths.

Currently (mahout version 0.5) LDA implementation is not completely integrated with the other Mahout clustering applications. There is no classification step to match an initial input document with a computed topic.

posted under Machine Learning