senaduka/old-nlp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d9c7848abbc1a9a2690c52de8d9dd4965eb55d28
old-nlp/venv/lib/python3.7/site-packages/nltk/__pycache__/probability.cpython-37.pyc

1197 lines
85 KiB
Plaintext
Raw Normal View History

Initial commit
2019-10-20 13:16:49 +02:00
B
D(<28>]a<00>@sdZddlmZmZmZddlZddlZddlZddlZddl m
Z
m Z ddl m Z ddlmZmZddlmZmZmZddlmZdd lmZed
<EFBFBD>ZejGd d <0C>d e <0B><03>Zee<0F>Gd d<0E>de<1D><03>ZejGdd<10>de<1E><03>ZejGdd<12>de<1E><03>Z ejGdd<14>de<1E><03>Z!ejGdd<16>de<1E><03>Z"ejGdd<18>de<1E><03>Z#ejGdd<1A>de#<23><03>Z$ejGdd<1C>de#<23><03>Z%ejGdd<1E>de<1E><03>Z&ejGdd <20>d e<1E><03>Z'ejGd!d"<22>d"e<1E><03>Z(ejGd#d$<24>d$e<1E><03>Z)Gd%d&<26>d&e<1E>Z*ejGd'd(<28>d(e<1E><03>Z+d)d*<2A>Z,d+d,<2C>Z-ejGd-d.<2E>d.e
<EFBFBD><03>Z.ejee<0F>Gd/d0<64>d0e/<2F><03><01>Z0Gd1d2<64>d2e0<65>Z1Gd3d4<64>d4e0<65>Z2e<05>3d5d6<64>Z4d7d8<64>Z5d9d:<3A>Z6Gd;d<<3C>d<e<1D>Z7Gd=d><3E>d>e7<65>Z8d?d@<40>Z9dAdB<64>Z:dCdD<64>Z;dOdGdH<64>Z<dIdJ<64>Z=e>dKk<02>r<>e<dEdL<64>e<dMdN<64>e=<3D>d.d2d0d d4ddd d$dd>dddd&d(dd<dd"d8d*d:d,gZ?dS)Pa<50>
Classes for representing and processing probabilistic information.
The ``FreqDist`` class is used to encode "frequency distributions",
which count the number of times that each outcome of an experiment
occurs.
The ``ProbDistI`` class defines a standard interface for "probability
distributions", which encode the probability of each outcome for an
experiment. There are two types of probability distribution:
- "derived probability distributions" are created from frequency
distributions. They attempt to model the probability distribution
that generated the frequency distribution.
- "analytic probability distributions" are created directly from
parameters (such as variance).
The ``ConditionalFreqDist`` class and ``ConditionalProbDistI`` interface
are used to encode conditional distributions. Conditional probability
distributions can be derived or analytic; but currently the only
implementation of the ``ConditionalProbDistI`` interface is
``ConditionalProbDist``, a derived distribution.
<EFBFBD>)<03>print_function<6F>unicode_literals<6C>divisionN)<02> defaultdict<63>Counter)<01>reduce)<02>ABCMeta<74>abstractmethod)<03>
itervalues<EFBFBD> text_type<70> add_metaclass)<01>compat)<01>raise_unorderable_typesz-1e300cseZdZdZd:dd<04>Zdd<06>Z<05>fdd<08>Z<06>fd d
<EFBFBD>Z<07>fd d <0C>Z<08>fd d<0E>Z dd<10>Z
dd<12>Z d;dd<14>Z d<dd<16>Z dd<18>Zdd<1A>Zdd<1C>Zdd<1E>Zdd <20>Zd!d"<22>Z<13>fd#d$<24>Z<14>fd%d&<26>Z<15>fd'd(<28>Z<16>fd)d*<2A>Zd+d,<2C>Zd-d.<2E>Zd/d.<2E>Zd0d.<2E>Zd1d2<64>Zd=d4d5<64>Zd>d6d7<64>Zd8d9<64>Z<1F>Z S)?<3F>FreqDista<74>
A frequency distribution for the outcomes of an experiment. A
frequency distribution records the number of times each outcome of
an experiment has occurred. For example, a frequency distribution
could be used to record the frequency of each word type in a
document. Formally, a frequency distribution can be defined as a
function mapping from each sample to the number of times that
sample occurred as an outcome.
Frequency distributions are generally constructed by running a
number of experiments, and incrementing the count for a sample
every time it is an outcome of an experiment. For example, the
following code will produce a frequency distribution that encodes
how often each word occurs in a text:
>>> from nltk.tokenize import word_tokenize
>>> from nltk.probability import FreqDist
>>> sent = 'This is an example sentence'
>>> fdist = FreqDist()
>>> for word in word_tokenize(sent):
... fdist[word.lower()] += 1
An equivalent way to do this is with the initializer:
>>> fdist = FreqDist(word.lower() for word in word_tokenize(sent))
NcCst<00>||<01>d|_dS)ab
Construct a new frequency distribution. If ``samples`` is
given, then the frequency distribution will be initialized
with the count of each object in ``samples``; otherwise, it
will be initialized to be empty.
In particular, ``FreqDist()`` returns an empty frequency
distribution; and ``FreqDist(samples)`` first creates an empty
frequency distribution, and then calls ``update`` with the
list ``samples``.
:param samples: The samples to initialize the frequency
distribution with.
:type samples: Sequence
N)r<00>__init__<5F>_N)<02>self<6C>samples<65>r<00>2/tmp/pip-install-4m6m_5d_/nltk/nltk/probability.pyr\s zFreqDist.__init__cCs|jdkrt|<00><02><00>|_|jS)z<>
Return the total number of sample outcomes that have been
recorded by this FreqDist. For the number of unique
sample values (or bins) with counts greater than zero, use
``FreqDist.B()``.
:rtype: int
N)r<00>sum<75>values)rrrr<00>Nqs
z
FreqDist.Ncsd|_tt|<00><02>||<02>dS)zO
Override ``Counter.__setitem__()`` to invalidate the cached N
N)r<00>superr<00> __setitem__)r<00>key<65>val)<01> __class__rrrszFreqDist.__setitem__csd|_tt|<00><02>|<01>dS)zO
Override ``Counter.__delitem__()`` to invalidate the cached N
N)rrr<00> __delitem__)rr)rrrr<00>szFreqDist.__delitem__csd|_tt|<00>j||<02>dS)zJ
Override ``Counter.update()`` to invalidate the cached N
N)rrr<00>update)r<00>args<67>kwargs)rrrr<00>szFreqDist.updatecsd|_tt|<00><02>||<02>dS)zN
Override ``Counter.setdefault()`` to invalidate the cached N
N)rrr<00>
setdefault)rrr)rrrr"<00>szFreqDist.setdefaultcCst|<00>S)a 
Return the total number of sample values (or "bins") that
have counts greater than zero. For the total
number of sample outcomes recorded, use ``FreqDist.N()``.
(FreqDist.B() is the same as len(FreqDist).)
:rtype: int
)<01>len)rrrr<00>B<>s z
FreqDist.Bcs<00>fdd<02><08>D<00>S)ze
Return a list of all samples that occur once (hapax legomena)
:rtype: list
csg|]}<01>|dkr|<01>qS)<01>r)<02>.0<EFBFBD>item)rrr<00>
<listcomp><3E>sz$FreqDist.hapaxes.<locals>.<listcomp>r)rr)rr<00>hapaxes<65>szFreqDist.hapaxescCs|<00>|<02>|S)N)<01>r_Nr)r<00>r<>binsrrr<00>Nr<4E>sz FreqDist.NrcCsJtt<01>}x |<00><02>D]}||d7<qW|dk r>||<00><03>nd|d<|S)a<>
Return the dictionary mapping r to Nr, the number of samples with frequency r, where Nr > 0.
:type bins: int
:param bins: The number of possible sample outcomes. ``bins``
is used to calculate Nr(0). In particular, Nr(0) is
``bins-self.B()``. If ``bins`` is not specified, it
defaults to ``self.B()`` (so Nr(0) will be 0).
:rtype: int
r%Nr)r<00>intrr$)rr,<00>_r_Nr<4E>countrrrr*<00>s
z FreqDist.r_Nrccs(d}x|D]}|||7}|Vq
WdS)a0
Return the cumulative frequencies of the specified samples.
If no samples are specified, all counts are returned, starting
with the largest.
:param samples: the samples whose frequencies should be returned.
:type samples: any
:rtype: list(float)
gNr)rr<00>cf<63>samplerrr<00>_cumulative_frequencies<65>s

 z FreqDist._cumulative_frequenciescCs |<00><00>}|dkrdS|||S)a0
Return the frequency of a given sample. The frequency of a
sample is defined as the count of that sample divided by the
total number of sample outcomes that have been recorded by
this FreqDist. The count of a sample is defined as the
number of times that sample outcome was recorded by this
FreqDist. Frequencies are always real numbers in the range
[0, 1].
:param sample: the sample whose frequency
should be returned.
:type sample: any
:rtype: float
r)r)rr2<00>nrrr<00>freq<65>sz FreqDist.freqcCs&t|<00>dkrtd<02><01>|<00>d<03>ddS)a<>
Return the sample with the greatest number of outcomes in this
frequency distribution. If two or more samples have the same
number of outcomes, return one of them; which sample is
returned is undefined. If no outcomes have occurred in this
frequency distribution, return None.
:return: The sample with the maximum number of outcomes in this
frequency distribution.
:rtype: any or None
rz?A FreqDist must have at least one sample before max is defined.r%)r#<00>
ValueError<EFBFBD> most_common)rrrr<00>max<61>s z FreqDist.maxc sRyddlm}Wntk
r,td<03><01>YnXt|<01>dkrDt<04><01>g}dd<05><00>j|<01>D<00>}t|dd<07>}t|dd<07>}|r<>t<07><01>|<04><01><01>d }|r<><72>fd
d<05><08>D<00><01>d }n<16>fd d<05>|D<00><01>d }|<03> <09>}|j
ddd<10>d|kr<>d|d<d|kr<>|<08> |d<00>|d=|j <0C>f|<02>|<08> tt|<04><01><01>|jdd<05>|D<00>dd<16>|<08>d<17>|<08>|<07>|<03><12>|S)a<>
Plot samples from the frequency distribution
displaying the most frequent sample first. If an integer
parameter is supplied, stop after this many samples have been
plotted. For a cumulative plot, specify cumulative=True.
(Requires Matplotlib to be installed.)
:param title: The title for the graph
:type title: str
:param cumulative: A flag to specify whether the plot is cumulative (default = False)
:type title: bool
rNzPThe plot function requires matplotlib to be installed.See http://matplotlib.org/cSsg|] \}}|<01>qSrr)r&r'<00>_rrrr(sz!FreqDist.plot.<locals>.<listcomp><3E>
cumulativeF<EFBFBD>percentszCumulative Countscs$g|]}|<01>t<00><00>dd<00>qS)r%<00>d)r#)r&<00>f)<01>freqsrrr(szCumulative Percentscsg|] }<01>|<00>qSrr)r&r2)rrrr(s<00>CountsT<73>silver)<01>color<6F> linewidth<74><00>titlecSsg|] }t|<01><01>qSr)r )r&<00>srrrr(.s<00>Z)<01>rotation<6F>Samples)<13>matplotlib.pyplot<6F>pyplot<6F> ImportErrorr6r#r7<00>
_get_kwarg<EFBFBD>listr3<00>gca<63>grid<69> set_title<6C>plot<6F>
set_xticks<EFBFBD>range<67>set_xticklabels<6C>
set_xlabel<EFBFBD>
set_ylabel<EFBFBD>show) rr r!<00>pltrr:r;<00>ylabel<65>axr)r>rrrQ<00>s@   
  

z FreqDist.plotcs<>t|<01>dkrt<00><00>g}dd<03><00>j|<01>D<00>}t|dd<05>}|rJt<03><00>|<03><01>}n<12>fdd<03>|D<00>}tdd<08>|D<00><01>}t|td d<08>|D<00><01><01>}x,tt|<03><01>D]}td
|||fd d <0C>q<>Wt<07>x,tt|<03><01>D]}td |||fd d <0C>q<>Wt<07>dS)a<>
Tabulate the given samples from the frequency distribution (cumulative),
displaying the most frequent sample first. If an integer
parameter is supplied, stop after this many samples have been
plotted.
:param samples: The samples to plot (default is all samples)
:type samples: list
:param cumulative: A flag to specify whether the freqs are cumulative (default = False)
:type title: bool
rcSsg|] \}}|<01>qSrr)r&r'r9rrrr(Dsz%FreqDist.tabulate.<locals>.<listcomp>r:Fcsg|] }<01>|<00>qSrr)r&r2)rrrr(Jscss|]}td|<00>VqdS)z%sN)r#)r&rErrr<00> <genexpr>Msz$FreqDist.tabulate.<locals>.<genexpr>css|]}td|<00>VqdS)z%dN)r#)r&r=rrrr[Nsz%*s<> )<01>endz%*dN)r#r7rLrMr3r8rS<00>print)rr r!rr:r><00>width<74>ir)rr<00>tabulate6s 
 zFreqDist.tabulatecCs
|<00>|<00>S)zY
Create a copy of this frequency distribution.
:rtype: FreqDist
)r)rrrr<00>copyWsz FreqDist.copycs|<00>tt|<00><02>|<01><01>S)z<>
Add counts from two counters.
>>> FreqDist('abbb') + FreqDist('bcc')
FreqDist({'b': 4, 'c': 2, 'a': 1})
)rrr<00>__add__)r<00>other)rrrrcaszFreqDist.__add__cs|<00>tt|<00><02>|<01><01>S)z<>
Subtract count, but keep only results with positive counts.
>>> FreqDist('abbbc') - FreqDist('bccd')
FreqDist({'b': 2, 'a': 1})
)rrr<00>__sub__)rrd)rrrrekszFreqDist.__sub__cs|<00>tt|<00><02>|<01><01>S)z<>
Union is the maximum of value in either of the input counters.
>>> FreqDist('abbb') | FreqDist('bcc')
FreqDist({'b': 3, 'c': 2, 'a': 1})
)rrr<00>__or__)rrd)rrrrfuszFreqDist.__or__cs|<00>tt|<00><02>|<01><01>S)z<>
Intersection is the minimum of corresponding counts.
>>> FreqDist('abbb') & FreqDist('bcc')
FreqDist({'b': 1})
)rrr<00>__and__)rrd)rrrrgszFreqDist.__and__cs<t<00>t<01>std<01><01><00>t<03><01><01><04><00>o:t<05><00>fdd<03><08>D<00><01>S)Nz<=c3s|]}<01>|<00>|kVqdS)Nr)r&r)rdrrrr[<00>sz"FreqDist.__le__.<locals>.<genexpr>)<06>
isinstancerr<00>set<65>issubset<65>all)rrdr)rdrr<00>__le__<5F>s
 zFreqDist.__le__cCs||k p||kS)Nr)rrdrrr<00><lambda><3E><00>zFreqDist.<lambda>cCs||ko||k S)Nr)rrdrrrrm<00>rncCs
||k S)Nr)rrdrrrrm<00>rncCs|<00><00>S)zZ
Return a string representation of this FreqDist.
:rtype: string
)<01>pformat)rrrr<00>__repr__<5F>szFreqDist.__repr__<5F>
cCst|j|d<01>|d<02>dS)z<>
Print a string representation of this FreqDist to 'stream'
:param maxlen: The maximum number of items to print
:type maxlen: int
:param stream: The stream to print to. stdout by default
)<01>maxlen)<01>fileN)r^ro)rrr<00>streamrrr<00>pprint<6E>szFreqDist.pprintcCs:dd<02>|<00>|<01>D<00>}t|<00>|kr*|<02>d<03>d<04>d<05>|<02><01>S)z<>
Return a string representation of this FreqDist.
:param maxlen: The maximum number of items to display
:type maxlen: int
:rtype: string
cSsg|]}dj|<01><00>qS)z {0!r}: {1!r})<01>format)r&r'rrrr(<00>sz$FreqDist.pformat.<locals>.<listcomp>z...zFreqDist({{{0}}})z, )r7r#<00>appendrv<00>join)rrr<00>itemsrrrro<00>s 
zFreqDist.pformatcCsdt|<00>|<00><01>fS)zZ
Return a string representation of this FreqDist.
:rtype: string
z*<FreqDist with %d samples and %d outcomes>)r#r)rrrr<00>__str__<5F>szFreqDist.__str__)N)N)N)rqN)rq)!<21>__name__<5F>
__module__<EFBFBD> __qualname__<5F>__doc__rrrrrr"r$r)r-r*r3r5r8rQrarbrcrerfrgrl<00>__ge__<5F>__lt__<5F>__gt__rprurorz<00> __classcell__rr)rrr>s:
     

9!

rc@s\eZdZdZdZedd<04><00>Zedd<06><00>Zdd<08>Zed d
<EFBFBD><00>Z ed d <0C><00>Z
d d<0E>Z dd<10>Z dS)<12> ProbDistIa<49>
A probability distribution for the outcomes of an experiment. A
probability distribution specifies how likely it is that an
experiment will have any given outcome. For example, a
probability distribution could be used to predict the probability
that a token in a document will have a given type. Formally, a
probability distribution can be defined as a function mapping from
samples to nonnegative real numbers, such that the sum of every
number in the function's range is 1.0. A ``ProbDist`` is often
used to model the probability distribution of the experiment used
to generate a frequency distribution.
TcCsdS)zN
Classes inheriting from ProbDistI should implement __init__.
Nr)rrrrr<00>szProbDistI.__init__cCsdS)a
Return the probability for a given sample. Probabilities
are always real numbers in the range [0, 1].
:param sample: The sample whose probability
should be returned.
:type sample: any
:rtype: float
Nr)rr2rrr<00>prob<6F>s
zProbDistI.probcCs"|<00>|<01>}|dkrt<01>|d<02>StS)z<>
Return the base 2 logarithm of the probability for a given sample.
:param sample: The sample whose probability
should be returned.
:type sample: any
:rtype: float
rrC)r<><00>math<74>log<6F>_NINF)rr2<00>prrr<00>logprob<6F>s
zProbDistI.logprobcCsdS)z<>
Return the sample with the greatest probability. If two or
more samples have the same probability, return one of them;
which sample is returned is undefined.
:rtype: any
Nr)rrrrr8<00>sz ProbDistI.maxcCsdS)z<>
Return a list of all samples that have nonzero probabilities.
Use ``prob`` to find the probability of each sample.
:rtype: list
Nr)rrrrr<00>szProbDistI.samplescCsdS)zi
Return the ratio by which counts are discounted on average: c*/c
:rtype: float
gr)rrrr<00>discountszProbDistI.discountcCsrt<00><00>}|}x*|<00><01>D]}||<00>|<03>8}|dkr|SqW|dkrD|S|jr`t<04>d|||f<00>t<00>t|<00><01><00><01>S)z<>
Return a randomly selected sample from this probability distribution.
The probability of returning each sample ``samp`` is equal to
``self.prob(samp)``.
rg-C<1C><>6?zTProbability distribution %r sums to %r; generate() is returning an arbitrary sample.)<08>randomrr<><00>
SUM_TO_ONE<EFBFBD>warnings<67>warn<72>choicerM)rr<>Zp_initr2rrr<00>generateszProbDistI.generateN) r{r|r}r~r<>r rr<>r<>r8rr<>r<>rrrrr<><00>s   

r<EFBFBD>c@s8eZdZdZdd<03>Zdd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd S) <0A>UniformProbDistz<74>
A probability distribution that assigns equal probability to each
sample in a given set; and a zero probability to all other
samples.
cCs>t|<01>dkrtd<02><01>t|<01>|_dt|j<03>|_t|j<03>|_dS)a4
Construct a new uniform probability distribution, that assigns
equal probability to each sample in ``samples``.
:param samples: The samples that should be given uniform
probability.
:type samples: list
:raise ValueError: If ``samples`` is empty.
rzAA Uniform probability distribution must have at least one sample.g<00>?N)r#r6ri<00>
_sampleset<EFBFBD>_probrM<00>_samples)rrrrrr2s

zUniformProbDist.__init__cCs||jkr|jSdS)Nr)r<>r<>)rr2rrrr<>DszUniformProbDist.probcCs
|jdS)Nr)r<>)rrrrr8GszUniformProbDist.maxcCs|jS)N)r<>)rrrrrJszUniformProbDist.samplescCsdt|j<01>S)Nz!<UniformProbDist with %d samples>)r#r<>)rrrrrpMszUniformProbDist.__repr__N) r{r|r}r~rr<>r8rrprrrrr<>*s r<>c@sDeZdZdZdd<03>Zedd<05><00>Zdd<07>Zdd <09>Zd
d <0B>Z d d <0A>Z
dS)<0F>RandomProbDistz<74>
Generates a random probability distribution whereby each sample
will be between 0 and 1 with equal probability (uniform random distribution.
Also called a continuous uniform distribution).
cCs4t|<01>dkrtd<02><01>|<00>|<01>|_t|j<03><05><00>|_dS)Nrz9A probability distribution must have at least one sample.)r#r6<00>unirand<6E>_probsrM<00>keysr<73>)rrrrrrYs
  zRandomProbDist.__init__cs<>t|<01>}dd<02>tt|<01><01>D<00><01>t<03><00>}x t<04><00>D]\}}||<00>|<q0Wt<03><00>}|dkrl<72>d|d8<t<05>fdd<06>t|<01>D<00><01>S)a
The key function that creates a randomized initial distribution
that still sums to 1. Set as a dictionary of prob values so that
it can still be passed to MutableProbDist and called with identical
syntax to UniformProbDist
cSsg|] }t<00><00><00>qSr)r<>)r&r`rrrr(jsz*RandomProbDist.unirand.<locals>.<listcomp>r%<00><><EFBFBD><EFBFBD><EFBFBD>c3s|]\}}|<02>|fVqdS)Nr)r&r`rE)<01>randrowrrr[usz)RandomProbDist.unirand.<locals>.<genexpr>)rirSr#r<00> enumerate<74>dict)<05>clsr<00>totalr`<00>xr)r<>rr<>aszRandomProbDist.unirandcCs.t|d<01>s(tdd<03>|j<02><03>D<00><01>d|_|jS)N<>_maxcss|]\}}||fVqdS)Nr)r&<00>vr<76>rrrr[ysz%RandomProbDist.max.<locals>.<genexpr>r%)<05>hasattrr8r<>ryr<>)rrrrr8ws
zRandomProbDist.maxcCs|j<00>|d<01>S)Nr)r<><00>get)rr2rrrr<>|szRandomProbDist.probcCs|jS)N)r<>)rrrrrszRandomProbDist.samplescCsdt|j<01>S)Nz'<RandomUniformProbDist with %d samples>)r#r<>)rrrrrp<00>szRandomProbDist.__repr__N) r{r|r}r~r<00> classmethodr<64>r8r<>rrprrrrr<>Qs r<>c@sBeZdZdZddd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd d <0A>Zdd<0F>Z dS)<11>DictionaryProbDistz<74>
A probability distribution whose probabilities are directly
specified by a given dictionary. The given dictionary maps
samples to probabilities.
NFc Cs |dk r|<01><00>ni|_||_|<03>rt|<01>dkr6td<03><01>|r<>tt|j<01><07><00><01>}|tkr<>t <09>
dt|<01>d<05>}xB|D]}||j|<qnWn*x<>|j<01> <0B>D]\}}|j||8<q<>Wnnt |j<01><07><00>}|dkr<>dt|<01>}xJ|D]}||j|<q<>Wn2d|}x(|j<01> <0B>D]\}}|j||9<q<>WdS)a<>
Construct a new probability distribution from the given
dictionary, which maps values to probabilities (or to log
probabilities, if ``log`` is true). If ``normalize`` is
true, then the probability values are scaled by a constant
factor such that they sum to 1.
If called without arguments, the resulting probability
distribution assigns zero probability to all values.
NrzOA DictionaryProbDist must have at least one sample before it can be normalized.g<00>?rC) rb<00>
_prob_dict<EFBFBD>_logr#r6<00>sum_logsrMrr<>r<>r<>ryr) rZ prob_dictr<74><00> normalizeZ value_sumZlogpr<70>r<>Z norm_factorrrrr<00>s,  
 
zDictionaryProbDist.__init__cCs4|jr"||jkrd|j|SdS|j<01>|d<02>SdS)NrCr)r<>r<>r<>)rr2rrrr<><00>szDictionaryProbDist.probcCsJ|jr|j<01>|t<03>S||jkr"tS|j|dkr4tSt<04>|j|d<02>SdS)NrrC)r<>r<>r<>r<>r<>r<>)rr2rrrr<><00>s
zDictionaryProbDist.logprobcCs.t|d<01>s(tdd<03>|j<02><03>D<00><01>d|_|jS)Nr<4E>css|]\}}||fVqdS)Nr)r&r<>r<>rrrr[<00>sz)DictionaryProbDist.max.<locals>.<genexpr>r%)r<>r8r<>ryr<>)rrrrr8<00>s
zDictionaryProbDist.maxcCs
|j<00><01>S)N)r<>r<>)rrrrr<00>szDictionaryProbDist.samplescCsdt|j<01>S)Nz<ProbDist with %d samples>)r#r<>)rrrrrp<00>szDictionaryProbDist.__repr__)NFF)
r{r|r}r~rr<>r<>r8rrprrrrr<><00>s
* r<>c@sBeZdZdZddd<04>Zdd<06>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Zd d<0E>Z dS)<10> MLEProbDista%
The maximum likelihood estimate for the probability distribution
of the experiment used to generate a frequency distribution. The
"maximum likelihood estimate" approximates the probability of
each sample as the frequency of that sample in the frequency
distribution.
NcCs
||_dS)a)
Use the maximum likelihood estimate to create a probability
distribution for the experiment used to generate ``freqdist``.
:type freqdist: FreqDist
:param freqdist: The frequency distribution that the
probability estimates should be based on.
N)<01> _freqdist)r<00>freqdistr,rrrr<00>s zMLEProbDist.__init__cCs|jS)z<>
Return the frequency distribution that this probability
distribution is based on.
:rtype: FreqDist
)r<>)rrrrr<><00>szMLEProbDist.freqdistcCs |j<00>|<01>S)N)r<>r5)rr2rrrr<><00>szMLEProbDist.probcCs
|j<00><01>S)N)r<>r8)rrrrr8<00>szMLEProbDist.maxcCs
|j<00><01>S)N)r<>r<>)rrrrr<00>szMLEProbDist.samplescCsd|j<00><01>S)z\
:rtype: str
:return: A string representation of this ``ProbDist``.
z!<MLEProbDist based on %d samples>)r<>r)rrrrrp<00>szMLEProbDist.__repr__)N)
r{r|r}r~rr<>r<>r8rrprrrrr<><00>s
 r<>c@sNeZdZdZdZddd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd d <0A>Z dd<0F>Z
dd<11>Z dS)<13>LidstoneProbDista8
The Lidstone estimate for the probability distribution of the
experiment used to generate a frequency distribution. The
"Lidstone estimate" is parameterized by a real number *gamma*,
which typically ranges from 0 to 1. The Lidstone estimate
approximates the probability of a sample with count *c* from an
experiment with *N* outcomes and *B* bins as
``c+gamma)/(N+B*gamma)``. This is equivalent to adding
*gamma* to the count for each bin, and taking the maximum
likelihood estimate of the resulting frequency distribution.
FNcCs<>|dks|dkr<|<01><00>dkr<|jjdd<03>}td|d<00><01>|dk r<>||<01><04>kr<>|jjdd<03>}td|d|dd |<01><04><00><01>||_t|<02>|_|j<05><00>|_|dkr<>|<01><04>}||_ |j|||_
|j
d
kr<>d|_d |_
dS) a<>
Use the Lidstone estimate to create a probability distribution
for the experiment used to generate ``freqdist``.
:type freqdist: FreqDist
:param freqdist: The frequency distribution that the
probability estimates should be based on.
:type gamma: float
:param gamma: A real number used to parameterize the
estimate. The Lidstone estimate is equivalent to adding
*gamma* to the count for each bin, and taking the
maximum likelihood estimate of the resulting frequency
distribution.
:type bins: int
:param bins: The number of sample values that can be generated
by the experiment that is described by the probability
distribution. This value must be correctly set for the
probabilities of the sample values to sum to one. If
``bins`` is not specified, it defaults to ``freqdist.B()``.
rNi<4E><69><EFBFBD><EFBFBD>zA %s probability distribution zmust have at least one bin.z)
The number of bins in a %s distribution z&(%d) must be greater than or equal to
z(the number of bins in the FreqDist used zto create it (%d).gr%) rrr{r6r$r<><00>float<61>_gammar<00>_bins<6E>_divisor)rr<><00>gammar,<00>namerrrrs$"
 
zLidstoneProbDist.__init__cCs|jS)z<>
Return the frequency distribution that this probability
distribution is based on.
:rtype: FreqDist
)r<>)rrrrr<>FszLidstoneProbDist.freqdistcCs|j|}||j|jS)N)r<>r<>r<>)rr2<00>crrrr<>Os
zLidstoneProbDist.probcCs
|j<00><01>S)N)r<>r8)rrrrr8SszLidstoneProbDist.maxcCs
|j<00><01>S)N)r<>r<>)rrrrrYszLidstoneProbDist.samplescCs|j|j}||j|S)N)r<>r<>r)r<00>gbrrrr<>\s zLidstoneProbDist.discountcCsd|j<00><01>S)z[
Return a string representation of this ``ProbDist``.
:rtype: str
z&<LidstoneProbDist based on %d samples>)r<>r)rrrrrp`szLidstoneProbDist.__repr__)N) r{r|r}r~r<>rr<>r<>r8rr<>rprrrrr<>s 
2 r<>c@s"eZdZdZddd<04>Zdd<06>ZdS)<08>LaplaceProbDista<74>
The Laplace estimate for the probability distribution of the
experiment used to generate a frequency distribution. The
"Laplace estimate" approximates the probability of a sample with
count *c* from an experiment with *N* outcomes and *B* bins as
*(c+1)/(N+B)*. This is equivalent to adding one to the count for
each bin, and taking the maximum likelihood estimate of the
resulting frequency distribution.
NcCst<00>||d|<02>dS)a<>
Use the Laplace estimate to create a probability distribution
for the experiment used to generate ``freqdist``.
:type freqdist: FreqDist
:param freqdist: The frequency distribution that the
probability estimates should be based on.
:type bins: int
:param bins: The number of sample values that can be generated
by the experiment that is described by the probability
distribution. This value must be correctly set for the
probabilities of the sample values to sum to one. If
``bins`` is not specified, it defaults to ``freqdist.B()``.
r%N)r<>r)rr<>r,rrrruszLaplaceProbDist.__init__cCsd|j<00><01>S)z\
:rtype: str
:return: A string representation of this ``ProbDist``.
z%<LaplaceProbDist based on %d samples>)r<>r)rrrrrp<00>szLaplaceProbDist.__repr__)N)r{r|r}r~rrprrrrr<>is

r<>c@s"eZdZdZddd<04>Zdd<06>ZdS)<08> ELEProbDista<74>
The expected likelihood estimate for the probability distribution
of the experiment used to generate a frequency distribution. The
"expected likelihood estimate" approximates the probability of a
sample with count *c* from an experiment with *N* outcomes and
*B* bins as *(c+0.5)/(N+B/2)*. This is equivalent to adding 0.5
to the count for each bin, and taking the maximum likelihood
estimate of the resulting frequency distribution.
NcCst<00>||d|<02>dS)a<>
Use the expected likelihood estimate to create a probability
distribution for the experiment used to generate ``freqdist``.
:type freqdist: FreqDist
:param freqdist: The frequency distribution that the
probability estimates should be based on.
:type bins: int
:param bins: The number of sample values that can be generated
by the experiment that is described by the probability
distribution. This value must be correctly set for the
probabilities of the sample values to sum to one. If
``bins`` is not specified, it defaults to ``freqdist.B()``.
g<00>?N)r<>r)rr<>r,rrrr<00>szELEProbDist.__init__cCsd|j<00><01>S)z[
Return a string representation of this ``ProbDist``.
:rtype: str
z!<ELEProbDist based on %d samples>)r<>r)rrrrrp<00>szELEProbDist.__repr__)N)r{r|r}r~rrprrrrr<><00>s

r<>c@sfeZdZdZdZddd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd d <0A>Z dd<0F>Z
dd<11>Z dd<13>Z dd<15>Z dd<17>ZdS)<19>HeldoutProbDistag
The heldout estimate for the probability distribution of the
experiment used to generate two frequency distributions. These
two frequency distributions are called the "heldout frequency
distribution" and the "base frequency distribution." The
"heldout estimate" uses uses the "heldout frequency
distribution" to predict the probability of each sample, given its
frequency in the "base frequency distribution".
In particular, the heldout estimate approximates the probability
for a sample that occurs *r* times in the base distribution as
the average frequency in the heldout distribution of all samples
that occur *r* times in the base distribution.
This average frequency is *Tr[r]/(Nr[r].N)*, where:
- *Tr[r]* is the total count in the heldout distribution for
all samples that occur *r* times in the base distribution.
- *Nr[r]* is the number of samples that occur *r* times in
the base distribution.
- *N* is the number of outcomes recorded by the heldout
frequency distribution.
In order to increase the efficiency of the ``prob`` member
function, *Tr[r]/(Nr[r].N)* is precomputed for each value of *r*
when the ``HeldoutProbDist`` is created.
:type _estimate: list(float)
:ivar _estimate: A list mapping from *r*, the number of
times that a sample occurs in the base distribution, to the
probability estimate for that sample. ``_estimate[r]`` is
calculated by finding the average frequency in the heldout
distribution of all samples that occur *r* times in the base
distribution. In particular, ``_estimate[r]`` =
*Tr[r]/(Nr[r].N)*.
:type _max_r: int
:ivar _max_r: The maximum number of times that any sample occurs
in the base distribution. ``_max_r`` is used to decide how
large ``_estimate`` must be.
FNcsd||_||_||<01><02>|_|<00><04>}|<01>|<03><01><00>fdd<02>t|jd<00>D<00>}|<02><07>}|<00>|||<06>|_ dS)a<>
Use the heldout estimate to create a probability distribution
for the experiment used to generate ``base_fdist`` and
``heldout_fdist``.
:type base_fdist: FreqDist
:param base_fdist: The base frequency distribution.
:type heldout_fdist: FreqDist
:param heldout_fdist: The heldout frequency distribution.
:type bins: int
:param bins: The number of sample values that can be generated
by the experiment that is described by the probability
distribution. This value must be correctly set for the
probabilities of the sample values to sum to one. If
``bins`` is not specified, it defaults to ``freqdist.B()``.
csg|] }<01>|<00>qSrr)r&r+)r*rrr(<00>sz,HeldoutProbDist.__init__.<locals>.<listcomp>r%N)
<EFBFBD> _base_fdist<73>_heldout_fdistr8<00>_max_r<5F> _calculate_Trr*rSr<00>_calculate_estimate<74> _estimate)r<00>
base_fdist<EFBFBD> heldout_fdistr,<00>Trr-rr)r*rr<00>s
zHeldoutProbDist.__init__cCsDdg|jd}x.|jD]$}|j|}|||j|7<qW|S)z<>
Return the list *Tr*, where *Tr[r]* is the total count in
``heldout_fdist`` for all samples that occur *r*
times in ``base_fdist``.
:rtype: list(float)
gr%)r<>r<>r<>)rr<>r2r+rrrr<>s
 
zHeldoutProbDist._calculate_TrcCsRg}xHt|jd<00>D]6}||dkr0|<04>d<03>q|<04>|||||<00>qW|S)a~
Return the list *estimate*, where *estimate[r]* is the probability
estimate for any sample that occurs *r* times in the base frequency
distribution. In particular, *estimate[r]* is *Tr[r]/(N[r].N)*.
In the special case that *N[r]=0*, *estimate[r]* will never be used;
so we define *estimate[r]=None* for those cases.
:rtype: list(float)
:type Tr: list(float)
:param Tr: the list *Tr*, where *Tr[r]* is the total count in
the heldout distribution for all samples that occur *r*
times in base distribution.
:type Nr: list(float)
:param Nr: The list *Nr*, where *Nr[r]* is the number of
samples that occur *r* times in the base distribution.
:type N: int
:param N: The total number of outcomes recorded by the heldout
frequency distribution.
r%rN)rSr<>rw)rr<>r-rZestimater+rrrr<>s   z#HeldoutProbDist._calculate_estimatecCs|jS)z<>
Return the base frequency distribution that this probability
distribution is based on.
:rtype: FreqDist
)r<>)rrrrr<>-szHeldoutProbDist.base_fdistcCs|jS)z<>
Return the heldout frequency distribution that this
probability distribution is based on.
:rtype: FreqDist
)r<>)rrrrr<>6szHeldoutProbDist.heldout_fdistcCs
|j<00><01>S)N)r<>r<>)rrrrr?szHeldoutProbDist.samplescCs|j|}|j|S)N)r<>r<>)rr2r+rrrr<>Bs
zHeldoutProbDist.probcCs
|j<00><01>S)N)r<>r8)rrrrr8GszHeldoutProbDist.maxcCs
t<00><00>dS)N)<01>NotImplementedError)rrrrr<>MszHeldoutProbDist.discountcCsd}||j<00><01>|j<02><01>fS)z\
:rtype: str
:return: A string representation of this ``ProbDist``.
z6<HeldoutProbDist: %d base samples; %d heldout samples>)r<>rr<>)rrErrrrpPszHeldoutProbDist.__repr__)N)r{r|r}r~r<>rr<>r<>r<>r<>rr<>r8r<>rprrrrr<><00>s)
"  r<>c@sDeZdZdZdZdd<04>Zdd<06>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Z d d<0E>Z
dS)<10>CrossValidationProbDistaA
The cross-validation estimate for the probability distribution of
the experiment used to generate a set of frequency distribution.
The "cross-validation estimate" for the probability of a sample
is found by averaging the held-out estimates for the sample in
each pair of frequency distributions.
FcCsL||_g|_x:|D]2}x,|D]$}||k rt|||<02>}|j<01>|<05>qWqWdS)a<>
Use the cross-validation estimate to create a probability
distribution for the experiment used to generate
``freqdists``.
:type freqdists: list(FreqDist)
:param freqdists: A list of the frequency distributions
generated by the experiment.
:type bins: int
:param bins: The number of sample values that can be generated
by the experiment that is described by the probability
distribution. This value must be correctly set for the
probabilities of the sample values to sum to one. If
``bins`` is not specified, it defaults to ``freqdist.B()``.
N)<04>
_freqdists<EFBFBD>_heldout_probdistsr<73>rw)r<00> freqdistsr,<00>fdist1<74>fdist2Zprobdistrrrres

 z CrossValidationProbDist.__init__cCs|jS)z<>
Return the list of frequency distributions that this ``ProbDist`` is based on.
:rtype: list(FreqDist)
)r<>)rrrrr<><00>sz!CrossValidationProbDist.freqdistscCsttdd<02>|jD<00>g<00><02>S)NcSsg|] }t|<01><01>qSr)rM)r&<00>fdrrrr(<00>sz3CrossValidationProbDist.samples.<locals>.<listcomp>)rirr<>)rrrrr<00>szCrossValidationProbDist.samplescCs0d}x|jD]}||<03>|<01>7}q W|t|j<00>S)Ng)r<>r<>r#)rr2r<>Zheldout_probdistrrrr<><00>s zCrossValidationProbDist.probcCs
t<00><00>dS)N)r<>)rrrrr<><00>sz CrossValidationProbDist.discountcCsdt|j<01>S)z[
Return a string representation of this ``ProbDist``.
:rtype: str
z!<CrossValidationProbDist: %d-way>)r#r<>)rrrrrp<00>sz CrossValidationProbDist.__repr__N) r{r|r}r~r<>rr<>rr<>r<>rprrrrr<>Ysr<>c@sJeZdZdZddd<04>Zdd<06>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Zd d<0E>Z dd<10>Z
dS)<12>WittenBellProbDista<74>
The Witten-Bell estimate of a probability distribution. This distribution
allocates uniform probability mass to as yet unseen events by using the
number of events that have only been seen once. The probability mass
reserved for unseen events is equal to *T / (N + T)*
where *T* is the number of observed event types and *N* is the total
number of observed events. This equates to the maximum likelihood estimate
of a new type event occurring. The remaining probability mass is discounted
such that all probability estimates sum to one, yielding:
- *p = T / Z (N + T)*, if count = 0
- *p = c / (N + T)*, otherwise
NcCs<>|dks$||<01><00>ks$td|<01><00><00><01>|dkr4|<01><00>}||_|j<02><00>|_||j<02><00>|_|j<02><05>|_|jdkrzd|j|_n|j|j|j|j|_dS)a)
Creates a distribution of Witten-Bell probability estimates. This
distribution allocates uniform probability mass to as yet unseen
events by using the number of events that have only been seen once. The
probability mass reserved for unseen events is equal to *T / (N + T)*
where *T* is the number of observed event types and *N* is the total
number of observed events. This equates to the maximum likelihood
estimate of a new type event occurring. The remaining probability mass
is discounted such that all probability estimates sum to one,
yielding:
- *p = T / Z (N + T)*, if count = 0
- *p = c / (N + T)*, otherwise
The parameters *T* and *N* are taken from the ``freqdist`` parameter
(the ``B()`` and ``N()`` values). The normalizing factor *Z* is
calculated using these values along with the ``bins`` parameter.
:param freqdist: The frequency counts upon which to base the
estimation.
:type freqdist: FreqDist
:param bins: The number of possible event types. This must be at least
as large as the number of bins in the ``freqdist``. If None, then
it's assumed to be equal to that of the ``freqdist``
:type bins: int
Nz4bins parameter must not be less than %d=freqdist.B()rg<00>?)r$<00>AssertionErrorr<72><00>_TZ_Zrr<00>_P0)rr<>r,rrrr<00>s  
zWittenBellProbDist.__init__cCs(|j|}|dkr"||j|jS|jS)Nr)r<>rr<>r<>)rr2r<>rrrr<><00>s
zWittenBellProbDist.probcCs
|j<00><01>S)N)r<>r8)rrrrr8<00>szWittenBellProbDist.maxcCs
|j<00><01>S)N)r<>r<>)rrrrr<00>szWittenBellProbDist.samplescCs|jS)N)r<>)rrrrr<><00>szWittenBellProbDist.freqdistcCs
t<00><00>dS)N)r<>)rrrrr<><00>szWittenBellProbDist.discountcCsd|j<00><01>S)z[
Return a string representation of this ``ProbDist``.
:rtype: str
z(<WittenBellProbDist based on %d samples>)r<>r)rrrrrp<00>szWittenBellProbDist.__repr__)N) r{r|r}r~rr<>r8rr<>r<>rprrrrr<><00>s
+r<>c@s<>eZdZdZdZd$dd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd d <0A>Z dd<0F>Z
dd<11>Z dd<13>Z dd<15>Z dd<17>Zdd<19>Zdd<1B>Zdd<1D>Zdd<1F>Zd d!<21>Zd"d#<23>ZdS)%<25>SimpleGoodTuringProbDista<
SimpleGoodTuring ProbDist approximates from frequency to frequency of
frequency into a linear line under log space by linear regression.
Details of Simple Good-Turing algorithm can be found in:
- Good Turing smoothing without tears" (Gale & Sampson 1995),
Journal of Quantitative Linguistics, vol. 2 pp. 217-237.
- "Speech and Language Processing (Jurafsky & Martin),
2nd Edition, Chapter 4.5 p103 (log(Nc) = a + b*log(c))
- http://www.grsampson.net/RGoodTur.html
Given a set of pair (xi, yi), where the xi denotes the frequency and
yi denotes the frequency of frequency, we want to minimize their
square variation. E(x) and E(y) represent the mean of xi and yi.
- slope: b = sigma ((xi-E(x)(yi-E(y))) / sigma ((xi-E(x))(xi-E(x)))
- intercept: a = E(y) - b.E(x)
FNcCs||dks(||<01><00>ks(td|<01><00>d<00><01>|dkr<|<01><00>d}||_||_|<00><04>\}}|<00>||<04>|<00>||<04>|<00>||<04>dS)ap
:param freqdist: The frequency counts upon which to base the
estimation.
:type freqdist: FreqDist
:param bins: The number of possible event types. This must be
larger than the number of bins in the ``freqdist``. If None,
then it's assumed to be equal to ``freqdist``.B() + 1
:type bins: int
Nz6bins parameter must not be less than %d=freqdist.B()+1r%)r$r<>r<>r<>r/<00> find_best_fit<69>_switch<63> _renormalize)rr<>r,r+<00>nrrrrrGs     z!SimpleGoodTuringProbDist.__init__cCs|j<00><01>}|d=|S)Nr)r<>r*)rr*rrr<00>_r_Nr_non_zero]s
z'SimpleGoodTuringProbDist._r_Nr_non_zerocCs$|<00><00>}|sggfStt|<01><03><00><01>S)zW
Split the frequency distribution in two list (r, Nr), where Nr(r) > 0
)r<><00>zip<69>sortedry)rZnonzerorrrr/bszSimpleGoodTuringProbDist._r_NrcCsJ|r|s dSg}xvtt|<01><01>D]f}|dkr6||dnd}|t|<01>dkrZd|||n
||d}d||||}|<03>|<07>qWdd<07>|D<00>}dd<07>|D<00>} d }
} t|<08>t|<08>} t| <09>t| <09>} x:t|| <09>D],\}}|
|| || 7}
| || d7} q<>W| dk<03>r|
| nd |_|jd
k<05>r6t<06>d <0B>| |j| |_dS) z<>
Use simple linear regression to tune parameters self._slope and
self._intercept in the log-log space based on count and Nr(count)
(Work in log space to avoid floating point underflow.)
Nrr%rCg@cSsg|]}t<00>|<01><01>qSr)r<>r<>)r&r`rrrr(<00>sz:SimpleGoodTuringProbDist.find_best_fit.<locals>.<listcomp>cSsg|]}t<00>|<01><01>qSr)r<>r<>)r&r`rrrr(<00>sgr<>z<>SimpleGoodTuring did not find a proper best fit line for smoothing probabilities of occurrences. The probability estimates are likely to be unreliable.) rSr#rwrr<><00>_sloper<65>r<><00>
_intercept)rr+r<>Zzr<7A>jr`<00>kZzr_Zlog_rZlog_zrZxy_covZx_varZx_meanZy_meanr<6E><00>yrrrr<>ls* , z&SimpleGoodTuringProbDist.find_best_fitc
Cs<>x<>t|<01>D]<5D>\}}t|<01>|dks6||d|dkr>||_P|j}|d||d<00>||<04>}|d||d||}t<04>|<00>|||||d<00><03>}t||<00>d|kr
||_Pq
WdS)zl
Calculate the r frontier where we must switch from Nr to Sr
when estimating E[Nr].
r%g\<5C><><EFBFBD>(\<5C>?N)r<>r#<00>
_switch_at<EFBFBD>
smoothedNrr<EFBFBD><00>sqrt<72> _variance<63>abs) rr+r<>r`<00>r_ZSrZ smooth_r_starZunsmooth_r_starZstdrrrr<><00>s$ z SimpleGoodTuringProbDist._switchcCs<t|<01>}t|<02>}t|<03>}|dd||dd||S)Ng<00>?rC)r<>)rr+r<>Znr_1rrrr<><00>sz"SimpleGoodTuringProbDist._variancecCsJd}x(t||<02>D]\}}|||<00>|<04>7}qW|rFd|<00>d<03>||_dS)ay
It is necessary to renormalize all the probability estimates to
ensure a proper probability distribution results. This can be done
by keeping the estimate of the probability mass for unseen items as
N(1)/N and renormalizing all the estimates for previously seen items
(as Gale and Sampson (1995) propose). (See M&S P.213, 1999)
gr%rN)r<><00> _prob_measure<72> _renormal)rr+r<>Zprob_covr<76>Znr_rrrr<><00>s
z%SimpleGoodTuringProbDist._renormalizecCst<00>|j|jt<00>|<01><00>S)z<>
Return the number of samples with count r.
:param r: The amount of frequency.
:type r: int
:rtype: float
)r<><00>expr<70>r<>r<>)rr+rrrr<><00>sz#SimpleGoodTuringProbDist.smoothedNrcCsV|j|}|<00>|<02>}|dkrH|j|j<00><03>kr2d}qR||j|j<00><03>}n
||j}|S)z<>
Return the sample's probability.
:param sample: sample of the event
:type sample: str
:rtype: float
rg)r<>r<>r<>r$r<>)rr2r0r<>rrrr<><00>s


zSimpleGoodTuringProbDist.probcCs<>|dkr|j<00><01>dkrdS|dkrF|j<00><01>dkrF|j<00>d<03>|j<00><01>S|j|krn|j<00>|d<00>}|j<00>|<01>}n|<00>|d<00>}|<00>|<01>}|d||}||j<00><01>S)Nrg<00>?r%)r<>rr-r<>r<>)rr0ZEr_1ZErZr_starrrrr<><00>s

z&SimpleGoodTuringProbDist._prob_measurecCsJd}x6tdt|j<02><01>D]"}||j||<00>|<02>|j7}qWtd|<01>dS)NgrzProbability Sum:)rSr#Z_Nrr<72>r<>r^)rZprob_sumr`rrr<00>check<63>s"zSimpleGoodTuringProbDist.checkcCs|<00>d<01>|j<01><02>S)z<>
This function returns the total mass of probability transfers from the
seen samples to the unseen samples.
r%)r<>r<>r)rrrrr<><00>sz!SimpleGoodTuringProbDist.discountcCs
|j<00><01>S)N)r<>r8)rrrrr8<00>szSimpleGoodTuringProbDist.maxcCs
|j<00><01>S)N)r<>r<>)rrrrrsz SimpleGoodTuringProbDist.samplescCs|jS)N)r<>)rrrrr<>sz!SimpleGoodTuringProbDist.freqdistcCsd|j<00><01>S)z[
Return a string representation of this ``ProbDist``.
:rtype: str
z.<SimpleGoodTuringProbDist based on %d samples>)r<>r)rrrrrpsz!SimpleGoodTuringProbDist.__repr__)N)r{r|r}r~r<>rr<>r/r<>r<>r<>r<>r<>r<>r<>r<>r<>r8rr<>rprrrrr<>0s$

)r<>c@sDeZdZdZddd<04>Zdd<06>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Zdd d<0E>Z dS)<12>MutableProbDistz<74>
An mutable probdist where the probabilities may be easily modified. This
simply copies an existing probdist, storing the probability values in a
mutable dictionary and providing an update method.
Tcs<><00>|_t<01>fdd<02>tt<03><00><01>D<00><01>|_t<05>td<03>dg<01>t<03><00>|_xBtt<03><00><01>D]2}|rn|<01><08>|<00>|j|<qP|<01> <09>|<00>|j|<qPW||_
dS)a
Creates the mutable probdist based on the given prob_dist and using
the list of samples given. These values are stored as log
probabilities if the store_logs flag is set.
:param prob_dist: the distribution from which to garner the
probabilities
:type prob_dist: ProbDist
:param samples: the complete set of samples
:type samples: sequence of any
:param store_logs: whether to store the probabilities as logarithms
:type store_logs: bool
c3s|]}<01>||fVqdS)Nr)r&r`)rrrr[&sz+MutableProbDist.__init__.<locals>.<genexpr><3E>dgN) r<>r<>rSr#<00> _sample_dict<63>array<61>str<74>_datar<61>r<><00>_logs)rZ prob_distrZ
store_logsr`r)rrrs zMutableProbDist.__init__cCstdd<02>|j<01><02>D<00><01>dS)Ncss|]\}}||fVqdS)Nr)r&r<>r<>rrrr[1sz&MutableProbDist.max.<locals>.<genexpr>r%)r8r<>ry)rrrrr8/szMutableProbDist.maxcCs|jS)N)r<>)rrrrr3szMutableProbDist.samplescCs6|j<00>|<01>}|dkrdS|jr,d|j|S|j|S)NgrC)r<>r<>r<>r<>)rr2r`rrrr<>7s zMutableProbDist.probcCs>|j<00>|<01>}|dkrtd<01>S|jr,|j|St<05>|j|d<02>S)Nz-infrC)r<>r<>r<>r<>r<>r<>r<>)rr2r`rrrr<>>s zMutableProbDist.logprobcCsT|j<00>|<01>}|dk st<02>|jr:|r&|n
t<04>|d<02>|j|<n|rFd|n||j|<dS)a<
Update the probability for the given sample. This may cause the object
to stop being the valid probability distribution - the user must
ensure that they update the sample probabilities such that all samples
have probabilities between 0 and 1 and that all probabilities sum to
one.
:param sample: the sample for which to update the probability
:type sample: any
:param prob: the new probability
:type prob: float
:param log: is the probability already logged
:type log: bool
NrC)r<>r<>r<>r<>r<>r<>r<>)rr2r<>r<>r`rrrrEs
  zMutableProbDist.updateN)T)T)
r{r|r}r~rr8rr<>r<>rrrrrr<>s
r<>c@sJeZdZdZddd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd d <0A>Zdd<0F>Z dd<11>Z
dS)<13>KneserNeyProbDista~
Kneser-Ney estimate of a probability distribution. This is a version of
back-off that counts how likely an n-gram is provided the n-1-gram had
been seen in training. Extends the ProbDistI interface, requires a trigram
FreqDist instance to train on. Optionally, a different from default discount
value can be specified. The default discount is set to 0.75.
N<><00>?cCs<>|s|<01><00>|_n||_||_i|_tt<05>|_||_tt<08>|_ tt<08>|_
tt<08>|_ xp|D]h\}}}|j||f||||f7<|j ||fd7<|j
|d7<|j ||fd7<qVWdS)a<>
:param freqdist: The trigram frequency distribution upon which to base
the estimation
:type freqdist: FreqDist
:param bins: Included for compatibility with nltk.tag.hmm
:type bins: int or float
:param discount: The discount applied when retrieving counts of
trigrams
:type discount: float (preferred, but can be set to int)
r%N) r$r<><00>_D<5F>_cacherr.<00>_bigrams<6D> _trigramsr<73><00>_wordtypes_after<65>_trigrams_contain<69>_wordtypes_before)rr<>r,r<><00>w0<77>w1<77>w2rrrr<00>s  



 zKneserNeyProbDist.__init__c
Cs<>t|<01>dkrtd<02><01>t|<01>}|\}}}||jkr:|j|S||jkrf|j||<00><05>|j||f}nr||f|jkr<>||f|jkr<>|j||f}|j||f}||<00><05>|j||f}||j ||} || }nd}||j|<|SdS)N<>z$Expected an iterable with 3 members.g)
r#r6<00>tupler<65>r<>r<>r<>r<>r<>r<>)
rZtrigramr<6D>r<>r<>r<>ZaftrZbfrZ leftover_prob<6F>betarrrr<><00>s$ 





zKneserNeyProbDist.probcCs|jS)zq
Return the value by which counts are discounted. By default set to 0.75.
:rtype: float
)r<>)rrrrr<><00>szKneserNeyProbDist.discountcCs
||_dS)z<>
Set the value by which counts are discounted to the value of discount.
:param discount: the new value to discount counts by
:type discount: float (preferred, but int possible)
:rtype: None
N)r<>)rr<>rrr<00> set_discount<6E>szKneserNeyProbDist.set_discountcCs
|j<00><01>S)N)r<>r<>)rrrrr<00>szKneserNeyProbDist.samplescCs
|j<00><01>S)N)r<>r8)rrrrr8<00>szKneserNeyProbDist.maxcCsd<01>|j<01><02><00>S)zV
Return a string representation of this ProbDist
:rtype: str
z(<KneserNeyProbDist based on {0} trigrams)rvr<>r)rrrrrp<00>szKneserNeyProbDist.__repr__)Nr<4E>) r{r|r}r~rr<>r<>r<>rr8rprrrrr<><00>s 
#$
r<>cs4t<00>t<01>rt<00>t<01>std<01><01>t<03><00>fdd<03><08>D<00><01>S)Nzexpected a ProbDist.c3s*|]"}<01><00>|<01>t<01><02><01>|<01>d<00>VqdS)rCN)r<>r<>r<>)r&rE)<02> actual_pdist<73>
test_pdistrrr[<00>sz!log_likelihood.<locals>.<genexpr>)rhr<>r6r)r<>r<>r)r<>r<>r<00>log_likelihood<6F>sr<>cs*<00>fdd<02><08><00><00>D<00>}tdd<02>|D<00><01> S)Nc3s|]}<01><00>|<01>VqdS)N)r<>)r&rE)<01>pdistrrr[szentropy.<locals>.<genexpr>css|]}|t<00>|d<00>VqdS)rCN)r<>r<>)r&r<>rrrr[s)rr)r<>Zprobsr)r<>r<00>entropysrc@s<>eZdZdZd!dd<04>Zdd<06>Zdd<08>Zd d
<EFBFBD>Zd d <0C>Zd d<0E>Z dd<10>Z
dd<12>Z dd<14>Z dd<16>Z dd<18>Zdd<1A>Zdd<1C>Zdd<1E>Zdd <20>ZdS)"<22>ConditionalFreqDista<74>
A collection of frequency distributions for a single experiment
run under different conditions. Conditional frequency
distributions are used to record the number of times each sample
occurred, given the condition under which the experiment was run.
For example, a conditional frequency distribution could be used to
record the frequency of each word (type) in a document, given its
length. Formally, a conditional frequency distribution can be
defined as a function that maps from each condition to the
FreqDist for the experiment under that condition.
Conditional frequency distributions are typically constructed by
repeatedly running an experiment under a variety of conditions,
and incrementing the sample outcome counts for the appropriate
conditions. For example, the following code will produce a
conditional frequency distribution that encodes how often each
word type occurs, given the length of that word type:
>>> from nltk.probability import ConditionalFreqDist
>>> from nltk.tokenize import word_tokenize
>>> sent = "the the the dog dog some other words that we do not care about"
>>> cfdist = ConditionalFreqDist()
>>> for word in word_tokenize(sent):
... condition = len(word)
... cfdist[condition][word] += 1
An equivalent way to do this is with the initializer:
>>> cfdist = ConditionalFreqDist((len(word), word) for word in word_tokenize(sent))
The frequency distribution for each condition is accessed using
the indexing operator:
>>> cfdist[3]
FreqDist({'the': 3, 'dog': 2, 'not': 1})
>>> cfdist[3].freq('the')
0.5
>>> cfdist[3]['dog']
2
When the indexing operator is used to access the frequency
distribution for a condition that has not been accessed before,
``ConditionalFreqDist`` creates a new empty FreqDist for that
condition.
NcCs:t<00>|t<02>|r6x$|D]\}}|||d7<qWdS)aZ
Construct a new empty conditional frequency distribution. In
particular, the count for every sample, under every condition,
is zero.
:param cond_samples: The samples to initialize the conditional
frequency distribution with
:type cond_samples: Sequence of (condition, sample) tuples
r%N)rrr)rZ cond_samples<65>condr2rrrr<s
zConditionalFreqDist.__init__cs&<00>fdd<02><08><00><00>D<00>}<01>jddd|fS)Nc3s|]}|<01>|fVqdS)Nr)r&r)rrrr[Msz1ConditionalFreqDist.__reduce__.<locals>.<genexpr>r)<02>
conditionsr)rZkv_pairsr)rr<00>
__reduce__LszConditionalFreqDist.__reduce__cCs t|<00><01><00>S)aT
Return a list of the conditions that have been accessed for
this ``ConditionalFreqDist``. Use the indexing operator to
access the frequency distribution for a given condition.
Note that the frequency distributions for some conditions
may contain zero sample outcomes.
:rtype: list
)rMr<>)rrrrrPs
zConditionalFreqDist.conditionscCstdd<02>t|<00>D<00><01>S)z<>
Return the total number of sample outcomes that have been
recorded by this ``ConditionalFreqDist``.
:rtype: int
css|]}|<01><00>VqdS)N)r)r&<00>fdistrrrr[csz(ConditionalFreqDist.N.<locals>.<genexpr>)rr
)rrrrr\szConditionalFreqDist.Ncs<>yddlm}Wntk
r,td<03><01>YnXt|dd<05>}t|dd<05>}<05>fdd<08>t|d <09><02><05><00>D<00>}t|d
d <0B>}t|d tt<07>fd d<0E>|D<00><01><01><01>}d|kr<>d|d<|<03><08>} t |<06>dk<03>r<>g<00>xv|D]n<>|<04>r<0E><01>
t <0B><02><00> |<08><01><01>d}
d} |<05>r0<72>fdd<08><08>D<00><01>d<d}
qˆ<01>
<EFBFBD><00>fdd<08>|D<00><01>d}
d} q<>Wd} x4<78>D],} || |d<| d7} | j | f|<01>|<02><00>q>W| j| d<1B>| jddd<1E>| <09>tt |<08><01><01>| jdd<08>|D<00>d d!<21>|<07>r<>| <09>|<07>| <09>d"<22>| <09>|
<EFBFBD>|<03><16>| S)#a<>
Plot the given samples from the conditional frequency distribution.
For a cumulative plot, specify cumulative=True.
(Requires Matplotlib to be installed.)
:param samples: The samples to plot
:type samples: list
:param title: The title for the graph
:type title: str
:param conditions: The conditions to plot (default is all)
:type conditions: list
rNzPThe plot function requires matplotlib to be installed.See http://matplotlib.org/r:Fr;csg|]}|<01>kr|<01>qSrr)r&r<>)rrrr(|sz,ConditionalFreqDist.plot.<locals>.<listcomp>rrD<00>rc3s |]}<01>|D]
}|VqqdS)Nr)r&r<>r<>)rrrr[sz+ConditionalFreqDist.plot.<locals>.<genexpr>rBrCzCumulative Countsz lower rightcs$g|]}|<01>t<00><00>dd<00>qS)r%r<)r#)r&r=)r>rrr(<00>sr<>zCumulative Percentscsg|]}<01><01>|<00>qSrr)r&r2)<02> conditionrrrr(<00>sr?z upper right<68>labelr%)<01>locTr@)rAcSsg|] }t|<01><01>qSr)r )r&rErrrr(<00>srF)rGrH)rIrJrKr6rLrr<>rirNr#rwrMr3rQZlegendrOrRrSrTrPrUrVrW)rr r!rXr:r;rrDrrZrYZ
legend_locr`r5r)rr>rrrQesV     

  


zConditionalFreqDist.plotc sXt|dd<02>}t|dt<01><01><02><00><01>}t|dtt<03>fdd<06>|D<00><01><01><01>}tdd<06>|D<00><01>}t<05>}x\|D]T<>|r<>t<06><01><00>|<05><01>|<07><n<18><00>fdd <09>|D<00>|<07><t|td
d<06>|<07>D<00><01><01>}q`Wtd d<06>|D<00><01>}td |d d <0A>x |D]} td|| fd d <0A>q<>Wt<08>xP|D]H<>td|<08>fd d <0A>x&|<07>D]}
td||
fd d <0A><00>q*Wt<08><00>qWdS)a~
Tabulate the given samples from the conditional frequency distribution.
:param samples: The samples to plot
:type samples: list
:param conditions: The conditions to plot (default is all)
:type conditions: list
:param cumulative: A flag to specify whether the freqs are cumulative (default = False)
:type title: bool
r:Frrc3s(|] }|<01>kr<02>|D]
}|VqqdS)Nr)r&r<>r<>)rrrr[<00>sz/ConditionalFreqDist.tabulate.<locals>.<genexpr>css|]}td|<00>VqdS)z%sN)r#)r&rErrrr[<00>scsg|]}<01><01>|<00>qSrr)r&r2)r<>rrrr(<00>sz0ConditionalFreqDist.tabulate.<locals>.<listcomp>css|]}td|<00>VqdS)z%dN)r#)r&r=rrrr[<00>scss|]}td|<00>VqdS)z%sN)r#)r&r<>rrrr[<00>sr\)r]z%*sz%*dN) rLr<>rrir8r<>rMr3r^) rr r!r:rrr_r>Zcondition_sizerEr=r)r<>rrra<00>s*  
 

zConditionalFreqDist.tabulatecCs<>t|t<01>stSt<01>}x,|<00><03>D] }||||}|r|||<qWxJ|<01><03>D]>}||<00><03>krLx,||<00><04>D]\}}|dkrj||||<qjWqLW|S)z;
Add counts from two ConditionalFreqDists.
r)rhr<00>NotImplementedrry)rrd<00>resultr<00> newfreqdist<73>elemr0rrrrc<00>s
  zConditionalFreqDist.__add__cCs<>t|t<01>stSt<01>}x,|<00><03>D] }||||}|r|||<qWxN|<01><03>D]B}||<00><03>krLx0||<00><04>D] \}}|dkrjd||||<qjWqLW|S)zM
Subtract count, but keep only results with positive counts.
r)rhrr
rry)rrdr rr r r0rrrre<00>s
  zConditionalFreqDist.__sub__cCs<>t|t<01>stSt<01>}x,|<00><03>D] }||||B}|r|||<qWxJ|<01><03>D]>}||<00><03>krLx,||<00><04>D]\}}|dkrj||||<qjWqLW|S)zP
Union is the maximum of value in either of the input counters.
r)rhrr
rry)rrdr rr r r0rrrrf<00>s
  zConditionalFreqDist.__or__cCsFt|t<01>stSt<01>}x,|<00><03>D] }||||@}|r|||<qW|S)zF
Intersection is the minimum of corresponding counts.
)rhrr
r)rrdr rr rrrrgs
 zConditionalFreqDist.__and__csHt<00>t<01>std<01><01><00>t<03><01><04><00><01><05><00><04><00>oFt<06><00>fdd<03><08><01><04>D<00><01>S)Nz<=c3s|]}<01>|<00>|kVqdS)Nr)r&r<>)rdrrrr[sz-ConditionalFreqDist.__le__.<locals>.<genexpr>)rhrrrirrjrk)rrdr)rdrrrls
 zConditionalFreqDist.__le__cCs&t|t<01>std||<01>||ko$||kS)N<><)rhrr)rrdrrrr<>s
 zConditionalFreqDist.__lt__cCst|t<01>std||<01>||kS)Nz>=)rhrr)rrdrrrr!s
 zConditionalFreqDist.__ge__cCst|t<01>std||<01>||kS)N<>>)rhrr)rrdrrrr<>&s
 zConditionalFreqDist.__gt__cCs dt|<00>S)zf
Return a string representation of this ``ConditionalFreqDist``.
:rtype: str
z(<ConditionalFreqDist with %d conditions>)r#)rrrrrp+szConditionalFreqDist.__repr__)N)r{r|r}r~rrrrrQrarcrerfrgrlr<>rr<>rprrrrr s /
  C)rc@s,eZdZdZedd<03><00>Zdd<05>Zdd<07>ZdS) <09>ConditionalProbDistIao
A collection of probability distributions for a single experiment
run under different conditions. Conditional probability
distributions are used to estimate the likelihood of each sample,
given the condition under which the experiment was run. For
example, a conditional probability distribution could be used to
estimate the probability of each word type in a document, given
the length of the word type. Formally, a conditional probability
distribution can be defined as a function that maps from each
condition to the ``ProbDist`` for the experiment under that
condition.
cCsdS)zY
Classes inheriting from ConditionalProbDistI should implement __init__.
Nr)rrrrrDszConditionalProbDistI.__init__cCs t|<00><01><00>S)z<>
Return a list of the conditions that are represented by
this ``ConditionalProbDist``. Use the indexing operator to
access the probability distribution for a given condition.
:rtype: list
)rMr<>)rrrrrJszConditionalProbDistI.conditionscCsdt|<00>jt|<00>fS)zf
Return a string representation of this ``ConditionalProbDist``.
:rtype: str
z<%s with %d conditions>)<03>typer{r#)rrrrrpTszConditionalProbDistI.__repr__N)r{r|r}r~r rrrprrrrr4s 
rc@s eZdZdZdd<03>Zdd<05>ZdS)<07>ConditionalProbDista<74>
A conditional probability distribution modeling the experiments
that were used to generate a conditional frequency distribution.
A ConditionalProbDist is constructed from a
``ConditionalFreqDist`` and a ``ProbDist`` factory:
- The ``ConditionalFreqDist`` specifies the frequency
distribution for each condition.
- The ``ProbDist`` factory is a function that takes a
condition's frequency distribution, and returns its
probability distribution. A ``ProbDist`` class's name (such as
``MLEProbDist`` or ``HeldoutProbDist``) can be used to specify
that class's constructor.
The first argument to the ``ProbDist`` factory is the frequency
distribution that it should model; and the remaining arguments are
specified by the ``factory_args`` parameter to the
``ConditionalProbDist`` constructor. For example, the following
code constructs a ``ConditionalProbDist``, where the probability
distribution for each condition is an ``ELEProbDist`` with 10 bins:
>>> from nltk.corpus import brown
>>> from nltk.probability import ConditionalFreqDist
>>> from nltk.probability import ConditionalProbDist, ELEProbDist
>>> cfdist = ConditionalFreqDist(brown.tagged_words()[:5000])
>>> cpdist = ConditionalProbDist(cfdist, ELEProbDist, 10)
>>> cpdist['passed'].max()
'VBD'
>>> cpdist['passed'].prob('VBD')
0.423...
cOs<||_||_||_x$|D]}|||f|<03>|<04>||<qWdS)a<>
Construct a new conditional probability distribution, based on
the given conditional frequency distribution and ``ProbDist``
factory.
:type cfdist: ConditionalFreqDist
:param cfdist: The ``ConditionalFreqDist`` specifying the
frequency distribution for each condition.
:type probdist_factory: class or function
:param probdist_factory: The function or class that maps
a condition's frequency distribution to its probability
distribution. The function is called with the frequency
distribution as its first argument,
``factory_args`` as its remaining arguments, and
``factory_kw_args`` as keyword arguments.
:type factory_args: (any)
:param factory_args: Extra arguments for ``probdist_factory``.
These arguments are usually used to specify extra
properties for the probability distributions of individual
conditions, such as the number of bins they contain.
:type factory_kw_args: (any)
:param factory_kw_args: Extra keyword arguments for ``probdist_factory``.
N)<03>_probdist_factory<72> _factory_args<67>_factory_kw_args)rZcfdistZprobdist_factoryZ factory_argsZfactory_kw_argsrrrrrs 
zConditionalProbDist.__init__cCs$|jt<01>f|j<02>|j<03>||<||S)N)rrrr)rrrrr<00> __missing__<5F>szConditionalProbDist.__missing__N)r{r|r}r~rrrrrrr]s !rc@s eZdZdZdd<03>Zdd<05>ZdS)<07>DictionaryConditionalProbDistz<74>
An alternative ConditionalProbDist that simply wraps a dictionary of
ProbDists rather than creating these from FreqDists.
cCs|<00>|<01>dS)z<>
:param probdist_dict: a dictionary containing the probdists indexed
by the conditions
:type probdist_dict: dict any -> probdist
N)r)rZ probdist_dictrrrr<00>sz&DictionaryConditionalProbDist.__init__cCst<00>||<||S)N)r<>)rrrrrr<00>s
z)DictionaryConditionalProbDist.__missing__N)r{r|r}r~rrrrrrr<00>srg<><67><EFBFBD><EFBFBD>KH<4B>9rCcCsN||tkr|S||tkr |St||<01>}|t<02>d||d||d<01>S)a
Given two numbers ``logx`` = *log(x)* and ``logy`` = *log(y)*, return
*log(x+y)*. Conceptually, this is the same as returning
``log(2**(logx)+2**(logy))``, but the actual implementation
avoids overflow errors that could result from direct computation.
rC)<04>_ADD_LOGS_MAX_DIFF<46>minr<6E>r<>)ZlogxZlogy<67>baserrr<00>add_logs<67>s   
rcCs(t|<00>dkr$tt|dd<00>|d<00>StS)Nrr%)r#rrr<>)Zlogsrrrr<><00>sr<>c@s8eZdZdZdd<03>Zdd<05>Zdd<07>Zdd <09>Zd
d <0B>Zd S) <0A>ProbabilisticMixIna<6E>
A mix-in class to associate probabilities with other classes
(trees, rules, etc.). To use the ``ProbabilisticMixIn`` class,
define a new class that derives from an existing class and from
ProbabilisticMixIn. You will need to define a new constructor for
the new class, which explicitly calls the constructors of both its
parent classes. For example:
>>> from nltk.probability import ProbabilisticMixIn
>>> class A:
... def __init__(self, x, y): self.data = (x,y)
...
>>> class ProbabilisticA(A, ProbabilisticMixIn):
... def __init__(self, x, y, **prob_kwarg):
... A.__init__(self, x, y)
... ProbabilisticMixIn.__init__(self, **prob_kwarg)
See the documentation for the ProbabilisticMixIn
``constructor<__init__>`` for information about the arguments it
expects.
You should generally also redefine the string representation
methods, the comparison methods, and the hashing method.
cKsVd|kr,d|krtd<03><01>qRt<01>||d<00>n&d|krFt<01>||d<00>n d|_|_dS)a<>
Initialize this object's probability. This initializer should
be called by subclass constructors. ``prob`` should generally be
the first argument for those constructors.
:param prob: The probability associated with the object.
:type prob: float
:param logprob: The log of the probability associated with
the object.
:type logprob: float
r<>r<>z.Must specify either prob or logprob (not both)N)<06> TypeErrorr<00>set_prob<6F> set_logprob<6F>_ProbabilisticMixIn__prob<6F>_ProbabilisticMixIn__logprob)rr!rrrr<00>s 
zProbabilisticMixIn.__init__cCs||_d|_dS)z<>
Set the probability associated with this object to ``prob``.
:param prob: The new probability
:type prob: float
N)r r!)rr<>rrrr
szProbabilisticMixIn.set_probcCs||_d|_dS)z<>
Set the log probability associated with this object to
``logprob``. I.e., set the probability associated with this
object to ``2**(logprob)``.
:param logprob: The new log probability
:type logprob: float
N)r!r )rr<>rrrr s zProbabilisticMixIn.set_logprobcCs*|jdkr$|jdkrdSd|j|_|jS)z\
Return the probability associated with this object.
:rtype: float
NrC)r r!)rrrrr<> s


 zProbabilisticMixIn.probcCs.|jdkr(|jdkrdSt<02>|jd<02>|_|jS)z
Return ``log(p)``, where ``p`` is the probability associated
with this object.
:rtype: float
NrC)r!r r<>r<>)rrrrr<>, s


zProbabilisticMixIn.logprobN) r{r|r}r~rrrr<>r<>rrrrr<00>s 
  rc@seZdZdd<02>Zdd<04>ZdS)<06>ImmutableProbabilisticMixIncCstd|jj<00><01>dS)Nz%s is immutable)r6rr{)rr<>rrrr; sz$ImmutableProbabilisticMixIn.set_probcCstd|jj<00><01>dS)Nz%s is immutable)r6rr{)rr<>rrrr> sz'ImmutableProbabilisticMixIn.set_logprobN)r{r|r}rrrrrrr": sr"cCs ||kr||}||=n|}|S)Nr)r!r<00>default<6C>argrrrrLE s
rLcCsPt<00>}xDt|<01>D]8}t<02>dd|d<00>t<02>d|d<00>}||d7<qW|S)z<>
Create a new frequency distribution, with random samples. The
samples are numbers from 1 to ``numsamples``, and are generated by
summing two numbers, each of which has a uniform distribution.
r%rCr)rrSr<><00>randint)<05>
numsamples<EFBFBD> numoutcomesrr<>r<>rrr<00>_create_rand_fdistS s r(cCs^t<00>}xNtdd|dd<00>D]4}x.td|dd<00>D]}|||d7<q6WqWt|<01>S)zp
Return the true probability distribution for the experiment
``_create_rand_fdist(numsamples, x)``.
r%rCr)rrSr<>)r&rr<>r<>rrr<00>_create_sum_pdistc s
r)<00><00><>c sdt||<01>}t||<01>}t||<01>}t|<02>t|d|<00>t|||<00>t|||<00>t|||g|<00>t|<02>t|d<02>t|<00>g}g}x@td|d<00>D].<2E>|<06>t <09>|<02>
<EFBFBD><00>g<02>fdd<05>|D<00><00><01>q|Wt d|||f<00>t dt |<05>d<00>d d
t |<05>dd }t |t d d <0A>|dd<0F>D<00><01><00>t dt |<05>d<00>ddt |<05>dd}x|D]}t ||<00><00>q@Wt t|<06><00>} dd<05>| dd<0E>D<00>}
t dt |<05>d<00>ddt |<05>d}t |t |
<EFBFBD><00>t dt |<05>d<00>t d|<00>dk<00>r<>t d|<00>t d|<00>t d|<00>t <0B>t d<1B>xP|D]H<>t<0F>fdd <0A>td<1D>D<00><01>} t d<1E>jjdd<1F>d| dd <20>f<00><00>qWt <0B>dS)!aK
A demonstration of frequency distributions and probability
distributions. This demonstration creates three frequency
distributions with, and uses them to sample a random process with
``numsamples`` samples. Each frequency distribution is sampled
``numoutcomes`` times. These three frequency distributions are
then used to build six probability distributions. Finally, the
probability estimates of these distributions are compared to the
actual probability of each sample.
:type numsamples: int
:param numsamples: The number of samples to use in each demo
frequency distributions.
:type numoutcomes: int
:param numoutcomes: The total number of outcomes for each
demo frequency distribution. These outcomes are divided into
``numsamples`` bins.
:rtype: None
g<00>?<3F>r%csg|]}|<01><00><00><01>qSr)r<>)r&r<>)r4rrr(<00> szdemo.<locals>.<listcomp>z=%d samples (1-%d); %d outcomes were sampled for each FreqDistz =========rCz FreqDist z%8s z | Actualcss|]}t|<01>dd<01>VqdS)r%<00> N)<01>repr)r&r<>rrrr[<00> szdemo.<locals>.<genexpr>Nr<4E>z ---------z %3d %8.6f z%8.6f z| %8.6fcSsg|] }t|<01><01>qSr)r)r&rrrrr(<00> szTotal z%s<>Fz fdist1: %sz fdist2: %sz fdist3: %sz Generating:c3s|]}<01><00><00>VqdS)N)r<>)r&r`)r<>rrr[<00> si<>z%20s %s<><00>7)r(r<>r<>r<>r<>r<>r)rSrwr<>r5r^r#rMr<>rrr{) r&r'r<>r<>Zfdist3Zpdists<74>valsZ FORMATSTRrZzvalsZsumsrr)r4r<>r<00>demoo sP






."
    
.r3cCs<>ddlm}|j<02>d<03>}t|<01>}t|<02>}tdd<00>dd<07>t|<02><08>dd <09>d
d <0B>D<00>}x(|D] }td ||||<03> |<05>f<00>qZWdS) Nr)<01>corpuszausten-emma.txtz%18s %8s %14s)<03>wordZfreqencyZSimpleGoodTuringcss|]\}}|VqdS)Nr)r&r<00>valuerrrr[<00> szgt_demo.<locals>.<genexpr>cSs|dS)Nr%r)r'rrrrm<00> rnzgt_demo.<locals>.<lambda>T)r<00>reversez%18s %8d %14e)
<EFBFBD>nltkr4Z gutenberg<72>wordsrr<>r^r<>ryr<>)r4Z
emma_wordsr<EFBFBD>ZsgtZfd_keys_sortedrrrr<00>gt_demo<6D> s    
r:<00>__main__rq<00>i<>)r*r+)@r~<00>
__future__rrrr<>r<>r<>r<><00> collectionsrr<00> functoolsr<00>abcrr <00>sixr
r r r8r Znltk.internalsrr<>r<><00>python_2_unicode_compatibler<00>objectr<74>r<>r<>r<>r<>r<>r<>r<>r<>r<>r<>r<>r<>r<>r<>rrr<>rrrr<>rrr<>rr"rLr(r)r3r:r{<00>__all__rrrr<00><module>'s<>   g&4N.d$%%F`pw 
*
'J  ` 
P



Reference in New Issue Copy Permalink