30 lines
3.3 KiB
Plaintext
30 lines
3.3 KiB
Plaintext
|
B
|
|||
|
|
#U<>]<5D> <00> @ s0 d dl Z d dlZddlmZ G dd<05> de<04>ZdS )<06> N<> )<01> ProbingStatec @ sn e Zd ZdZddd<04>Zdd<06> Zedd<08> <00>Zd d
|
|||
|
|
<EFBFBD> Zed d <0C> <00>Z d
d<0E> Z
|
|||
|
|
e dd<10> <00>Z e dd<12> <00>Z
e dd<14> <00>ZdS )<16>
CharSetProbergffffff<66>?Nc C s d | _ || _t<02>t<04>| _d S )N)<06>_state<74> lang_filter<65>logging<6E> getLogger<65>__name__<5F>logger)<02>selfr <00> r <00>:/tmp/pip-install-zuyyrlx8/chardet/chardet/charsetprober.py<70>__init__' s zCharSetProber.__init__c C s t j| _d S )N)r <00> DETECTINGr )r r r r
<00>reset, s zCharSetProber.resetc C s d S )Nr )r r r r
<00> charset_name/ s zCharSetProber.charset_namec C s d S )Nr )r <00>bufr r r
<00>feed3 s zCharSetProber.feedc C s | j S )N)r )r r r r
<00>state6 s zCharSetProber.statec C s dS )Ng r )r r r r
<00>get_confidence: s z CharSetProber.get_confidencec C s t <00>dd| <00>} | S )Ns ([ -])+<2B> )<02>re<72>sub)r r r r
<00>filter_high_byte_only= s z#CharSetProber.filter_high_byte_onlyc C s` t <00> }t<01>d| <00>}xH|D ]@}|<01>|dd<03> <00> |dd<02> }|<04><04> sN|dk rNd}|<01>|<04> qW |S )u9
|
|||
|
|
We define three types of bytes:
|
|||
|
|
alphabet: english alphabets [a-zA-Z]
|
|||
|
|
international: international characters [-ÿ]
|
|||
|
|
marker: everything else [^a-zA-Z-ÿ]
|
|||
|
|
|
|||
|
|
The input buffer can be thought to contain a series of words delimited
|
|||
|
|
by markers. This function works to filter all words that contain at
|
|||
|
|
least one international character. All contiguous sequences of markers
|
|||
|
|
are replaced by a single space ascii character.
|
|||
|
|
|
|||
|
|
This filter applies to all scripts which do not use English characters.
|
|||
|
|
s% [a-zA-Z]*[<5B>-<2D>]+[a-zA-Z]*[^a-zA-Z<>-<2D>]?N<><4E><EFBFBD><EFBFBD><EFBFBD><EFBFBD> <00>r )<05> bytearrayr <00>findall<6C>extend<6E>isalpha)r <00>filtered<65>words<64>word<72> last_charr r r
<00>filter_international_wordsB s
|
|||
|
|
z(CharSetProber.filter_international_wordsc C s<> t <00> }d}d}x~tt| <00><01>D ]n}| ||d <00> }|dkr>d}n |dkrJd}|dk r |<05><03> s ||kr<>|s<>|<01>| ||<04> <00> |<01>d<08> |d }q W |s<>|<01>| |d <09> <00> |S )
|
|||
|
|
a<EFBFBD>
|
|||
|
|
Returns a copy of ``buf`` that retains only the sequences of English
|
|||
|
|
alphabet and high byte characters that are not between <> characters.
|
|||
|
|
Also retains English alphabet and high byte characters immediately
|
|||
|
|
before occurrences of >.
|
|||
|
|
|
|||
|
|
This filter can be applied to all scripts which contain both English
|
|||
|
|
characters and extended ASCII characters, but is currently only used by
|
|||
|
|
``Latin1Prober``.
|
|||
|
|
Fr r <00> ><3E> <Tr r N)r <00>range<67>lenr r )r r <00>in_tag<61>prev<65>curr<72>buf_charr r r
<00>filter_with_english_lettersg s"
|
|||
|
|
z)CharSetProber.filter_with_english_letters)N)r <00>
|
|||
|
|
__module__<EFBFBD> __qualname__<5F>SHORTCUT_THRESHOLDr r <00>propertyr r r r <00> staticmethodr r$ r- r r r r
r # s
|
|||
|
|
%r )r r <00>enumsr <00>objectr r r r r
<00><module> s
|