old-svevijesti/pyth/__pycache__/scrapingsingle.cpython-310.pyc

o
<00>f<EFBFBD>e<EFBFBD><00>@s<>dd
lm
Z

ddlZddlmZ
ddlmZ
ddlZddlm	Z	
ddl
mZmZm
Z
mZ
ddlZddlmZ
ddlZddlmZ
e<11>
e<0E>
e<07>d	<09>
Ze<06>Ze	<09>Zgd
<EFBFBD>
Zddi
Zd5dedefdd<11>Zdd<13>Zdd<15>Zdd<17>Z dd<19>Z!e"<22>Z#e"<22>Z$dd<1B>Z%e"<22>Z&eD]Z'e%e'e&<26>Z(e(r<>e#<23>)e(<28>

q<>dd<1D>e#D<00>
Z*e"e
<0A><00>
Z+e*e+Z,e,Z*e"e*<2A>
Z*e!e*<2A>
Z*e-dk<02>
ree*D]<5D>Z.e<02>/e.e<1A>Z0e
e0j1d<1F>Z2e2<65>3gd <20>
<01>
Z4d!<21>5d"d#<23>e4D<00>
<01>
Z6e2<65>3d$g
<01>
Z7d!<21>5d%d#<23>e7D<00>
<01>
Z8e8Z8e6Z6e e6<65>
Z6ee8<65>
Z8e ee8<65>
<01>
Z8ee8<65>
Z9e9d&kr<>ee6<65>
Z6zKej:j;j<d
d'd(d)<29>d*d+e6<65>d,e8<65>d-<2D>d)<29>gd.<2E>Z=e=j>dj?j@ZAeeA<65>
ZAe<0F>BeA<65>
ZCeCd/ZDeCd0Z1e<18>EeA<65>
ZFeeDe1e.eFd1d2<64><05>
sId3ZGeeDe1e.eFeG<65>
Wq<>eH<65>
yd
ZI
z
eJd4eI<65><00><02>

WYdZI[Iq<49>dZI[Iw
wdSdS)6<>)
<01>
BeautifulSoupN)
<01>urljoin)
<01>OpenAI)
<01>OpenAIEmbeddings)<04>insert_data<74>is_similar_data<74>
get_all_links<6B>	cleansing)
<01>load_dotenv)
<01>repair_json<6F>OPENAI_API_KEY)zhttps://klix.bazhttps://srpskainfo.comzhttps://bljesak.infoz
User-Agentz<74>Mozilla/5.0 (Linux; Android 5.1.1; SM-G928X Build/LMY47X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.83 Mobile Safari/537.36<EFBFBD>
gpt-3.5-turbo<62>string<6E>returncCst<00>
|
<01>
}t|<02>|<00>
<01>
S)
N)<04>tiktoken<65>encoding_for_model<65>len<65>encode)r<00>model<65>encoding<6E>r<00>7/home/asabani/Desktop/svevijesti/pyth/scrapingsingle.py<70>num_tokens_from_strings


rc
C<00>Hd
}
d}t<00>
|
<01>
}|<03>|<00>
}t|<04>
|k
r|g
S|d|<02>}|<03>|<05>
}|S)Nr
i<><00>rrrr<00>decode<64><07>text<78>
encoding_name<6D>
max_tokensr<00>tokens<6E>
sliced_tokens<6E>sliced_textrrr<00>slice_text_at_2k_tokens<00>











r#c
Cr)Nr
<00>drrrrr<00>slice_title_if_needed'r$r&c
s d
<01>d<02><00>f
dd<04>|D<00>
<01>
}
|
S)NuYABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzČčĆćDždžĐ𩹮ž0123456789 <20>c
3s <00>|]}
|
<01>vr
|
n
dV
qd
S)<02>
 Nr)<02>.0<EFBFBD>char<61>
<01>
allowed_charsrr<00>	<genexpr>4s<02>z&replace_with_spaces.<locals>.<genexpr>)
<01>join)r<00>cleaned_textrr+r<00>replace_with_spaces2s


r0c
Cs>t<00>}
|D]}d
|vr|<02>
dd<03>}|
<01>|<03>

q|
<01>|<02>

q|
S)N<>wwwzwww.r')<03>set<65>replace<63>add)<04>	links_set<65>modified_links<6B>link<6E>
modified_linkrrr<00>	fix_links7s





r9c
Cs<>t<00>
|t<02>}|jd
kr@t|jd<02>}|<03>d<03>
}g}|D]#}|jddd<06>}|D]}t||d<00>}	|	|
v
r<|<05>|	<09>

|
<01>	|	<09>

q%q|SdS)N<><4E><00>html.parser<65>article<6C>
aT)
<01>hrefr>)
<EFBFBD>requests<74>get<65>headers<72>status_coderr<00>find_allr<00>appendr4)
<EFBFBD>url<72>already_checked<65>response<73>soup<75>articles<65>
link_storer<<00>linksr7<00>
link_valuerrr<00>get_article_linksDs 














<02><02><04>rMc
Csh|]}
|
r|
<01>qSrr)r)<00>itemrrr<00>	<setcomp>ZsrO<00>__main__r;)<03>h2<68>h1<68>h3r(c
C<00>g|]}
|
jdd
<01>
<01>qS<00>T)
<01>strip<69>
<01>get_text)r)<00>titlerrr<00>
<listcomp>i<00>rZ<00>
pc
CrTrUrW)r)rrrrrZlr[il<00>systemz+Data analytic, Journalist and News reporter)<02>role<6C>content<6E>userz>Extract relevant information from the following input: Title: z, Text: z<>. Remove any non-news element related to the current text and title, and provide the cleaned data make sure that its on Bosnian language and valid JSON object with 'title' field and 'content' field.)r<00>messagesrYr_g\<5C><><EFBFBD>(\<5C>?)
<01>	threshold<6C>NOzError in completion: )
r
)K<>bs4rr?<00>urllib.parser<00>openair<00>os<6F>langchain.embeddingsr<00>vectDatarrrr	<00>json<6F>dotenvr
r<00>json_repairr<00>getenvr<00>client<6E>
embeddings<EFBFBD>dlinksrA<00>str<74>intrr#r&r0r9r2<00>total_links<6B>collected_newsrMrF<00>dlink<6E>
temp_links<EFBFBD>update<74>final_links<6B>db_links<6B>	new_links<6B>__name__r7r@rGrrHrC<00>titlesr.<00>
title_text<EFBFBD>texts<74>	text_text<78>ttk<74>chat<61>completions<6E>create<74>
completion<EFBFBD>choices<65>messager_<00>generated_text<78>loads<64>
response_datarY<00>embed_query<72>vector<6F>	similar_d<5F>	Exception<6F>
e<>printrrrr<00><module>
s<>




















<02>

















<02><06>






<04>

<08><02><04>