senaduka/old-web
1
0
Fork 0
Code Issues Pull Requests 5 Actions Packages Projects Releases Wiki Activity

Saljic crawler changed substring call. #95

Merged
RabbIT09-n merged 2 commits from saljic-crawler-fix into master 2020-02-26 16:09:29 +01:00
Conversation 11 Commits 2 Files Changed 1 +19 -18
RabbIT09-n commented 2020-02-14 23:47:16 +01:00 (Migrated from gitlab.com)
Copy Link

Changed places where substring is called on undefined - possibly. Need to check on staging crawler.

Changed places where substring is called on undefined - possibly. Need to check on staging crawler.
edazdarevic commented 2020-02-17 10:17:04 +01:00 (Migrated from gitlab.com)
Copy Link

@RabbIT09-n Kontam zasto uopste parsiramo agency id kada znamo da je u pitanju samo jedna agencija?

@RabbIT09-n Kontam zasto uopste parsiramo agency id kada znamo da je u pitanju samo jedna agencija?
edazdarevic commented 2020-02-17 11:05:54 +01:00 (Migrated from gitlab.com)
Copy Link

Imas li primjer kako izgleda taj URL obicno i kako je moguce da bude null?

Imas li primjer kako izgleda taj URL obicno i kako je moguce da bude `null`?
edazdarevic commented 2020-02-17 11:07:31 +01:00 (Migrated from gitlab.com)
Copy Link

Mozes li malo pojasniti kako ovo parsiranje radi? Da li je bolje koristiti regex u ovom slucaju?

Mozes li malo pojasniti kako ovo parsiranje radi? Da li je bolje koristiti regex u ovom slucaju?
RabbIT09-n commented 2020-02-17 15:53:48 +01:00 (Migrated from gitlab.com)
Copy Link

Parsiramo agencyObjectId a to je broj tj id pod kojim se nekretnina vodi u agenciji. Jeste uvijek Saljic agencija ali npr je kod njih https://www.saljicnekretnine.ba/v2/nekretnina/746 ovo je nekretnina 746 i to parse. Eh nisam ja sigurna da je ovaj substring bacao onu gresku sto je Senad nasao jer ima jos substring poziva nize. Kad god sam gledala log nije mi se pojavljivala greska. Pa sam sad u ovom commit-u dodala da loguje kad god scrape radi neke nekretnine da kad deployamo na staging jos malo vidim sta nije ok tj da li je to url ili neceg drugog. Mozda i jos logova treba dodati...

Parsiramo agencyObjectId a to je broj tj id pod kojim se nekretnina vodi u agenciji. Jeste uvijek Saljic agencija ali npr je kod njih https://www.saljicnekretnine.ba/v2/nekretnina/746 ovo je nekretnina 746 i to parse. Eh nisam ja sigurna da je ovaj substring bacao onu gresku sto je Senad nasao jer ima jos substring poziva nize. Kad god sam gledala log nije mi se pojavljivala greska. Pa sam sad u ovom commit-u dodala da loguje kad god scrape radi neke nekretnine da kad deployamo na staging jos malo vidim sta nije ok tj da li je to url ili neceg drugog. Mozda i jos logova treba dodati...
RabbIT09-n commented 2020-02-17 16:03:38 +01:00 (Migrated from gitlab.com)
Copy Link

Ovo bi trebalo da nadje lon i lat iz linka sa google mapa, taj link bude npr. `


`

Jedino gdje sam nasla da procitam lat i long su kod ovog marker dijela linka is src atributa. Eh sad provjerim da li moze nekako elegantnije da se ovo izdvoji :)

Ovo bi trebalo da nadje lon i lat iz linka sa google mapa, taj link bude npr. `<div class="gmap bottommargin" style="min-height: 360px; width:100%;"> <iframe src="https://www.openstreetmap.org/export/embed.html?bbox=18.19220066070557%2C43.8330408429302%2C18.236403465271%2C43.85882283079319&amp;layer=mapnik&amp;marker=43.845948705143044%2C18.21430206298828" width="100%" height="360" frameborder="0" webkitAllowFullScreen mozallowfullscreen allowFullScreen></iframe> </div>` Jedino gdje sam nasla da procitam lat i long su kod ovog marker dijela linka is src atributa. Eh sad provjerim da li moze nekako elegantnije da se ovo izdvoji :)
RabbIT09-n commented 2020-02-17 16:06:30 +01:00 (Migrated from gitlab.com)
Copy Link

Ne znam gdje mi je nestoa dio poruke:
src="https://www.openstreetmap.org/export/embed.html?bbox=18.19220066070557%2C43.8330408429302%2C18.236403465271%2C43.85882283079319&layer=mapnik&marker=43.845948705143044%2C18.21430206298828"

Ne znam gdje mi je nestoa dio poruke: src="https://www.openstreetmap.org/export/embed.html?bbox=18.19220066070557%2C43.8330408429302%2C18.236403465271%2C43.85882283079319&amp;layer=mapnik&amp;marker=43.845948705143044%2C18.21430206298828"
RabbIT09-n commented 2020-02-17 21:55:43 +01:00 (Migrated from gitlab.com)
Copy Link

changed this line in version 2 of the diff

changed this line in [version 2 of the diff](/saburly/marketalarm/web/-/merge_requests/95/diffs?diff_id=75916923&start_sha=addd8c134490bad63557a8762a93a8810f230e9c#82687aaefc2a467eb955a1337d91a76ac9a35ebe_288_277)
RabbIT09-n commented 2020-02-17 21:55:43 +01:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

  • f56cd5b5 - More elegant scrape of lat and long.

Compare with previous version

added 1 commit <ul><li>f56cd5b5 - More elegant scrape of lat and long.</li></ul> [Compare with previous version](/saburly/marketalarm/web/-/merge_requests/95/diffs?diff_id=75916923&start_sha=addd8c134490bad63557a8762a93a8810f230e9c)
RabbIT09-n commented 2020-02-17 22:00:41 +01:00 (Migrated from gitlab.com)
Copy Link

Promijenila sam kod tako da malo elegantnije scrape-a lat i long za nekretninu sa mape. I mislim da na tome pada ovo gore substring a ne na url-u. Pada im mapa ova neka koju koriste npr meni trenutno ne radi i onda je ne moze scrape Screenshot_from_2020-02-17_21-59-58

Promijenila sam kod tako da malo elegantnije scrape-a lat i long za nekretninu sa mape. I mislim da na tome pada ovo gore substring a ne na url-u. Pada im mapa ova neka koju koriste npr meni trenutno ne radi i onda je ne moze scrape ![Screenshot_from_2020-02-17_21-59-58](/uploads/b068b845ebb2c5d96f062f5e2fca87bd/Screenshot_from_2020-02-17_21-59-58.png)
RabbIT09-n commented 2020-02-26 16:09:29 +01:00 (Migrated from gitlab.com)
Copy Link

merged

merged
RabbIT09-n commented 2020-02-26 16:09:29 +01:00 (Migrated from gitlab.com)
Copy Link

mentioned in commit 44402a9cc4

mentioned in commit 44402a9cc46307a72a1ab588104b930808411412
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
senaduka
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: senaduka/old-web#95
Delete Branch "saljic-crawler-fix"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?