Commit Graph

48 Commits

Author SHA1 Message Date
Bilal Catic
cff7cc2e9c apply prettier 2020-02-20 19:46:39 +01:00
Naida Vatric
5bdc8e149a Prostor VIP ads fixed. 2020-02-14 22:41:51 +01:00
Naida Vatric
b82134e280 Fixed saljic bug for heroku. 2020-02-09 19:09:00 +01:00
Naida Vatric
be378883c8 Just another fix try. 2020-02-08 00:47:00 +01:00
Naida Vatric
8a87b9e253 Another fix. 2020-02-08 00:27:26 +01:00
Naida Vatric
43bc23b164 Another fix. Defined more var. 2020-02-07 22:27:01 +01:00
Naida Vatric
fc6351af46 Added columns and logs for types. 2020-02-07 22:12:53 +01:00
Naida Vatric
97724a47a1 Removed debugg logs. Smal fixes. 2020-02-06 22:40:56 +01:00
Naida Vatric
d29b3eb1b3 Merge branch 'crawler-saljicnekretnine' into 'master'
Crawler saljicnekretnine

See merge request saburly/marketalarm/web!90
2020-02-04 13:09:10 +00:00
Naida Vatric
1ba7cf8531 Added crawler for Saljic nekretnine. 2020-01-31 22:03:39 +01:00
Naida Vatric
7a7aecb3ee WIP Scraped no of rooms, floors etc. 2020-01-31 00:55:24 +01:00
Naida Vatric
78c4054cde WIP Scraped title, price and location. 2020-01-30 16:24:34 +01:00
Naida Vatric
94ffc2d6d2 WIP Started saljic crawler. 2020-01-29 23:22:39 +01:00
Naida Vatric
b11f18696f Prepared config files. 2020-01-29 01:09:53 +01:00
Naida Vatric
511b290096 Login to prostor.ba befoure crawl. 2020-01-13 12:05:33 +01:00
Naida Vatric
ba43fa0713 WIP Changed cookies. 2020-01-13 11:02:26 +01:00
Naida Vatric
e70901d369 WIP Changed login to crawler. 2020-01-13 09:12:03 +01:00
Naida Vatric
8505282670 WiP Login of crawler prostor. 2020-01-12 01:22:50 +01:00
Naida Vatric
64e4835899 Changed redirecting for VIP ads. 2020-01-10 22:52:50 +01:00
Naida Vatric
1658325c4b WIP Fake vip ads. 2020-01-10 19:20:26 +01:00
Naida Vatric
49161c1b60 WIP Changed redirecting for VIP ads. 2020-01-09 12:19:19 +01:00
Naida Vatric
0c2d218d29 Changed floor numbers and basement-attic tag. 2020-01-02 00:10:31 +01:00
Naida Vatric
fed2dc00dc Changed number of rooms. 2019-12-29 23:42:39 +01:00
Bilal Catic
af42d2c448 improve OLX ad status detection 2019-11-14 08:47:48 +01:00
Bilal Catic
5148f88a62 improve Rental and Aktido ad status detection 2019-11-14 08:31:57 +01:00
Bilal Catic
a7cd75653d improve OLX ad status detection 2019-11-14 08:04:58 +01:00
Bilal Catic
168b2186e7 add more fields to the Prostor real estates crawler 2019-11-14 07:23:23 +01:00
Bilal Catic
c13857bc09 add additional fields to the Prostor crawler 2019-11-14 02:09:42 +01:00
Bilal Catic
3b3e2eda07 refactor Prostor crawler 2019-11-13 16:54:16 +01:00
Bilal Catic
a63671959b improve real estate properties detection for Rental 2019-11-12 22:53:16 +01:00
Bilal Catic
b6d68db3a3 improve real estate properties detection for aktido 2019-11-12 21:39:28 +01:00
Bilal Catic
c91e56c46e add additional real estate fields for Aktido crawler 2019-11-11 19:34:43 +01:00
Bilal Catic
e871550ba6 add two more heating types for Rental crawler 2019-11-11 18:46:01 +01:00
Bilal Catic
debdd01b28 add new fields to the Rental crawler 2019-11-11 17:15:46 +01:00
Bilal Catic
b6024af2cb add new fields for OLX crawler 2019-11-08 17:05:51 +01:00
Bilal Catic
e3e0ddd508 stop logging scrape action for Rental crawler 2019-11-01 01:02:45 +01:00
Bilal Catic
2e3ddbac95 fix request ad type bug 2019-11-01 00:01:02 +01:00
Bilal Catic
4318fa8a2d extend AD_TYPE enum in specific crawler files 2019-10-31 19:06:44 +01:00
Bilal Catic
ecc5b174a0 implement RENT option for Aktido; implement force crawl option 2019-10-30 17:23:43 +01:00
Bilal Catic
fa712ce97d implement RENT option for Rental; implement force crawl option 2019-10-30 15:53:11 +01:00
Bilal Catic
3abbed183e implement RENT and REQUEST option for OLX; implement force crawl option 2019-10-30 15:03:59 +01:00
Bilal Catic
1e36cb8423 add ALL category option for Rental agency 2019-10-28 09:24:08 +01:00
Bilal Catic
2c2fcd648f remove scrapeAd logging 2019-10-28 09:23:51 +01:00
Bilal Catic
5b6886f52b add ALL categories option for Aktido agency 2019-10-28 09:20:03 +01:00
Bilal Catic
f899c96dc6 add crawler and crawler config for Aktido agency 2019-10-28 09:14:45 +01:00
Bilal Catic
747ebb88e5 add debugging log switch for crawler process 2019-10-25 11:08:52 +02:00
Bilal Catic
7e3b0bfcd5 implement crawler for Prostor agency 2019-10-25 10:54:08 +02:00
Bilal Catic
935ae60ae1 move specific crawler config to the separated files 2019-10-24 16:57:23 +02:00