Commit Graph

610 Commits

Author SHA1 Message Date
=
bf8d131025 Increase timeout to 20 secs 2020-09-15 01:34:07 -07:00
=
698acb010a Add timeout to fetch wrapper 2020-09-15 01:27:20 -07:00
=
ade3eb307d Add try/catch to generator 2020-09-14 14:54:39 -07:00
=
8031f6f8a3 Moar attempts 2020-09-14 14:46:21 -07:00
=
d7a680a3ac Modify procfile instead of package.json for --inspect 2020-09-14 07:17:37 -07:00
=
8018caab47 Enable inspector and debugger 2020-09-14 06:53:35 -07:00
Edin Dazdarevic
d871d9ad1f More debugging, disable all crawlers 2020-09-13 23:45:08 +02:00
=
dfbefc20cd Remove PromisePool from olx 2020-09-13 04:57:01 -07:00
=
a481ecfe37 Debug 2020-09-13 04:48:11 -07:00
Senad Uka
8df94da48c Saljic fix fix 2020-09-11 05:03:37 +02:00
Senad Uka
d4fcd1950d Fix saljic 2020-09-11 04:51:54 +02:00
=
e8115a9215 moar fixes 2020-09-10 14:26:48 -07:00
=
160efdf6ab More fixes 2020-09-10 14:06:09 -07:00
=
c9b8c2e1a5 Fiks 2020-09-10 13:31:57 -07:00
Senad Uka
855b93ca41 Scraped data 2020-09-10 19:46:25 +02:00
Senad Uka
70779b24c0 Scraped data 2020-09-10 19:42:50 +02:00
Senad Uka
ba873f9f4e package json 2020-09-10 19:40:51 +02:00
Senad Uka
e4775158fc Promise pool 2020-09-10 19:39:13 +02:00
Senad Uka
26377c485c Reverting 2020-09-10 18:30:14 +02:00
Senad Uka
b30b0f45a6 trying different approach 2020-09-10 18:23:01 +02:00
Senad Uka
9c1a029ff1 Debugging ba 2020-09-10 18:17:16 +02:00
Senad Uka
9b49759485 Pause is not enough 2020-09-10 18:13:47 +02:00
Senad Uka
0c2f8d11ee Add random timeout up to 500ms 2020-09-10 18:10:42 +02:00
Senad Uka
b27d9d3499 Add debugging info 2020-09-10 18:00:13 +02:00
Senad Uka
dd3f30ef0e Comment agent header 2020-09-10 17:57:21 +02:00
Senad Uka
78c6056db4 Change for scraperapi 2020-09-10 17:37:19 +02:00
Bilal Catic
ecf27f2ba1 Merge branch 'improve-olx-scrapper' into 'master'
Improve olx scrapper

See merge request saburly/marketalarm/web!113
2020-06-16 20:37:37 +00:00
Bilal
1229b3fa6c Add more debug logs 2020-06-16 19:51:13 +02:00
Edin
542ff56123 Merge branch 'improve-wom' into 'master'
Improve styling of WOM message

See merge request saburly/marketalarm/web!112
2020-06-05 16:24:31 +00:00
=
0aa851015b Improve styling of WOM message 2020-06-05 09:07:54 -07:00
Senad Uka
c033b2e47c Merge branch 'word-of-mouth-email' into 'master'
Add word of mouth requests

See merge request saburly/marketalarm/web!111
2020-06-05 14:55:44 +00:00
Senad Uka
0895654db2 Update emailContentGenerator.js 2020-06-05 14:55:00 +00:00
Senad Uka
8925eb9f4e Update emailContentGenerator.js 2020-06-05 14:54:22 +00:00
=
52201af3ba Add word of mouth requests 2020-06-05 07:52:52 -07:00
Senad Uka
1505c07363 Merge branch 'improve-crawler' into 'master'
Improve crawler

See merge request saburly/marketalarm/web!110
2020-05-18 09:30:24 +00:00
Bilal
159fedbc2d handle failed page fetch 2020-05-18 03:53:08 +02:00
Bilal
65068932ad handle failed page fetch; detect discounted price 2020-05-18 03:43:49 +02:00
Senad Uka
820227827e Merge branch 'fix-crawler-errors' into 'master'
Fix crawler errors

See merge request saburly/marketalarm/web!109
2020-05-14 21:34:17 +00:00
Bilal
d35a113baa Fix saljic crawler 2020-05-14 19:01:19 +02:00
Bilal
ba60f8749d Fix Prostor crawler - use new JSON location in page body 2020-05-14 15:38:15 +02:00
Senad Uka
f1d45fed26 Merge branch 'move-scraper-url-to-the-env' into 'master'
Move scraper api url to the ENV

See merge request saburly/marketalarm/web!108
2020-05-12 11:55:12 +00:00
Bilal
ff923605ad Move scraper api base url to the ENV; send URL as base64 string 2020-05-12 13:44:09 +02:00
Naida Vatric
692577fb8c Merge branch 'after-scraper-fix' into 'master'
After scraper fix

See merge request saburly/marketalarm/web!104
2020-02-28 16:14:43 +00:00
Naida Vatric
df5e38092d Decomment and saljic smal fix. 2020-02-28 17:12:55 +01:00
Naida Vatric
feb2d04ed6 Olx debugg log. 2020-02-28 15:43:35 +01:00
Naida Vatric
90e171d07d Olx scraper debugging. 2020-02-28 15:41:20 +01:00
Naida Vatric
747f56941a Logged olx price scraping. 2020-02-28 14:21:53 +01:00
Naida Vatric
2a13ab55ed Merge branch 'after-scraper-fix' into 'master'
Olx price parsing changed.

See merge request saburly/marketalarm/web!103
2020-02-27 22:38:00 +00:00
Naida Vatric
441f905b29 Olx price parsing changed. 2020-02-27 23:35:18 +01:00
Naida Vatric
39f9383ae2 Merge branch 'after-scraper-fix' into 'master'
Commented checkup out and Prostor fetch changed.

See merge request saburly/marketalarm/web!102
2020-02-27 20:31:14 +00:00