Commit Graph

21 Commits

Author SHA1 Message Date
Bilal
0772c2ba1b Add timeout on proxy connections 2020-05-22 16:08:59 +02:00
Bilal
5fb5d22ff2 Load proxy list before client connections and reload periodically 2020-05-22 16:00:25 +02:00
Senad Uka
491fbfd1f1 Merge branch 'skip-proxy-on-failure' into 'master'
Add custom proxies from ENV

See merge request saburly/kiviscraplib!4
2020-05-19 17:27:47 +00:00
Bilal
4779f0a91e handle invalid response from proxy-list; add custom proxies from ENV 2020-05-19 19:14:21 +02:00
Senad Uka
eb6bcb981d Merge branch 'reconnect-after-waiting-too-long' into 'master'
Reconnect after waiting too long for request

See merge request saburly/kiviscraplib!3
2020-05-14 07:37:51 +00:00
Bilal
a709ac2dc5 Reconnect after waiting too long for request 2020-05-13 18:23:56 +02:00
Senad Uka
e9b6aec79d Merge branch 'accept-url-as-base64-string' into 'master'
Accept URL as base64 string in scrape request

See merge request saburly/kiviscraplib!2
2020-05-12 11:56:07 +00:00
Bilal
3629e8442c Accept URL as base64 string in scrape request 2020-05-12 13:48:22 +02:00
Bilal Catic
c7268cd89e Merge branch 'implement-worker-client' into 'master'
Implement worker client

See merge request saburly/kiviscraplib!1
2020-05-08 10:18:10 +00:00
Bilal
451c3477bc Use config values instead of hardcoded values 2020-05-08 10:43:55 +02:00
Bilal
aa982dafb8 Use config values instead of hardcoded values 2020-05-08 10:42:11 +02:00
Bilal
9a2425d245 Update config with server configuration 2020-05-08 10:41:49 +02:00
Bilal
6d626ccecc Include https proxies 2020-05-08 09:51:09 +02:00
Bilal
e279dda637 Replace hardcoded values with config in worker client 2020-05-08 07:51:02 +02:00
Bilal
87b18c7f7e Handle config as struct with values from ENV file 2020-05-08 07:50:27 +02:00
Bilal
7e12d2819e Encode/decode request from load balancer server to the worker client 2020-05-08 06:32:04 +02:00
Bilal
a77b46aad6 Adapt worker server for base64 client response 2020-05-07 10:03:03 +02:00
Bilal
6011863525 Implement worker client 2020-05-07 10:02:49 +02:00
Senad Uka
0882dc4b15 Add worker server / move timeout to 100 seconds 2020-03-18 20:42:43 +01:00
Senad Uka
62f2e7c3fd Http server and structures ready 2020-03-16 22:24:37 +01:00
Senad Uka
203db78950 Initial commit 2020-03-14 07:27:39 +00:00