senaduka/old-kiviscraplib
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement worker client #1

Merged
bilal.catic merged 9 commits from implement-worker-client into master 2020-05-08 12:18:10 +02:00
Conversation 11 Commits 9 Files Changed 5 +346 -11
bilal.catic commented 2020-05-07 10:07:15 +02:00 (Migrated from gitlab.com)
Copy Link

Implemented for now :

  • Create N connections to the worker server
  • Wait for request message containing URL for scraping
  • When request message is received, fetch list of SOCKS5 and HTTPS proxies, select one randomly and fetch requested URL through selected proxy
Implemented for now : * Create N connections to the worker server * Wait for request message containing URL for scraping * When request message is received, fetch list of SOCKS5 and HTTPS proxies, select one randomly and fetch requested URL through selected proxy
senaduka commented 2020-05-07 11:19:49 +02:00 (Migrated from gitlab.com)
Copy Link

Treba i u serveru i u klijentu uraditi base64 encoding ovog requesta ja sam to bio krenuo ali nisam pushao

Treba i u serveru i u klijentu uraditi base64 encoding ovog requesta ja sam to bio krenuo ali nisam pushao
bilal.catic commented 2020-05-08 06:32:59 +02:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

  • 7e12d281 - Encode/decode request from load balancer server to the worker client

Compare with previous version

added 1 commit <ul><li>7e12d281 - Encode/decode request from load balancer server to the worker client</li></ul> [Compare with previous version](/saburly/kiviscraplib/-/merge_requests/1/diffs?diff_id=89467567&start_sha=a77b46aad64fa5885a5bb84325c01f7cf8fce666)
bilal.catic commented 2020-05-08 07:51:38 +02:00 (Migrated from gitlab.com)
Copy Link

changed this line in version 3 of the diff

changed this line in [version 3 of the diff](/saburly/kiviscraplib/-/merge_requests/1/diffs?diff_id=89473475&start_sha=7e12d2819e8e6512e8b2850efb0b1ac5d8c79930#8384cd7bb107cd1a7e1929940b5fe5fb8cb7261e_63_60)
bilal.catic commented 2020-05-08 07:51:38 +02:00 (Migrated from gitlab.com)
Copy Link

added 2 commits

  • 87b18c7f - Handle config as struct with values from ENV file
  • e279dda6 - Replace hardcoded values with config in worker client

Compare with previous version

added 2 commits <ul><li>87b18c7f - Handle config as struct with values from ENV file</li><li>e279dda6 - Replace hardcoded values with config in worker client</li></ul> [Compare with previous version](/saburly/kiviscraplib/-/merge_requests/1/diffs?diff_id=89473475&start_sha=7e12d2819e8e6512e8b2850efb0b1ac5d8c79930)
bilal.catic commented 2020-05-08 09:51:45 +02:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

Compare with previous version

added 1 commit <ul><li>6d626cce - Include https proxies</li></ul> [Compare with previous version](/saburly/kiviscraplib/-/merge_requests/1/diffs?diff_id=89488510&start_sha=e279dda637602eb1e230d7054df0f88ecdf3293c)
bilal.catic commented 2020-05-08 10:42:49 +02:00 (Migrated from gitlab.com)
Copy Link

added 3 commits

  • 9a2425d2 - Update config with server configuration
  • aa982daf - Use config values instead of hardcoded values
  • 1bb83ac2 - Load client config when starting client daemon

Compare with previous version

added 3 commits <ul><li>9a2425d2 - Update config with server configuration</li><li>aa982daf - Use config values instead of hardcoded values</li><li>1bb83ac2 - Load client config when starting client daemon</li></ul> [Compare with previous version](/saburly/kiviscraplib/-/merge_requests/1/diffs?diff_id=89496589&start_sha=6d626ccecc3e0b443a6defb076cc2b4b156b0847)
bilal.catic commented 2020-05-08 10:44:26 +02:00 (Migrated from gitlab.com)
Copy Link

added 1 commit

  • 451c3477 - Use config values instead of hardcoded values

Compare with previous version

added 1 commit <ul><li>451c3477 - Use config values instead of hardcoded values</li></ul> [Compare with previous version](/saburly/kiviscraplib/-/merge_requests/1/diffs?diff_id=89496841&start_sha=1bb83ac2e86876c86230dcb537a9c161edf6b650)
bilal.catic commented 2020-05-08 10:47:22 +02:00 (Migrated from gitlab.com)
Copy Link

unmarked as a Work In Progress

unmarked as a **Work In Progress**
bilal.catic commented 2020-05-08 10:47:22 +02:00 (Migrated from gitlab.com)
Copy Link

changed the description

changed the description
bilal.catic commented 2020-05-08 12:18:11 +02:00 (Migrated from gitlab.com)
Copy Link

merged

merged
bilal.catic commented 2020-05-08 12:18:11 +02:00 (Migrated from gitlab.com)
Copy Link

mentioned in commit c7268cd89e

mentioned in commit c7268cd89eaeac2370f022715c225b48bf4921ee
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
senaduka
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: senaduka/old-kiviscraplib#1
Delete Branch "implement-worker-client"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?