kiviscrapworker

Kivi scraping worker

Setup

Copy example.env to the .env and set desired values

Deployment procedure

1. Compile source code with go build

2. Transfer generated binary to the clients

   Command for sending files to the remote computer :

   scp -i path/to/the/key.pem path/to/the/binary username@host:/path/to/destination

   Path on the server where client resides is : /home/ubuntu/kivi_scraper/client

3. Add .env file next to the binary on the server and set correct values

4. If monit is not installed on the machine where client worker will be installed, install it (sudo apt install monit)

5. Create starth.sh with contents

    #!/bin/sh
   
    cd /home/ubuntu/kivi_scraper/client
    ./kiviscrapworker
   

6. Create stop.sh with contents

    #!/bin/sh
   
    cd /home/ubuntu/kivi_scraper/client
    sudo kill `cat sample.pid`
   

7. Edit monit configuration with sudo nano /etc/monit/monitrc

8. Around line 208 is example how to set process check. Add new line below example

9. Add to the file :

   check process kivi_scraper_worker_client pidfile /home/ubuntu/kivi_scraper/client/sample.pid
     start program = "/home/ubuntu/kivi_scraper/client/start.sh"
     stop program  = "/home/ubuntu/kivi_scraper/client/stop.sh"
     if 5 restarts within 5 cycles then timeout
   

10. Reboot with sudo reboot

Notes

• To check monit logs, check /var/log/monit.log