This assumes you already have [homebrew](https://brew.sh) installed as well as an [ssh key added to your github account](https://help.github.com/articles/generating-an-ssh-key/).
This also assumes your username is pivotal, see the [circleci configuration](circleci.yml) for examples of how to add a pivotal user to postgres.
This installs python3, the [invoke utility](https://github.com/pyinvoke/invoke), xcode, node, imagemagick, redis, and postgres. It also sets up a virtual environment for the calculator, installs all the python dependencies into the virtual environment, and installs all the node dependencies.
Note that invoke should be installed after the virtual env is installed to ensure that the python 3.x version of invoke is used (instead of the system version, which is typically 2.7.x).
#### Linux/Debian-based systems
While most of this README assumes you're running macOS, for linux/debian based systems (e.g. ubuntu), follow these instructions. See the mac section for an explanation.
##### PostgreSQL
Postgres needs to be installed and configured before we do anything else.
# Modify /etc/postgresql/common/*/pg_hba.conf and set local connections to trust
sudo -u postgres createdb test
sudo -u postgres createdb pivotal
# create pivotal role in postgres, grant it access to the test/pivotal databases.
echo "CREATE ROLE pivotal WITH UNENCRYPTED PASSWORD 'password';" | psql -U postgres
echo "ALTER ROLE pivotal WITH LOGIN;" | psql -U postgres
echo "GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA test.public TO pivotal;" | psql -U postgres
echo "GRANT CREATE, CONNECT ON DATABASE test TO pivotal;" | psql -U postgres
echo "GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA pivotal.public TO pivotal;" | psql -U postgres
echo "GRANT CREATE, CONNECT ON DATABASE pivotal TO pivotal;" | psql -U postgres
```
This installs postgres and creates the database and roles that helix looks for in test/local usage. Because ubuntu is systemd, you can start/stop/restart the server by running something along the lines of `sudo systemctl [start|stop|restart] postgresql.service`. Please see [this ubuntu postgres documentation page](https://help.ubuntu.com/community/PostgreSQL) for more information.
The comment for the second line tells you how to enable connections to postgres without requiring a password. See http://dba.stackexchange.com/questions/83164/remove-password-requirement-for-user-postgres for more information
Note that it doesn't work to use the nodejs-provided phantom, nor does apt-get install phantomjs work. Yes, it's annoying.
### Adding Dependencies
Because of how we're separating production dependencies and test dependencies, it's recommended that each new dependency be manually added to either the [requirements.txt](requirements.txt) or the [requirements.test.txt](requirements.test.txt), depending on whether it's a production or test dependency, respectively.
### Adding database migrations
To manage the database, we use [SQLAlchemy](http://www.sqlalchemy.org/).
To manage database migrations, we use [Alembic](http://alembic.zzzcomputing.com/en/latest/).
First, make your changes to the sql models (in the models/sql) package to reflect your new vision of the sql schema. Next, we need to take those changes and generate an alembic migration from them. Run `PYTHONPATH=. alembic revision -m "(migration description)"` to generate that migration. Now, add the changes (and revert instructions!) to the generated migration file. Finally, run `invoke db_migrate` to run that migration.
##### Downgrading a migration
There currently is not an invoke task to revert migrations. Instead, just directly use the alembic cli to do this, something along the lines of `PYTHONPATH=.:$PYTHONPATH alembic downgrade -1`. Setting PYTHONPATH to include the current directory will ensure that the correct sql models get picked up when alembic does the migration. To redo a migration using alembic, run `PYTHONPATH=.:$PYTHONPATH alembic upgrade +1`.
Please refer to the [Alembic Documentation](http://alembic.zzzcomputing.com/en/latest/) for more information on using Alembic.
### Running Tests
First, install PhantomJS (`brew install phantomjs`) (this should already be done as part of the setup).
Run `invoke test` from the project directory will run every test.
Note that we use [Python's built-in unittest library](https://docs.python.org/3/library/unittest.html) for our python tests. Additionally, there are a few other helper libraries we use in tests. See the [requirements.test.txt](requirements.test.txt) file for those.
##### Running Javascript Tests
`invoke test` will, by default, run both the python tests and the javascript tests. Because the python tests can take a while, it's advantageous to be able to run the javascript tests (used for the array visualization) separately, especially when working entirely on the visualization frontend.
To do this, run `invoke test_js` from your command. This spins up an instance of [karma](https://karma-runner.github.io/0.13/index.html) attached to (by default) Chrome which watches the [spec](spec) and the [helix/javascript](helix/javascript) directories for changes, and (usually) re-runs the javascript tests if anything changes.
Note that we use [Jasmine](http://jasmine.github.io/2.4/introduction.html) for our javascript specs.
### Building JS
Part of the application (as of now only files in `helix/javascript/array_summary`) are built using Webpack and leveraging ES6.
To run the build use this command from the root of the repository
```bash
./node_modules/.bin/webpack
```
### Running Locally
First, install redis (`brew install redis`), and then (in a separate terminal) run `redis-server /usr/local/etc/redis.conf` to run redis locally.
Next, install postgres (`brew install postgres`), and then start it (`brew services start postgres`), follow the instructions at the top for additional configuration information.
Then, tell alembic to migrate your postgres db to the latest (`invoke db_migrate`).
Run `invoke serve` from the project directory to run locally.
This will run the project on [port 5000](http://localhost:5000/).
##### Running in debug mode
Run `invoke serve_debug` to start the project in debug mode.
##### The image tests
Note that you may see image comparison tests failing for erroneous reasons. These are flaky and machine-dependent. If you've done anything to the image generation code, please verify visually that you're getting the output you want.
### Circle CI
[CI (Continuous Integration)](https://en.wikipedia.org/wiki/Continuous_integration) is a service we use that automatically runs all the tests whenever a new commit is pushed. This is important because it runs the code in a context-free (the container the code runs in is set up for each run, then destroyed afterwards) environment. We're using [Circle CI](http://circleci.com/) here because Rachel recommended it. In addition to automatically running tests, circle also automatically deploys successful builds to cloud foundry for us.
### Deploying to Heroku (locally)
Install the Heroku CLI by running `brew install heroku` from your terminal.
Add heroku git endpoints for the different environments you want to push to. Typically, you'll just want to push to staging (especially seeing that preprod/production are automatically pushed to heroku for you). The heroku git endpoints are:
Finally, push to heroku using git! (Something like `git push heroku master`)
Note that circle ci does all that for us!
#### Deploying to preprod
To deploy to preprod, rebase master (up to the commit you want) onto preprod, then push preprod to github. CI will automatically deploy to preprod assuming that tests pass.
To push the current state of master to preprod:
```bash
git pull -r # Pull from origin (assuming origin is github)
git checkout preprod # switch to preprod branch
git rebase master # rebase head of master onto preprod
git push origin preprod # push preprod to github
```
To push a past commit to preprod:
```bash
git pull -r # Pull from origin (assuming origin is github)
git log
# [grab the sha of the commit you want]
git checkout preprod # switch to preprod branch
git rebase f00d # replace f00d with the sha of the commit you want
git push origin preprod # push preprod to github
```
#### Deploying to production
To deploy to production, tag a commit as `release-<number>` (for example, `release-2`) and push the commit (`git push origin release-2`). CI will automatically deploy to production assuming that tests pass.
Please increment the release count each time.
### Setting up PyCharm
[PyCharm](https://www.jetbrains.com/pycharm/) is an IDE from Jetbrains for python. It can be installed by running `brew cask install pycharm`.
Open pycharm, input a license, and select to install the command line tool. Then, open the project in pycharm by running `charm .` from the project directory.
Right now, you can start to use pycharm as an editor, but it's not very useful as an IDE, there's a bit more configuration stuff before it's useful as an IDE.
#### Use the correct Python
By default, PyCharm assumes you're using Python 2.7.x, we need to tell it to not only use 3.5.x, but use the 3.5.x for our local virtual environment.
To do this, open the preferences window, and select `Project: Helix_Roof_Calculator` (our project directory is named helix instead of Helix_Roof_Calculator, so the screenshots will be different than what you see), then select Project Intepreter and change that to `3.5.2 virtualenv at [projectdirectory]/env`.
Make sure to change the python tests configuration for nosetests and unittests (target selection menu (top-right) -> Edit Configurations -> Python Tests -> Nosetests) so that the working directory is the project directory (not the test directory!).
Additionally, you can set up an 'All Tests' target, just click the + button, name the target something like 'All Tests', and tell it to run all in folder, and point it at the test subdirectory of the project. Your configuration should look something like the attached screenshot
##### Running local server in PyCharm
Again, we need to edit the target configurations. This time add (or change, if it already exists) a python target named `helix`. It should be configured to run the `helix/main.py` script (on pivotal's machine, that field is filled with `/Users/pivotal/workspace/helix/helix/main.py`), with a working directory of the project directory (on pivotal's machine, that would be `/Users/pivotal/workspace/helix`). Additionally, the `FLASK_DEBUG` environment variable should be set to `1`. 
#### PhantomJS w/ PyCharm
You may need to ensure that phantomjs is in your PyCharm PATH. One way to do this is to symlink it into the project's env/bin folder
(for example `ln -s /usr/local/bin/phantomjs env/bin/`).
### Using Git
See [Github's documentation](https://help.github.com/). We prefer to commit directly to master, with small-ish commits that always have passing tests.
### Environment set-up using Docker
In order to set up complete environment using `docker-compose` you need to execute following line from project root directory:
```
docker-compose up -d
```
On Windows remember that repository should be inside Users dir. Additionally you need to set envvar:
export COMPOSE_CONVERT_WINDOWS_PATHS=1
Dependencies (including `node.js` packages) are resolved when building an image.
Environment will be exposed via ssh and accessible via following command:
```
ssh -p 2222 root@localhost
```
ssh password is: `screencast`
It can be seamlessly used via PyCharm by setting proper python interpreter:
