Quickstart

Setup your account

  1. Go to the Benchify signup page and create an account.
  2. Connect your Github account to Benchify and authorize it to access whichever repositories you want it to review.
  3. Complete the signup process to view your (empty) dashboard. At this point, Benchify is set up and ready to review your code.

Create a pull request

Next time you create a pull request, Benchify will automatically review your code. Like this.
Or this.

In general, Benchify will “just work” if you have a requirements.txt in the root of your repository and running pip install -r requirements.txt && pytest . succeeds. If this is not the case, then Benchify will go through a cascading series of heuristics to try and run your code, including: scanning for a Pipfile, poetry.lock, pyproject.toml, or setup.py anywhere in your project, and installing the dependencies listed therein; scanning your repository for hints as to what PYTHONPATH should be set to and what Python version you’re using; and finally, running pytest in the root of your repository.

Benchify fails silently, which means that if it can’t build your code for some reason, it won’t comment on your PR or otherwise provide any sign of life other than the Check at the bottom of the PR. We can change this for you on request (email max@benchify.com) but for now, we think it’s nice to keep things quiet until we’re able to provide value. Behind the scenes, we read the stack trace from each run failure and then improve our service to fix the problems you run into, until you start getting reports on your PRs.

Safety Considerations

Benchify actually runs your code, and at a pretty reasonable scale (hundreds or even potentially thousands of individual function executions per review).
Hence, you should not push your secrets or real API keys.

Note that Benchify ignores any file containing the string This file is automatically generated. Do not edit.. In addition, Benchify will not test any file which is over 100,000 characters in length. For reference, the US constitution is about a quarter that length, and it’s a pretty long document. If you have an individual file that’s over 100,000 characters, you should probably refactor it anyway.