Quantum software project evaluation

Most recent results of our monthly evaluation of curated open quantum software projects.


Become a supporter

The evaluation is not completely automated yet. Current evaluation results are identical to the results reported in the PLoS ONE collection review on "Open source software in quantum computing". We're currently working hard on fully automating the evaluation such that we can publish monthly updates soon.

Project selection

We only evaluate a subset of the projects presented in our exhaustive list of open quantum software repositories. The following decision tree outlines the process which is used to select free and open software projects for evaluation. The main goal is to identify projects that have already built a community around their project. The acronym PR stands for pull request which is a form of code contribution on software hosting websites.

Flow diagram showing quantum project selection


These rules are not set in stone and we're always open for suggestions. If you have some thoughts, comments or concerns, please open an issue on our GitHub page.

Repository evaluation

Evaluation results for the static analysis of each project and its source code. We report the version control and issue tracking systems as well as the total number, attention rate and average response time for all open and closed issues and pull requests (PRs). We define attention rate as 1-I where I is the fraction of ignored issues and pull requests with respect to the total number of issues and pull requests. An ideal project never ignores any of its user or developer questions or contributions and would have an attention rate of 1.0. The average response time measures how long it takes a core contributor (or an employee of the company hosting the project) to respond to issues or pull requests. Next, we analyze the existence of a test suite and report the resulting code coverage for most projects. Code complexity is only reported for projects written in Python since other languages do not allow for fast retrieval of this metric (if you know how to do this please let us know through a GitHub issue or email).

Name Version control system Issue tracking system Issues/PRs Attention rate Average response time (days) Test suite Code coverage Cyclomatic Complexity
Cirq Git GitHub 448/686 0.54 2.6 94% 2.99
Cliffords.jl Git GitHub 6/12 0.33 <1 - -
dimod Git GitHub 110/201 0.30 5.3 94% 2.96
dwave-system Git GitHub 54/72 0.24 8.2 87% 3.47
FermiLib Git GitHub 24/134 0.31 <1 99% 2.43
Forest - Grove Git GitHub 53/130 0.51 17.7 72% 3.25
Forest - pyQuil Git GitHub 293/385 0.41 10.6 88% 2.65
OpenFermion Git GitHub 137/345 0.61 1.3 100% 2.46
ProjectQ Git GitHub 84/198 0.75 4.0 100% 4.02
PyZX Git GitHub 6/2 0.80 <1 51% 4.42
QGL.jl Git GitHub 17/13 0.75 130.6 - -
Qbsolv Git GitHub 50/85 0.17 22.2 95% -
Qiskit Aqua Git GitHub 43/141 0.20 1.8 67% 3.04
Qiskit Terra Git GitHub 526/713 0.11 16.0 76% 2.56
Qiskit Tutorials Git GitHub 94/274 0.40 8.6 - -
Qiskit.js Git GitHub 19/8 0.33 4.4 66% -
Qrack Git GitHub 7/78 0.07 8.7 87% -
Quantum Fog Git GitHub 17/1 1.00 <1 0% 3.32
Quantum++ Git GitHub 8/45 0.88 <1 72% -
Qubiter Git GitHub 14/3 0.75 <1 0% -
Quirk Git GitHub 286/131 0.96 <1 - -
reference-qvm Git GitHub 6/14 0.44 75.6 80% 3.99
ScaffCC Git GitHub 15/11 0.18 10.1 - -
Strawberry Fields Git GitHub 16/20 0.73 1.2 97% 2.70
XACC Git GitHub 65/14 0.65 <1 - -
XACC VQE Git GitHub 22/4 0.33 8.8 - -

Documentation evaluation

The image below shows the detailed results of our qualitative documentation analysis in form of a colour coded heatmap with scores ranging from 1 (bad) to 5 (good). We evaluated each project based on the quality of their source code documentation, README files, changelogs, user documentation and tutorials. The detailed rubrik used for scoring each of the five aspects can be found in the Supporting Information of the review paper “Open source software in quantum computing”.

Heatmap


Community evaluation

Lastly, the following table shows the evaluation results for the community analysis. For each project, we analyse if a public development roadmap exists and if the software is published in form of releases. Additionally, we report the GitHub community profile score, the total number of contributors, the type of user- and developer-centric discussion channel and the type of public code review process – specifically if it applies to internal (I) and/or external (E) contributors.

Project Roadmap Releases Contributors User-discussion channels Developer-discussion channels Public review process Community Profile
Cirq 28 Stack Exchange - E+I 4/7
Cliffords.jl 7 - - E 3/7
dimod 11 Forum - E+I 5/7
dwave-system 6 Forum - E+I 4/7
FermiLib 10 - - E+I 3/7
Forest - Grove 24 Slack Slack E+I 3/7
Forest - pyQuil 46 Slack Slack E+I 3/7
OpenFermion 26 - - E+I 3/7
ProjectQ 10 - - E+I 3/7
PyZX 3 - - - 3/7
QGL.jl 3 - - E+I 3/7
Qbsolv 18 Forum - E+I 5/7
Qiskit Aqua 14 Forum - E+I 7/7
Qiskit Terra 67 Forum, Slack Slack E+I 7/7
Qiskit Tutorials 37 - - E+I 3/7
Qiskit.js 4 Forum - E 7/7
Qrack 2 - - E+I 3/7
Quantum Fog 2 - - E 3/7
Quantum++ 3 Gitter - E 5/7
Qubiter 2 - - E 3/7
Quirk 3 - - E 4/7
reference-qvm 8 - - E+I 3/7
ScaffCC 7 - - E 3/7
Strawberry Fields 5 Slack Slack E+I 7/7
XACC 6 - - E 4/7
XACC VQE 2 - - E 3/7

Let us know if you disagree with some of these results. If you're maintaining one of the projects above and you think that we did a mistake in our evaluation then we want to hear from you! Please open an issue on our GitHub page.