Train Benchmark

🚂 Summary. The Train Benchmark is a framework for measuring the performance of continuous model transformations, with a particular emphasis on the performance of incremental query reevaluation. The benchmark is actively developed since 2011.

📖 Documentation. If you are interested in implementing the benchmark on your tool, please visit the documentation.

🎥 Presentation. For a short summary, check out this presentation, delivered at the Linked Data Benchmark Council's 9th Technical User Community meeting.

📚 Publications. The definitive publication on the benchmark is our journal paper The Train Benchmark: cross-technology performance evaluation of continuous model queries. For use cases, also check out the related publications.

💻 Technologies. The framework is implemented in Java 8 (for the main components) and Groovy (for scripts). The visualization is handled by R scripts. Both the build and the benchmark process in governed by Gradle.

👋 Contributions welcome. If you would like to implement the benchmark on your tool, we recommend to read the documentation and also please do not hesitate to get in touch!

⚠️ Warning. The Train Benchmark is designed to run in an isolated server environment, e.g. virtual machines in the cloud. Some implementations may shut down or delete existing databases, so only run it on your developer workstation if you understand the consequences. See also issue #75.

⚠️ Warning. The Train Benchmark has a fork for the 2015 Transformation Tool Contest, primarily targeting EMF tools. That fork is no longer maintained. You should use this repository, containing the full, cross-technology Train Benchmark (also supporting RDF, SQL and property graph databases).

📄 Citing the benchmark. For referencing the benchmark, please cite the paper in Software and Systems Modeling with the following BibTeX snippet.

📐 Models. Pre-generated models for scale factors 1 to 4096 are available as a tar.zst package.

@article{DBLP:journals/sosym/SzarnyasIRV18,
  author    = {G{\'{a}}bor Sz{\'{a}}rnyas and
               Benedek Izs{\'{o}} and
               Istv{\'{a}}n R{\'{a}}th and
               D{\'{a}}niel Varr{\'{o}}},
  title     = {The Train Benchmark: cross-technology performance evaluation of continuous
               model queries},
  journal   = {Software and System Modeling},
  volume    = {17},
  number    = {4},
  pages     = {1365--1393},
  year      = {2018},
  url       = {https://doi.org/10.1007/s10270-016-0571-8},
  doi       = {10.1007/s10270-016-0571-8},
  timestamp = {Fri, 07 Sep 2018 14:25:47 +0200},
  biburl    = {https://dblp.org/rec/bib/journals/sosym/SzarnyasIRV18},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}

License

The project uses the Eclipse Public License 1.0 and was supported by the MONDO EU FP7 (EU ICT-611125) project. It is primarily maintained by the MTA-BME Lendület Research Group on Cyber-Physical Systems.