☁️ 🚀 📊 📈 Evaluating state of the art in AI
Alternatives To Evalai
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Allennlp11,300117677 months ago264April 14, 202294apache-2.0Python
An open-source NLP research library, built on PyTorch.
515 years ago12March 05, 2017mitPython
Universe: a software platform for measuring and training an AI's general intelligence across the world's supply of games, websites and other applications.
8 hours ago432September 25, 2022115apache-2.0Go
An open-source Identity and Access Management (IAM) / Single-Sign-On (SSO) platform powered by Casbin and AI gateway with web UI supporting OAuth 2.0, OIDC, SAML and OpenAI ChatGPT
Wasmedge6,09158 hours ago6September 01, 2022346apache-2.0C++
WasmEdge is a lightweight, high-performance, and extensible WebAssembly runtime for cloud native, edge, and decentralized applications. It powers serverless apps, embedded functions, microservices, smart contracts, and IoT devices.
7 months ago85July 18, 20171apache-2.0Jsonnet
PipelineAI Kubeflow Distribution
11 days ago41otherCSS
ApacheCN 开源组织:公告、介绍、成员、活动、交流方式
3 months ago143agpl-3.0Python
A modern, web-based photo management server. Run it on your home server and it will let you find the right photo from your collection on any device. Smart filtering is made possible by object recognition, face recognition, location awareness, color analysis and other ML algorithms.
Elyra1,6051013 days ago89July 07, 2022258apache-2.0Python
Elyra extends JupyterLab with an AI centric approach.
a month ago35JavaScript
🦄 The Enterprise™ programming language
16 hours ago302otherPython
:cloud: :rocket: :bar_chart: :chart_with_upwards_trend: Evaluating state of the art in AI
Alternatives To Evalai
Select To Compare

Alternative Project Comparisons

Join the chat at https://gitter.im/Cloud-CV/EvalAI Build Status codecov Coverage Status Code style: black Requirements Status Code Climate Documentation Status Twitter Follow

EvalAI is an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence (AI) algorithms at scale.

In recent years, it has become increasingly difficult to compare an algorithm solving a given task with other existing approaches. These comparisons suffer from minor differences in algorithm implementation, use of non-standard dataset splits and different evaluation metrics. By providing a central leaderboard and submission interface, we make it easier for researchers to reproduce the results mentioned in the paper and perform reliable & accurate quantitative analysis. By providing swift and robust backends based on map-reduce frameworks that speed up evaluation on the fly, EvalAI aims to make it easier for researchers to reproduce results from technical papers and perform reliable and accurate analyses.


  • Custom evaluation protocols and phases: We allow creation of an arbitrary number of evaluation phases and dataset splits, compatibility using any programming language, and organizing results in both public and private leaderboards.

  • Remote evaluation: Certain large-scale challenges need special compute capabilities for evaluation. If the challenge needs extra computational power, challenge organizers can easily add their own cluster of worker nodes to process participant submissions while we take care of hosting the challenge, handling user submissions, and maintaining the leaderboard.

  • Evaluation inside environments: EvalAI lets participants submit code for their agent in the form of docker images which are evaluated against test environments on the evaluation server. During evaluation, the worker fetches the image, test environment, and the model snapshot and spins up a new container to perform evaluation.

  • CLI support: evalai-cli is designed to extend the functionality of the EvalAI web application to your command line to make the platform more accessible and terminal-friendly.

  • Portability: EvalAI is designed with keeping in mind scalability and portability of such a system from the very inception of the idea. Most of the components rely heavily on open-source technologies – Docker, Django, Node.js, and PostgreSQL.

  • Faster evaluation: We warm-up the worker nodes at start-up by importing the challenge code and pre-loading the dataset in memory. We also split the dataset into small chunks that are simultaneously evaluated on multiple cores. These simple tricks result in faster evaluation and reduces the evaluation time by an order of magnitude in some cases.


Our ultimate goal is to build a centralized platform to host, participate and collaborate in AI challenges organized around the globe and we hope to help in benchmarking progress in AI.

Installation instructions

Setting up EvalAI on your local machine is really easy. You can setup EvalAI using docker: The steps are:

  1. Install docker and docker-compose on your machine.

  2. Get the source code on to your machine via git.

    git clone https://github.com/Cloud-CV/EvalAI.git evalai && cd evalai
  3. Build and run the Docker containers. This might take a while.

    docker-compose up --build
  4. That's it. Open web browser and hit the URL Three users will be created by default which are listed below -

    SUPERUSER- username: admin password: password
    HOST USER- username: host password: password
    PARTICIPANT USER- username: participant password: password

If you are facing any issue during installation, please see our common errors during installation page.

Citing EvalAI

If you are using EvalAI for hosting challenges, please cite the following technical report:

    title   =  {EvalAI: Towards Better Evaluation Systems for AI Agents},
    author  =  {Deshraj Yadav and Rishabh Jain and Harsh Agrawal and Prithvijit
                Chattopadhyay and Taranjeet Singh and Akash Jain and Shiv Baran
                Singh and Stefan Lee and Dhruv Batra},
    year    =  {2019},
    volume  =  arXiv:1902.03570


EvalAI is currently maintained by Rishabh Jain, Deshraj Yadav, and Ram Ramrakhya. A non-exhaustive list of other major contributors includes: Akash Jain, Taranjeet Singh, Shiv Baran Singh, Harsh Agarwal, Prithvijit Chattopadhyay, Devi Parikh and Dhruv Batra.

Contribution guidelines

If you are interested in contributing to EvalAI, follow our contribution guidelines.


Popular Artificial Intelligence Projects
Popular Docker Projects
Popular Artificial Intelligence Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Machine Learning
Artificial Intelligence
Reproducible Research