A general purpose recommender metrics library for fair evaluation.
Alternatives To Rexmex
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Pytorch Metric Learning5,282124 hours ago193June 29, 202250mitPython
The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.
13 hours ago88April 24, 2021234apache-2.0C++
Instant Kubernetes-Native Application Observability
Ignite4,27731522 days ago1,038July 07, 2022152bsd-3-clausePython
High-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.
4 hours ago172October 11, 2021271apache-2.0Python
Aim 💫 — An easy-to-use & supercharged open-source AI metadata tracker (experiment tracking, AI agents tracing)
Serve3,4911119 hours ago14May 13, 2022284apache-2.0Java
Serve, optimize and scale PyTorch models in production
2 months ago99apache-2.0Python
mean Average Precision - This code evaluates the performance of your neural net for object recognition.
Aif3602,043284 hours ago10March 04, 2021170apache-2.0Python
A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.
Pytorch Nlp1,929982 years ago19November 04, 201916bsd-3-clausePython
Basic Utilities for PyTorch Natural Language Processing (NLP)
7 hours ago63apache-2.0Python
Torchmetrics - Machine learning metrics for distributed, scalable PyTorch applications.
2 years ago14mitPython
Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models
Alternatives To Rexmex
Select To Compare

Alternative Project Comparisons

Version License repo size build badge codecov

reXmeX is recommender system evaluation metric library.

Please look at the Documentation and External Resources.

reXmeX consists of utilities for recommender system evaluation. First, it provides a comprehensive collection of metrics for the evaluation of recommender systems. Second, it includes a variety of methods for reporting and plotting the performance results. Implemented metrics cover a range of well-known metrics and newly proposed metrics from data mining (ICDM, CIKM, KDD) conferences and prominent journals.


If you find RexMex useful in your research, please consider adding the following citation:

       title = {{rexmex: A General Purpose Recommender Metrics Library for Fair Evaluation.}},
       author = {Benedek Rozemberczki and Sebastian Nilsson and Piotr Grabowski and Charles Tapley Hoyt and Gavin Edwards},
       year = {2021},

An introductory example

The following example loads a synthetic dataset which has the mandatory y_true and y_score keys. The dataset has binary labels and predictied probability scores. We read the dataset and define a defult ClassificationMetric instance for the evaluation of the predictions. Using this metric set we create a score card and get the predictive performance metrics.

from rexmex import ClassificationMetricSet, DatasetReader, ScoreCard

reader = DatasetReader()
scores = reader.read_dataset()

metric_set = ClassificationMetricSet()

score_card = ScoreCard(metric_set)

report = score_card.get_performance_metrics(scores["y_true"], scores["y_score"])

An advanced example

The following more advanced example loads the same synthetic dataset which has the source_id, target_id, source_group and target group keys besides the mandatory y_true and y_score. Using the source_group key we group the predictions and return a performance metric report.

from rexmex import ClassificationMetricSet, DatasetReader, ScoreCard

reader = DatasetReader()
scores = reader.read_dataset()

metric_set = ClassificationMetricSet()

score_card = ScoreCard(metric_set)

report = score_card.generate_report(scores, grouping=["source_group"])


A rexmex score card allows the reporting of recommender system performance metrics, plotting the performance metrics and saving those. Our framework provides 7 rating, 38 classification, 18 ranking, and 2 coverage metrics.

Metric Sets

Metric sets allow the users to calculate a range of evaluation metrics for a label - predicted label vector pair. We provide a general MetricSet class and specialized metric sets with pre-set metrics have the following general categories:

  • Ranking
  • Rating
  • Classification
  • Coverage

Ranking Metric Set

Expand to see all ranking metrics in the metric set.

Rating Metric Set

These metrics assume that items are scored explicitly and ratings are predicted by a regression model.

Expand to see all rating metrics in the metric set.

Classification Metric Set

These metrics assume that the items are scored with raw probabilities (these can be binarized).

Expand to see all classification metrics in the metric set.

Coverage Metric Set

These metrics measure how well the recommender system covers the available items in the catalog and possible users. In other words measure the diversity of predictions.

Documentation and Reporting Issues

Head over to our documentation to find out more about installation and data handling, a full list of implemented methods, and datasets.

If you notice anything unexpected, please open an issue and let us know. If you are missing a specific method, feel free to open a feature request. We are motivated to constantly make RexMex even better.

Installation via the command line

RexMex can be installed with the following command after the repo is cloned.

$ pip install .

Use -e/--editable when developing.

Installation via pip

RexMex can be installed with the following pip command.

$ pip install rexmex

As we create new releases frequently, upgrading the package casually might be beneficial.

$ pip install rexmex --upgrade

Running tests

Tests can be run with tox with the following:

$ pip install tox
$ tox -e py


If you use RexMex in a scientific publication, we would appreciate citations. Please see GitHub's built-in citation tool.


Popular Metrics Projects
Popular Machine Learning Projects
Popular Operations Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Machine Learning
Deep Learning
Recommender System