Awesome Open Source
Awesome Open Source
Sponsorship

Awesome MLOps Awesome Made With Love

MLOps. You Desing It. Your Train It. You Run It.

An awesome list of references for MLOps - Machine Learning Operations 👉 ml-ops.org

Table of Content

MLOps Core MLOps Communities
MLOps Books MLOps Articles
MLOps: Feature Stores MLOps: Data Engineering (DataOps)
MLOps: Model Deployment and Serving MLOps: Testing, Monitoring and Maintenance
MLOps: Infrastructure MLOps Papers
Talks About MLOps Existing ML Systems
Machine Learning Software Engineering
Product Management for ML/AI The Economics of ML/AI
Model Governance, Ethics, Responsible AI Newsletters About MLOps, Machine Learning, Data Science and Co.

MLOps Core

  1. Machine Learning Operations: You Design It, You Train It, You Run It!
  2. MLOps SIG Specification
  3. ML in Production
  4. Awesome production machine learning: State of MLOps Tools and Frameworks
  5. Udemy “Deployment of ML Models”
  6. Full Stack Deep Learning
  7. Engineering best practices for Machine Learning
  8. 🚀 Putting ML in Production
  9. Stanford MLSys Seminar Series
  10. IBM ML Operationalization Starter Kit

MLOps Communities

  1. MLOps.community
  2. CDF Special Interest Group - MLOps
  3. RsqrdAI - Robust and Responsible AI

MLOps Books

  1. “Machine Learning Engineering” by Andriy Burkov, 2020
  2. "ML Ops: Operationalizing Data Science" by David Sweenor, Steven Hillion, Dan Rope, Dev Kannabiran, Thomas Hill, Michael O'Connell
  3. "Building Machine Learning Powered Applications" by Emmanuel Ameisen
  4. "Building Machine Learning Pipelines" by Hannes Hapke, Catherine Nelson, 2020, O’Reilly
  5. "Managing Data Science" by Kirill Dubovikov
  6. "Accelerated DevOps with AI, ML & RPA: Non-Programmer's Guide to AIOPS & MLOPS" by Stephen Fleming
  7. "Evaluating Machine Learning Models" by Alice Zheng
  8. Agile AI. 2020. By Carlo Appugliese, Paco Nathan, William S. Roberts. O'Reilly Media, Inc.
  9. "Machine Learning Logistics". 2017. By T. Dunning et al. O'Reilly Media Inc.
  10. "Machine Learning Design Patterns" by Valliappa Lakshmanan, Sara Robinson, Michael Munn. O'Reilly 2020
  11. "Serving Machine Learning Models: A Guide to Architecture, Stream Processing Engines, and Frameworks" by Boris Lublinsky, O'Reilly Media, Inc. 2017
  12. "Kubeflow for Machine Learning" by Holden Karau, Trevor Grant, Ilan Filonenko, Richard Liu, Boris Lublinsky
  13. "Clean Machine Learning Code" by Moussa Taifi. Leanpub. 2020

MLOps Articles

  1. Continuous Delivery for Machine Learning (by Thoughtworks)

  2. What is MLOps? NVIDIA Blog

  3. MLSpec: A project to standardize the intercomponent schemas for a multi-stage ML Pipeline.

  4. State of Enterprise ML 2019: PDF and Interactive

  5. Organizing machine learning projects: project management guidelines.

  6. Rules for ML Project (Best practices)

  7. ML Pipeline Template

  8. Data Science Project Structure

  9. Reproducible ML

  10. ML project template facilitating both research and production phases.

  11. Machine learning requires a fundamentally different deployment approach. As organizations embrace machine learning, the need for new deployment tools and strategies grows.

  12. Why is DevOps for Machine Learning so Different?

  13. Lessons learned turning machine learning models into real products and services – O’Reilly

  14. MLOps: Model management, deployment and monitoring with Azure Machine Learning

  15. Guide to File Formats for Machine Learning: Columnar, Training, Inferencing, and the Feature Store

  16. Architecting a Machine Learning Pipeline How to build scalable Machine Learning systems

  17. Why Machine Learning Models Degrade In Production

  18. Concept Drift and Model Decay in Machine Learning

  19. Bringing ML to Production

  20. A Tour of End-to-End Machine Learning Platforms

  21. MLOps: Continuous delivery and automation pipelines in machine learning

  22. AI meets operations

  23. What would machine learning look like if you mixed in DevOps? Wonder no more, we lift the lid on MLOps

  24. Forbes: The Emergence Of ML Ops

  25. Cognilytica Report "ML Model Management and Operations 2020 (MLOps)"

  26. Introducing Cloud AI Platform Pipelines

  27. A Guide to Production Level Deep Learning

  28. The 5 Components Towards Building Production-Ready Machine Learning Systems

  29. Deep Learning in Production (references about deploying deep learning-based models in production)

  30. Machine Learning Experiment Tracking

  31. The Team Data Science Process (TDSP)

  32. MLOps Solutions (Azure based)

  33. Monitoring ML pipelines

  34. Deployment & Explainability of Machine Learning COVID-19 Solutions at Scale with Seldon Core and Alibi

  35. Demystifying AI Infrastructure

  36. Organizing machine learning projects: project management guidelines.

  37. The Checklist for Machine Learning Projects (from Aurélien Géron,"Hands-On Machine Learning with Scikit-Learn and TensorFlow")

  38. Data Project Checklist by Jeremy Howard

  39. MLOps: not as Boring as it Sounds

  40. 10 Steps to Making Machine Learning Operational. Cloudera White Paper

  41. MLOps is Not Enough. The Need for an End-to-End Data Science Lifecycle Process.

  42. Data Science Lifecycle Repository Template

  43. Template: code and pipeline definition for a machine learning project demonstrating how to automate an end to end ML/AI workflow.

  44. Nitpicking Machine Learning Technical Debt

  45. The Best Tools, Libraries, Frameworks and Methodologies that Machine Learning Teams Actually Use – Things We Learned from 41 ML Startups

  46. Software Engineering for AI/ML - An Annotated Bibliography

  47. Intelligent System. Machine Learning in Practice

  48. CMU 17-445/645: Software Engineering for AI-Enabled Systems (SE4AI)

  49. Machine Learning is Requirements Engineering

  50. Machine Learning Reproducibility Checklist

  51. Machine Learning Ops. A collection of resources on how to facilitate Machine Learning Ops with GitHub.

  52. Task Cheatsheet for Almost Every Machine Learning Project A checklist of tasks for building End-to-End ML projects

  53. Web services vs. streaming for real-time machine learning endpoints

  54. How PyTorch Lightning became the first ML framework to run continuous integration on TPUs

  55. The ultimate guide to building maintainable Machine Learning pipelines using DVC

  56. Continuous Machine Learning (CML) is CI/CD for Machine Learning Projects (DVC)

  57. What I learned from looking at 200 machine learning tools

  58. Big Data & AI Landscape

  59. Deploying Machine Learning Models as Data, not Code — A better match?

  60. “Thou shalt always scale” — 10 commandments of MLOps

  61. Three Risks in Building Machine Learning Systems

  62. Blog about ML in production (by maiot.io)

  63. Back to the Machine Learning fundamentals: How to write code for Model deployment. Part 1, Part 2, Part 3

  64. MLOps: Machine Learning as an Engineering Discipline

  65. ML Engineering on Google Cloud Platform (hands-on labs and code samples)

  66. Deep Reinforcement Learning in Production. The use of Reinforcement Learning to Personalize User Experience at Zynga

  67. What is Data Observability?

  68. A Practical Guide to Maintaining Machine Learning in Production

  69. Continuous Machine Learning. Part 1, Part 2. Part 3 is coming soon.

  70. The Agile approach in data science explained by an ML expert

  71. Here is what you need to look for in a model server to build ML-powered services

  72. The problem with AI developer tools for enterprises (and what IKEA has to do with it)

  73. Streaming Machine Learning with Tiered Storage

  74. Best practices for performance and cost optimization for machine learning (Google Cloud)

  75. Lean Data and Machine Learning Operations

  76. A Brief Guide to Running ML Systems in Production Best Practices for Site Reliability Engineers

  77. AI engineering practices in the wild - SIG | Getting software right for a healthier digital world

  78. SE-ML | The 2020 State of Engineering Practices for Machine Learning

  79. Awesome Software Engineering for Machine Learning (GitHub repository)

  80. Sampling isn’t enough, profile your ML data instead

  81. Reproducibility in ML: why it matters and how to achieve it

  82. 12 Factors of reproducible Machine Learning in production

  83. MLOps: More Than Automation

  84. Lean Data Science

  85. Engineering Skills for Data Scientists

MLOps: Feature Stores

  1. Feature Stores for Machine Learning Medium Blog
  2. MLOps with a Feature Store
  3. Feature Stores for ML
  4. Hopsworks: Data-Intensive AI with a Feature Store
  5. Feast: An open-source Feature Store for Machine Learning
  6. What is a Feature Store?
  7. ML Feature Stores: A Casual Tour
  8. Comprehensive List of Feature Store Architectures for Data Scientists and Big Data Professionals

MLOps: Data Engineering (DataOps)

  1. The state of data quality in 2020 – O’Reilly
  2. Why We Need DevOps for ML Data
  3. Data Preparation for Machine Learning (7-Day Mini-Course)
  4. Best practices in data cleaning: A Complete Guide to Everything You Need to Do Before and After Collecting Your Data.
  5. 17 Strategies for Dealing with Data, Big Data, and Even Bigger Data
  6. DataOps Data Architecture
  7. Data Orchestration — A Primer
  8. 4 Data Trends to Watch in 2020
  9. CSE 291D / 234: Data Systems for Machine Learning
  10. A complete picture of the modern data engineering landscape
  11. Continuous Integration for your data with GitHub Actions and Great Expectations. One step closer to CI/CD for your data pipelines
  12. Emerging Architectures for Modern Data Infrastructure
  13. Awesome Data Engineering. Learning path and resources to become a data engineer

MLOps: Model Deployment and Serving

  1. AI Infrastructure for Everyone: DeterminedAI
  2. Deploying R Models with MLflow and Docker
  3. What Does it Mean to Deploy a Machine Learning Model?
  4. Software Interfaces for Machine Learning Deployment
  5. Batch Inference for Machine Learning Deployment
  6. AWS Cost Optimization for ML Infrastructure - EC2 spend
  7. CI/CD for Machine Learning & AI
  8. Itaú Unibanco: How we built a CI/CD Pipeline for machine learning with online training in Kubeflow
  9. 101 For Serving ML Models
  10. Deploying Machine Learning models to production — Inference service architecture patterns
  11. Serverless ML: Deploying Lightweight Models at Scale

MLOps: Testing, Monitoring and Maintenance

  1. Building dashboards for operational visibility (AWS)
  2. Monitoring Machine Learning Models in Production
  3. Effective testing for machine learning systems
  4. Unit Testing Data: What is it and how do you do it?
  5. How to Test Machine Learning Code and Systems (Accompanying code)
  6. Wu, T., Dong, Y., Dong, Z., Singa, A., Chen, X. and Zhang, Y., 2020. Testing Artificial Intelligence System Towards Safety and Robustness: State of the Art. IAENG International Journal of Computer Science, 47(3).
  7. Multi-Armed Bandits and the Stitch Fix Experimentation Platform
  8. A/B Testing Machine Learning Models
  9. Data validation for machine learning. Polyzotis, N., Zinkevich, M., Roy, S., Breck, E. and Whang, S., 2019. Proceedings of Machine Learning and Systems
  10. Testing machine learning based systems: a systematic mapping
  11. Explainable Monitoring: Stop flying blind and monitor your AI
  12. WhyLogs: Embrace Data Logging Across Your ML Systems
  13. Evidently AI. Insights on doing machine learning in production. (Vendor blog.)

MLOps: Infrastructure

  1. Rise of the Canonical Stack in Machine Learning. How a Dominant New Software Stack Will Unlock the Next Generation of Cutting Edge AI Apps
  2. AI Infrastructure Alliance. Building the canonical stack for AI/ML
  3. Linux Foundation AI Foundation
  4. ML Infrastructure Tools for Production | Part 1 — Production ML — The Final Stage of the Model Workflow | Part 2 — Model Deployment and Serving
  5. The MLOps Stack Template

MLOps Papers

  1. Studer, S., Bui, T.B., Drescher, C., Hanuschkin, A., Winkler, L., Peters, S. and Mueller, K.R., 2020. "Towards CRISP-ML (Q): A Machine Learning Process Model with Quality Assurance Methodology". arXiv
  2. Building a Reproducible Machine Learning Pipeline
  3. A Systems Perspective to Reproducibility in Production Machine Learning Domain
  4. Hidden Technical Debt in Machine Learning Systems
  5. Scaling Machine Learning as a Service (Uber)
  6. What’s your ML Test Score? A rubric for ML production systems
  7. Adversarial Machine Learning Reading List
  8. From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices
  9. Workshop on MLOps Systems. 2020 Third Conference on Machine Learning and Systems (MLSys)
  10. sensAI: Fast ConvNets Serving on Live Data via Class Parallelism. Guanhua Wang, Zhuang Liu, Siyuan Zhuang, Brandon Hsieh, Joseph Gonzalez and Ion Stoica.
  11. Towards Automated ML Model Monitoring: Measure, Improve and Quantify Data Quality. Tammo Rukat, Dustin Lange, Sebastian Schelter and Felix Biessmann.
  12. Towards Automating the AI Operations Lifecycle. Matthew Arnold, Jeff Boston, Michael Desmond, Evelyn Duesterwald, Benjamin Elder, Anupama Murthi, Jiri Navratil and Darrell Reimer.
  13. Efficient Scheduling of DNN Training on Multitenant Clusters. Deepak Narayanan, Keshav Santhanam, Amar Phanishayee and Matei Zaharia.
  14. Towards Complaint-driven ML Workflow Debugging. Weiyuan Wu, Lampros Flokas, Eugene Wu and Jiannan Wang.
  15. PerfGuard: Deploying ML-for-Systems without Performance Regressions. H M Sajjad Hossain, Lucas Rosenblatt, Gilbert Antonius, Irene Shaffer, Remmelt Ammerlaan, Abhishek Roy, Markus Weimer, Hiren Patel, Marc Friedman, Shi Qiao, Peter Orenberg, Soundarajan Srinivasan and Alekh Jindal.
  16. Implicit Provenance for Machine Learning Artifacts. Alexandru A. Ormenisan, Mahmoud Ismail, Seif Haridi and Jim Dowling.
  17. Addressing the Memory Bottleneck in AI Model-Training. David Ojika, Bhavesh Patel, G Anthony Reina, Trent Boyer, Chad Martin and Prashant Shah.
  18. Simulating Performance of ML Systems with Offline Profiling. Hongming Huang, Peng Cheng, Hong Xu and Yongqiang Xiong.
  19. A Viz Recommendation System: ML Lifecycle at Tableau. Kazem Jahanbakhsh, Eric Borchu, Mya Warren, Xiang-Bo Mao and Yogesh Sood.
  20. CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking. Grigori Fursin, Herve Guillou and Nicolas Essayan.
  21. Towards split learning at scale: System design. Iker Rodríguez, Eduardo Muñagorri, Alberto Roman, Abhishek Singh, Praneeth Vepakomma and Ramesh Raskar.
  22. MLBox: Towards Reproducible ML. Victor Bittorf, Xinyuan Huang, Peter Mattson, Debojyoti Dutta, David Aronchick, Emad Barsoum, Sarah Bird, Sergey Serebryakov, Natalia Vassilieva, Tom St. John, Grigori Fursin, Srini Bala, Sivanagaraju Yarramaneni, Alka Roy, David Kanter and Elvira Dzhuraeva.
  23. Conversational Applications and Natural Language Understanding Services at Scale. Minh Tue Vo Thanh and Vijay Ramakrishnan.
  24. Towards Distribution Transparency for Supervised ML With Oblivious Training Functions. Moritz Meister, Sina Sheikholeslami, Robin Andersson, Alexandru Ormenisan and Jim Dowling.
  25. Tools for machine learning experiment management. Vlad Velici and Adam Prügel-Bennett.
  26. MLPM: Machine Learning Package Manager. Xiaozhe Yao.
  27. Common Problems with Creating Machine Learning Pipelines from Existing Code. Katie O’Leary, Makoto Uchida.
  28. Overton: A Data System for Monitoring and Improving Machine-Learned Products, Apple.
  29. Reliance on Metrics is a Fundamental Challenge for AI
  30. "Assuring the machine learning lifecycle: Desiderata, methods, and challenges." Ashmore, Rob, Radu Calinescu, and Colin Paterson. (2019)
  31. "Machine learning testing: Survey, landscapes and horizons." Zhang, Jie M., et al. IEEE Transactions on Software Engineering (2020).
  32. "Teaching Software Engineering for AI-Enabled Systems." Kästner, Christian, and Eunsuk Kang. arXiv (2020).
  33. "Explainable machine learning in deployment." Bhatt, Umang, et al. Proceedings of the Conference on Fairness, Accountability, and Transparency. 2020.
  34. "Studying software engineering patterns for designing machine learning systems." Washizaki, Hironori, Hiromu Uchida, Foutse Khomh, and Yann-Gaël Guéhéneuc. In 2019 10th International Workshop on Empirical Software Engineering in Practice (IWESEP)
  35. Chen, A., Chow, A., Davidson, A., DCunha, A., Ghodsi, A., Hong, S.A., Konwinski, A., Mewald, C., Murching, S., Nykodym, T. and Ogilvie, P., 2020, June. Developments in MLflow: A System to Accelerate the Machine Learning Lifecycle. In Proceedings of the Fourth International Workshop on Data Management for End-to-End Machine Learning
  36. Karlaš, B., Interlandi, M., Renggli, C., Wu, W., Zhang, C., Mukunthu, D., Babu, I., Edwards, J., Lauren, C., Xu, A. and Weimer, M., Building Continuous Integration Services for Machine Learning. KDD 2020
  37. Workshop at ICML 2020: "Challenges in Deploying and Monitoring Machine Learning Systems" (Accepted Papers)
  38. Bosch, J., Crnkovic, I. and Olsson, H.H., 2020. Engineering AI Systems: A Research Agenda. arXiv preprint arXiv. 2020
  39. Ribeiro, M.T., Wu, T., Guestrin, C. and Singh, S., 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL
  40. Klaise, J., Van Looveren, A., Cox, C., Vacanti, G. and Coca, A., 2020. Monitoring and explainability of models in production. arXiv preprint arXiv:2007.06299.
  41. Serban, A., van der Blom, K., Hoos, H. and Visser, J., 2020. Adoption and Effects of Software Engineering Best Practices in Machine Learning. arXiv preprint arXiv:2007.14130.
  42. Gama, João, Indrė Žliobaitė, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. "A survey on concept drift adaptation." ACM computing surveys (CSUR) (2014)
  43. Belle, V. and Papantonis, I., 2020. Principles and Practice of Explainable Machine Learning. arXiv preprint arXiv:2009.11698.
  44. H.Z. Zhang, Y.M. Li, Y.Z. Huang, Y.G. Wen, J.X. Yin, and K. Guan. 2020. MLModelCI: An Automatic Cloud Platform for Efficient MLaaS. In Proceedings of the 28th ACM International Conference on Multimedia (MM '20).
  45. Karmarkar, A., Altay, A., Zaks, A., Polyzotis, N., Ramesh, A., Mathes, B., Vasudevan, G., Giannoumis, I., Wilkiewicz, J., Simsa, J. and Hong, J., 2020. Towards ML Engineering: A Brief History Of TensorFlow Extended (TFX). arXiv preprint arXiv:2010.02013.

Talks About MLOps

  1. DeliveryConf 2020. "Continuous Delivery For Machine Learning: Patterns And Pains" by Emily Gorcenski
  2. MLOps Conference: Talks from 2019
  3. A CI/CD Framework for Production Machine Learning at Massive Scale (using Jenkins X and Seldon Core)
  4. MLOps Virtual Event (Databricks)
  5. MLOps NY conference 2019
  6. MLOps.community YouTube Channel
  7. MLinProduction YouTube Channel
  8. Introducing MLflow for End-to-End Machine Learning on Databricks. Spark+AI Summit 2020. Sean Owen
  9. MLOps Tutorial #1: Intro to Continuous Integration for ML
  10. Machine Learning At Speed: Operationalizing ML For Real-Time Data Streams (2019)
  11. Damian Brady - The emerging field of MLops
  12. MLOps - Entwurf, Entwicklung, Betrieb (INNOQ Podcast in German)
  13. Instrumentation, Observability & Monitoring of Machine Learning Models
  14. Efficient ML engineering: Tools and best practices
  15. Beyond the jupyter notebook: how to build data science products
  16. An introduction to MLOps on Google Cloud (First 19 min are vendor-, language-, and framework-agnostic. @visenger)
  17. How ML Breaks: A Decade of Outages for One Large ML Pipeline
  18. Clean Machine Learning Code: Practical Software Engineering

Existing ML Systems

  1. Introducing FBLearner Flow: Facebook’s AI backbone
  2. TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
  3. Getting started with Kubeflow Pipelines
  4. Meet Michelangelo: Uber’s Machine Learning Platform
  5. Meson: Workflow Orchestration for Netflix Recommendations
  6. What are Azure Machine Learning pipelines?
  7. Uber ATG’s Machine Learning Infrastructure for Self-Driving Vehicles
  8. An overview of ML development platforms
  9. Snorkel AI: Putting Data First in ML Development
  10. A Tour of End-to-End Machine Learning Platforms
  11. Introducing WhyLabs, a Leap Forward in AI Reliability
  12. Project: Ease.ml (ETH Zürich)

Machine Learning

  1. Book, Aurélien Géron,"Hands-On Machine Learning with Scikit-Learn and TensorFlow"
  2. Foundations of Machine Learning
  3. Best Resources to Learn Machine Learning
  4. Awesome TensorFlow
  5. "Papers with Code" - Browse the State-of-the-Art in Machine Learning
  6. Zhi-Hua Zhou. 2012. Ensemble Methods: Foundations and Algorithms. Chapman & Hall/CRC.
  7. Feature Engineering for Machine Learning. Principles and Techniques for Data Scientists. By Alice Zheng, Amanda Casari
  8. Google Research: Looking Back at 2019, and Forward to 2020 and Beyond
  9. O’Reilly: The road to Software 2.0
  10. Machine Learning and Data Science Applications in Industry
  11. Curated papers, articles, and blogs on data science & machine learning in production.
  12. Deep Learning for Anomaly Detection
  13. Federated Learning for Mobile Keyboard Prediction
  14. Federated Learning. Building better products with on-device data and privacy on default
  15. Federated Learning: Collaborative Machine Learning without Centralized Training Data
  16. Yang, Q., Liu, Y., Cheng, Y., Kang, Y., Chen, T. and Yu, H., 2019. Federated learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 13(3). Chapters 1 and 2.
  17. Book: Molnar, Christoph. "Interpretable machine learning. A Guide for Making Black Box Models Explainable", 2019
  18. Book: Hutter, Frank, Lars Kotthoff, and Joaquin Vanschoren. "Automated Machine Learning". Springer,2019.
  19. ML resources by topic, curated by the community.
  20. An Introduction to Machine Learning Interpretability, by Patrick Hall, Navdeep Gill, 2nd Edition. O'Reilly 2019
  21. Examples of techniques for training interpretable machine learning (ML) models, explaining ML models, and debugging ML models for accuracy, discrimination, and security.
  22. Paper: "Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence", by Sebastian Raschka, Joshua Patterson, and Corey Nolet. 2020
  23. Distill: Machine Learning Research
  24. AtHomeWithAI: Curated Resource List by DeepMind
  25. Awesome Data Science
  26. Intro to probabilistic programming. A use case using Tensorflow-Probability (TFP)
  27. Dive into Snorkel: Weak-Superversion on German Texts. inovex Blog
  28. Dive into Deep Learning. An interactive deep learning book with code, math, and discussions. Provides NumPy/MXNet, PyTorch, and TensorFlow implementations
  29. Data Science Collected Resources (GitHub repository)
  30. A resource list for causality in statistics, data science and physics
  31. Set of illustrated Machine Learning cheatsheets

Software Engineering

  1. The Twelve Factors
  2. Book "Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations", 2018 by Nicole Forsgren et.al
  3. Book "The DevOps Handbook" by Gene Kim, et al. 2016
  4. State of DevOps 2019
  5. Clean Code concepts adapted for machine learning and data science.

Product Management for ML/AI

  1. What you need to know about product management for AI. A product manager for AI does everything a traditional PM does, and much more.
  2. Bringing an AI Product to Market. Previous articles have gone through the basics of AI product management. Here we get to the meat: how do you bring a product to market?
  3. The People + AI Guidebook
  4. User Needs + Defining Success
  5. Building machine learning products: a problem well-defined is a problem half-solved.
  6. Talk: Designing Great ML Experiences (Apple)
  7. Machine Learning for Product Managers

The Economics of ML/AI

  1. Book: "Prediction Machines: The Simple Economics of Artificial Intelligence"
  2. Book: "The AI Organization" by David Carmona
  3. Book: "Succeeding with AI". 2020. By Veljko Krunic. Manning Publications
  4. A list of articles about AI and the economy
  5. Gartner AI Trends 2019
  6. Global AI Survey: AI proves its worth, but few scale impact
  7. Getting started with AI? Start here! Everything you need to know to dive into your project
  8. 11 questions to ask before starting a successful Machine Learning project
  9. What AI still can’t do
  10. Demystifying AI Part 4: What is an AI Canvas and how do you use it?
  11. A Data Science Workflow Canvas to Kickstart Your Projects
  12. Is your AI project a nonstarter? Here’s a reality check(list) to help you avoid the pain of learning the hard way
  13. What is THE main reason most ML projects fail?
  14. Designing great data products. The Drivetrain Approach: A four-step process for building data products.
  15. The New Business of AI (and How It’s Different From Traditional Software)
  16. The idea maze for AI startups
  17. The Enterprise AI Challenge: Common Misconceptions
  18. Misconception 1 (of 5): Enterprise AI Is Primarily About The Technology
  19. Misconception 2 (of 5): Automated Machine Learning Will Unlock Enterprise AI
  20. Three Principles for Designing ML-Powered Products
  21. A Step-by-Step Guide to Machine Learning Problem Framing
  22. AI adoption in the enterprise 2020
  23. How Adopting MLOps can Help Companies With ML Culture?
  24. Weaving AI into Your Organization
  25. What to Do When AI Fails
  26. Introduction to Machine Learning Problem Framing
  27. Structured Approach for Identifying AI Use Cases
  28. Book: "Machine Learning for Business" by Doug Hudgeon, Richard Nichol, O'reilly
  29. Why Commercial Artificial Intelligence Products Do Not Scale (FemTech)
  30. Google Cloud’s AI Adoption Framework (White Paper)
  31. Data Science Project Management
  32. Book: "Competing in the Age of AI" by Marco Iansiti, Karim R. Lakhani. Harvard Business Review Press. 2020
  33. Laszlo Sragner Newsletter
  34. The Three Questions about AI that Startups Need to Ask. The first is: Are you sure you need AI?
  35. Taming the Tail: Adventures in Improving AI Economics
  36. Managing the Risks of Adopting AI Engineering
  37. Get rid of AI Saviorism
  38. Collection of articles listing reasons why data science projects fail
  39. How to Choose Your First AI Project by Andrew Ng
  40. How to Set AI Goals
  41. Expanding AI's Impact With Organizational Learning

Model Governance, Ethics, Responsible AI

  1. Book: "Practical Fairness". 2020. By Aileen Nielsen. O'Reilly Media, Inc.
  2. Book: "Fairness and machine learning: Limitations and Opportunities." Barocas, S., Hardt, M. and Narayanan, A., 2018.
  3. What are model governance and model operations? A look at the landscape of tools for building and deploying robust, production-ready machine learning models
  4. Specialized tools for machine learning development and model governance are becoming essential. Why companies are turning to specialized machine learning tools like MLflow.
  5. What are model governance and model operations? – O’Reilly
  6. AI Fairness 360, A Step Towards Trusted AI - IBM Research
  7. Responsible AI
  8. Learn how to integrate Responsible AI practices into your ML workflow using TensorFlow
  9. ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT)
  10. Programming Fairness in Algorithms. Understanding and combating issues of fairness in supervised learning.
  11. Secure, privacy-preserving and federated machine learning in medical imaging
  12. Artifical intelligence and machine learning security (by Microsoft) The references therein are useful.
  13. Evtimov, Ivan, Weidong Cui, Ece Kamar, Emre Kiciman, Tadayoshi Kohno, and Jerry Li. "Security and Machine Learning in the Real World." arXiv (2020).
  14. Explainable AI (Gartner Prediction for 2023)
  15. What We've Learned to Control. By Ben Recht
  16. State of AI Ethics June 2020 Report by the Montreal AI Ethics Institute
  17. Practical Data Ethics
  18. Vasudevan, Sriram and Kenthapadi, Krishnaram. "LiFT: A Scalable Framework for Measuring Fairness in ML Applications" (2020) - Code: The LinkedIn Fairness Toolkit (LiFT)
  19. Four Principles of Explainable Artificial Intelligence (NIST Draft). Phillips, P.J., Hahn, A.C., Fontana, P.C., Broniatowski, D.A. and Przybocki, M.A., 2020.
  20. Data Ethics Canvas. Helps identify and manage ethical issues – at the start of a project that uses data, and throughout. Also see Ethics Canvas for broader scope.
  21. ABOUT ML - Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles.
  22. Mitchell, Margaret and Wu, Simone and Zaldivar, Andrew and Barnes, Parker and Vasserman, Lucy and Hutchinson, Ben and Spitzer, Elena and Raji, Inioluwa Deborah and Gebru, Timnit. "Model Cards for Model Reporting" (2019) - Code: Model Card Toolkit

Newsletters About MLOps, Machine Learning, Data Science and Co.

  1. ML in Production newsletter
  2. MLOps.community
  3. Andriy Burkov newsletter
  4. Decision Intelligence by Cassie Kozyrkov
  5. Laszlo's Newsletter about Data Science
  6. Data Elixir newsletter for a weekly dose of the top data science picks from around the web. Covering machine learning, data visualization, analytics, and strategy.
  7. The Data Science Roundup by Tristan Handy
  8. Vicki Boykis Newsletter about Data Science
  9. KDnuggets News
  10. Analytics Vidhya, Any questions on business analytics, data science, big data, data visualizations tools and techniques
  11. Data Science Weekly Newsletter: A free weekly newsletter featuring curated news, articles and jobs related to Data Science
  12. The Machine Learning Engineer Newsletter
  13. Gradient Flow helps you stay ahead of the latest technology trends and tools with in-depth coverage, analysis and insights. See the latest on data, technology and business, with a focus on machine learning and AI
  14. Your guide to AI by Nathan Benaich. Monthly analysis of AI technology, geopolitics, research, and startups.
  15. O'Reilly Data & AI Newsletter
  16. deeplearning.ai’s newsletter by Andrew Ng
  17. Deep Learning Weekly
  18. Import AI is a weekly newsletter about artificial intelligence, read by more than ten thousand experts. By Jack Clark.
  19. AI Ethics Weekly
  20. Announcing Projects To Know, a weekly machine intelligence and data science newsletter
  21. TWIML: This Week in Machine Learning and AI newsletter
  22. featurestore.org: Monthly Newsletter on Feature Stores for ML

Twitter Follow


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
machine-learning (3,233
data-science (778
devops (521
ai (420
ml (160
engineering (70
software-engineering (60
federated-learning (23
mlops (23

Find Open Source By Browsing 7,000 Topics Across 59 Categories