Loma Buena Peculiar Inklings

This Blog Post: Are you Certain About How Uncertain You Should Be?  © 2024  by  Lynd Bacon & Loma Buena Associates  is licensed under   CC BY-SA 4.0  TL;DR?  Here's the BLUF: It has been said that love is a many-faceted thing.  So is uncertainty.  Because it is, the multiple sources of it in any data science or analytics effort should be well documented and expressed as precisely as possible.  Doing otherwise is not good practice, and is not in the best interest of supporting the best possible decision-making based on results. Life is chock full of uncertainty.  So are research, prediction, and data analytics. I've recently been considering some challenges in expressing and communicating uncertainty. I've also been thinking about the critical elements of reproducible data science workflows that I need to emphasize for my students and colleagues. I find myself once again reminded of how difficult it can be to understand uncertai...

This Blog Post: Risk Controlled Conformal Prediction of Outcomes for Customers, Patients, or Other Entities  © 2024  by  Lynd Bacon & Loma Buena Associates  is licensed under   CC BY-SA 4.0  TL; DR?  Here's the BLUF: Risk-controlled conformal prediction methods subsume conformal prediction methods as a special case.  They allow controlling for the risk of specified types of prediction errors.   Single or multiple kinds of risk can be accommodated.  Like conformal prediction methods, risk-controlling methods are model agnostic, and they can provide finite sample statistical certainty guarantees.  A simple, and mostly conceptual, description of risk controlled conformal prediction, follows. As noted elsewhere and by many, conformal prediction methods can provide a statistically valid, finite sample certainty guarantees about whether predictions for individual new cases, contain, or cover, the "true" quantity or value to be ...

This Blog Post: Conformal Prediction of Customer Segment Memberships by Customer Type; Uncertainty, Quantified and Unquantified © 2024 by Lynd Bacon & Loma Buena Associates is licensed under CC BY-NC-SA 4.0 TL;DR? Here’s the BLUF: Conformal prediction methods can provide statistically valid certainty guarantees when predicting class (e.g., segment) memberships, or other kinds of outcomes, that machine learning models are often used for predicting. Depending on how these methods are used, the quality of the results can depend on characteristics of the objects (e.g., customers) that predictions are for. When such characteristics consist of a priori known types or groups, investigating the quality of conformal prediction results for them is very straightforward. Conformal prediction is for quantifying uncertainty. Some kinds of uncertainty that decision-makers face may not be quantifiable.   This conformal prediction niblet is a follow up to a pr...

This Blog Post about Conformal Prediction © 2023 by Lynd Bacon and Loma Buena Associates, is licensed under CC BY-SA 4.0    TL;DR? Here’s the BLUF:  "Conformal prediction methods can provide you with statistically valid certainty guarantees regarding the likely values of predictions using “new” data, data that you haven’t observed outcomes for, or that you just didn’t use for training and validating models." A simple classification example with Python code snippets follows below, along with some suggested follow-up resources.    Suppose you’ve trained, tuned, and validated a predictive model that you will use to predict outcomes for new “objects,” new cases or observations. Your model might be simple regression or classification model, an ensemble model of some sort, or a deep neural network. Your objects might be customers and their responses or their types, drug molecules and their efficacy, or any other sort of thing that you use a model to p...