Alex Jones Ph.D

Bayesian Ordinal Regression in PyMC

Mon, 10 Jul 2023 23:00:00 GMT

Ordinal data is everywhere in social and behavioural science. People indicate their responses to questionnaire items by selecting a response out of a set of available options, such as disagree, neutral, agree, or rate their preferences on a numeric scale with a few discrete options (e.g. 1, 2, 3, 4). The longstanding approach to analysing this data is to simply treat it as though it were truly continuous, applying the usual techniques that assume Gaussian distributions, like OLS and ANOVA. This ignores the very real possibility that the distances between the categories may not be fixed and equal. Its easy to imagine situations where the gap between a 3 and a 4 on a scale is small, but a big jump exists between selecting 4 over 5.

Accordingly, there’s a lot of work over the last few years that convincingly demonstrates that treating ordinal data as continuous is not a good idea, and can often lead to bias in the conclusions or even getting the sign of relationships incorrect. Ordinal data is modelled much more accurately using an appropriate likelihood function that can express the ordinal nature of the observed variable. Unfortunately, fitting and interpreting these models can be difficult to do, and is something I’ve struggled with for quite a while.

Using Bayesian inference, its straightforward to express a model that expresses ordindal data, and then extend that to including predictors, which is almost always the needed use case in applied statistics. We’ll work through some use cases with PyMC, laying out the bare bones of the model, different approaches, and then move onto full regression modelling with an ordinal outcome through a few different datasets.

Are the latent variables in the room with us?

First though, some theory. Understanding this part can be tricky, but is key to grasping the way ordinal models work. Ordinal models assume a latent, continuous variable that is unobserved. This is the variable of interest - a persons tendency to endorse the statement, find a face attractive, their level of agreement to the statement, and so on. We make assumptions about the kind of distribution this variable can take - typically its the normal or logistic distribution. But to realise the discrete and ordered nature of the responses, we must impose a set of thresholds or cutpoints onto this distribution to slice it into categories. We can then work out the probability density within a category, and use those probabilities to assess the frequency with which a response will appear. If this is confusing, don’t worry. Writing down the steps and plotting some data will hopefully clear things up.

Simulating ordinal data with PyMC from scratch

First, lets import some packages, and then make some assumptions about what we want to see. Imagine we are trying to simulate a dataset where 1000 participants indicate their response to the question “I like chocolate”. We will imagine there is a latent normal distribution that governs this response, and it has a mean of zero, and a standard deviation of 1 (i.e., the standard normal). People are however forced to respond to this statement on a 1-5 scale, indicating their agreement (1 = strong dislike, 5 = strong like).

What might a distribution of ordinal responses to this question look like? For me, I’d imagine a lot of people might respond with a four or five - i.e., a four and five have a high probability of occuring. A smaller minority might select three, and not a lot of people might select one, two, or three. Lets see if we can have the normal distribution help us realise that, using SciPy.

# Imports
import arviz as az
import numpy as np
import pandas as pd
import pymc as pm
import pytensor.tensor as pt
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import seaborn.objects as so
import statsmodels.formula.api as smf

from formulae import design_matrices as dm

rng = np.random.default_rng(35)
plt.style.use('bmh')
plt.rcParams['axes.axisbelow'] = True

%matplotlib inline

Lets remind ourselves of the normal distributions appearance:

# Plot the normal
x = np.linspace(-3, 3, 1000) # Values to evaluate
standard_normal = st.norm(0, 1) # Our standard normal, mean 0, SD 1

# Plot the probability density function of normal
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.plot(x, standard_normal.pdf(x), lw=2, color='black');

The standard normal has most of its probability mass around zero, and less in the tails. How might we carve it up so that responses of 1 and 2 are less frequent, 3 is a bit more common, and 4 and 5 are most popular? We had to add add the thresholds in a sensible position to chop the distribution up.

In ordinal regression, there are always K-1 cutpoints. If we want 5 responses, we must place 4 thresholds. But where do they go? How would they translate to probabilities of responses? Well, we can use the cumulative density function of the standard normal to figure things out. Lets see what we mean by this. Imagine we place a cutpoint at -2, and a cutpoint at -1, which would correspond to the responses of 1 and 2 (strong dislike to dislike of chocolate) respectively. That would look like this:

# Add two cutpoints
ax.axvline(-2, color='black', lw=0.5) # -2
ax.axvline(-1, color='black', lw=0.5) # -1
fig

Looking at these cutpoints, we can get a sense that for the area up to the first one, there’s not much probability. Between the first and second one there’s quite a bit of a bigger amount, too. But precisely how much probability is there? Remember, this probability is going to exactly correspond to the probability of seeing a 1 and a 2 in our chocolate data. We can work out this using the cumulative density function (CDF) of the normal distribution. This tells us the probability of all the possible responses up to and including the threshold, and essentially is calculating the area under the curve. For cutpoint 1 (-2), the cdf can be obtained easily:

# Standard normal knows the answer
cut1_cdf = standard_normal.cdf(-2)
print(cut1_cdf)

0.022750131948179195

We are essentially computing the area in the shaded region below:

# Fill in the area
ax.fill_between(x, standard_normal.pdf(x), where=standard_normal.cdf(x) <= standard_normal.cdf(-2), color='blue')
fig

There we have it - that’s the probability of seeing a 1 in our data, if we keep the threshold there. If we repeat the same thing for second threshold, we probability of all the possible responses up to and including the second threshold, we have this:

# Standard normal still knows the answer
cut2_cdf = standard_normal.cdf(-1)
print(cut2_cdf)

# Fill in the area
ax.fill_between(x, standard_normal.pdf(x), where=standard_normal.cdf(x) <= standard_normal.cdf(-1), color='red')
fig

0.15865525393145707

So the probability of a 2 in our data is going to be about 16%, right? Not so fast. Looking at the shaded region its clear that this is actually overshadowing the first area. What we need is the area between -2 and -1, between the first and second cutpoints! How do we find this? Easily enough - just subtract the cdf up to -2 from the cdf up to -1!

# Probability of a 2
print(cut2_cdf - cut1_cdf)
ax.fill_between(x, standard_normal.pdf(x), where=standard_normal.cdf(x) <= standard_normal.cdf(-1), color='red')
ax.fill_between(x, standard_normal.pdf(x), where=standard_normal.cdf(x) <= standard_normal.cdf(-2), color='blue')
fig

0.13590512198327787

So the probability of a 2 is actually about 14%, and we’ve now correctly identified the area we want. This point highlights a key aspect of ordinal models. They carve up a latent distribution, and probabilities of a given response (e.g., 1, 2) are generated by subtracting the cdf of cutpoint c-1 from cutpoint c.

Lets now throw in the rest of the thresholds.

# Define thresholds as a list
thresholds = [-2, -1, -0.5, 0.25]

# Plot the set of thresholds on a new figure
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.plot(x, standard_normal.pdf(x), lw=2, color='black')

# Shade the cutpoints and add them
for alpha, cut in zip(np.linspace(1, .1, num=4), thresholds):
    ax.axvline(cut, color='black', lw=0.5)
    ax.fill_between(x, standard_normal.pdf(x), 
                    where=standard_normal.cdf(x) <= standard_normal.cdf(cut),
                    alpha=alpha)

What are the corresponding probabilities here? Visually, its clear that the fourth shaded area (response 4’s portion of the probability pie) and the unshaded area (response 5) have larger areas. We can compute them by subtracting each category from its previous one, which is easy using NumPy slicing.

# get the cdf of each threshold
cat_probs = standard_normal.cdf(thresholds)

# Subtract each probability from its previous 
cat_probs[1:] - cat_probs[:-1]

array([0.13590512, 0.14988228, 0.29016879])

Not enough probabilities here. We need 5, not three! We can solve this issue with a moment’s reflection. The probability of the very first category is equal to its cumulative density function. So we dont need to subtract a thing from that. The final probability - of response 5 in our case - should be the remaining probability left over after the rest of the probabilities have been computed. After all, these values will all sum to one, since we are dividing all of the probability of responses among the available responses. A simple computational trick we can use here is to book-end the cat_probs variable with a zero (left) and a one (right). That way, the slice operation will allow all the probabilities to be computed easily. In fact, this trick is used in PyMC to calculate the probabilites in a proper model, so its worth stating here. Here we go:

# Bookend
cat_probs = np.concatenate(
    [np.zeros(1),
     cat_probs,
     np.ones(1)]
)

# Get the response probabilities
response_probs = cat_probs[1:] - cat_probs[:-1]
response_probs

array([0.02275013, 0.13590512, 0.14988228, 0.29016879, 0.40129367])

And, finally, we can now get what we turned up for. A set of ordinal responses for our imaginary participants on how much they like chocolate. How to get this? Well, we can use a categorical distribution. This is the simplest kind of distribution. We simply provide a set of probabilities, the number of which indicate how likely each option is. PyMC has a nice Categorical distribution that we will now sample from, and compare the counts of the data to the response probabilities generated above.

# Actually get the responses, convert to Series
chocolate = pm.draw(pm.Categorical.dist(p=response_probs), draws=1000, random_seed=rng)
chocolate = pd.DataFrame(chocolate, columns=['chocolate_preference'])

# Show
sns.countplot(data=chocolate, x='chocolate_preference');
print(chocolate.value_counts(normalize=True).sort_index())

chocolate_preference
0                       0.016
1                       0.153
2                       0.154
3                       0.294
4                       0.383
Name: proportion, dtype: float64

That is a pretty close match to the normal distribution cutpoints we generated! You will notice the scale doesn’t run from 1-5 but zero to four. This does not matter at all, if we wanted we can add one to the output without changing anything. When working with ordinal models in PyMC we make sure the data runs from zero to K, the max response value. Add or subtract whatever you need to make sure the lowest possible response is zero.

Recovering parameters of an ordinal distribution with PyMC - the Hard Way

So far, we’ve generated some data from scratch, working backwards from the latent distribution to the observed data. But how would we model this data if we collected it in the wild, where we can only work from the observed ordinal data to make inference about the latent distribution? PyMC will make this easy for us, but we first need to make some assumptions and pin some things down, in a statistical sense.

When we are working with ordinal models, we have several parameters we may want to estimate: - The K-1 thresholds. With responses that can run from zero to K (e.g. 0, 1, 2, 3, 4, K being 5), we can estimate K-1 ordered thresholds. - The location or mean of the underlying latent distribution. - The scale or standard deviation of the underlying distribution.

This was hard for me to get my head around, but if you think about it for a while, you realise that you cannot estimate all of these at once. The model will be undetermined if you try to estimate all of these parameters. Without fixing down any two of these parameters, the model can’t be estimated, because each parameter set is estimated relative to another. For example, when we set the mean and standard deviation to zero and one above, we could place the thresholds wherever we liked to get the right probabilities within the bounds set by the distribution. As we’ll see later, we can also fix the first and last threshold to a constant value too, and then the distribution and remaining parameters will shift around those constraints to accommodate the probabilities.

In many examples of ordinal models, a standard normal cumulative density function is used, which implicity sets the mean and standard deviation to zero. Thats OK sometimes, but not always an assumption you’d want to make. Below, we write a very explicit PyMC model that will try to recover the thresholds we set above on the chocolate dataset. We’ll rely on the dims and coords arguments to make thing easy to manage. Note that this model will look very complex, because its explicitly layout out all the probability calculations, distributions to use, and so on. This will be simplifed later but this version could be adapted for complex problems - e.g., assuming the latent distribution is a t-distribution or similar.

# Figure out K
K = 5 # the number of possible responses
Kc = K - 1 # Number of cutpoints
latent_mean = 0 # Fixed mean
latent_sigma = 1 # Fixed sigma

# Set the coordinates
c = {'Kc': range(Kc), 
     'N': range(len(chocolate))
    }

with pm.Model(coords=c) as ordinal_chocolate:
    
    # First, priors on cutpoints, which are normally distributed themselves, 
    # and transformed to be ordered. 
    # We set them along the range of the latent distribution - note that we are doing this relative
    # to the knowledge we are fixing mean and sigma
    # Notice also the small SD, which says these cuts should not vary too much around the means, revisit as needed
    cutpoints = pm.Normal('cutpoints',
                          mu=np.linspace(-1, 1, num=Kc),
                          sigma=2,
                          transform=pm.distributions.transforms.univariate_ordered,
                          dims='Kc')
    
    
    # Working out the threshold CDFs from a distribution of choice, we set here to be normal with parameters fixed
    # if we wanted to, we could replace this with another distribution or swap out the fixed constants for priors
    cdf_getter = pm.Normal.dist(latent_mean, latent_sigma)
    
    # Get the CDFs of the cutpoints using logcdf, then exponentiate
    cdfs = pm.logcdf(cdf_getter, cutpoints).exp() # These are all the cumulative density probabilities now
    
    # Bookend the cdfs before subtraction using PyTensor - exactly the same as NumPy, just works for PyMC tensor variables
    cdfs = pt.concatenate(
        [pt.zeros(1),
         cdfs,
         pt.ones(1)]
    )
    
    # Compute probabilities
    p = cdfs[1:] - cdfs[:-1]
    
    # Place into a categorical distribution likelihood
    pm.Categorical('y', p=p, observed=chocolate['chocolate_preference'].values, dims='N')
    
    # Sample the posterior and then the posterior predictive distribution
    idata = pm.sample(random_seed=rng)
    idata.extend(pm.sample_posterior_predictive(idata))

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [cutpoints]
/Users/alexjones/opt/anaconda3/envs/pyTen/lib/python3.10/site-packages/pytensor/compile/function/types.py:970: RuntimeWarning: invalid value encountered in add
  self.vm()

/Users/alexjones/opt/anaconda3/envs/pyTen/lib/python3.10/site-packages/pytensor/compile/function/types.py:970: RuntimeWarning: invalid value encountered in accumulate
  self.vm()

100.00% [8000/8000 00:16<00:00 Sampling 4 chains, 0 divergences]

/Users/alexjones/opt/anaconda3/envs/pyTen/lib/python3.10/site-packages/pytensor/compile/function/types.py:970: RuntimeWarning: invalid value encountered in add
  self.vm()
/Users/alexjones/opt/anaconda3/envs/pyTen/lib/python3.10/site-packages/pytensor/compile/function/types.py:970: RuntimeWarning: invalid value encountered in add
  self.vm()
Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 35 seconds.
Sampling: [y]

100.00% [4000/4000 00:17<00:00]

That’s a lot to take in, but the model has sampled with a few warnings. Lets examine both the posterior of the cutpoints, as well as the posterior predictive distribution.

# Get distribution of cutpoints
az.plot_posterior(idata,
                  hdi_prob=.95, 
                  lw=2, 
                  color='black',
                  ref_val=thresholds);

az.plot_ppc(idata);

While the model didn’t quite nail the thresholds, it managed to capture them within the 95% credible intervals of each parameter. Reassuringly though, the model has a good approximation of the data, with the posterior predictive check aligning almost perfectly with the observed data.

Recovering parameters - the easy way

The above model is a lot to take in. While we will always specify priors over the cutpoints, specifying the distribution to use to calculate the cumulative density function as well as computing the category probabilities manually is a lot! Fortunately, PyMC has a convenience likelihood distribution that allows us to bypass a lot of the complexity of the above model in fewer lines of code. The above parameterisation is worth knowing though, as it allows us to expand our capabilities to more complex ordinal data. Before we leave behind the chocolate data, I’ll rewrite the model above to expand on the fantastic pm.OrderedProbit distribution which will encompass a lot of the code in the above model. Note that ‘probit’ simply refers to the use of a normal distribution to model ordinal data, as we’ve been doing.

# Rewrite to leverage orderedProbit
with pm.Model(coords=c) as probit_chocolate:
    
    # First, priors on cutpoints
    cutpoints = pm.Normal('cutpoints',
                          mu=np.linspace(-1, 1, num=Kc),
                          sigma=2,
                          transform=pm.distributions.transforms.univariate_ordered,
                          dims='Kc')
    
    
    # Likelihood of orderedProbit
    # Eta is the mean of the latent normal
    # Sigma is the SD
    # Cutpoints are the cutpoints
    # This function internally sets the distribution to a normal with eta and sigma
    # Evaluates the CDF, 
    # and computes probabilities to pass to a Categorical distribution
    pm.OrderedProbit('y', eta=0, sigma=1,
                     cutpoints=cutpoints,
                     observed=chocolate['chocolate_preference'].values
                    )
    
    # Sample the posterior and then the posterior predictive distribution
    idata = pm.sample(random_seed=rng)
    idata.extend(pm.sample_posterior_predictive(idata))

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [cutpoints]

100.00% [8000/8000 00:18<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 34 seconds.
Sampling: [y]

100.00% [4000/4000 00:17<00:00]

# Show results again
fig, ax = plt.subplots(1, 5, figsize=(30, 5))

# Get distribution of cutpoints
az.plot_posterior(idata,
                  hdi_prob=.95, 
                  lw=2, 
                  color='black',
                  ref_val=thresholds,
                  ax=ax[:-1])

# Plot PPC
az.plot_ppc(idata, ax=ax[-1]);

The results look more or less the same, as you can see - but there’s no warnings from the model this time, as the PyMC developers have much cleaner functions behind the scenes to ensure stable computations.

A flag in the ground - fixing cutpoints to estimate the location and scale

We’ll now examine a dataset in which the above approach - fixing mean and sigma to 0 and 1, respectively - simply doesn’t work, and highlight an approach to orindal modelling that I am most convinced by. The example data here is from the excellent book Doing Bayesian Data Analysis 2nd Edition, by John Kruschke. In the chapter on ordinal modelling, Kruschke provides a dataset (OrdinalProbitData-1grp-1.csv, get the book/data here) in which he takes the approach of fixing the top and bottom cutpoints of the model, and estimates the mean and sigma of the latent distribution (as well as the remaining thresholds). This is a particularly convincing approach, at least to me, because it allows for clever use cases such as comparing the latent means of two ordinal distributions (no more t-test style analyses), as well letting the mean be estimated by a linear combination of predictor variables with an error term, as in linear regression. First, lets read in Kruschke’s data, and see what our usual approach gets us, fixing the mean and sigma to zero and one.

# Read in data, subtract one from the observed data
one_group = (pd.read_csv('OrdinalProbitData-1grp-1.csv')
             .assign(y=lambda x: x['Y'].sub(1))
            )

display(one_group.head())
sns.countplot(data=one_group, x='y');

	Y	y
0	1	0
1	1	0
2	1	0
3	1	0
4	1	0

Lets try to model this using the approach we’ve seen so far. We’ll see if this generates the posterior-predictive fit we want.

# Set number of values
K = 7
Kc = K - 1

# Coords
c = {'Kc': range(Kc),
     'N': range(len(one_group))
    }

with pm.Model(coords=c) as dbda:
    
    # Priors on the cutpoints as before - note they are in relation to a standard normal
    cutpoints = pm.Normal('cutpoints',
                          mu=np.linspace(-2, 2, num=Kc),
                          sigma=2,
                          transform=pm.distributions.transforms.univariate_ordered,
                          dims='Kc')
    
    # Fix 0 and 1 in the orderedProbit
    pm.OrderedProbit('y', eta=0, sigma=1,
                     cutpoints=cutpoints,
                     compute_p=False,
                     observed=one_group['y'].values)
    
    idata = pm.sample(random_seed=rng)
    idata.extend(pm.sample_posterior_predictive(idata))

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [cutpoints]

100.00% [8000/8000 00:15<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 33 seconds.
Sampling: [y]

100.00% [4000/4000 00:03<00:00]

# Plot the PPC
az.plot_ppc(idata);

Overall, not too bad. The model captures pretty well the observed frequencies in the data. What about the cutpoints themselves?

# Get cutpoints
az.plot_posterior(idata);

The first cutpoint has an estimate of around 0, and they get progressively higher. How to make sense of this? Remember, we can interpret this on the latent normal distribution. Lets pull out the mean of each cutpoint, put them on a normal distribution, and compare it to the observed frequencies of the data:

# Get cutpoint means
cut_ests = idata['posterior']['cutpoints'].mean(['chain', 'draw']).to_numpy()

# Plot a normal
x_eval = np.linspace(-5, 5, 1000)
fig, (ax1, ax) = plt.subplots(1, 2, figsize=(12, 4))
ax.plot(x_eval, standard_normal.pdf(x_eval), lw=2, color='black')

# Add the cutpoints and shade
for alpha, c in zip(np.linspace(1, .1, num=Kc), cut_ests):
    ax.axvline(c, color='black', lw=0.5)
    
    ax.fill_between(x_eval, standard_normal.pdf(x_eval),
                    where=standard_normal.cdf(x_eval) <= standard_normal.cdf(c),
                    alpha=alpha)
    
sns.countplot(one_group, x='y', ax=ax1, color='grey', edgecolor='black');

This is what we discussed earlier. If you pin down two parameters - here the mean and the sigma - the cutpoints have to shift around that distribution to find the accommodating regions. Clearly, a huge chunk of the distribution is needed to fit the large number of zero responses, while the tiny amount of response 6 is in the very far reaches of the right tail of the distribution, where almost no probability exists.

For prediction purposes, there is nothing wrong with this parameterisation and distribution, as the predictive fit shows. But if we wanted to interpret the underlying latent normal more clearly, or indeed compare its mean to something else, we need to actually estimate that mean. Kruschke’s solution is elegant - pin down the first and last cutpoints on the scale of the data, by adding about 0.5 to the first observed response in the data, and subtracting 0.5 from the final response in the data. The way we can do this is by estimating the internal cutpoints, book-ending them with constants, and then placing priors on the mean and sigma values of the distribution. Lets see how this works in PyMC:

# Recap - number of responses
K = 7

# Number of cutpoints we'd *usually estimate*
Kt = K - 1

# Number of cutpoints we're going to pin now is 2, so Kc is actually
Kc = Kt - 2

# Coords
c = {'Kc': range(Kc),
     'Kt': range(Kt),
     'N': range(len(one_group))
    }

with pm.Model(coords=c) as dbda_probit:
    
    # Priors on the cutpoints we will estimate. Notice their means now traverse the values of the ordinal scale!
    # Also notice the sigma is small, keeping them focused around the stated mean
    cutpoints_raw = pm.Normal('cutpoints_raw',
                              mu=np.linspace(2, 5, num=Kc),
                              sigma=.2,
                              transform=pm.distributions.transforms.univariate_ordered,
                              dims='Kc')
    
    # Now booked and store into a Deterministic 
    cutpoints = pm.Deterministic('cutpoints',
                                 pt.concatenate(
                                     [
                                         pt.ones(1)*1.5, # The lowest observed data - remember that was 1, not zero!
                                         cutpoints_raw,
                                         pt.ones(1)*6.5 # The highest cutpoint 
                                     ]
                                 ), 
                                 dims='Kt'
                                )
    
    # Priors for mu and sigma - on the scale of the data
    μ = pm.Normal('μ', mu=4, sigma=2)
    σ = pm.HalfNormal('σ', sigma=3)
    
    # Into orderedProbit
    pm.OrderedProbit('y', eta=μ, sigma=σ,
                     cutpoints=cutpoints,
                     observed=one_group['y'].values)
    
    idata2 = pm.sample(random_seed=rng)
    idata2.extend(pm.sample_posterior_predictive(idata2))

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [cutpoints_raw, μ, σ]

100.00% [8000/8000 00:18<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 35 seconds.
Sampling: [y]

100.00% [4000/4000 00:02<00:00]

# Check the posterior predictive
az.plot_ppc(idata2);

And take a look at the parameters, which closely match those of Kruschke’s reported data.

az.plot_posterior(idata2, var_names=['μ', 'σ', 'cutpoints_raw']);

The key insight here is that we have now estimated that the mean of the latent distribution is around 1, and the sigma is 2.1. Lets take a look at the means of the posterior and visualise this distribution we have estimated:

# Aggregate parameters
params = idata2['posterior'][['μ', 'σ', 'cutpoints']].mean(['chain', 'draw'])

# Build a normal that fits the estimates
x_eval = np.linspace(-6, 8, 1000)
new_normal = st.norm(params['μ'], params['σ'])

# Plot
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
ax.plot(x_eval, new_normal.pdf(x_eval), lw=3, color='black')

# Add the cuts
[ax.axvline(c, lw=0.5, color='black') for c in params['cutpoints']]

# Shade
for alpha, c in zip(np.linspace(1, .1, num=Kt), params['cutpoints']):
    
    ax.fill_between(x_eval, new_normal.pdf(x_eval),
                    where=new_normal.cdf(x_eval) <= new_normal.cdf(c),
                    alpha=alpha)

# Clip the axis to emphasise range of data, where the original data lived
ax.set(xlim=(0, 8), xticks=range(1, 8));

This looks similar to the one we saw in the initial model. But its clear the distribution has ‘shifted’ along the X axis, and is spread out. Either parameterisation is OK, but this one has a lot of benefits, and extends naturally to the linear regression case, as we will see. Kruschke highlights further examples in his chapter, such as comparing the latent means/SDs of two ordinal measures, seamlessly using Bayesian inference to derive distributions of latent mean-difference effect sizes.

When looking hot means not feeling cold

We’re now at a point where we’ve pinned down a method of working with ordinal data to estimate the parameters of the underlying latent distribution. Extending this to a regression context is now natural. Consider a standard linear regression, where the goal is to estimate the expectancy of a normal distribution from a set of linear predictors, as well as the noise of the measure (i.e., the residuals).

To highlight the use case of a true ordinal regression, I am going to use a dataset from social psychology, a paper titled “When looking hot means not feeling cold”. In this study, the authors photographed women on a night out during a cold period in Florida.

The authors asked the women how cold they felt, responding on a 1 (not at all cold) to a 6 (extremely cold) scale. They also measured the tendency for their participants to self-objectify, i.e., show more concern about how they look, rather than how they feel. From the photographs, the researchers also quantified the amount of skin the women showed. Using standard linear regression, the authors reported a statistically significant interaction between the level of self-objectification and skin exposure (after adjusting for the actual temperature), such that only women low in self-objectification with high levels of skin exposure reported feeling colder - those with high levels of self-objectification did not “feel the cold”.

Its worth a read. Its an interesting paper and the authors did a good job testing their theory in an creative way. When the paper came out, the authors sadly received huge backlash because of the marginally significant p-value of the interaction, and lots of people jumped in to re-analyse their data with more covariates, simulations, and more, desperately trying to show the effect was not significant. In my view, all of these criticisms missed the point, which was that the data was probably better suited to being modelled by an ordinal outcome. As an aside, this is the issue with frequentist null-hypothesis significance testing approaches - focusing on that arbitrary threshold of “existence” is silly, and people can be really motivated to knock something to either side of it. With the Bayesian approach, we can incorporate some skepticism into the model as well as modelling it appropriately and see what we find.

I’ve got the data from the Open Science Framework, and am reading it in here. Huge thanks to the authors for sharing their data.

# Read in data and subset out the columns and rows we need
# Also Z-score the predictor variables
hotcold = (pd.read_spss('field data_coded_clean.sav')
           .query('`filter_$` == "Selected"')
           .filter(items=['ID', 'Cold', 'Self_Obj', 'Temp_True', 'Clothes_RF'])
           .pipe(lambda df_: df_.assign(**df_[['Self_Obj', 'Temp_True', 'Clothes_RF']].apply(st.zscore, ddof=1).add_suffix('_Z')))
          )

hotcold.head()

	ID	Cold	Self_Obj	Temp_True	Clothes_RF	Self_Obj_Z	Temp_True_Z	Clothes_RF_Z
0	170.0	1.0	4.625	50.0	2.0	0.942003	-0.694347	-0.542569
1	26.0	1.0	5.375	52.0	2.0	1.745924	-0.140646	-0.542569
2	27.0	3.0	3.375	50.0	0.0	-0.397866	-0.694347	-1.567422
3	67.0	3.0	3.000	52.0	4.0	-0.799827	-0.140646	0.482284
4	68.0	6.0	5.250	52.0	6.0	1.611937	-0.140646	1.507137

Lets first recreate their analyses using good ol’ NHST and OLS, which is the prediction of Cold by the interaction of self objectification and skin exposure, adjusting for actual temperature.

# Fit an OLS model
ols_mod = smf.ols('Cold ~ Temp_True_Z + Self_Obj_Z * Clothes_RF_Z', data=hotcold).fit()

# display summary
ols_mod.summary(slim=True)

Cold      R-squared:             0.060

         OLS      Adj. R-squared:        0.040

OLS Regression Results
Dep. Variable:
Model:
No. Observations:	186	F-statistic:	2.909
Covariance Type:	nonrobust	Prob (F-statistic):	0.0230

             3.4489     0.103    33.615  0.000     3.246     3.651

          -0.1755     0.100    -1.758  0.080    -0.373     0.021

           -0.0069     0.105    -0.066  0.947    -0.213     0.200

          0.1458     0.104     1.400  0.163    -0.060     0.351

Intercept
	coef	std err	t	P>\|t\|	[0.025]
Temp_True_Z
Self_Obj_Z
Clothes_RF_Z
Self_Obj_Z:Clothes_RF_Z	-0.2063	0.098	-2.111	0.036	-0.399	-0.013

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

This highlights the contentious interaction that attracted so much social media ire. Lets have this linear model make a prediction to understand the interaction a bit more, which we will also end up doing with a PyMC model. We’ll estimate the feeling of cold for hypothetical individuals with ±1 SD on the clothes variable, and ±1 SD on the self objectivation variable.

# Prediction dataset
predmat = pd.DataFrame(
    {
        'Self_Obj_Z': [-1, -1, 1, 1],
        'Clothes_RF_Z': [-1, 1, -1, 1],
        'Temp_True_Z': [0] * 4
    }
)

# get estimates
estimates = ols_mod.get_prediction(predmat).summary_frame().filter(items=['mean', 'mean_ci_lower', 'mean_ci_upper'])

# Quick plot
(
    so.Plot(data=predmat.assign(**estimates), 
            x='Clothes_RF_Z', y='mean', 
            ymin='mean_ci_lower', ymax='mean_ci_upper', 
            linestyle='Self_Obj_Z')
    .add(so.Lines(color='black'), so.Dodge())
    .add(so.Dot(color='black'), so.Dodge())
    .add(so.Range(color='black', linewidth=1), so.Dodge())
    .scale(x=so.Nominal())
    .label(x='Skin Exposure', y='Predicted Cold Feeling', linestyle='Self Objectification')
    .theme(plt.style.library['bmh'])
)

This recreates the overall finding, such that low self-objectifiers feel colder with more skin exposure, high self objectifiers show no evidence for a change. Let’s now build an ordinal regression in PyMC to refit the data with a) a proper ordered likelihood, and b) a bit of regularisation on the priors to limit noisy inference. I only use a basic type of regularisation in the below example, but there are more elaborate ways of doing this.

The key thing to remember here is, when using a normal latent variable, is that the coefficients represent changes in the underlying latent distribution - i.e., if the coefficient were 1, it would indicate a 1-unit increase in the predictor leads to a 1 unit change in the underlying latent distribution.

# Use formulae to build the design matrix
model_design = dm('Cold ~ Temp_True_Z + Self_Obj_Z * Clothes_RF_Z', data=hotcold)

# Obtain X and y
y = np.array(model_design.response) - 1 # Notice subtraction
X = model_design.common.as_dataframe().drop(columns='Intercept') # Remove the intercept column

# Set up the coordinate dictionary
K = 6 # max number
Kc = K - 1 # The number of cutpoints
Kt = Kc - 2 # We are pinning two

# Coord dict
c = {'Kt': range(Kt), # cutpoints to estimate
     'Kc': range(Kc), # total cutpoint
     'coefs': X.columns, # Coefficients
     'N': range(len(y))
    }

# Build the model
with pm.Model(coords=c) as hotcold_ordinal:
    
    # Set the X data as mutable, we will later make predictions
    Xdata = pm.MutableData('Xdata', np.array(X))
    ydata = pm.MutableData('ydata', y)
    
    # First, set cupoints we will estimate
    cut_raw = pm.Normal('cut_raw',
                        mu=np.linspace(2, 5, num=Kt), # on range of data
                        sigma=0.25,
                        transform=pm.distributions.transforms.univariate_ordered,
                        dims='Kt')
    
    # Cutpoints
    cutpoints = pm.Deterministic('cutpoints',
                                 pt.concatenate(
                                     [
                                         pt.ones(1)*1.5, # First cutpoint
                                         cut_raw,
                                         pt.ones(1)*5.5 # Second cutpoint
                                     ]
                                 ),
                                 dims='Kc')
    
    # Model terms 
    λ = pm.Gamma('λ', alpha=3, beta=3) # The regulariser
    β0 = pm.Normal('β0', mu=3, sigma=1) # Intercept
    β = pm.Normal('β', mu=0, sigma=λ, dims='coefs') # Coefficients with a hierarchical variability
    σ = pm.HalfNormal('σ', sigma=1) # Model error
    
    # Take linear combination
    μ = β0 + Xdata @ β
    
    # Likelihood
    pm.OrderedProbit('y', eta=μ, sigma=σ,
                     cutpoints=cutpoints,
                     compute_p=False,
                     observed=ydata)

Automatically removing 1/187 rows from the dataset.

Before we sample this, lets take a look at the model structure. We’re using an ordinal likelihood but the same model structure as the standard OLS used by the authors, but we are adding some regularisation to the parameters to constrain the coefficients some.

# View the model
pm.model_to_graphviz(hotcold_ordinal)

This further highlights that ordinal regression is really a categorical prediction problem. Lets now obtain the posterior and predictive distribution of the model.

# Obtain posterior
with hotcold_ordinal:
    idata = pm.sample(random_seed=rng) # Posterior
    idata.extend(pm.sample_posterior_predictive(idata)) # Posterior predictive
    idata.extend(pm.sample_prior_predictive()) # Also get prior for later

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [cut_raw, λ, β0, β, σ]

100.00% [8000/8000 00:29<00:00 Sampling 4 chains, 1 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 46 seconds.
There were 1 divergences after tuning. Increase `target_accept` or reparameterize.
Sampling: [y]

100.00% [4000/4000 00:05<00:00]

Sampling: [cut_raw, y, β, β0, λ, σ]

Let’s take a look at the posterior distribution of the interaction, the effect of interest in the paper, as well as the posterior predictive check. We will also submit the interaction coefficient to a Bayes Factor hypothesis test.

# Visualise
fig, ax = plt.subplots(1, 3, figsize=(16, 6))

# Plot posterior of interaction
az.plot_posterior(idata, 
                  var_names='β', coords={'coefs': 'Self_Obj_Z:Clothes_RF_Z'},
                  hdi_prob=.95, lw=3, color='black',
                  ref_val=0, ax=ax[0])

# Plot ppc
az.plot_ppc(idata, ax=ax[1]);

# Bayes Factor plot
# Extract data
bf = az.from_dict(
    posterior={'interaction': az.extract(idata, var_names='β', group='posterior').sel(coefs='Self_Obj_Z:Clothes_RF_Z')},
    prior={'interaction': az.extract(idata, var_names='β', group='prior').sel(coefs='Self_Obj_Z:Clothes_RF_Z')}
)

# Plot BF
az.plot_bf(bf, var_name="interaction", ref_val=0, ax=ax[2]);

/Users/alexjones/opt/anaconda3/envs/pyTen/lib/python3.10/site-packages/IPython/core/events.py:89: UserWarning: Creating legend with loc="best" can be slow with large amounts of data.
  func(*args, **kwargs)
/Users/alexjones/opt/anaconda3/envs/pyTen/lib/python3.10/site-packages/IPython/core/pylabtools.py:152: UserWarning: Creating legend with loc="best" can be slow with large amounts of data.
  fig.canvas.print_figure(bytes_io, **kw)

According to the posterior distribution of the coefficient, there is an almost 97% probability of the coefficient being less than zero - it doesn’t quite exclude that magical parameter, but based on the model and the data, we’d deem a negative relationship is likely. The predictive check looks good, mirroring the data. The Bayes Factor plot suggests that the value of zero is similarly likely under both the posterior and prior distribution of the coefficient, which most Bayes Factor proponents would deem an inconclusive finding.

Let’s move beyond these hypothesis tests and see if we can use the model to really dig into the predictions. As our model is Bayesian, the predictions it makes for a datapoint will be a distribution over the possible scale response options. What we’ll now do is ask the model to predict the hypothetical individuals the standard linear model predicted, and thus we can see the likely response profile of feeling cold a hypothetical individual might make. First, let’s make the model return the predictions on new data.

# Have formulae create our design matrix
newX = (model_design
        .common.evaluate_new_data(predmat) # the data we made earlier for OLS
        .as_dataframe()
        .drop(columns='Intercept')
       )


with hotcold_ordinal:
    # Set new data
    pm.set_data({'Xdata': np.array(newX), 
                 'ydata': np.zeros(len(newX)) # placeholder, not important
                })
    
    # Predict
    predictions = pm.sample_posterior_predictive(idata, extend_inferencedata=False, predictions=True)

Sampling: [y]

100.00% [4000/4000 00:01<00:00]

Now we have the predictions - which are draws from the posterior predictive distribution, conditional on the hypothetical model inputs - we can get a sense of how the pattern of responses changes at these different levels of skin exposure and self objectification. These predictions are a bit unwieldy at the moment, so it makes sense to aggregate them a bit. For each of the four predicted observations, we can take the frequency of each of the responses in the posterior:

# Get frequencies and add in prediction data, and reshape
freq_counts_posterior = (predictions
                         ['predictions']
                         .to_dataframe()
                         .add(1) # adding one to resemble original data
                         .groupby(['y_dim_2']) # Over each observation
                         ['y']
                         .value_counts(normalize=True)
                         .unstack('y')
                         .pipe(lambda x: pd.concat([predmat, x], axis='columns'))
                         .melt(id_vars=['Self_Obj_Z', 'Clothes_RF_Z', 'Temp_True_Z'],
                                        var_name='response', value_name='pp')
                         .replace({'Self_Obj_Z': {-1: 'Low Self Objectify', 1: 'High Self Objectify'},
                                   'Clothes_RF_Z': {-1: 'Low Skin Exposure', 1: 'High Skin Exposure'}})
                        )

freq_counts_posterior.head()

	Self_Obj_Z	Clothes_RF_Z	response	pp
0	Low Self Objectify	Low Skin Exposure	1	0.14525
1	Low Self Objectify	High Skin Exposure	1	0.06600
2	High Self Objectify	Low Skin Exposure	1	0.09750
3	High Self Objectify	High Skin Exposure	1	0.11025
4	Low Self Objectify	Low Skin Exposure	2	0.13850

And lets see if we can plot these implied predictions across the categories.

(
    so.Plot(data=freq_counts_posterior, x='response', y='pp', color='Self_Obj_Z')
    .add(so.Bar(alpha=.2), so.Dodge())
    .facet('Clothes_RF_Z')
    .scale(x=so.Nominal(values=[1, 2, 3, 4, 5, 6]))
    .theme(plt.style.library['bmh'])
    .layout(size=(10, 5))
)

Here we can see the implied probabilites of a response across the conditions the model predicted. For high skin exposure, high self objectifiers are more likely to respond a 1, its about equal for 2, more likely to respond a 3, and beyond that, there is higher probability of a low self objectifier using the higher categories. For low skin exposure, its almost the opposite pattern - high self objectifiers seem to have a higher probability of responding to being colder than low self objectifiers. The highest likelihood is with the responses 3/4 in both cases.

Its possible to simplify (complicate?) this a bit further, by taking the odds ratio of a condition against the other. For example, we can take the odds of a “1” response for a low self objectifying individual under the low skin exposure condition, divide it by the odds of a “1” response for a high self objectifying individual, under the same condition. Lets see what that looks like:

# Compute odds ratios
odd = (freq_counts_posterior
       .assign(odds=lambda x: x['pp'] / (1 - x['pp']))
       .groupby(['Clothes_RF_Z', 'response']) # comparing self objectifer here, as we will merge over those
       .apply(lambda x: x.iloc[1, -1] / x.iloc[0, -1]) # odds of a high self objectifier to a low
       .to_frame('odds')
       .reset_index()
      )

# Plot
(
    so.Plot(data=odd, x='response', y='odds', color='Clothes_RF_Z')
    .add(so.Bar(), so.Dodge())
    .theme(plt.style.library['bmh'])
    .label(y='Odds of High Objectifier to Low Objectifier\nselecting response')
)

We can see a clear pattern here. For high skin exposure, the odds a high self-objectifier selects a 1 is much greater than a low self objectifier, and the pattern reverses as the response increases up to feeling very cold.

A final note on priors for cutpoints

Throughout the examples used above, I’ve been placing normally distributed priors on each of the cutpoints. This has been mostly fine, though the standard deviation of those normal distributions has sometimes been quite small to make the distribution about each cutpoint narrow. Some may object to this, and indeed, in some of the examples above, reduction of the prior width was required to get the model to estimate.

The Bayesian modelling community have been advocating the use of a Dirichlet prior over the cutpoints (see here). The Dirichlet is a multivariate generalisation of the Beta distribution, generating a set of probabilities that all sum to one. With some tricks and transforms, its possible to use the Dirichlet to set strictly ordered cutpoints.

The trick is to take the cumulative sum across the draws from a Dirichlet, and add and multiply values to it to expand its range (which is naturally always between 0-1). We’ll revisit the data from Kruschke a final time to put Dirichlet priors on the cutpoints. First, lets see a draw from a Dirichlet.

# Get a draw for 5 cutpoints from the Dirichlet
draw1 = pm.draw(pm.Dirichlet.dist(a=np.ones(5)), 1)
print('Five values in the range of 0-1', draw1.round(2))

# How can we constrain order? Use a cumulative sum
print('Five ordered values, range 0-1:', draw1.cumsum().round(2))

# Now expand them to the range of 2.5 to 5.5, like we did earlier for the normal priors
2.5 + draw1.cumsum() * (5.5 - 2.5)

Five values in the range of 0-1 [0.06 0.19 0.16 0.58 0.02]
Five ordered values, range 0-1: [0.06 0.25 0.41 0.98 1.  ]





array([2.67595003, 3.24229597, 3.71680105, 5.44940491, 5.5       ])

The final step above uses a mathematical sleight-of-hand to expand the range. We can expand 0-1 numbers in the range of [a, b] by the simple formula of a + x * (b - a).

More confusion awaits however. As a result of the cumulative sum, the final cutpoint will always be a constant. Before transforming, it will be fixed at 1, so will always take on the value b after transforming. If we want to do the Kruschke-style top and bottom pinned cutpoints, we can actually omit the top cutpoint, as it will fall out of the cumulative-sum Dirichlet, and simply pin the bottom.

Let’s see how to implement this in PyMC using the one_group data.

# Reminder of one-group
K = 7 # number of options
Kc = K - 1 # number of cutpoints in total
Kt = Kc - 1 # We want now to only pin ONE

# Create coordinates
c = {'Kc': range(Kc),
     'Kt': range(Kt)}

with pm.Model(coords=c) as dirichlet_prior:
    
    # Cutpoints to estimate
    cut_raw = pm.Dirichlet('cut_raw',
                           a=np.ones(Kt),
                           dims='Kt')
    
    # Pin lower and transform
    cutpoints = pm.Deterministic('cutpoints',
                                 pt.concatenate(
                                     [
                                         np.ones(1)*1.5, # Fixing the first cut
                                         2.5 + cut_raw.cumsum() * (6.5 - 2.5) # The transform of the cumulative sum, top (which we want fixed) - lower
                                     ]
                                 ),
                                 dims='Kc')
    
    # Mu and sigma
    μ = pm.Normal('μ', mu=4, sigma=2)
    σ = pm.HalfNormal('σ', sigma=3)
    
    # Into orderedProbit
    pm.OrderedProbit('y', eta=μ, sigma=σ,
                     cutpoints=cutpoints,
                     compute_p=False,
                     observed=one_group['y'].values)
    
    idata3 = pm.sample(random_seed=rng)

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [cut_raw, μ, σ]

100.00% [8000/8000 00:18<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 35 seconds.

You definitely get a speedup in sampling! Let’s examine the estimates of the cutpoints:

# Check the output between both models
pd.concat(
    [
        az.summary(idata3, var_names=['cutpoints', 'μ', 'σ'], kind='stats').assign(model='Dirichlet Prior'),
        az.summary(idata2, var_names=['cutpoints', 'μ', 'σ'], kind='stats').assign(model='Normal Prior')
    ]
)

	mean	sd	hdi_3%	hdi_97%	model
cutpoints[0]	1.500	0.000	1.500	1.500	Dirichlet Prior
cutpoints[1]	2.718	0.179	2.500	3.043	Dirichlet Prior
cutpoints[2]	3.750	0.314	3.189	4.335	Dirichlet Prior
cutpoints[3]	4.778	0.404	4.057	5.532	Dirichlet Prior
cutpoints[4]	5.671	0.389	4.945	6.317	Dirichlet Prior
cutpoints[5]	6.500	0.000	6.500	6.500	Dirichlet Prior
μ	1.044	0.357	0.361	1.693	Dirichlet Prior
σ	2.605	0.397	1.886	3.322	Dirichlet Prior
cutpoints[0]	1.500	0.000	1.500	1.500	Normal Prior
cutpoints[1]	2.144	0.110	1.943	2.354	Normal Prior
cutpoints[2]	3.048	0.148	2.767	3.318	Normal Prior
cutpoints[3]	4.035	0.159	3.730	4.331	Normal Prior
cutpoints[4]	5.106	0.181	4.763	5.442	Normal Prior
cutpoints[5]	6.500	0.000	6.500	6.500	Normal Prior
μ	1.035	0.289	0.483	1.543	Normal Prior
σ	2.131	0.280	1.649	2.679	Normal Prior

The results look broadly similar here, which is reassuring. But, the parameters of the Dirichlet prior model are closer to those reported in the text. While an ordered transform of the normal distribution certainly works, in cases where we know the generating parameters, a model formulation that brings us closer to that is probably to be preferred, so its really vital to understand this formulation, even if at first it is tricky to comprehend.

%load_ext watermark
%watermark --iversions

statsmodels: 0.13.5
numpy      : 1.25.0
scipy      : 1.11.1
pymc       : 5.6.0
pytensor   : 2.12.3
arviz      : 0.15.1
matplotlib : 3.7.2
pandas     : 2.0.3
seaborn    : 0.12.2

Out-of-sample predictions with PyMC: A hieararchical linear model case study

Thu, 29 Jun 2023 23:00:00 GMT

PyMC is an awesome probabilistic programming language that makes it easy to build Bayesian statistical models, and I’ve used it exclusively for several years for data analysis problems. Despite its flexibility, one of the few sticking points I’ve encountered is trying to generate out-of-sample predictions. In the age of machine learning and predictive modelling, simply fitting a model to data and assessing the quality of predictions is no longer the gold standard - rather, our models should be able to predict data that they have never seen before. Python libraries like scikit-learn place the predictive utility of models front and centre, but it is not so straightforward to do this in PyMC. This is a shame, because Bayesian models come with free uncertainty around predictions - an incredibly powerful feature when uncertainty is so important to quantify, and yet point estimates or confusing frequentist confidence intervals are the main currency of prediction.

This problem is especially compounded with hierarchial models, because they several kinds of choices when doing out-of-sample predictions! For example: - A hierarchical model could predict new, unobserved data for an “average” person, omitting the group-specific effects entirely - this is common in many packages as the default, such as in lme4 in R. - A hierarchical model could predict on new, unobserved data for a known group-specific (or random effect) value. An example would be using the model to predict new scores on an unobserved quantity for an individual already in the dataset. - A hierarchical model might predict unobserved data for an unknown group-specific effect. An example of this would be predicting scores for unseen, new groups.

Doing this in PyMC is possible, but not immediately obvious. Here, I’ll try some examples of out of sample predictions using PyMC. Much of this was inspired by the excellent webinar on out of sample predictions by Ricardo Vieira and Alex Andorra, on the Learning Bayesian Statistics podcast. Its also worth noting that the excellent bambi package abstracts much of this away and makes prediction very easy, but there are still many occasions when I need to write a model myself that bambi doesn’t yet support or I need more specific control and detail.

Let’s go!

# Imports
import bambi as bmb
import pymc as pm
import pandas as pd
import numpy as np
import arviz as az
import seaborn as sns
import seaborn.objects as so
import matplotlib.pyplot as plt

from itertools import product

rng = np.random.default_rng(42)
plt.style.use('bmh')

As a working example, I’ll use the sleepstudy dataset that comes with bambi. This simple dataset is hiearchical in nature, showing reaction times on a task for people who had participated in a sleep deprivation study. Participants are measured over several days, providing reaction times once a day. Thus, the reaction times are nested within the participants, making for an ideal case of hiearchical modelling.

# Load sleep
sleep = bmb.load_data(dataset='sleepstudy')
display(sleep.head())

# Visualise the reaction times for each day
sns.kdeplot(data=sleep, x='Reaction', hue='Days');

	Reaction	Days	Subject
0	249.5600	0	308
1	258.7047	1	308
2	250.8006	2	308
3	321.4398	3	308
4	356.8519	4	308

Reaction time gets a bit noisier and slower, as the length of deprivation increases, as we’d expect!

Out of sample predictions, approach one - `pm.MutableData`

In situations outside of the hierarchical model, PyMC has a data container class that allows for simple switching out of predictor and observed datasets inside a model. The model is first estimated to get the posterior, and then the data is swapped out for new observations, and a posterior-predictive sample can be drawn.

Below, a non-hiearchical (or ‘no pooling’) model is fitted to the data. This obviously ignores the nesting withing participants, so is done only to illustrate how easy pm.MutableData makes swapping out predictions. First, let us build a simple linear regression, predicitng reaction time from the observed number of days, and setting the Days variable as a mutable object, as well as setting the dependent variable Reaction as one, too. The priors will be roughly informative given the scale of the data we see in the above plot.

# Build the model
with pm.Model() as no_pool:
    
    # Set Days as mutable data
    X = pm.MutableData('X', sleep['Days'].to_numpy())
    y = pm.MutableData('y', sleep['Reaction'].to_numpy())
    
    # Prior for intercept, roughly informative
    β0 = pm.Normal('β0', mu=0, sigma=300)
    
    # Prior for slope of Days, also roughly informative
    β1 = pm.Normal('β1', mu=0, sigma=20)
    
    # Noise variability
    σ = pm.HalfCauchy('σ', beta=10)
    
    # Linear predictor
    μ = β0 + X*β1
    
    # Likelihood is normal
    pm.Normal('likelihood', mu=μ, sigma=σ, observed=y)
    
    # Sample
    idata = pm.sample()

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [β0, β1, σ]

100.00% [8000/8000 00:11<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 31 seconds.

With the posterior in hand, we can examine distributions of the parameters if we want to. We can also sample the posterior-predictive distribution, which will inform us of the distribution of the likely future values, integrating over the uncertainty in the parameters of the model. To be very strict with definitions, this is more of a retrodiction than a prediction, because the model will generate likely values for the data about which it has seen. Lets generate the posterior predictive (for the in-sample data), and see how it compares to the original data.

# Sample the posterior predictive
with no_pool:
    idata.extend(pm.sample_posterior_predictive(idata))

# Visualise 
az.plot_ppc(idata, num_pp_samples=500);

Sampling: [likelihood]

100.00% [4000/4000 00:00<00:00]

This looks reasonable, but the predictions could be better, but we will move on!

Now lets imagine a scenario where some unpleasant experimenter wants to know what happens if we were to deprive people of sleep for 15 days - what would the reaction time look like for that? We can simply add in a new datapoint (remember our model is entirely blind to different participants) using pm.set_data, for our X mutable data, and resample the posterior predictive distribution. We will also need to add a “placeholder” value for y. In many cases you may genuine observations for Y that you are predicting, but we don’t here, so we simply add a new variable as a stopgap.

# Generate a prediction for reaction time for 20 days of deprivation
with no_pool:
    
    # Set the data, and add in a new coordinate for it
    pm.set_data({'X': np.array([15]),
                 'y': np.array([-99])
                })
    
    # Sample posterior predictive, and set predictions = True
    idata.extend(pm.sample_posterior_predictive(idata, predictions=True))

Sampling: [likelihood]

100.00% [4000/4000 00:00<00:00]

Lets see what happens by extracting the data point, and comparing it to the observed data

# Plot the data
with sns.plotting_context('poster'):
    
    # Canvas
    fig, ax = plt.subplots(1, 1, figsize=(10, 5))
    
    # Scatter the observed data
    sns.scatterplot(data=sleep, x='Days', y='Reaction', ax=ax, color='white', edgecolor='black', label='Observed')
    
    # Add in the predicted datapoints mean by aggregating it across chains and draws
    ax.plot(15, idata['predictions']['likelihood'].mean(('chain', 'draw')), color='black', marker='s', label='Predicted')
    
    # Compute the HDI
    hdi = az.hdi(idata['predictions']['likelihood'],
                 hdi_prob=.95).to_array().to_numpy()
    
    # Plot it
    ax.plot([15, 15], hdi.flatten(), color='black')
    
    # Clean up axis
    ax.set(ylabel='Reaction Time', 
           xticks=range(0, 16))
    
    # Turn on legend()
    ax.legend(frameon=False)

The model implies, with some uncertainty, that the reaction time will be pretty slow after 20 days in the study. Certainly it will be in the range of some of the highest observed reaction times in the study.

Fitting the hierarchical model

We can now turn to predictions with a hierarchical model. For the purposes of the sleepstudy data, this involves a group-specific effect for each participant who provides multiple ratings. This is achievable by taking advantages of the labelled coordinates PyMC accepts, and indexing the prior for each participant using fancy-indexing like any normal NumPy array. First we build the model, using the .factorize() method of a pd.Series object to obtain the indexes and labels for each participant.

# Get index locations and labels
subj_ind, subj_label = sleep['Subject'].factorize()

print(subj_ind)
print(subj_label)

[ 0  0  0  0  0  0  0  0  0  0  1  1  1  1  1  1  1  1  1  1  2  2  2  2
  2  2  2  2  2  2  3  3  3  3  3  3  3  3  3  3  4  4  4  4  4  4  4  4
  4  4  5  5  5  5  5  5  5  5  5  5  6  6  6  6  6  6  6  6  6  6  7  7
  7  7  7  7  7  7  7  7  8  8  8  8  8  8  8  8  8  8  9  9  9  9  9  9
  9  9  9  9 10 10 10 10 10 10 10 10 10 10 11 11 11 11 11 11 11 11 11 11
 12 12 12 12 12 12 12 12 12 12 13 13 13 13 13 13 13 13 13 13 14 14 14 14
 14 14 14 14 14 14 15 15 15 15 15 15 15 15 15 15 16 16 16 16 16 16 16 16
 16 16 17 17 17 17 17 17 17 17 17 17]
Index([308, 309, 310, 330, 331, 332, 333, 334, 335, 337, 349, 350, 351, 352,
       369, 370, 371, 372],
      dtype='int64')

The first variable keeps track of where responses belong to which group (i.e., participant), and the second is the label. We use the first to index the prior in the model, and the second to give the prior instantiation the number of individual priors, as PyMC supports multidimensional priors:

# First set up the coords, passed as a dictionary
c = {'subject': subj_label} # PyMC infers number of labels

with pm.Model(coords=c) as hierarchical:
    
    # Add in X and y (though this is not strictly needed
    X = pm.MutableData('X', sleep['Days'].to_numpy())
    y = pm.MutableData('y', sleep['Reaction'].to_numpy())
    
    # Hyperprior for the variability of the subjects
    subj_σ = pm.HalfCauchy('subj_σ', beta=50)
    subj_offset = pm.ZeroSumNormal('subj_offset', sigma=subj_σ, dims='subject') # Zero sum simplifies hiearchical modelling, and dims infers number of individual priors
    
    # Roughly informative coefficients prior
    β0 = pm.Normal('β0', mu=0, sigma=300)
    β1 = pm.Normal('β1', mu=0, sigma=20)
    
    # Noise variability
    σ = pm.HalfCauchy('σ', beta=50)
    
    # Linear predictor now includes indexing of the offset with `subj_ind`
    μ = β0 + subj_offset[subj_ind] + X*β1
    
    # Likelihood is normal
    pm.Normal('likelihood', mu=μ, sigma=σ, observed=y)
    
    # Sample
    hierarchical_idata = pm.sample()

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [subj_σ, subj_offset, β0, β1, σ]

100.00% [8000/8000 00:18<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 1_000 draw iterations (4_000 + 4_000 draws total) took 35 seconds.

# Examine the trace, as we'll be using this model a lot
az.plot_trace(idata, figsize=(10, 15));

Out of sample predictions

Type 1 - predicting for an “average” person, or omitting the group-specific terms

With a fitted hierarchical model, we can start trying to make predictions. The first kind of prediction we’ll examine is the most common - predicting with only the common, or “fixed” effects. In the case of this model, that would be relying on the intercept and slope terms (β0 and β1 respectively) to take in a new predictor value and output a prediction, with no reference to the group specific effects.

If we try the method above - setting new data for X and y - we will immediately encounter an error. The model will take in the new data and try to pass it through the linear predictor formula, which includes the group-specific indexing of subj_offset, and it won’t work.

The solution, which is not immediately obvious, is to build another model with an identical structure, add in a new variable we would like to sample, and have the model sample the posterior predictive distribution of it. Lets say we want to predict, for the average person, the effects of 10, 11, and 12 days of taking part in the study, but we want to omit the group specific effect entirely. There will likely be little difference in this toy example out of sample predictions in this fashion and the no-pooled model, but more complex hieararchical models would definitely show a different pattern.

# First, set the new data we want to predict
newX = np.array([10, 11, 12])

# Build a new model that is the same as the original in its structure, but omits the group-specific effect from the linear predictor
with pm.Model(coords=c) as no_group_specific:
    
    # Prior for scale of random effects then the random effect itself - as earlier. We could even delete these lines
    ppt_sigma = pm.HalfCauchy('ppt_sigma', beta=50)
    ppt_ranef = pm.ZeroSumNormal('ppt_ranef', sigma=ppt_sigma, dims='subject')
    
    # Prior for the model coefficients - as earlier
    β0 = pm.Normal('β0', mu=0, sigma=300)
    β1 = pm.Normal('β1', mu=0, sigma=20)
    σ = pm.HalfCauchy('σ', beta=50)
    
    # Take the linear combination with NO group-specific and using the newX
    μ = β0 + newX*β1

So far, the model is a repeat of the initial model, even including group specific effects that aren’t used. We now add in a new term and sample the posterior predictive distribution of it, by passing the InferenceData of the initial model. Surprisingly - and this is the non-obvious part - the new model will use the posterior contained in the inference data for all the values in the model, aside from the new one which will be sampled. The group specific effects could be deleted from the code if we wanted to; they will be ignored.

with no_group_specific:
    
    # This is the new variable we want to sample
    no_group_estimate = pm.Normal('no_group_estimate', mu=μ, sigma=σ)
    
    # First set of out of sample predictions
    predictions1 = pm.sample_posterior_predictive(hierarchical_idata, var_names=['no_group_estimate'], predictions=True)

Sampling: [no_group_estimate]

100.00% [4000/4000 00:00<00:00]

Lets take a look at these predictions, generated without any recourse to individual groups:

# Plot the data
with sns.plotting_context('poster'):
    
    # Canvas
    fig, ax = plt.subplots(1, 1, figsize=(10, 5))
    
    # Scatter the observed data
    sns.scatterplot(data=sleep, x='Days', y='Reaction', ax=ax, color='white', edgecolor='black', label='Observed')
    
    # Add in the predicted datapoints mean by aggregating it across chains and draws
    ax.plot(newX, predictions1['predictions']['no_group_estimate'].mean(('chain', 'draw')), color='black', 
            marker='s', linestyle='None', label='Predicted')
    
    # Nicely shade the area that the points might fall in
    az.plot_hdi(newX, y=predictions1['predictions']['no_group_estimate'])
    
    # Clean up axis
    ax.set(ylabel='Reaction Time', 
           xticks=range(0, 13), 
           axisbelow=True)
    
    # Turn on legend()
    ax.legend(frameon=False)

Type 2 - predicting new scores for specific, observed groups

One benefit of hierarchical models is that they capture differences in specific groups; in social science or psychology research, this is typically leveraged to study individual differences. Here, we could use the model to predict scores for days 10, 11, and 12, for all individuals, accounting for their variability in baseline reaction times. This can be achieved with a creation of a new dataframe that ties the new values to be predicted to the existing group labels, and using the .factorize() approach to reindex the priors. Let’s now predict for days 10, 11, and 12, but this time including all group specific effects.

# Build a new DataFrame, tying the days to predict to each unique subject ID
# use of product to cleanly create the long-format dataframe
newDF = pd.DataFrame([(subject, day) for subject, day in product(sleep['Subject'].unique(), newX)], 
                     columns=['Subject', 'Days'])

newInd, _ = newDF['Subject'].factorize() # Get the new indexes of the subject locations in newDF

with pm.Model(coords=c) as include_group_specific:
    
    # Hyperprior for the variability of the subjects - like before
    subj_σ = pm.HalfCauchy('subj_σ', beta=50)
    subj_offset = pm.ZeroSumNormal('subj_offset', sigma=subj_σ, dims='subject') 
    
    # Roughly informative coefficients prior - like before
    β0 = pm.Normal('β0', mu=0, sigma=300)
    β1 = pm.Normal('β1', mu=0, sigma=20)
    
    # Noise variability - like before
    σ = pm.HalfCauchy('σ', beta=50)
    
    # Linear predictor is identical but uses the new data and indexes the offset with newInd
    μ = β0 + subj_offset[newInd] + newDF['Days'].values*β1
    
    # And we want to sample this unobserved variable
    group_predictions = pm.Normal('group_predictions', mu=μ, sigma=σ)
    
    # Sample and provide the inference data
    predictions2 = pm.sample_posterior_predictive(hierarchical_idata, var_names=['group_predictions'], predictions=True)

Sampling: [group_predictions]

100.00% [4000/4000 00:00<00:00]

Let’s examine how the predictions work across each participant. First we will precompute the mean and HDI of each prediction:

# Add these back to the original dataframe
mu = predictions2['predictions']['group_predictions'].mean(('chain', 'draw'))
hd = az.hdi(predictions2, hdi_prob=.95, group='predictions', var_names='group_predictions').to_dataframe().unstack('hdi').droplevel(0, axis='columns')

# Add them in
newDF = newDF.assign(mu=mu.values, **hd)

# Custom function to plot predictions and original data, per subject
def custom_plot(**kwargs):
    
    # get data, axis
    data = kwargs.pop('data')
    ax = plt.gca()
    
    # Plot the original data
    sns.scatterplot(data=data, x='Days', y='Reaction', color='white', edgecolor='black', label='Observed', ax=ax, zorder=1)
    
    # Subset newDF and plot it plus bands
    thisSubj = newDF.loc[newDF['Subject'] == data["Subject"].unique()[0]]
    
    # Plot the predictions
    ax.plot('Days', 'mu', data=thisSubj, marker='s', color='black', linestyle='None', label='Predicted')
    ax.fill_between('Days', 'lower', 'higher', color='black', alpha=.2, data=thisSubj)
    
    # Tidy
    ax.set(xticks=range(13), axisbelow=True)
    ax.grid(alpha=.3)
    
with sns.plotting_context('talk', font_scale=0.8):
    
    (
        sns.FacetGrid(data=sleep, col='Subject', col_wrap=6, aspect=1.3)
        .map_dataframe(custom_plot)
        .add_legend()
    )

The model makes predictions for unseen days for each of the participants in the study, and the uncertainty is naturally handled by the posterior distribution.

Type 3 - Predicting scores for unobserved groups

Finally, one useful property of hierarchical Bayesian models is that they can comfortably generate predictions for unseen group-level data. Once we learn the variability of the group-level effects, then new groups can be thought of as draws from this population, fed to the model, and predictions made accordingly. This sort of prediction isn’t widely seen in psychology (whereas Type 1 is), but really highlights the kind of uncertainty we’d expect to see in new data, accounting for individual variability. We can do this in PyMC below, following a similar approach as above. We can instantiate a new prior for the offsets, sample from it, and pass it to the model.

However, its generally a good idea to inject some prior knowledge into the unseen data. Given the model set up, drawing new groups will tend to collapse to zero (reducing the model to a noisier version of the no-group-specific effect prediction). Instead, if we pass some expected group-specific values, we can recover some predictions. An example here would be knowing a-priori that one of two new participants tend to have quite fast reaction times (a negative offset - faster than the average) and the other quite slow (a positive offset). This is easy to model in PyMC, and follows the standard approach of adding in new variables to sample.

First, lets make a new DataFrame containing the variables to predict - we’ll ask for each of the “Days” to be predicting for imaginary new participants, and then pass it onto the model.

# New content
days = np.arange(10)
ids = ['new1', 'new2']

# Data
unseen = pd.DataFrame(
    [(subject, day) for subject, day in product(ids, days)],
    columns=['Subject', 'Days'])

# Creates the new indexers
unseen_ind, unseen_sub = unseen['Subject'].factorize()

# Build a new model, and this time set new coordinates to ease dims
with pm.Model(coords={'new_subject': unseen_sub}) as unseen_group:
     
    # Model takes the hyperprior that has been learned
    subj_σ = pm.HalfCauchy('subj_σ', beta=50)
    
    # But we add in a new unseen-offset variable, which injects corresponding prior knowledge that 
    # One participant is fast, and another is slow
    unseen_offset = pm.Normal('unseen_offset', mu=[-50, 50], sigma=subj_σ, dims='new_subject') 
    
    # Same coefficients as before
    β0 = pm.Normal('β0', mu=0, sigma=300)
    β1 = pm.Normal('β1', mu=0, sigma=20)
    
    # Noise variability - like before
    σ = pm.HalfCauchy('σ', beta=50)
    
    # Linear predictor is again identical, relying on the new indexer to the group specific
    μ = β0 + unseen_offset[unseen_ind] + unseen['Days'].values*β1
    
    # And we want to sample this unobserved variable
    unseen_group_prediction = pm.Normal('unseen_group_prediction', mu=μ, sigma=σ)
    
    # Sample and provide the inference data, taking draws for both the predictions and the offsets
    predictions3 = pm.sample_posterior_predictive(hierarchical_idata, var_names=['unseen_group_prediction', 'unseen_offset'], predictions=True)

Sampling: [unseen_group_prediction, unseen_offset]

100.00% [4000/4000 00:00<00:00]

From this model, we can examine the variability of the offsets themselves, as well as the predictions. Lets first examine the offsets, which show that they centre roughly around the means we specified with corresponding high variability (there is no real data to update them, after all):

# Comparison of original model and new draws
az.plot_forest([predictions3['predictions'], hierarchical_idata],
               combined=True, hdi_prob=.95,
               model_names=['Unseen Groups', 'Fitted Model'],
               var_names=['subj_offset', 'unseen_offset']);

Finally, we can examine the predictions for the unseen participants. Lets recover their predictions from the posterior and plot them as above.

# Estimate mean and HDI
unseen_mu = predictions3['predictions']['unseen_group_prediction'].mean(('chain', 'draw')).values
unseen_hd = az.hdi(predictions3, var_names='unseen_group_prediction', group='predictions', hdi_prob=.95).to_dataframe().unstack('hdi').droplevel(0, axis='columns')

# Add these to the created DataFrame
unseen = unseen.assign(mu=unseen_mu, **unseen_hd)

# Another facet grid plot
with sns.plotting_context('poster'):
    
    (
        sns.FacetGrid(data=unseen, col='Subject', height=5, aspect=1.5)
        .map_dataframe(sns.scatterplot, x='Days', y='mu', color='white', edgecolor='black')
        .map(plt.fill_between, 'Days', 'lower', 'higher', color='black', alpha=.2)
        .set(xticks=range(10), ylabel='Predicted Reaction Time', axisbelow=True)
    )

There is a huge range of different plausible datasets generated by the model, implying the steady increase (slowing) in reaction time estimated from the original data. The summary is shown here, but equally a few draws from the posterior could have been shown here for the two new participants:

# Shift the posterior into a dataframe and tack on the unseen data
psd = (predictions3['predictions']['unseen_group_prediction']
       .to_dataframe()
       .unstack(('chain', 'draw'))
       .droplevel([0, 1], axis='columns')
       .sample(n=6, axis='columns', random_state=rng)
       .add_prefix('Dataset: ')
       .pipe(lambda x: pd.concat((x, unseen[['Subject', 'Days']]), axis='columns'))
       .melt(id_vars=['Subject', 'Days'], 
             var_name='simulated', value_name='Reaction')
      )

with sns.plotting_context('poster'):
    (
        sns.FacetGrid(data=psd, col='simulated', col_wrap=3, height=5, aspect=1.1)
        .map_dataframe(sns.scatterplot, x='Days', y='Reaction', edgecolor='black', color='white')
        .set(xticks=range(10), ylabel='Predicted Reaction Time', axisbelow=True)
        .set_titles('{col_name}')
    )

Finally, its worth reiterating the huge flexibility of PyMC. Using the “re-model” approach to generating predictions, other assumptions can be played with depending on the situation - for example, its possible to swap out the pm.Normal assumption for the observed values, and insert a Student’s T distribution. Or we could perhaps fix the variance of some of the distributions if we want to restrict our assumptions about future data more tightly. The flexibility in analysis is one of the truly great things about Bayesian inference and PyMC!

%load_ext watermark
%watermark --iversions

seaborn   : 0.12.2
arviz     : 0.15.1
pandas    : 2.0.0rc0
matplotlib: 3.6.3
pymc      : 5.5.0
bambi     : 0.10.0
numpy     : 1.24.2

Customer segmentation with Bayes, Spectral Embedding, and a second-place win

Tue, 23 Aug 2022 23:00:00 GMT

To get a break from academic work over the summer, I took part in another data science competition. This one was difficult, involving finding clusters of different customers in a multi-table database, and invovled some tricky feature engineering and data imputation. It also struck me how effective the general approach of dimension-reduction and clustering is; I’ve used it in some papers but the logic here was the same. One thing I picked up here was the utility of non-linear dimension reduction approaches like spectral embedding, which work by maintaing distances between points as the the higher dimensions are projected onto manifolds. I also had some great Altair plots in this, but on submission they all got stripped. So I had to remake some static plots last minute. The final version is here, but I much preferred this one. Anyway, second place isn’t so bad for a stab at a very applied problem.

# Imports
import arviz as az
import altair as alt
import numpy as np
import pandas as pd
import pymc as pm
import seaborn as sns

import matplotlib.pyplot as plt
import scipy.stats as st

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
from sklearn.manifold import SpectralEmbedding
from sklearn.impute import KNNImputer
from statsmodels.distributions.empirical_distribution import ECDF

Data-driven segmentation reveals three doctor profiles

Executive Summary

This report highlights a new, data-driven perspective on the customer base of the company. The historical approach of segmenting doctors by their region has been shown to be a poor predictor of their support or purchasing behaviour. The analysis describes here reveals three new segments of customers with distinct profiles that will inform product and marketing development alongside customer service needs.

The report focuses on three key questions: 1. What is the composition of each region in terms of number of doctors and their average number of purchases? 2. Is there a relationship between purchases and number of complaints? 3. Can new segments be found, and what variables best describe these new segments if they exist?

Key Insights

The number of doctors in each area varies a lot, from one to 34, though on average there are around 10. The number of purchases also varies, from 13 to 129. There is no discernible pattern between these, and one small region has a very high purchase rate.
Given the data, there is a 75% probability that doctors who complain more, purchase more, though this effect is relatively small, with higher-than-average numbers of complaints being associated with just a 6% increase in purchases. The analysis suggests the effect could be as large as a 25% increase, or reduce purchases by 11%.
Segmentation analysis suggests there are three distinct groups of doctors, with a unique relationship to the company.

New doctor segments:

Segment One - Doctors here have low experience with the company, lower rates of purchases, R and incidence rates, and give fewer complaints. They also treat fewer unique conditions, and have only average reported satisfaction. There are more general practitioner in this segment.
Segment Two - Doctors here have high R rates, give more complaints, and have a high level of experience with the company. They make relatively fewer purchases and seem to treat fewer unique conditions. This segment, the largest in number, is diffuse in terms of its members and their rank and category.
Segment Three. Doctors here make a lot of purchases and have high incidence rates, are relatively satisfied, and treat the most number of unique conditions. They make fewer complaints but have less experience with the company. They are comprised of a lot of Ambassadors.

Key recommendations

Abandon use of region as a predictor of doctor behaviour.
Acknowledge the association between purchases and complaints, and its uncertainty.
Label Segment One doctors as Inexperienced and Indifferent doctors. This segment would benefit from marketing and raising awareness of the company to on-board doctors and increase satisfaction.
Label Segment Two doctors as Experienced but Unhappy doctors. While these doctors have a longstanding relationship with the company, they have unaddressed needs and make many complaints. These would be a good audience for focus groups to understand their needs and increase satisfaction.
Label Segment Three doctors as Returning, Happy, but less experienced doctors. These individuals are overall positive, treating a range of conditions, and make many purchases, but have less experience with the company. Further engagement with this audience will be important to retain them as a loyal set of customers.
Caveats: Despite a large dataset, doctors with matching observations across the range of tables were relatively few, with an effective sample size of 74 doctors. Statistical methods were used to help here but it is worth noting.

Doctor-by-region analysis; interactively

The main data sources here are split across four tables, which briefly comprise the following information: - doctors includes information on doctor category, rank, experience with the company, satisfaction, purchases, and incidence/rework rates. - orders illustrates the number of orders an individual doctor has placed. - complaints considers doctors and the number of complaints and complaint types they make. - instructions highlights whether a doctor has leaves specific instructions on the order.

There is missing data in the doctors dataset about satisfaction, which will be imputed after highlighting the doctor-by-region analysis.

# Read in data files
doctors = pd.read_csv('data/doctors.csv')            # doctor details
orders = pd.read_csv('data/orders.csv')              # order details
complaints = pd.read_csv('data/complaints.csv')      # complaint details
instructions = pd.read_csv('data/instructions.csv')  # instruction details (y/n)

# Aggregate the doctors dataset
query1 = (doctors
          .groupby(by='Region', as_index=False)
          .agg(number_of_doctors=('DoctorID', pd.Series.nunique),
               average_purchase_number=('Purchases', 'mean'))
          .round(3))

## Display interactively with Altair ##
input_dropdown = alt.binding_select(options=query1['Region'].unique(), name='Select Region! ')
selection = alt.selection_single(fields=['Region'], bind=input_dropdown)

query1_chart = (alt.Chart(query1, width=900, height=400)
                .mark_point(filled=True, size=300, opacity=.75)
                .encode(x=alt.X('number_of_doctors', title='Number of Doctors in Region'),
                        y=alt.Y('average_purchase_number', title='Average Number of Purchases in Region', scale=alt.Scale(domain=(0, 140))),
                        color=alt.condition(selection, alt.Color('Region:N', legend=None), alt.value('grey')),
                        tooltip=[alt.Tooltip('Region'),
                                 alt.Tooltip('number_of_doctors', title='Number of Doctors'),
                                 alt.Tooltip('average_purchase_number', title='Average Purchase #')])
                .configure_axis(labelFontSize=16,
                                titleFontSize=16)
                .configure_legend(labelFontSize=16,
                                  titleFontSize=16)   
               ).add_selection(selection)
# Render
query1_chart

Hover over a point to view detailed information, or use the dropdown to isolate a specific region. A brief glance at this chart highlights why region is a poor predictor of the number of purchases that can be made. Overall, there is no relationship between number of doctors and their average number of purchases across regions, aside from a large outlier.

Assessing the purchase ~ complaint association probabilistically

While there may be a simple association between purchases and complaints, the relationship may be driven by other variables (e.g. satisfaction). Using a combination of careful feature engineering and a Bayesian linear model, the influence of other variables in the dataset (see Technical Annexe for details) can be controlled, and the relationship between complaint number as a predictor of purchase number examined. Before modelling, 55 missing datapoints in the satisfaction variable were imputed, and a final sample of 74 doctors were analysed.

## Feature engineering
# Doctors cleaning
doctors_processed = (doctors
                     .assign(**pd.get_dummies(doctors[['Category', 'Rank']], drop_first=True))     # One hot encodes category and rank
                     .assign(Satisfaction=pd.to_numeric(doctors['Satisfaction'], errors='coerce')) # Fixes the string column of Satisfaction
                     .set_index('DoctorID')
                     .filter(regex='rate|Satisfaction|Experience|Purchases|Rank_|Category_')       # Select relevant variables
                    )

# Orders cleaning - calculate unique-condition-count - number of unique treatments
orders_processed = (orders
                    .set_index('DoctorID')               # Push doctor ID into index
                    .filter(regex='Condition [A-I]')     # Keep conditions A-I; J seems related to when devices are administered
                    .groupby('DoctorID')                 # Groupby Doctor as some doctors appear more than once
                    .apply(lambda x: x.sum(axis='rows')) # Take the sum across rows - each doctor then has a count of how many times they treat a condition
                    .gt(0)                               # where that is above zero, indicate true
                    .sum(axis='columns')                 # The sum of those booleans indicates a condition has been treated by this doctor
                    .to_frame('Unique Condition Index')
                   )

# Complaints - count the total number of complaints a given doctor gives, across complaint types
complaints_processed = (complaints
                        .groupby('DoctorID')
                        .agg(Complaints=('Qty', 'sum'))
                       )

# Merge the data and scale appropriately
z_score_these = ['Incidence rate', 'R rate', 'Satisfaction', 
                 'Experience', 'Complaints', 'Unique Condition Index']

# Merge the data together
df = (complaints_processed
      .join(doctors_processed, how='inner')
      .join(orders_processed, how='left')
     )

# Impute the missing data in Satisfaction
imputed_data = KNNImputer(n_neighbors=4).fit_transform(df)

# Recreate a dataframe, scale necessary variables
df = pd.DataFrame(imputed_data, index=df.index, columns=df.columns).apply(lambda z: st.zscore(z) if z.name in z_score_these else z)

# Test association using a PyMC negative binomial model
X = df.drop(columns='Purchases')
y = df['Purchases'].values

with pm.Model(coords={'coefs': X.columns}) as bayesian_negbinom_regression:
    
    # Define priors over intercept and all coefficients
    β0 = pm.Normal('β0', mu=0, sigma=1)
    β = pm.Normal('β', mu=0, sigma=1, dims='coefs')
    
    # Compute linear predictor
    λ = β0 + pm.math.dot(X.values, β)
    
    # Take the exponent
    λ_exp = pm.Deterministic('λ_exp', pm.math.exp(λ))
    
    # Send to a Poisson likelihood which models the count of a variable
    α = pm.Exponential('nb', 0.01)
    pm.NegativeBinomial('likelihood', mu=λ_exp, alpha=α, observed=y)
    
    # Sample
    trace = pm.sample(2000, tune=2000, target_accept=.90, random_seed=42)

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [β0, β, nb]

100.00% [16000/16000 00:19<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 2_000 tune and 2_000 draw iterations (8_000 + 8_000 draws total) took 43 seconds.

# Extract the posterior distribution of the "complaints" predictor
coef_trace = np.exp(trace['posterior'].sel(coefs='Complaints')['β'].values.flatten())

# Take its empirical cumulative distribution
ecdf = ECDF(coef_trace)([coef_trace]).ravel()

# and its survival function
esf = 1 - ecdf

# Build a kernel density estimate and its probability density function of the coefficient
kde = st.gaussian_kde(coef_trace).pdf(coef_trace)

# Store as a df
traceplot = pd.DataFrame({'trace': coef_trace, 'density': kde, 'ecdf': ecdf, 'esf': esf}).round(3)

# The kdeplot
alt.data_transformers.enable('default', max_rows=8_000)
kdeplot = (alt.Chart(traceplot, width=500, height=500)
           .mark_area(opacity=.5, color='black', width=10, point=alt.OverlayMarkDef(color='black'))
           .encode(x=alt.X('trace:Q', title='Change in rate of purchasing with increasing complaints'),
                   y=alt.Y('density:Q', title='', axis=alt.Axis(labels=False)),
                   tooltip=[alt.Tooltip('trace:Q', title='Plausible purchase change rate'),
                            alt.Tooltip('ecdf:O', title='Probability effect is less than X'),
                            alt.Tooltip('esf:O', title='Probability effect is bigger than X')])
          )

# Text labels
text_μ = (alt.Chart({'values': [{'x': 1.40, 'y': 1.84}]})
          .mark_text(text=f'Average effect = {coef_trace.mean():.2f}', size=16)
          .encode(x='x:Q', y='y:Q'))
text_θ = (alt.Chart({'values': [{'x': 1.40, 'y': 1.60}]})
          .mark_text(text='95% interval [{0:.2f}, {1:.2f}]'.format(*az.hdi(coef_trace, .95)), size=16)
          .encode(x='x:Q', y='y:Q'))

# Add together in a layer, adjust font size
(kdeplot + text_μ + text_θ).configure_axis(labelFontSize=16, titleFontSize=16)

Hover over the above plot to examine plausible rates of change in purchases with increasing numbers of complaints, and an associated probability. For example, by hovering over one, which indicates no change in purchases as a result of increasing complaints, we see that the probability the effect is bigger than this - that is, increasing complaints means increasing purchases - is 75%. Similarly, the probability is negative is 25%. The plot highlights clearly that the most likely rate of increase is 1.06, or 6%, but the rate is plausibly within the region of 0.89 to 1.25. From this we are able to infer the effect of increased complaints on purchases across a variety of contexts.

Customer segmentation - revealing three profiles

With the same dataset used above, machine learning techniques (see Technical Annexe for details) can uncover the latent structure in the data that can then be grouped together, revealing an optimal grouping of doctors. The below chart, also interactive, allows for the exploration of each segment and their distinct customer profile. Doctors were segmented using a two-feature representation of a wider feature space, specifically: - Number of complaints - Incidence and R rates - Satisfaction with company - Experience with company - Category; Specialist or GP - Rank; from Silver through to Titanium Plus - Unique Condition Index; a count, per doctor, of the number of unique conditions they treat.

# First scale the Qty measure
df = df.assign(Purchases=st.zscore(df['Purchases']))

# Use Spectral embedding to preserve local distances in a 2D representation
spectra = SpectralEmbedding(n_components=2, random_state=42).fit_transform(df)

# Use silhouette scoring to select K with K-means
sc = {}
for k in range(2, 10):
    
    # Fit KMeans
    km = KMeans(n_clusters=k, random_state=42).fit(spectra)
    
    # Extra and score silhouette
    sc[k] = silhouette_score(spectra, km.labels_)

# Refit KMeans with the max silhouette score
best_K, best_sc = max(sc.items(), key=lambda y: y[-1])
final_k = KMeans(n_clusters=best_K, random_state=42).fit(spectra)

# Reveal dimension reduction and clusters in an interactive plot, with additional data on mouseover!
spectra_df = (pd.DataFrame(spectra, columns=['spec_1', 'spec_2'], index=df.index)
              .assign(Segment=final_k.labels_+1)
              .join(doctors[['DoctorID', 'Category', 'Rank', 'Region']].set_index('DoctorID'))
              .join(df)
              .reset_index()
             )

# Per group, take mean of continuous features, count of Category/Rank
average = ['Complaints', 'Incidence rate', 'R rate', 'Satisfaction', 'Experience', 'Purchases', 'Unique Condition Index']

# Groupby object
base = spectra_df.groupby('Segment')
chart = (base.agg({x: 'mean' for x in average})
         .join(base['Category'].value_counts().unstack('Category'))
         .join(base['Rank'].value_counts().unstack('Rank'))
         .fillna(0)
         .reset_index()
         .melt(id_vars='Segment', var_name='Attribute', value_name='Aggregate')
         .assign(prep=lambda x: np.where(x['Attribute'].isin(average), 'con', 'cat'))
        )

# Build continuous/cat plots
left = (alt.Chart(chart.query('prep == "con"'), width=450, height=400)
        .mark_bar()
        .encode(x='Aggregate', 
                y=alt.Y('Attribute', title='Metric variable'),
                color='Segment:N')
       )

right = (alt.Chart(chart.query('prep == "cat"'), width=450, height=400)
         .mark_bar()
         .encode(x=alt.X('Aggregate', title='Number of doctors'),
                 y=alt.Y('Attribute', title='Rank and Category'),
                 color='Segment:N')
       )

# Dropdown selectors 
input_dropdown = alt.binding_select(options=list(chart['Segment'].unique()), name='SELECT SEGMENT:   ')
selection = alt.selection_single(fields=['Segment'], bind=input_dropdown)

# Add to plots
left = left.add_selection(selection).transform_filter(selection)
right = right.add_selection(selection).transform_filter(selection)

# Stack, and configure appearance
(left | right).configure_axis(labelFontSize=16, titleFontSize=16).configure_legend(labelFontSize=16, titleFontSize=16)

# Segment profile
(spectra_df['Segment']
 .value_counts(normalize=True)
 .sort_index()
 .round(2)
 .to_frame('Proportion of sample')
 .rename_axis(index='Segment Number'))

	Proportion of sample
Segment Number
1	0.20
2	0.53
3	0.27

Using the interactive dropdown above, it is easy to explore the new segments and their important features. These are described simply below: - Segment One. Doctors here have low experience with the company, lower rates of purchases, R and incidence rates, and give fewer complaints They also treat very few unique conditions, and have very average reported satisfaction. There are more general practitioners here, and some individuals at the higher ranks. This would be a great group to target for more positive engagement and on-boarding marketing materials. - Segment Two. Doctors here have high R rates, give more complaints, and have a high level of experience with the company. They make relatively fewer purchases and seem to treat fewer unique conditions. This segment, the largest in number, is diffuse in terms of its members and their rank and category. This set of doctors seem to be longstanding customers but are unhappy, and are making fewer purchases - a good target for focus groups to obtain detailed data on what is going wrong, and if this is perhaps related to the smaller number of conditions they treat. - Segment Three. Doctors here make a lot of purchases and have high incidence rates, are relatively satisfied, and treat the most number of unique conditions. They make fewer complaints but have less experience with the company. They are comprised of a lot of Ambassadors. This group appears to be a relatively content customer base, and its important they stay that way through understanding what the company is doing right.

A simple, memorable mapping could be: - Segment One: Inexperienced and indifferent doctors. - Segment Two: Experienced but unhappy doctors. - Segment Three: Returning, happy, but less experienced doctors.

Technical Annexe

Significantly greater technical detail on the above sections are provided below, for interested readers.

Feature engineering

The features used across the analysis were generated from the doctors, complaints, and orders datasets. Despite a large number of observations in the doctors dataset (437), relatively few of these were in the complaints dataset (just 16%). Given that complaints and purchases were a primary query, the smaller dataset was analysed. Features were engineered from across the datasets before being merged into one:

From doctors, Incidence rate, R rate, Satisfaction, Experience, and Purchases were standardised and scaled as continuous features. The Rank and Category variables were one-hot-encoded, using “General Practitioner” and “Ambassador” as the reference categories, respectively - the former having two levels, and the latter eight.
From complaints, the Qty variable was summed across complaint types per doctor, yielding a single count per doctor.
The orders data was processed such that across the orders each doctor placed, the sum of unique conditions was calculated - doctors who treat more varied conditions may have more needs, and this feature could help discriminate this.

The final dataset contained 74 unique doctors, and 16 features. After merging, there were 55 missing data points for the satisfaction feature. This was imputed using K-nearest-neighbours imputation.

Assessing purchase ~ complaint association probabilistically

To assess the relationship between complaints and purchases, PyMC was used to fit a Bayesian negative-binomial regression, complaints alongside the 14 other features described above (purchases being the target variable). Negative-binomial regressions allow for the prediction of count data precisely, especially in situations where the variance of the count data is greater than the mean (here, the average purchase count is 10, while the variance is 130). Priors were set to be restrictive to help with the small dataset, avoiding overfitting, following a distribution.

The central feature is of course the complaint predictor, but the full model summary is printed below, highlighting those predictors which have a high (or indeed low) probability of being associated with increasing purchases, which may be valuable information. For example, the unique condition index is much more likely to have a negative association with purhcases; doctors who treat more conditions make fewer purchases. Those with higher incidence rates make more purchases. There is uncertainty in many of these estimates but they are provided for further insight.

# Tabulate results
(az
 .summary(trace, var_names='β', 
          filter_vars='like', 
          kind='stats', 
          hdi_prob=.95,
          stat_funcs={'Purchase Rate Change': lambda x: np.mean(np.exp(x)),
                      '95% lower': lambda x: az.hdi(np.exp(x), .95)[0],
                      '95% upper': lambda x: az.hdi(np.exp(x), .95)[1],
                      'Probability Positive': lambda x: np.mean(np.exp(x) > 1)})
 .drop(index=['β0'])
 .assign(Feature=lambda x: x.index.str.replace('β|\\[|\\]', '', regex=True).str.replace('_', ': '))
 .set_index('Feature')
 .filter(items=['Purchase Rate Change', '95% lower', '95% upper', 'Probability Positive'])
 .style
 .applymap(lambda x: 'color: blue' if x < .20 or x > .80 else None, subset=['Probability Positive'])
 .format('{:.2f}')
)

	Purchase Rate Change	95% lower	95% upper	Probability Positive
Feature
Complaints	1.06	0.90	1.24	0.76
Incidence rate	1.09	0.92	1.26	0.84
R rate	1.03	0.83	1.25	0.61
Satisfaction	1.08	0.89	1.27	0.79
Experience	1.15	0.94	1.36	0.92
Category: Specialist	2.26	0.97	3.85	0.99
Rank: Gold	0.38	0.08	0.80	0.02
Rank: Gold Plus	0.21	0.05	0.42	0.00
Rank: Platinum	0.29	0.15	0.46	0.00
Rank: Platinum Plus	0.32	0.16	0.51	0.00
Rank: Silver	1.61	0.03	4.89	0.50
Rank: Silver Plus	0.38	0.16	0.68	0.00
Rank: Titanium	0.49	0.30	0.71	0.00
Rank: Titanium Plus	1.63	0.03	5.03	0.50
Unique Condition Index	0.93	0.79	1.08	0.17

Customer segmentation - revealing three profiles

The full 16-feature dataset was reduced to two dimensions using spectral embedding. This was chosen as, unlike other feature reduction methods, it focuses on maintaining local distances between observations in the reduced space. Following the embedding, K-Means was used to determine clusters. K was chosen by iterating over candidate cluster numbers and assessing the silhouette coefficient, which assesses the average distance between a point and its within-cluster neighbours compared to its between-cluster neighbours. Three clusters emerged as having the highest coefficient, 0.47.

An interactive scatter plot of the resulting two-dimensional embedding and clusters is shown below, highlighting how the embedding retained distances that K-Means identified easily. Hover over each datapoint to visualise its underying full-dimensional data.

# Show scatter plot
query2_chart = (alt.Chart(spectra_df, width=600, height=400)
                .mark_point(filled=True, size=200, opacity=.50)
                .encode(x=alt.X('spec_1', title='Embedding 1'),
                        y=alt.Y('spec_2', title='Embedding 2'),
                        color=alt.Color('Segment:N'),
                        tooltip=[alt.Tooltip('DoctorID'),
                                 alt.Tooltip('Purchases'),
                                 alt.Tooltip('Complaints', title='Complaint Count'),
                                 alt.Tooltip('Category'),
                                 alt.Tooltip('Rank'),
                                 alt.Tooltip('Region'),
                                 alt.Tooltip('Incidence rate'),
                                 alt.Tooltip('R rate'),
                                 alt.Tooltip('Satisfaction'),
                                 alt.Tooltip('Experience')])
                .configure_axis(labelFontSize=16,
                                titleFontSize=16)
                .configure_legend(labelFontSize=16,
                                  titleFontSize=16)
               )

# Render
query2_chart

%load_ext watermark
%watermark --iversions

pymc      : 4.0.0
arviz     : 0.12.1
seaborn   : 0.12.0rc0
altair    : 4.2.0
numpy     : 1.21.6
matplotlib: 3.5.2
scipy     : 1.8.1
pandas    : 1.4.2

Multi100 Collaboration Project: Is Eco-Friendly Unmanly?

Thu, 07 Jul 2022 23:00:00 GMT

Earlier this year I responded to an invitation to take part in the MultiAnalysis Collaboration Project, run by the Centre for Open Science. The project aims to obtain at least five different analyses for 100 different papers published, with open data, in psychology. I opted for the below paper, published in 2016 — Is Eco-Friendly Unmanly? The Green-Feminine Stereotype and Its Effect on Sustainable Consumption. The organisers selected a single claim from each paper that appears to be the ‘central’ conclusion, and participants in the project have to test it — that’s more or less it. The claim in this paper is:

…consumers who engaged in green behavior were perceived …. as more feminine than consumers who engaged in nongreen behavior. (p. 571.)

I used a Bayesian hierarchical model here, and because the project required it, I had to figure out how to carry out a Bayes Factor hypothesis test with Python. This led me down a rabbit hole of reading about the Savage-Dickey density ratio, which is surprisingly (magically, even) quite easy to compute in comparison to the horror of marginal likelihoods of complex models. That said, I’m still not certain this is all correct, but I’m satisfied the posterior probabilities, if not the Bayes Factor, are good. The analysis and data is all on the OSF.

Multi100 Analysis — NF503

Is Eco-Friendly Unmanly? The Green-Feminine Stereotype and Its Effect on Sustainable Consumption

# Imports
import arviz as az
import bambi as bmb
import numpy as np
import pandas as pd
import pymc as pm
import seaborn as sns

import matplotlib.pyplot as plt
import scipy.stats as st

# Set matplotlib style
plt.style.use('bmh')

# Read in data
data = pd.read_csv('Phase 2 Reproduction Materials/Study 2 Data September 2015.csv')
data.head()

	SubPool	V1	V8	V9	Intro	M_NG_19	M_NG_20	M_NG_23	M_NG_24	M_NG_25	…	F_G_32	F_G_33	F_G_34	Q13	Age	Gender	GenderIden_1	Dating_1	Dating_2	Status
0	ND	R_a2xOgJn7aY967yJ	9/25/2015 7:00	9/25/2015 7:02	1	1.0	2.0	4.0	3.0	3.0	…	NaN	NaN	NaN	how we perceive ourselves	33	2	84.0	100.0	0.0	1
1	ND	R_9KB7Aqj8Dw5NYAR	9/25/2015 7:00	9/25/2015 7:02	1	NaN	NaN	NaN	NaN	NaN	…	3.0	4.0	1.0	Environmental awareness	21	1	76.0	0.0	100.0	1
2	ND	R_78605PJdv9gl2g5	9/25/2015 7:00	9/25/2015 7:02	1	1.0	1.0	1.0	1.0	1.0	…	NaN	NaN	NaN	perceptions	27	1	65.0	0.0	100.0	2
3	ND	R_6o5G3vGXbEi8LTT	9/25/2015 7:00	9/25/2015 7:03	1	NaN	NaN	NaN	NaN	NaN	…	NaN	NaN	NaN	attitudinal perceptions to reusable bags	29	1	75.0	2.0	99.0	2
4	ND	R_4IYda5xCDnq0Kln	9/25/2015 7:01	9/25/2015 7:03	1	NaN	NaN	NaN	NaN	NaN	…	1.0	4.0	1.0	Perception of recycling.	29	1	83.0	0.0	100.0	2

5 rows × 56 columns

There is no coding scheme evident for Gender, i.e. it is not clear whether 1 or 2 is female/male. The paper reports 45.9% were male, so we should be able to infer from that:

# What is what?
data['Gender'].value_counts(normalize=True).round(2)

2    0.54
1    0.46
Name: Gender, dtype: float64

‘1’ is probably male.

Step 1

First we must tidy the data up into a long-form dataset, and clear up the variable names (e.g. M_NG_19) to get a clearer sense of conditions. We also use the variable coding for each perception from Study 2 Survey September 2015.docx to more clearly label the perceptual trait ratings the participants gave.

# Variable coding from 'Study 2 Survey September 2015.docx'
recoding = {34: 'wasteful', 32: 'masculine',
            27: 'macho', 28: 'aggressive',
            24: 'eco-friendly', 25: 'feminine',
            33: 'sensitive', 26: 'gentle',
            19: 'athletic', 20: 'attractive',
            23: 'curious'
           }

# Convert to long format and do some cleaning up
data_l = (data
          .melt(id_vars=['SubPool', 'V1', 'Age', 'Gender', 'GenderIden_1', 'Dating_1', 'Dating_2', 'Status'],
                value_vars=data.loc[:,'M_NG_19':'F_G_34'].columns,
                var_name='condition', value_name='rating')
          .pipe(lambda df_: df_.join(df_['condition'].str.split('_', expand=True)))
          .rename(columns={0: 'target_sex', 1: 'green', 2: 'perception', 'V1': 'pid'})
          .rename(columns=str.lower)
          .assign(perception=lambda x: x['perception'].astype(int))
          .replace({'perception': recoding,
                    'target_sex': {'M': 'male', 'F': 'female'},
                    'green': {'NG': 'non-green', 'G': 'green'},
                    'gender': {1: 'male', 2: 'female'}})
          .drop(columns='condition')
         )

# Drop missing values in 'rating' - between-participants design means not everyone rated all combinations
data_l = data_l.dropna(subset='rating').reset_index(drop=True)

display(data_l.head())
display('Participant counts', data_l.groupby('green')['pid'].nunique())
display('Missing data', data_l.isna().any())

	subpool	pid	age	gender	genderiden_1	dating_1	dating_2	status	rating	target_sex	green	perception
0	ND	R_a2xOgJn7aY967yJ	33	female	84.0	100.0	0.0	1	1.0	male	non-green	athletic
1	ND	R_78605PJdv9gl2g5	27	male	65.0	0.0	100.0	2	1.0	male	non-green	athletic
2	ND	R_cNrHv7l4bzAPzo1	19	female	68.0	100.0	0.0	1	1.0	male	non-green	athletic
3	ND	R_cIl6ai9lZGVTEih	28	male	87.0	0.0	100.0	2	2.0	male	non-green	athletic
4	ND	R_0xi4MaYTLcWON7v	25	male	66.0	0.0	100.0	1	2.0	male	non-green	athletic

'Participant counts'

green
green         94
non-green    100
Name: pid, dtype: int64

There is also missing data in the genderiden_1 and dating_ variables. These will be used as additional predictors, so these missing values will be dropped — i.e. a complete-case design will be used.

# Complete cases - drop all missing in the three named variables
data_l = data_l.dropna(subset=['genderiden_1', 'dating_1', 'dating_2']).reset_index(drop=True)
data_l.pid.nunique()

Step 2

We now ready to specify a model, and lay out an inference plan.

The aim is to assess whether consumers who engaged in green behaviour are perceived as more feminine, with participants making ratings of vignettes. The claim is straightforward; however the data-generating process is more complex, with participants being allocated to one of four conditions (e.g. male-green, female-nongreen), and provided multiple ratings of the same vignette. In addition, there are variables that may affect these ratings (e.g., participant gender, interest in dating males or females) that could feasibly impact the ratings.

The original paper created a ‘femininity index’ by averaging together several traits (feminine, gentle, sensitive). This is perhaps fine, but it seems more direct to work with the rating of femininity itself, and not to open up other measurement issues from averaging variables (e.g., Flake & Fried, 2020). However, it is sensible to acknowledge that all trait ratings are correlated (e.g., Sutherland et al., 2013), and that other trait ratings could impact femininity, and thus estimate the effects jointly with other variables.

The approach taken here is to build a hierarchical Bayesian model that adjusts for other covariates, and accounts for correlations within-participants given repeated trait ratings, and to examine the posterior distributions of the estimated marginal means.

The model includes main effects of gender, gender identity, and male and female dating intention. The interaction between target sex (male or female) and green behaviour (green or non-green) is estimated for each of the 11 trait ratings the participant provided (i.e., a three way interaction). While it would be simple to examine a given coefficient that represents the differences between green-nongreen for the feminine perception, this becomes difficult with the inclusion of the target_sex variable, as this difference will be tied to either female or male targets given the nature of dummy coding. The clearest answer to the claim question is averaged over the target_sex variable — the claim makes no reference to target sex, but it is a core component of the design.

As such, the solution is to fit the model and have it make marginal mean predictions, from which differences can be drawn from the posterior.

As the inclusion of the interaction with perception increases the size of the predictor space, regularising priors are chosen to constrain estimates. Coefficient priors are set to a prior, representing a state of information that most effects will be no larger than half a rating-scale unit change.

# Set out a model formula and orders of levels for categoricals
attr_levels = (y := list(data_l['perception'].unique()))[1::-1] + y[2:]
green_levels = ['non-green', 'green']
sex_levels = ['male', 'female']

mdspec = """rating 
            ~ C(perception, levels=attr_levels)
            * C(green, levels=green_levels)
            * C(target_sex, levels=sex_levels)
            + gender
            + scale(genderiden_1)
            + scale(dating_1)
            + scale(dating_2)
            + (1|pid)
            """

# Instantiate the model with regularising N(0, 0.25) priors
model = bmb.Model(mdspec, 
                  priors={'common': bmb.Prior('Normal', mu=0, sigma=0.25)},
                  data=data_l)

# Sample the posterior
posterior_idata = model.fit(draws=3000, random_seed=34)

# Marginal mean predictions
model.predict(posterior_idata, kind='mean')

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
Sampling 4 chains for 1_000 tune and 3_000 draw iterations (4_000 + 12_000 draws total) took 198 seconds.

Step 3

Below we recover the estimated means from the model and aggregate them across the green/non-green for each perception, obtaining a posterior distribution for the estimated marginal means.

# Extract aggregate marginal mean for the femininity/green-nongreen comparison
feminine_emm = (posterior_idata['posterior']
                ['rating_mean']
                .to_dataframe()
                .reset_index()
                .merge(data_l, left_on='rating_obs', right_index=True)
                .groupby(['perception', 'green', 'chain', 'draw'])
                .agg(mu=('rating_mean', 'mean'))
                .query('perception == "feminine"')
                .unstack('green')
                .droplevel(0, axis='columns')
                .reset_index(drop=True)
                .assign(difference=lambda x: x['green'] - x['non-green'])
              )

# Plot
with sns.plotting_context('poster'):
    
    fig, ax = plt.subplot_mosaic("""AAB""", gridspec_kw={'wspace': 0.05}, **{'figsize': (30, 10)})
    sns.despine(fig)
    
    az.plot_dist(feminine_emm['green'], color='green', ax=ax['A'], rug=True, fill_kwargs={'alpha': .2}, rug_kwargs={'alpha': .05}, label='Green')
    az.plot_dist(feminine_emm['non-green'], color='black', ax=ax['A'], rug=True, fill_kwargs={'alpha': .2}, rug_kwargs={'alpha': .05}, label='Non-Green')
    ax['A'].set(title='Feminine\nGreen|Non-Green', xlabel='Estimated Distribution of Femininity Ratings')
    
    az.plot_posterior(feminine_emm['difference'].values,
                      color='green', lw=3, 
                      ref_val=0, hdi_prob=.95,
                      point_estimate='mean',
                      ax=ax['B'])
    ax['B'].set(title='Feminine\nGreen - Non-green', xlabel='θ Difference between conditions')

The posterior distribution implies:

The average difference is 0.33 rating scale points.
The lower and upper 95% credible intervals are [0.11, 0.53], indicating effects of around a tenth of, and up to half, of a rating scale point are plausible difference sizes.
The probability the effect is positive — that is, that higher levels of femininity are ascribed to those with green behaviour — is 99.9%.

It is also possible to estimate a Bayes Factor to test the null hypothesis, using the Savage-Dickey density ratio method. The difference to test was observed from marginal means, but the prior distribution is the same as what is implied in the model for the coefficients, .

# Estimate a Bayes Factor and plot
prior = st.norm(0, 0.25)
post = st.gaussian_kde(feminine_emm['difference'].values)

# Compute BF
BF = 1 / (post.pdf(0) / prior.pdf(0))

eval_ = np.linspace(-1, 1, 5000)

with sns.plotting_context('poster'):
    
    fig, ax = plt.subplots(1, 1, figsize=(15, 10))
    sns.despine(fig)
    ax.plot(eval_, prior.pdf(eval_), color='grey', lw=4, linestyle='--', label='Prior')
    ax.plot(eval_, post.pdf(eval_), color='black', lw=4, label='Posterior')
    ax.plot([0, 0], [prior.pdf(0), *post.pdf(0)], 'ko', markersize=15)
    ax.plot([0, 0], [prior.pdf(0), *post.pdf(0)], 'k-')
    ax.legend()
    ax.set(ylabel='Density', xlabel='Green - Nongreen\nDifference')
    ax.text(-0.9, 3.2, f'$BF_{{{10}}}$ = {BF[0]:.2f}')

Step 4

Thus far, the conclusion is that there is a positive effect of green behaviour on perceptions of femininity in general — the probability that the ratings of femininity are higher for green behaviour is 99.9%, and the BF for the alternative is 48.87. While this indicates the presence of an effect, the overall effect is relatively small, with the upper bound being only around half a rating scale point. This may — or may not — be psychologically practical or relevant.

It is also possible to examine the posterior predictive distribution of the data, which takes into account both the uncertainty in the sampling of participants, as well as the uncertainty in the parameters of the model. By generating the posterior predictive distribution, it will be possible to examine the differences between green/nongreen behaviour that could be expected in future, as opposed to a focus on the average effect. Examining this may give more insight into the expected size of the effect in actual data-point terms, as opposed to the mean. We conduct this analysis below, focused on the green/non-green difference for femininity ratings.

# Take a posterior predictive distribution
model.predict(posterior_idata, kind='pps')

# Extract posterior predictive
ppc = (posterior_idata['posterior_predictive']
       .to_dataframe()
       .reset_index()
       .merge(data_l, left_on='rating_dim_0', right_index=True, suffixes=('_pp', '_observed'))
       .groupby(['perception', 'green', 'chain', 'draw'])
       .agg({'rating_pp': 'mean'})
       .query('perception == "feminine"')
       .unstack('green')
       .droplevel(0, axis='columns')
       .assign(difference=lambda x: x['green'] - x['non-green'])
      )

# Plot
with sns.plotting_context('poster'):
    
    fig, ax = plt.subplot_mosaic("""AAB""", gridspec_kw={'wspace': 0.05}, **{'figsize': (30, 10)})
    sns.despine(fig)
    
    az.plot_dist(ppc['green'], color='green', ax=ax['A'], rug=True, fill_kwargs={'alpha': .2}, rug_kwargs={'alpha': .05}, label='Green')
    az.plot_dist(ppc['non-green'], color='black', ax=ax['A'], rug=True, fill_kwargs={'alpha': .2}, rug_kwargs={'alpha': .05}, label='Non-Green')
    ax['A'].set(title='Posterior Predictive\nFeminine - Green|Non-Green', xlabel='Posterior Predictive Distribution of Femininity Ratings')
    
    az.plot_posterior(ppc['difference'].values,
                      color='green', lw=3, 
                      ref_val=0, hdi_prob=.95,
                      point_estimate='mean',
                      ax=ax['B'])
    
    ax['B'].set(title='Posterior Predictive Green - Nongreen', xlabel='θ Difference between conditions')

There is slightly greater uncertainty in the difference, and thus more overlap between the conditions. However, it is still very probable that green behaviour will be rated as more feminine, though this is not a large effect.

Conclusion

Given this data, there is convincing evidence that green behaviours, as described in the study, are associated with higher ratings of femininity, and the magnitude of this effect is around a tenth to half a rating scale point.

Predicting Concrete Strength - and winning competitions with Bayes

Thu, 07 Jul 2022 23:00:00 GMT

I recently took part in a data-science competition that involved building a linear model to estimate the strength of concrete. Perhaps not the most thrilling of topics, but it was a good place to apply some Bayesian inference. The bambi package makes fitting Bayesian linear models in Python very simple, and I’ve been using it as part of my workflow on the majority of projects. I’m pleased to say I won, and I’m posting the solution below as a reminder. In the final steps I had to translate standardised coefficients back to the original scale, which was tricky.

Can you predict the strength of concrete?

📖 Background

You work in the civil engineering department of a major university. You are part of a project testing the strength of concrete samples.

Concrete is the most widely used building material in the world. It is a mix of cement and water with gravel and sand. It can also include other materials like fly ash, blast furnace slag, and additives.

The compressive strength of concrete is a function of components and age, so your team is testing different combinations of ingredients at different time intervals.

The project leader asked you to find a simple way to estimate strength so that students can predict how a particular sample is expected to perform.

💾 The data

The team has already tested more than a thousand samples (source):

Compressive strength data:

“cement” - Portland cement in kg/m3
“slag” - Blast furnace slag in kg/m3
“fly_ash” - Fly ash in kg/m3
“water” - Water in liters/m3
“superplasticizer” - Superplasticizer additive in kg/m3
“coarse_aggregate” - Coarse aggregate (gravel) in kg/m3
“fine_aggregate” - Fine aggregate (sand) in kg/m3
“age” - Age of the sample in days
“strength” - Concrete compressive strength in megapascals (MPa)

Acknowledgments: I-Cheng Yeh, “Modeling of strength of high-performance concrete using artificial neural networks,” Cement and Concrete Research, Vol. 28, No. 12, pp. 1797-1808 (1998).

💪 Challenge

Provide your project leader with a formula that estimates the compressive strength. Include:

The average strength of the concrete samples at 1, 7, 14, and 28 days of age.
The coefficients , to use in the following formula:

Starting Note

The bambi package is needed here to help build a Bayesian regression model that will give an informative, probabilistic estimate of compressive strength. !pip install bambi was used to acquire it, PyMC, and other Bayesian workflow packages for model building and exploration.

Step 0 - Imports

import re                       # Regular expressions
import arviz as az              # Bayesian plotting
import bambi as bmb             # Bayesian model building
import numpy as np              # numeric python
import pandas as pd             # dataframes
import seaborn as sns           # plots
import scipy.stats as st        # statistical functions
import matplotlib.pyplot as plt # Plotting 

# Set seed for reproducibility
np.random.seed(34)

Step 1 - Exploratory Data Analysis

First we check for missing values, examine the datatypes of the dataframe, and examine the distribution of the target variable - strength.

# Read in the data
df = pd.read_csv('concrete_data.csv')

# Any missing values?
display(df.isna().any())

# What are the data types?
display(df.dtypes)

# Plot the strength variable
sns.displot(df['strength'])

cement              False
slag                False
fly_ash             False
water               False
superplasticizer    False
coarse_aggregate    False
fine_aggregate      False
age                 False
strength            False
dtype: bool



cement              float64
slag                float64
fly_ash             float64
water               float64
superplasticizer    float64
coarse_aggregate    float64
fine_aggregate      float64
age                   int64
strength            float64
dtype: object

There are no missing values, which is really helpful. Moreover, the data is all numeric, and the strength variable looks approximately normally distributed. We can more or less proceed straight to building our Bayesian model!

Step 2 - Building a Bayesian model

The brief clearly lays out the foundations of a linear model for the relationship between strength and the other given variables. There are many approaches to linear models (and many other kinds of modelling strategies that could be used here), but the aim seems to be to estimate the β coefficients in the model and use those to predict the strengths of different ages of concrete.

Bayesian inference is a natural method here because it allows us to estimate these coefficients with uncertainty, and best of all - we get uncertainty in our estimates for free, which would be absolutely vital in civil engineering (in fact check out episode number 59 of the Learning Bayesian Statistics podcast!)

Bayesian inference does require a prior for each of the coefficients. There isn’t any clear indication of these effects in the brief, so we will use very wide, general priors. First though, let us set up the model using bambi.

# Models are built in bambi using a formula string, much like statsmodels or R
# We lay out this model according to the brief!
modelspec = 'strength ~ cement + slag + fly_ash + water + superplasticizer + coarse_aggregate + fine_aggregate + age'

However, unlike frequentist estimators, Bayesian inference is very picky about the scale of the data. The above model would cause the MCMC sampler to struggle given the hugely different scales of the variables. The solution is to scale the predictors. A z-score standardisation works particularly well here. In fact, bambi is clever enough to allow this in the model specification - we simply wrap each term in a scale function. We do this below, and build a model instance

# Set a scaled model
modelspec = """strength
               ~ scale(cement) + scale(slag) 
               + scale(fly_ash) + scale(water)
               + scale(superplasticizer) + scale(coarse_aggregate)
               + scale(fine_aggregate) + scale(age)
               """

# Set up a model - the priors are by default very weak, so there is no need to specify them explicitly here, though we will examine them
model = bmb.Model(modelspec, df)
model.build() # creates the model graph using theano

Step 3 - Examining the prior beliefs

With the model built (but not yet fitted), it is time to check the priors. First, let us examine a plot of the prior distribution of each of the coefficients - a good way to think of this is what the model thinks the coefficients could be before it sees the data.

# Show the priors
model.plot_priors(draws=10_000, hdi_prob=.95, kind='hist', bins=100);

This is great. It shows us that the most likely effects for the main predictors are hovering around zero, but that the effect could be as big or small as about ±80 strength units. Sometimes people accuse priors of ‘influencing’ the data; but this is clearly not happening much here as those effects are very broad. Next, we check a ‘prior predictive’ distribution, which will give us an idea of what the model thinks the strength variable itself looks like, based on - The observed data for our predictors, and - Our prior distributions for the coefficients

In simple terms - we can ask the model to make some predictions before its seen the data, just to get a sense of what it thinks.

# Build prior predictive
prpc = model.prior_predictive(draws=10_000)

# And pass this to the plot_ppc function in Arviz
az.plot_ppc(prpc, group='prior', num_pp_samples=1000, backend_kwargs={'figsize': (20, 10)})

OK, its pretty clear at this point our model gets the shape of the distribution correct, but its guesses are extremely spread out - it thinks concrete strength of -250 to 250 is plausible. Obviously that is not true, and we can sharpen things up by letting the model observe the strength data.

Step 4 - Fitting the model

Inspired by sklearn, we can fit a Bayesian model in bambi using MCMC with the .fit() method. We do this below and sample from the posterior distribution (which we can then use to predict on new data).

# Fit the model using the NUTS sampler in PyMC
posterior = model.fit(draws=2000, tune=1000)

# Once fitted, let us examine the posterior distributions of the coefficients
az.plot_posterior(posterior, kind='hist',
                  color='black',
                  bins=100,
                  var_names='~sigma', 
                  filter_vars='like', 
                  hdi_prob=.95);

Auto-assigning NUTS sampler...
Initializing NUTS using jitter+adapt_diag...
Multiprocess sampling (4 chains in 4 jobs)
NUTS: [Intercept, scale(cement), scale(slag), scale(fly_ash), scale(water), scale(superplasticizer), scale(coarse_aggregate), scale(fine_aggregate), scale(age), strength_sigma]

100.00% [12000/12000 00:25<00:00 Sampling 4 chains, 0 divergences]

Sampling 4 chains for 1_000 tune and 2_000 draw iterations (4_000 + 8_000 draws total) took 58 seconds.

This contains all the information about the coefficients we may need, but we need to bear in mind scaling the predictor variables now means that - for exmaple - a 1 standard deviation increase in the cement variable is associated with a 13 point increase in strength; and we’re 95% certain the effect is between 11 and 14 units. This is an issue for age in particular as we need to predict specific ages, and now we have age on the scale of the standard deviation. First, let us examine the predictions of the model, and see if it maps closely to the data. We do that using a posterior predictive check:

# Conduct a posterior predictive check
model.predict(posterior, kind='pps')

# And plot
az.plot_ppc(posterior, num_pp_samples=500, backend_kwargs={'figsize': (20, 10)})

After seeing the data, the models predictions look pretty good - we can rest knowing our model makes sensible predictions.

Step 5 - Delivering the answers

Now we turn to answering the question. The project leader wants: - The average strength of the concrete samples at 1, 7, 14, and 28 days of age. - The coefficients

We almost have the second part, but the first part involves making some specific predictions - we want the model to tell us specifically what it thinks the strength value will be at 1, 7, 14, and 28 days. This will invovle building a new dataset for the model to predict on. What we then want is to send to the model a dataset with four new observations for age (set at the desired levels). For all other predictors, we will hold them constant at their mean levels. bambi will transform this data and give us our predictions!

# First, compute the means of the other predictors (not age, or strength), and repeat them four times
new_data = (df
            .drop(columns=['age', 'strength'])
            .mean()
            .to_frame()
            .T
           )

# Repeat it, then assign the age data
new_data = (pd.concat([new_data] * 4, ignore_index=True)
            .assign(age = [1, 7, 14, 28])
           )

# Examine
new_data.head()

	cement	slag	fly_ash	water	superplasticizer	coarse_aggregate	fine_aggregate	age
0	281.165631	73.895485	54.187136	181.566359	6.203112	972.918592	773.578883	1
1	281.165631	73.895485	54.187136	181.566359	6.203112	972.918592	773.578883	7
2	281.165631	73.895485	54.187136	181.566359	6.203112	972.918592	773.578883	14
3	281.165631	73.895485	54.187136	181.566359	6.203112	972.918592	773.578883	28

This can now be handed to the model.

# Get predictions of the mean of those datapoints
mean_predictions = model.predict(posterior, kind='mean', data=new_data, inplace=False)

# Extract them to a dataframe, and join with the 'age' column of new_data, and do some tidying up
means = (mean_predictions['posterior']['strength_mean']
         .stack(draws=('chain', 'draw'))
         .to_dataframe()
         .reset_index()
         .merge(new_data[['age']], left_on='strength_obs', right_index=True)
         .pivot(index=['chain', 'draw'], columns='age', values='strength_mean')
        )

means.head()

	age	1	7	14	28
chain	draw
0	0	30.414769	31.088415	31.874336	33.446179
	1	30.112034	30.830555	31.668829	33.345377
	2	30.747711	31.412350	32.187762	33.738587
	3	30.955742	31.603543	32.359310	33.870844
	4	30.530722	31.261242	32.113516	33.818064

Our posterior predictions for the mean concrete strength for different ages are in hand. Let us analyse and plot these: - Averages and highest density intervals - Density plots for visualisation purposes

# Create the summary per age
def summarise_bayesian(col, coverage=.95):
    
    # Mean
    mu = col.mean()
    
    # Density
    lwr, upr = az.hdi(col.values, hdi_prob=coverage)
    
    return pd.Series([mu, lwr, upr], index=['Mean Strength', 'Lower Strength Credible Limit 95%', 'Upper Strength Credible Limit 95%'])

# Show the means and credible intervals
display(
    (means
     .apply(summarise_bayesian)
     .round(2)
     .rename_axis(index='Age (in days)')
     .T
    ))

# Illustrate via a plot
with sns.plotting_context('poster'):
    
    fig, ax = plt.subplots(1, 1, figsize=(20, 10))
    sns.despine(fig)
    
    sns.kdeplot(data=means, ax=ax, fill=True, palette='Greys', linewidth=5)
    ax.set(xlabel='Concrete Strength')
    ax.get_legend().set_title('Age (in days)')

Age (in days)	Mean Strength	Lower Strength Credible Limit 95%	Upper Strength Credible Limit 95%
age
1	30.72	29.90	31.52
7	31.41	30.63	32.17
14	32.21	31.48	32.94
28	33.80	33.11	34.45

From the above it is clear that, holding all other predictors constant, the average strength of concrete increases over time. Using Bayesian methods meant our predictions of this average comes with uncertainty - for 28 days for example, the average is 33.80, but we are 95% certain it is between 33.12 and 34.45. We can also easily make probabilistic claims, such as what is the probability a 28 day old concrete sample is stronger than a 14 day old?

# Whats the probability 28 day old is stronger on average than 14? According to the model - certain!
display(means[28].ge(means[14]).mean())

1.0

The only thing left to do is to provide the project leader with the coefficient terms for our regression. Given we are using Bayesian methods, we can provide three - one for the lower 95% credibliliy, one for the mean, and another for the upper 95% credibility, just so our project leader can quantify uncertainty! To save them the trouble of transforming variables, all we need to is transform the posterior coefficient estimates to the original scale and compute a summary.

Back transformation is a little tricky but not impossible, following the formula:

for the intercept, and

for the coefficients, where represents the standard deviation of the predictor.

# As it is simpler, let us convert the coefficients back first
# First get the coefficient distributions
posterior_draws = (posterior['posterior']
                   .drop_vars(['strength_sigma', 'strength_mean', 'Intercept'])
                   .to_dataframe()
                   .rename(columns=lambda x: re.sub(r'scale|[\\(\\)]', '', x)) # little regex to remove the 'scale(var_name)' formatting
                   .apply(lambda x, df_=df: x / df_[x.name].std()) # for each column, grab the corresponding column from df, compute SD, and divide posterior distribution by it
                  )

# # Then target the intercept which is a little trickier
# First scale the coefficients differently, and sum
sum_scaled_coefs = (posterior['posterior']
                    .drop_vars(['strength_sigma', 'strength_mean', 'Intercept'])
                    .to_dataframe()
                    .rename(columns=lambda x: re.sub(r'scale|[\\(\\)]', '', x)) # little regex to remove the 'scale(var_name)' formatting
                    .apply(lambda x, df_=df: x * (df_[x.name].mean() / df_[x.name].std())) # now grab corresponding column compute mean/sd and multiply by estimate
                    .sum(axis='columns')
                    .reset_index(drop=True)
                    .squeeze()
                   )

# Compute the rest of the equation 
Intercept = (posterior['posterior']['Intercept']
             .to_dataframe()
             .reset_index(drop=True)
             .squeeze()
             .sub(sum_scaled_coefs)
             .to_frame()
             .set_index(posterior_draws.index)
            )

# and add it to posterior_draws
posterior_draws.insert(0, 'Intercept', Intercept)

posterior_draws.head()

		Intercept	cement	slag	fly_ash	water	superplasticizer	coarse_aggregate	fine_aggregate	age
chain	draw
0	0	-23.620205	0.115327	0.107009	0.086837	-0.142423	0.260784	0.019041	0.018877	0.112220
	1	-15.596581	0.118582	0.106121	0.086538	-0.157617	0.304718	0.012568	0.018382	0.119695
	2	-26.222202	0.114074	0.098117	0.079643	-0.125431	0.376820	0.018306	0.020487	0.110719
	3	-21.705891	0.115148	0.089371	0.083510	-0.144183	0.453397	0.023125	0.012821	0.107914
	4	-35.762022	0.132922	0.113358	0.102077	-0.175410	0.137964	0.021733	0.031982	0.121694

With some pandas wrangling, we now have the posterior distributions on the original scale. This is one of the downsides of Bayesian modelling - sometimes you need the coefficients on the scale of the input variables, but MCMC sampling struggles with varying scales, so most estimation is done on a standardised scale. But we have it; and now we can give the three regression equations for the mean and upper/lower uncertainty:

# State the regression equation for the project lead, as well as upper/lower credible bounds
display(posterior_draws.apply(summarise_bayesian).round(3))

	Intercept	cement	slag	fly_ash	water	superplasticizer	coarse_aggregate	fine_aggregate	age
Mean Strength	-22.412	0.120	0.104	0.088	-0.151	0.289	0.018	0.020	0.114
Lower Strength Credible Limit 95%	-75.285	0.103	0.084	0.063	-0.230	0.108	-0.002	-0.002	0.104
Upper Strength Credible Limit 95%	29.179	0.136	0.124	0.113	-0.071	0.470	0.036	0.040	0.125

Conclusion

Using Bayesian inference, we have estimated a model that solves the problem the project leader asked for. For each of the predictor variables, we obtained a coefficient (actually a distribution of credible coefficients) that represents how the variable changes with concrete strength. One of the amazing things about Bayesian estimation is that we have free uncertainty in our estimates. As such, we are able to provide not just a point estimate (the mean-strength formula above) but also the lower and upper 95% credible values for each variable, which essentially represents actually-possible regression equations. The students would be wise to incorporate this uncertainty.

We also used the model to provide estimates for concrete strength at different levels of age, in days; 1, 7, 14, and 28 days. Again, because of the power of Bayesian estimation, we were able to provide credible limits for these predictions and were able to make statements about the superiority of concrete strength - for example, our model was 100% certain that, all other variables being equal, 28 day old concrete is stronger than 14 day old concrete.

Step [-1] - BONUS ANALYSIS

What is stated above is a fairly standard example of Bayesian linear regression. In linear regression, we aim to predict the mean of the distribution of observed data as a linear combination of factors. Formally, it is set out as:

However, from a probabilistic perspective, a linear model can be written like this:

where

That is, our observed data is expected to be normally distributed with the mean parameter generated by the linear combination of predictors, and a standard deviation parameter.

What is not obvious from looking at this formula is that it implies that there is a distribution per-observation governed by the model, and not a distribution around the mean parameter, which is what is focused on in the above analysis. In fact, using Bayesian estimation, it is possible to obtained a distribution of a given data point, known as the posterior predictive distribution, which is the distribution of future data points, given observed data points, integrating out the parameters in the model.

While this sounds complex, in reality what it gives us is simple: a probability distribution for a single datapoint for what it may look like in future, after taking the model into account. This is really important for the application at hand, given that concrete strength is critical for a lot of work! Specifically we can look past the mean of the distributions - that is, what is the average concrete strength for a given 14 day old sample - but instead estimate likely values we may see in future. Let’s examine this below and see what the distributions look like for the different ages using an empirical cumulative distribution plot. As will be seen, density plots will be too messy here.

# Sample the posterior predictive distribution
ppd = model.predict(posterior, kind='pps', data=new_data, inplace=False)

# Examine it
ppd = (ppd['posterior_predictive']
       .to_dataframe()
       .unstack('strength_dim_0')
       .rename(columns=new_data.age)
       .droplevel(0, axis='columns')
      )

with sns.plotting_context('poster'):
    
    plt.style.use('bmh')
    
    fig, ax = plt.subplots(1, 1, figsize=(20, 10))
    sns.despine(fig)
    
    sns.ecdfplot(data=ppd, ax=ax, palette='Greys', linewidth=5)
    ax.set(xlabel='Concrete Strength', facecolor='white')
    ax.get_legend().set_title('Age (in days)')
    ax.set_xticks(labs := np.arange(0, 110, 10), labels=labs)
    ax.set_yticks(labs := np.arange(0, 1.1, 0.1), labels=[round(x,1) for x in labs])

This gives a surprising result. While we can see that 28 day old concrete, for the most part, is stronger than the others. But the variability here is much greater than when just estimating the mean. In fact, there are similar overlaps in strength across all ages. For example, about 70% of 28 day year old concrete will be weaker than 40, but around 77% of 14 day old will be weaker, a difference of just 7% points. Above all, this highlights that future variability is much greater than intuition suggests.

COVID-19 Vaccine Side Effects — a rough night, in data

Wed, 06 Jul 2022 23:00:00 GMT

In April 2021 I got my first COVID-19 vaccination. The side effects were pretty hard going — for about 24 hours I was laid out, and woke up in the middle of the night to my smart watch giving me an “extreme heart rate warning”, which is just the kind of thing you want to see after getting injected with a vaccine for a virus sweeping the globe.

Once I felt more alive, I had the idea of extracting the heart rate data from my watch. As it turns out, there is an excellent Python package, fitparse, that allows the parsing of the .FIT files from Fitbit watches. I won’t post the raw data here and the associated code, but it was pretty simple to get a few days’ worth of data from the watch and plot it. Easy to see how bad things were — which made me extremely grateful for getting vaccinated and not the real thing.

I’m posting this because the plot took a little while to do, and has some functions I don’t use often and am always looking up. The trickiest part was adding annotations to the x-axis — because it’s a datetime object, positions need to be expressed as datetimes too. This took some creativity to make a readable attempt: while annotate gives flexibility in how you specify the text locations, raw data coordinates are the easiest approach, and the raw string representation of dates messes up the temporal ordering of the plot.

# Imports
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

# Data preprocessing
hr = (pd.read_csv('https://raw.githubusercontent.com/alexjonesphd/alexjonesphd.github.io/master/assets/covid/vaccination_experience.csv')
      .assign(date=lambda x: pd.to_datetime(x['timestamp'], format='%Y-%m-%d'))
      .sort_values(by='date')
      .dropna(subset='heart_rate', how='any')
      .query('heart_rate > 50')
      .assign(rolling_30=lambda x: x['heart_rate'].rolling(30).mean())
      .reset_index(drop=True)
     )

hr.head()

	timestamp	stress_level	heart_rate	date	rolling_30
0	2021-04-22 23:00:00	30	84.0	2021-04-22 23:00:00	NaN
1	2021-04-22 23:01:00	21	68.0	2021-04-22 23:01:00	NaN
2	2021-04-22 23:02:00	18	67.0	2021-04-22 23:02:00	NaN
3	2021-04-22 23:03:00	19	63.0	2021-04-22 23:03:00	NaN
4	2021-04-22 23:04:00	15	62.0	2021-04-22 23:04:00	NaN

There were also a bunch of things that happened I want to annotate on the plot, so first I’ll make a dictionary containing them along with their associated time points and my heart rate at the time (I worked backwards from the plot to figure these out, but it makes the code tidier to include here). These will be used as coordinates for the annotate method of an axis object.

# Event log dictionary, using datetime spec for X locations - the strings will need parsing into datetimes
# Format follows XX, YY...
events = {'Morning dog walk': [('2021-04-23 10:45', 105), ('2021-04-23 10:45', 109)],
          'Vaccinated!': [('2021-04-23 17', 125), ('2021-04-23 18', 135)],
          'Evening dog walk': [('2021-04-23 19:30', 108), ('2021-04-23 21', 125)],
          'Suspiciously high for\njust watching TV...': [('2021-04-23 22:30', 98), ('2021-04-23 22:30', 115)],
          'Sleep': [('2021-04-24 00', 68), ('2021-04-24 00', 55)],
          'Wake up with chills': [('2021-04-24 01', 83), ('2021-04-24 01', 65)],
          'Abnormal heart rate warning': [('2021-04-24 03:55', 134), ('2021-04-24 03:58', 140)],
          'Move downstairs': [('2021-04-24 12:30', 155), ('2021-04-24 13:15', 160)]
         }

Plot and annotate some of the events of the 24 hours and what I was doing, and do it in xkcd style.

Vaccines are great, and having a heart rate over 150bpm just walking down the stairs is worth it.

# Set up plotting context, style, and fig/ax
sns.set_context('poster', font_scale=2)
with plt.xkcd(scale=0.9, length=100, randomness=200):
    
    # Fig/axis
    fig, ax = plt.subplots(1, 1, figsize=(50, 20))
    sns.despine(fig) # Turn off up/right axis

    # Plot both raw and rolling lines separately for easier control
    sns.lineplot(data=hr, ax=ax,
                 x='date', y='heart_rate', 
                 alpha=.2, color='black', 
                 label='Raw Heart Rate')

    sns.lineplot(data=hr, ax=ax,
                 x='date', y='rolling_30', 
                 color='black', linestyle='--',
                 linewidth=8,
                 label='Rolling 30 minute average')

    # Add a horizontal line at 100bpm, shift legend, add a shaded block for the 23rd, annotate the 23rd/24th
    ax.axhline(100, color='black', alpha=.5)
    ax.legend(loc='upper left')
    ax.axvspan('2021-04-23 00:00:00', '2021-04-24 00:00:00', color='silver', alpha=.05)
    ax.text(.27, .80, '23rd', transform=ax.transAxes, fontsize=80)
    ax.text(.85, .80, '24th', transform=ax.transAxes, fontsize=80)
    
    # Add the annotations - the X coord needs converting to datetime
    for event, time in events.items():
        
        # Convert the first element - cleanest way I could think of
        time_ = [(pd.to_datetime(t[0]), t[1]) for t in time]
        
        # Add annotation
        ax.annotate(event, *time_, arrowprops={'width': 1}, fontsize=30)

    # Clean up axes, setting an x tick every four hours in date-time format, then relabelling
    ax.set(ylabel='Heart Rate\nbpm', xlabel='Time', ylim=(50, 160),
           xticks=[f'2021-04-{d} {h:02}:00:00' for d in [23, 24] for h in [0, 4, 8, 12, 16, 20]],
           xticklabels=[f'{x:02}:00' for x in [0, 4, 8, 12, 16, 20]]*2)

Bayesian Bookshelf

Tue, 05 Jul 2022 23:00:00 GMT

The pandemic lockdowns were a time for exploring new interests for a lot of people — baking sourdough, gardening, photography… Perhaps foolishly, I decided to get into Bayesian statistics.

I’d read a lot in the psychological methods literature for some time that the Bayesian approach was a natural and easily interpretable approach to inference, could give evidence for null effects, and there was no need to open a textbook to get the definition of a confidence interval every single time. I found plenty of published tutorial papers starting with these promises, but by the second paragraph almost all of them contained either deeply philosophical views on probability or introduced terrifying formulas and distributions I’d never heard of.

Even worse, there seemed to be two distinct schools of thought on Bayesian inference — an estimation and uncertainty approach, and the Bayes Factor hypothesis testing school. The former didn’t like the latter, and the latter seemed disconnected to the former and felt suspiciously like frequentist p-values, with journals requiring minimum thresholds for publication and discussions around the probability a Bayes Factor would be “significant”.

After two years of reading as many books as I could afford on the topic, I genuinely do think Bayesian inference has all of the benefits you read about. It is a natural way to reason about data, build models, and think about statistics. But after years of using frequentist approaches, it is not easy. Ironically, I learned much more about frequentist statistics by reading about Bayesian inference than I did in nearly a whole decade of using frequentist statistics.

If you’re interested in learning more about Bayesian inference, I’ve found these books very useful. They are a blend of theoretical and practical — I’m most comfortable in Python, but many of the ideas, if not the implementation, translate to other languages easily. Your experience may vary, but if you have a background in social or psychological sciences, my prior is that these books will help you like they helped me.

1. Bayesian Statistics The Fun Way

Will Kurt

This book is the perfect place to start. It gently introduces the idea of probability, probability distributions, and explains Bayes’ rule with a lot of clear examples. There’s chapters on priors and how to update them, and discussion of using Bayes Factors to understand individual degrees of belief. There’s some R code too to help calculate some of the trickier aspects that are necessary for inference, and everything is presented in a clear and engaging way.

2. Bayesian Statistics for Beginners: A step-by-step approach

Therese Donovan and Ruth Mickey

This book is amazing. It was after reading this that things finally started to make sense. The authors — who describe themselves as lifetime frequentists — go through a series of chapters that introduce probability distributions, the concepts of priors and likelihoods, and how to use Bayes’ rule to calculate posterior distributions with simple datasets. The thing I love the most about this book is that it is not afraid to repeat itself. In fact, the second half of the book revisits all the concepts of the first half, but repeats the examples with flavours of how modern Bayesian inference is done (using Markov-chain Monte Carlo methods), and introduces some basic models estimated with Bayesian methods. It’s very clear, accessible, and I got the sense the authors knew what I was thinking before I did when it came to trickier parts.

Outline of the book here.

3. Bayes Rules! An introduction to applied Bayesian Modelling

Alicia Johnson, Miles Ott, and Mine Dogucu

There’s a lot to cover learning Bayesian stats before you can even think about doing something as simple as a correlation. This book gives very clear examples of fitting regression models (all done in R) using Bayesian methods, and introduces concepts like prior predictive checks (what does my model think before it sees the data?) and how mixed models are thought of from a Bayesian perspective. There’s a lot of excellent content here — all types of linear models are covered, from simple linear regression through to generalised mixed models. This is an ideal hands-on book, especially if you’re comfortable with R. It’s an extremely accessible read.

You can find this book online here.

4. Statistical Rethinking, Second Edition

Richard McElreath

I would guess this is one of the most influential modern books on Bayesian statistics, and for good reason. McElreath works from the first principles of probability through to causal inference with directed acyclic graphs. As the name suggests, the information is presented alongside deep perspectives on inference and what it means to learn from data, and how we can often fool ourselves. This is a challenging but rewarding read — things made a lot more sense to me after some gentler texts, but it’s never far away when I am fitting models. It also comes with its own R package to facilitate model building, and is as hands-on as it is theoretical.

More info on Rethinking here.

5. Doing Bayesian Data Analysis

John Kruschke

From an applied statistics perspective, this is about as comprehensive a guide to Bayesian statistics as you would hope to find. Kruschke has an accessible style and teaches Bayesian inference in comparison to frequentist approaches. Once I was comfortable with the idea of priors and model building this book helped to connect the dots between what frequentist models were doing and their Bayesian equivalents, and what the differences meant. There is also some insightful (and incisive) discussion between the two approaches to statistical inference here, and the level of detail is excellent. Kruschke has really brought the estimation approach to the fore in psychology, and demonstrates how techniques like regions of practical equivalence can give evidence for the null. This book also helped connect Bayes Factors and Bayesian estimation, with some convincing criticisms of the former approach.

John Kruschke also has an associated blog here.

6. Bayesian Methods in Statistics: From Concepts to Practice

Mel Slater

One of the things I like about this book is that it has some excellent chapters on probability distributions which have been indispensable in figuring out different types of priors in analyses. It also has a wealth of examples in how to fit many models in Stan, a probabilistic programming language (read: code that does Bayesian inference) that interfaces with R and Python. It’s also got some nice examples of how to compute Bayes Factors from fitted models, which a lot of books tend to avoid. This book has a lot of formulas, which if you’re unaccustomed to reading can prove difficult. Disclaimer: I was a reviewer of this book!

Outline here.

7. Bayesian Analysis with Python, Second Edition

Osvaldo Martin

This book changed my statistical analysis practices enormously. Written by one of the developers of the amazing PyMC library, Python’s probabilistic programming package. It contains a range of examples of how to use the package, and translates a number of statistical models (linear models, generalized linear models, and hierarchical models) into Python and PyMC code. These examples are excellent. If like me you can read code faster than formulas, some big gains in understanding are to be found here. The book also introduces some of the other approaches in Bayesian statistics beyond the traditional linear model approaches, including Gaussian processes — which give you probability distributions over functions, rather than model parameters! Incredible stuff.

Codebase and discussion here.

8. Bayesian Modeling and Computation in Python

Osvaldo Martin, Ravin Kumar, and Junpeng Lao

An advancement on Bayesian Analysis with Python, this features updated approaches to Bayesian inference in Python, using both PyMC and other probabilistic programming languages. There’s some really advanced topics in this book that showcase what Python can do in this realm, and there’s perhaps one of the clearest demonstrations of hierarchical models from a Bayesian perspective written in code, as well as great examples of model-checking practices like Bayesian p-values. Not a beginner text, but one to expand the skill-set in Python.

You can find this book online here.

9. A Student’s Guide to Bayesian Statistics

Ben Lambert

I have some mixed feelings on this book, and depending on your journey you may find it more or less useful. I encountered it pretty early on, and found it difficult. The exercises at the end of each chapter are hard and that can be off-putting, and the focus is on the theoretical approaches (with some minor applications with R), rather than actual analysis. Looking back though, the book does a really thorough job of going through each part of the famous formula — there’s a chapter on priors, , one on likelihoods, , and on the marginal likelihood, , and how these come together. One of the standout sections is on the algorithms used in modern Bayesian computing and how they work. I’d say these are the most detailed and accessible descriptions of these “inference engines” I’ve seen.

10. Bayesian Data Analysis, 3rd Edition

Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin

There’s a reason this book is known as the “Bayesian bible”. It’s written by some of the leaders in the field of Bayesian statistics, as well as some of the core developers of the Stan language. The book is theoretical in scope, and lays out the mathematics behind Bayesian statistics comprehensively. I’ve definitely not read this cover to cover, but it’s a good litmus test of building knowledge — any time I pick it up and read through, I know I am progressing if I am following along. Not a beginner’s book but an essential addition.

You can find this book here.

Other notable mentions

Regression and Other Stories — Andrew Gelman, Jennifer Hill, and Aki Vehtari. This is the Trojan horse of Bayesian statistics. Presented as a thorough guide to how linear models are crafted and interpreted, with some solid advance on abandoning statistical significance and a gentle demonstration of how prior information is necessary.

Bayesian Methods for Hackers — Cameron Davidson-Pilon. This book (found online and continually updated) showcases many uses of Bayesian inference, using PyMC. Interestingly, this book is pretty much devoid of formulas and takes a strong code-only approach. The examples here are pretty sophisticated, but the code focus shows how to do some creative statistics with Bayes.

Bernoulli’s Fallacy — Aubrey Clayton. No statistics here, but an inspiring historical and contemporary discussion of the Bayesian view of probability. By the time I read the book I didn’t need convincing, but if you’re curious about the philosophy of Bayesian inference, start here. And then read it again if you’re not convinced.

Alex Jones Ph.D

Bayesian Ordinal Regression in PyMC

Are the latent variables in the room with us?

Simulating ordinal data with PyMC from scratch

Recovering parameters of an ordinal distribution with PyMC - the Hard Way

Recovering parameters - the easy way

A flag in the ground - fixing cutpoints to estimate the location and scale

When looking hot means not feeling cold

A final note on priors for cutpoints

Out-of-sample predictions with PyMC: A hieararchical linear model case study

Out of sample predictions, approach one - pm.MutableData

Fitting the hierarchical model

Out of sample predictions

Type 1 - predicting for an “average” person, or omitting the group-specific terms

Type 2 - predicting new scores for specific, observed groups

Type 3 - Predicting scores for unobserved groups

Customer segmentation with Bayes, Spectral Embedding, and a second-place win

Data-driven segmentation reveals three doctor profiles

Executive Summary

Key Insights

New doctor segments:

Key recommendations

Doctor-by-region analysis; interactively

Assessing the purchase ~ complaint association probabilistically

Customer segmentation - revealing three profiles

Technical Annexe

Feature engineering

Assessing purchase ~ complaint association probabilistically

Customer segmentation - revealing three profiles

Multi100 Collaboration Project: Is Eco-Friendly Unmanly?

Multi100 Analysis — NF503

Is Eco-Friendly Unmanly? The Green-Feminine Stereotype and Its Effect on Sustainable Consumption

Step 1

Step 2

Step 3

Step 4

Conclusion

Predicting Concrete Strength - and winning competitions with Bayes

Can you predict the strength of concrete?

📖 Background

💾 The data

Compressive strength data:

💪 Challenge

Starting Note

Step 0 - Imports

Step 1 - Exploratory Data Analysis

Step 2 - Building a Bayesian model

Step 3 - Examining the prior beliefs

Step 4 - Fitting the model

Step 5 - Delivering the answers

Conclusion

Step [-1] - BONUS ANALYSIS

COVID-19 Vaccine Side Effects — a rough night, in data

Bayesian Bookshelf

1. Bayesian Statistics The Fun Way

2. Bayesian Statistics for Beginners: A step-by-step approach

3. Bayes Rules! An introduction to applied Bayesian Modelling

4. Statistical Rethinking, Second Edition

5. Doing Bayesian Data Analysis

6. Bayesian Methods in Statistics: From Concepts to Practice

7. Bayesian Analysis with Python, Second Edition

8. Bayesian Modeling and Computation in Python

9. A Student’s Guide to Bayesian Statistics

10. Bayesian Data Analysis, 3rd Edition

Other notable mentions

Out of sample predictions, approach one - `pm.MutableData`