Posts on Bryan Shalloway's Blog

Identifying the Owners of Unclaimed Property

Tue, 23 Jun 2026 00:00:00 +0000

TL;DR: I used Claude to build a pipeline that identifies likely owners of unclaimed property (lost or dormant assets held by the state) and just messaged the likely owners of ~$10M currently held by the state of CA. Initial notes on the project/approach are at brshallo/escheat-finder.

There’s that classic economist joke:

Two economists walk down a road and see a twenty dollar bill lying on the sidewalk.

One asks, “Is that a twenty dollar bill?”

The other answers, “It can’t be, because someone would have picked it up already.”

And they keep walking.

With the emergence of AI, it feels like we’re in that in-between period where there’s lots of money that has yet to be picked up. Navigating the tangle of transactions of unclaimed property seems like one such problem that should just be mostly “solved” by agents scouring public records. Until the data aggregators or the state finish the job, I thought I’d explore throwing some intelligence at it.

The process started as one of a few candidates for: “What’s something genuinely useful I could set up to throw excess personal Claude tokens at before my limit resets each week?”

After some initial tests and positive responses, I felt good enough about my system to reach out to this first batch of people. Through a bunch of iterations, I now have this initial list of people I’ve reviewed (doctors, professors, non-profit coordinators, …) and feel a real weight thinking about what their reaction might be when/if they see my message. Will they be…

Excited about an asset they didn’t know they had?
Disappointed about securities they thought were appreciating but have instead been liquidated and sitting in a 0% government account?
Annoyed at some random person they think is trying to scam them? (Or that contacted them by mistake.)

I’m also still calibrating where to set the threshold for unsolicited outreach. How confident should I be before contacting someone? Is it more acceptable to reach out on a weaker match if the potential claim is much larger? How many times should I try contacting them. What’s the most effective medium (text, social media, email, mail). I’m interested to see how that changes as I start getting feedback from the people I contact in this first batch of emails.

We’ll see how this initial run goes.

Hoping people read their inboxes 🤞🤞🤞!

As a PSA, everyone should check their name (and their friends' and families' names) at https://missingmoney.com/. It’s free to search and file claims (the site is legitimate and run by the National Association of State Treasurers and State Unclaimed Property Administrators).

Statistical Distributions of Shooting Drills

Fri, 31 Jan 2025 00:00:00 +0000

TL;DR: A quick look at the binomial and negative binomial distributions through the lens of basketball shooting drills.

A friend and I were doing basketball shooting drills. To make things interesting, we bet on each round: the loser paid a dollar for every shot they lost by. For the first drill we took 30 shots. E.g. if I made 16 and he made 14, I’d win $2. For the next drill you shot until you made 15, with the loser paying a dollar for each extra miss. So if I took 27 shots to make my 15 (12 misses) and it took them 31 (16 misses), I’d win $4.

If we each shoot 50%, both games have the same expected outcome: 30 shots each, 15 for 30, and the bet’s a wash. But while the expected values match, the spread of outcomes is quite different.

Scenario 1: difference of binomial distributions

The distribution of outcomes for the first scenario can be modeled as the difference of two binomial distributions. To unpack this a little: the chance of making a single shot can be represented by what’s termed a bernoulli distribution¹, which just assigns a probability to each of the two outcomes (make or miss). A binomial distribution then gives the distribution of successes² across some number of bernoulli trials³. E.g. given a 50% chance of making each shot and 30 attempts, what’s the chance you make 0, 1, 2, … up to 30 shots.

Here’s what that distribution looks like for a 50% shooter taking 30 shots:

library(tidyverse)

tibble(makes = 0:30) |>
  mutate(probability = dbinom(makes, 30, 0.5)) |>
  ggplot(aes(x = makes, y = probability))+
  geom_col()+
  theme_bw()+
  scale_y_continuous(labels = scales::percent)

However I care about modeling pay-outs, i.e. the distribution of the difference in makes between me and my friend. In this simple case you could derive that distribution analytically⁴, but often your situation is too complicated to derive, so instead you simulate from the underlying components. Each round here is just the difference of two binomial distributions (each 30 shots at 50%). I’ll run this a million times and look at the distribution of payouts⁵:

set.seed(123)
# simulate a million rounds of 30 v 30 shots
output <- tibble(outcomes = rbinom(1000000, 30, 0.5) - rbinom(1000000, 30, 0.5))

output |>
  count(outcomes) |>
  mutate(prop = n / sum(n)) |>
  ggplot(aes(x = outcomes, y = prop))+
  geom_col()+
  theme_bw()+
  scale_y_continuous(labels = scales::percent)

For spread, let’s look at the magnitude of payouts irrespective of who wins.

output_abs <- output |> mutate(outcomes_abs = abs(outcomes))

output_abs |>
  count(outcomes_abs) |>
  mutate(prop = n / sum(n)) |>
  ggplot(aes(x = outcomes_abs, y = prop))+
  geom_col()+
  theme_bw()+
  scale_y_continuous(labels = scales::percent)

The typical (50th percentile) outcome is someone winning by three shots; the 90th percentile is six.

output_abs |>
  with(quantile(outcomes_abs, c(0.5, 0.9)))

## 50% 90% 
##   3   6

Scenario 2: difference in negative binomial distributions

In our second scenario we no longer have a fixed number of shots and an uncertain number of makes – we’ve fixed the makes at 15 and want the distribution of misses accumulated on the way there. We’re asking the inverse, or negative, of the earlier question, so perhaps it’s no surprise the relevant distribution is called the negative binomial. It too can be thought of as a series of bernoulli trials, but now we keep shooting until we hit a target number of makes, counting the failures along the way⁶ ⁷ ⁸.

Here’s the distribution of misses on the way to 15 makes for a 50% shooter.

tibble(misses = 0:50) |>
  mutate(probability = dnbinom(misses, 15, 0.5)) |>
  ggplot(aes(x = misses, y = probability))+
  geom_col()+
  theme_bw()+
  scale_y_continuous(labels = scales::percent)

A couple things to notice:

it’s wider than our initial binomial distribution – the variance here is 30⁹ vs 7.5¹⁰ for the binomial.
it’s right skewed¹¹.

So it’s not surprising that simulating the difference of two negative binomials also shows a wider spread:

set.seed(123)
nb_output <- tibble(outcomes = rnbinom(1000000, 15, 0.5) - rnbinom(1000000, 15, 0.5))

nb_output |>
  count(outcomes) |>
  mutate(prop = n / sum(n)) |>
  ggplot(aes(x = outcomes, y = prop))+
  geom_col()+
  theme_bw()+
  xlim(c(-40, 40))+
  scale_y_continuous(labels = scales::percent)

In terms of magnitude of payouts:

nb_output_abs <- nb_output |> mutate(outcomes_abs = abs(outcomes))

nb_output_abs |>
  count(outcomes_abs) |>
  mutate(prop = n / sum(n)) |>
  ggplot(aes(x = outcomes_abs, y = prop))+
  geom_col()+
  theme_bw()+
  xlim(c(0, 40))+
  scale_y_continuous(labels = scales::percent)

Our typical payout is now $5 and our 90th percentile payout is $13 (compared to $3 and $6 respectively in our first game).

nb_output_abs |>
  with(quantile(outcomes_abs, c(0.5, 0.9)))

## 50% 90% 
##   5  13

So if you hold the successes constant you can expect larger payouts between rounds in our second game¹² ¹³.

Which game should you play if you’re better?

Say you have an edge on your friend. Which game is better for you?

Let’s plot the expected value and standard deviation of payouts as your friend’s shooting percentage drops from 50% to 40%.

diffs_of_dists <- function(prob2, prob1 = .5, dist = rbinom, n = 30, size = 1000000){
  dist(size, n, prob1) - dist(size, n, prob2)
}

game_stats <- map_dfr(seq(.40, .50, by = 0.01), function(prob_friend){
  binom_diffs <- diffs_of_dists(prob_friend)
  # in scenario 2 fewer misses is better, so flip the sign to get *your* payout
  nbinom_diffs <- -diffs_of_dists(prob_friend, n = 15, dist = rnbinom)

  tibble(
    friend_shooting_pct = prob_friend,
    game = c("scenario 1: fixed shots", "scenario 2: shoot until 15 makes"),
    expected_payout = c(mean(binom_diffs), mean(nbinom_diffs)),
    sd_payout = c(sd(binom_diffs), sd(nbinom_diffs))
  )
})

game_stats |>
  pivot_longer(c(expected_payout, sd_payout)) |>
  mutate(name = ifelse(name == "expected_payout", "expected payout", "standard deviation of payout")) |>
  ggplot(aes(x = friend_shooting_pct, y = value, colour = game))+
  geom_line()+
  facet_wrap(~name, ncol = 1, scales = "free_y")+
  expand_limits(y = 0)+
  theme_bw()+
  theme(legend.position = "bottom")+
  scale_x_continuous(labels = scales::percent, breaks = c(.40, .45, .50))+
  scale_y_continuous(labels = scales::dollar, breaks = seq(0, 10, by = 2))+
  labs(x = "friend's shooting percentage", y = NULL)

Scenario 2 has the better expected payout but more uncertainty – and its uncertainty grows with the skill gap, whereas scenario 1’s variability stays about the same either way.

So the two games are only “the same” on average. The data generating process doesn’t just set what you expect to win – it also sets the spread of what actually happens and how that spread moves with the skill gap. Pick your game accordingly.

Appendix

Hypothesis tests on observational data

The two games can also mess with a statistician using observational data with no control of sample size. Pretend someone at the community center likes to watch players doing shooting drills and use the observational data to run hypothesis tests on whether they’re equally skilled. The observer doesn’t control the games – they just analyze whatever shots happen. Let’s simulate type 1 (detects a difference when there isn’t one) and type 2 (misses a difference when there is one) errors under a few scenarios.

One tweak first: we’ll have the players run longer versions of both games – 100 shots in the fixed game, shoot until 50 makes in the other¹⁴.

The observer would most likely use:

difference of proportions z test
chi-square test
fisher’s exact test

I’ll simulate the outputs from each.

run_tests <- function(makes, misses){
  shot_table <- rbind(makes, misses)

  tibble(
    `z test` = prop.test(makes, makes + misses, correct = FALSE)$p.value,
    `chi-square test` = chisq.test(shot_table)$p.value,
    `fisher's exact test` = fisher.test(shot_table)$p.value
  )
}

simulate_tests <- function(prob1, prob2, game, n_sims = 10000){
  map_dfr(1:n_sims, function(sim){
    if(game == "scenario 1"){
      makes <- c(rbinom(1, 100, prob1), rbinom(1, 100, prob2))
      misses <- 100 - makes
    } else {
      misses <- c(rnbinom(1, 50, prob1), rnbinom(1, 50, prob2))
      makes <- c(50, 50)
    }
    run_tests(makes, misses)
  }) |>
    summarise(across(everything(), ~mean(.x < 0.05))) |>
    mutate(game = game, .before = 1)
}

set.seed(123)
bind_rows(
  simulate_tests(0.5, 0.5, "scenario 1") |> mutate(skill = "equal (both 50%)", .before = 1),
  simulate_tests(0.5, 0.5, "scenario 2") |> mutate(skill = "equal (both 50%)", .before = 1),
  simulate_tests(0.5, 0.4, "scenario 1") |> mutate(skill = "different (50% v 40%)", .before = 1),
  simulate_tests(0.5, 0.4, "scenario 2") |> mutate(skill = "different (50% v 40%)", .before = 1)
) |>
  mutate(game = ifelse(game == "scenario 1", "scenario 1: fixed 100 shots", "scenario 2: shoot until 50 makes")) |>
  knitr::kable()

skill	game	z test	chi-square test	fisher’s exact test
equal (both 50%)	scenario 1: fixed 100 shots	0.0565	0.0401	0.0401
equal (both 50%)	scenario 2: shoot until 50 makes	0.0482	0.0351	0.0401
different (50% v 40%)	scenario 1: fixed 100 shots	0.3163	0.2671	0.2671
different (50% v 40%)	scenario 2: shoot until 50 makes	0.3274	0.2794	0.3007

The table shows the proportion of rounds where each test flagged a skill difference: for “equal” rows that’s the type 1 error rate (should be near our 5% significance level), for “different” rows it’s the power of the test (the complement of the type 2 error rate).

With equal players all three tests hold up in either game (z close to nominal, chi-square and fisher’s exact a touch conservative). But when there is a skill gap, every test is more likely to catch it in the shoot-until game. Same shooters, same tests but the observer reaches different conclusions at different rates depending on which game happened to be played.

The culprit is mostly sample size: reaching 50 makes at a 40% clip takes ~125 shots vs the fixed 100, so the shoot-until game generates more data on the weaker shooter. With observational data your sample is often at the whims of the unknown data generating process which can impact your conclusions in observational studies¹⁵.

though this can also be thought of as a special case of a binomial distribution where number of rounds is equal to 1.↩︎
In our case, given the probability of making each shot and the number of attempts, its probability mass function assigns a probability to each possible number of makes.↩︎
I.e. rounds of selecting from a bernoulli distribution.↩︎
using fancy math and convolution↩︎
I’m pretending we played a million rounds and showing the proportion of rounds that land in each payout bucket. Downsides of simulation: it can be slow, and it can struggle to accurately capture the tails of the distribution. E.g. if you care about a 1 in 100,000 outcome, a simulation of a million rounds will only produce a handful of draws that far out, so the simulated tail is noisy. In those cases you can apply methods that smooth information across the rare outcomes (or lean on analytic results) to get a better estimate of the tails.↩︎
Equivalently, you can think of it as a sum of geometric distributions: chop the sequence of shots at each make and the misses between consecutive makes follow a geometric distribution, so our negative binomial is just the sum of 15 geometrics. This sum-of-independent-pieces view is also why the normal approximation discussed later works.↩︎
Here’s a video with a little more detail on these distributions and the relation between them: https://www.youtube.com/watch?v=BPlmjp2ymxw ↩︎
Conveniently, R’s dnbinom() and rnbinom() are parameterized in terms of the number of failures before hitting a target number of successes – which maps exactly onto counting misses before 15 makes.↩︎
(r * (1 - p)) / p^2 = (15 * 0.5) / 0.5^2 = 30↩︎
n * p * (1 - p) = 30 * 0.5 * 0.5 = 7.5↩︎
I cut the chart at 50 misses but the distribution goes on to infinity.↩︎
You could actually skip the simulations here: both payout distributions are approximately normal (a binomial is a sum of bernoulli trials, a negative binomial is a sum of geometrics, so the CLT applies – and differencing two identical copies cancels the negative binomial’s right skew). Scenario 1 is roughly N(0, 2 * n * p * (1 - p)) and scenario 2 roughly N(0, 2 * r * (1 - p) / p^2), and if you hold the expected number of shots equal (n = r / p) the ratio of the standard deviations works out to exactly 1/p. At 50% shooting the shoot-until game has exactly double the spread.↩︎
This roughly doubled spread isn’t an artifact of double counting: in the first game paying on the difference in makes is the same as paying on the difference in misses (shots are fixed at 30, so every extra make is exactly one fewer miss). Both games are settling-up on misses, the shoot-until-15 game just genuinely has about twice the spread. You can see the same relationship in the expected (absolute) payouts: $6.12 for the shoot-until-15 game vs $3.08 for fixed shots.↩︎
At the shorter lengths from earlier, the counts are small enough that these tests all get a bit conservative and noisy, which muddies the comparison we care about.↩︎

E.g. if you only took the first 50 observations then it doesn’t matter which game was being played – the first 50 shots of either game are just the same coin flips, since a stopping rule can’t affect shots taken before it kicks in:

z_pvalue <- function(makes1, makes2, n = 50){
  prop.test(c(makes1, makes2), c(n, n), correct = FALSE)$p.value
}

# play out a shoot-until-50-makes game, but only score the first 50 shots
first_50 <- function(prob){
  shots <- rbinom(500, 1, prob)
  sum(shots[1:50])
}

set.seed(123)
map_dfr(1:10000, function(sim){
  tibble(
    `first 50 of fixed 100` = z_pvalue(rbinom(1, 50, 0.5), rbinom(1, 50, 0.4)),
    `first 50 of shoot until 50` = z_pvalue(first_50(0.5), first_50(0.4))
  )
}) |>
  summarise(across(everything(), ~mean(.x < 0.05))) |>
  knitr::kable()

first 50 of fixed 100	first 50 of shoot until 50
0.186	0.1874

↩︎

How You Should Keep Score in Pickup Basketball

Tue, 27 Aug 2024 00:00:00 +0000

TLDR: For full court pickup basketball you should play by 2s and 3s. If you’re playing make-it-take-it, you can calculate the expected value of a possession with a geometric series and it turns out 1s and 2s actually makes for more balanced scoring.

Steph Curry is sometimes blamed for ruining pickup basketball:

The argument is that people in pickup games see Steph launching 3’s, emulate him – but aren’t as talented – so games become a mess of people bricking long range shots with hardly any real basketball going on.

Another idea is that Steph is less to blame for this phenomena than the incentives embedded in the most common scoring system in pickup basketball: 1’s and 2’s. In 1s and 2s if you can hit a long distance shot at least half as often as an interior shot, it makes sense to take it because the expected value (/points) is greater than for an interior shot. You can find people on Reddit arguing these and related points.

I lean towards the incentives based perspective. This post will be a review of the “balance” of games under different scoring and possession systems. I’m defining “balanced” games as those where the expected points from interior and long range shots are similar to one another.

To make this explicit, say the average team¹ shooting percentage for interior shots is 45% and the average for long-range shots is 30% (for my examples the specific shooting percentages don’t matter so much as that the ratio between interior : long range shooting stays close to 0.45 : 0.30)². If you’re playing 2s and 3s, the expected value of an interior shot is $2 \times 0.45 = 0.9$ and the expected value of a long range shot is $3 \times 0.3 = 0.9$ – i.e. the expected value of both types of shots is equivalent (perfectly balanced). Given these shooting percentages (and ignoring offensive rebounds³) the expected value in 1s and 2s would be $1 \times 0.45 = 0.45$ for an interior shot and $2 \times 0.3 = 0.6$ from distance. In this scenario a long range shot comes with 33% more value (29% if using a symmetric measure⁴ of percent difference).

The 2s and 3s scoring system is the minority position in most pickup games⁵ but is somewhat more popular among analytics oriented players and common practice among the converted⁶. Hence the conventional sports nerd wisdom is to advocate for playing by 2s and 3s (see How to Fix Pickup Basketball…). However that analysis doesn’t take into account possession.

What if you’re playing make-it-take-it?

2s and 3s is only more balanced when you alternate possession after each score. While alternating possession is the norm in full court pickup basketball, in half court the more common play style is for the scoring team to retain possession – called “make-it-take-it.” Under these conditions⁷, calculating expected points is a little more complicated as you need to consider not just the expected points of the shot attempt but also the value associated with the chance of retaining possession (and making future shots) if you score.

Calculating the expected points for a shot is no longer just:

$\text{(points)} \times \text{(shooting %)}$

but:

$\left(\text{points}\right) \times \left(\text{shooting %}\right) \\+ \left(\text{chance last shot was made}\right) \times \left(\text{points}\right) \times \left(\text{shooting %}\right) \\+ \left(\text{chance last two shots were made}\right) \times \left(\text{points}\right) \times \left(\text{shooting %}\right) \\+ \ldots \left(\text{and so on and so on}\right) \ldots$

If we consider the expected points of an interior shot (when playing 1s and 2s) the series can be approximated as:

\[ 1 \times 0.45 + 1 \times 0.45^2 + \ldots + 1 \times 0.45^n \]

You might recognize this from calculus as a geometric series and notice that because each additional term in the series gets smaller and smaller towards 0, the total value of the sum will converge to a constant. The value of a converging infinite geometric series can be represented by the simple equation⁸:

\[ \frac{\text{(start value)}}{(1 - \text{rate})} \]

We can use this to approximate the expected points for a shot attempt when playing make-it-take-it (see But you don’t play pickup basketball to infinity for why this is just an approximation). If we plug in our values for a 1 pointer with 45% chance of being made:

\[ \frac{(0.45 \times 1)}{(1 - 0.45)} = 0.82 \]

and compare this to the rough expected points of a 2 pointer at 30%:

\[ \frac{(0.30 \times 2)}{(1 - 0.30)} = 0.86 \]

This shows a less than 5% difference in approximate expected points if taking an interior or long range shot when playing make-it-take⁹. This is pretty balanced, which is good news because 1s and 2s make-it-take-it is the norm for half court pickup basketball. Let’s compare this relative balance to the relationship between the expected values when playing make-it-take by 2s and 3s. Approximate expected points for a 2 pointer in make-it-take-it:

\[ \frac{(0.45 \times 2)}{(1 - 0.45)} = 1.64 \]

vs a 3 pointer:

\[ \frac{(0.3 \times 3)}{(1 - 0.3)} = 1.29 \]

Under these conditions, an interior shot has almost 27% greater expected value than taking a long range shot (24% symmetric percent difference). In this case 2s and 3s is almost as imbalanced for interior shots as the 33% advantage we saw for distance shooting when playing 1s and 2s with alternating possession. So if you’re playing make-it-take-it, you actually shouldn’t do scoring by 2s and 3s¹⁰.

Summary

The two game conditions that promote similar expected values for interior and long range shots are 2s and 3s with alternating possession after scores as well as make-it-take-it by 1s and 2s¹¹.

# Load the gt package
library(gt)
library(dplyr)

# Function to calculate percentage difference and format the string
format_with_difference <- function(value1, value2) {
  diff <- round(abs(value2 - value1) / ((value1 + value2) / 2) * 100, 0)
  return(paste0(value1, " : ", value2, "<br>(", diff, "% difference)"))
}

# Update the data frame with formatted strings
data <- tibble(ShotType = c("1pts : 2pts", "2pts : 3pts"),
               MakeItTakeIt = c(format_with_difference(0.82, 0.86), format_with_difference(1.64, 1.29)),
               Alternating = c(format_with_difference(0.45, 0.60), format_with_difference(0.90, 0.90)))

# Create a gt table
table <- gt(data) %>%
  # Add a title and subtitle
  tab_header(
    title = "Relative differences in expected points of shot type",
    subtitle = md("Given shooting percentages of... *interior: 45% ; long range: 30%*")
  ) %>%
  # Add spanner column label with italicized text
  tab_spanner(
    label = md("**Possession after a score**"),
    columns = c(MakeItTakeIt, Alternating)
  ) %>%
  tab_spanner(
    label = md("**Points assigned**"),
    columns = c(ShotType)
  ) %>%
  # Customize column labels
  cols_label(
    ShotType = md("*interior : long-range*"),
    MakeItTakeIt = md("*make-it-take-it*"),
    Alternating = md("*alternating*")
  ) %>%
  # Add vertical lines and other styles
  tab_style(
    style = cell_borders(
      sides = "right",
      color = "lightgray",
      weight = px(1)
    ),
    locations = list(
      cells_body(columns = MakeItTakeIt),
      cells_column_labels(columns = MakeItTakeIt)
    )
  ) %>%
  tab_style(
    style = cell_borders(
      sides = "left",
      color = "lightgray",
      weight = px(2)
    ),
    locations = cells_body(columns = c(MakeItTakeIt))
  ) %>%
  tab_style(
    style = cell_borders(
      sides = "right",
      color = "lightgray",
      weight = px(2)
    ),
    locations = cells_column_labels(columns = c(ShotType))
  ) %>%
  tab_style(
    style = cell_text(
      align = "center",
      style = "italic"
    ),
    locations = cells_body(columns = c(ShotType))
  ) %>%
  tab_style(
    style = cell_text(
      color = "darkblue"
    ),
    locations = cells_title(groups = "title")
  ) %>%
  # Render the markdown in the cells
  fmt_markdown(columns = c(MakeItTakeIt, Alternating)) %>%
  tab_footnote("Symmetric absolute percent difference used as metric for % difference.") %>%
  tab_style(
    style = cell_text(size = px(10)),
    locations = cells_footnotes()
  )

# Print the table
table

Points assigned	Possession after a score
Relative differences in expected points of shot type
Given shooting percentages of… interior: 45% ; long range: 30%
interior : long-range	make-it-take-it	alternating
1pts : 2pts	0.82 : 0.86 (5% difference)	0.45 : 0.6 (29% difference)
2pts : 3pts	1.64 : 1.29 (24% difference)	0.9 : 0.9 (0% difference)
Symmetric absolute percent difference used as metric for % difference.

Taking action

You might think that the expected values of different shots would naturally balance out. For example, if you’re playing by 1s and 2s with alternating possession (where deep shots are more valuable) players could defend long range shots more aggressively which would lead to the shooting percentage on those shots dropping relative to interior shots. In practice though, this realignment does not happen as much as would be expected by fully rational players¹². Instead, many people anchor their play style to whatever brand of basketball they are accustomed and do not fully adjust to the scoring and possession system in place. Given these tendencies, what should you do?

Advocate for a balanced scoring system¹³

This means if you’re playing full court (with alternating possessions after scores), push for playing by 2s and 3s to 21 (rather than the more common 1s and 2s to 15). If you’re playing half court, you don’t need to do anything as games are typically played make-it-take by 1s and 2s, which has mostly fine incentives¹⁴.

Arbitrage

In many cases you will not be able to influence the scoring system. Other players not fully adapting to unbalanced values of shots is an opportunity. If the system advantages long range scoring, get your best shooter as many decent looks as possible and push your team to play tough exterior defense¹⁵.

Other considerations

The conclusion that “1s and 2s make-it-take-it” and “2s and 3s alternating possession” make for the most balanced games depends on my guesses for standard shooting percentages¹⁶. 45% : 30% for interior and long range shots are reasonable estimates for competitive pickup games but may not apply universally¹⁷.

The chart below shows, across a range of possible interior and long range shooting percentages, the relative imbalance in expected points under each of the four {possession} x {scoring} conditions¹⁸. See Table of imbalances in the appendix to look-up these values at any point.

library(dplyr)
library(tidyr)
library(ggplot2)
library(purrr)

shooting_pcts <- crossing(
  tibble(pct_long = seq(0.10, 0.40, by = 0.01)),
  tibble(pct_short = seq(0.20, 0.60, by = 0.01))
)

shots_pts <- tibble(pts_long = c(2,3),
       pts_short = c(1, 2)
)

exp_value_alternating <- function(pts, percent) pts * percent

exp_value_makeitakeit <- function(pts, percent) (percent * pts) / (1 - percent)

total_pct_difference <- function(x, y) (x - y) / ((x + y) / 2)

data_imbalance <- crossing(shooting_pcts, shots_pts) %>% 
  mutate(pts_type = paste0(pts_short, "&", pts_long)) %>% 
  # expected value of each shot
  mutate(exp_short_alt = exp_value_alternating(pts_short, pct_short),
         exp_short_miti = exp_value_makeitakeit(pts_short, pct_short),
         exp_long_alt = exp_value_alternating(pts_long, pct_long),
         exp_long_miti = exp_value_makeitakeit(pts_long, pct_long)) %>% 
  # percentage imbalances
  mutate(alt = total_pct_difference(exp_short_alt, exp_long_alt),
         miti = total_pct_difference(exp_short_miti, exp_long_miti))

data_chart_raw <- data_imbalance %>% 
  pivot_longer(one_of("alt", "miti"), names_to = "possession_type", values_to = "pct_imbalance") %>% 
  mutate(possession_type = ifelse(possession_type == "alt", "alternating", "make-it-take-it")) %>% 
  mutate(type_combine = paste0(pts_type, "; ", possession_type)) %>% 
  mutate(near_0 = near(pct_imbalance, 0, 0.05)) %>% 
  mutate(type_combine = forcats::fct_relevel(type_combine, "1&2; make-it-take-it", "1&2; alternating", "2&3; make-it-take-it", "2&3; alternating"))

data_chart <- data_chart_raw %>% 
  mutate(pct_imbalance = abs(pct_imbalance))

data_near <- data_chart %>% 
  filter(near_0) %>% 
  mutate(near_0 = "+/- 0.05 imbalance")

data_reviewed <- data_chart %>% 
  filter(near(pct_long, 0.30), near(pct_short, 0.45)) %>% 
  mutate(label = "Standard shooting conditions:\n45% shooting from interior;\n30% shooting from long range")

# equations for perfect balance
alt_1_2 <- function(x) x / 2
alt_2_3 <- function(x) x / 1.5
miti_1_2 <- function(x) x / (2 - x)
miti_2_3 <- function(x) 2 * x / (3 - x)

perfect_balance <- list(alt_1_2, alt_2_3, miti_1_2, miti_2_3) %>% 
  map(~{tibble(pct_short = seq(0.20, 0.60, by = 0.01)) %>% mutate(pct_long = .x(pct_short))}) %>% 
  map2(list("1&2; alternating", "2&3; alternating", "1&2; make-it-take-it", "2&3; make-it-take-it"),
      ~{mutate(.x, type_combine = .y)}) %>% 
  bind_rows() %>% 
  mutate(type_combine = forcats::fct_relevel(type_combine, "1&2; make-it-take-it", "1&2; alternating", "2&3; make-it-take-it", "2&3; alternating"))


data_chart %>% 
  ggplot(aes(x = pct_short, y = pct_long))+
  geom_tile(aes(fill = pct_imbalance))+
  geom_tile(aes(fill = pct_imbalance, colour = near_0), alpha = 0, data = data_near)+
  geom_line(color = "red", data = perfect_balance)+
  geom_point(aes(shape = label), data = data_reviewed, colour = "violet")+
  scale_color_manual(values = c("red"))+
  facet_wrap(~type_combine)+
  theme_bw()+
  labs(
    title = "Imbalance of expected points by shooting percentages",
    # subtitle = "For 45%; 30%, full court 2s and 3s or make-it-take-it 1s and 2s are most balanced",
    # subtitle = "By scoring and possession rules",
    fill = "Proportion imbalance",
    x = "Interior shooting %", 
    y = "Long range shooting %",
    caption = "Symmetric absolute difference used as metric for proportion imbalance.")+
  theme(legend.key = element_rect(colour="white"),
        legend.position = "right")+
  guides(color = guide_legend(title = NULL, order = 2),
         fill = guide_legend(order = 1),
         shape = guide_legend(title = NULL))+
  scale_y_continuous(limits = c(NA, 0.4), labels = scales::percent)+
  scale_x_continuous(labels = scales::percent)+
  coord_fixed()

The greater range in color hue for make-it-take-it settings demonstrates how, as you deviate from balanced shooting percentage conditions, expected value imbalance grows more quickly than when playing with alternating possession. Hence make-it-take-it comes with greater downside risks of incentives getting out of whack¹⁹.

The red indicates the shooting percentages for balanced incentives. Everything above the red “perfect balance” lines (or curves in the cases of make-it-take) can be interpreted as shooting percentages where long range shooting has an expected points advantage. Everything below them represents shooting percentages where interior shooting has an advantage²⁰. For example, in games where interior shooting percentages are still ~45% but long range shooting is a bit worse (e.g. percentages in the low 20s) 1s and 2s with alternating possession after a score can actually be more balanced than 2s and 3s. However you would be setting-up conditions for a lot of missed shots.

Sometimes it’s better to just play by all 1s.

Appendix

But you don’t play pickup basketball to infinity

The solutions in What if you’re playing make-it-take-it? are with infinite series but in pickup basketball you typically play to some fixed value (e.g. 7, 11, 15, 16, 21, 25). Evaluating Finite geometric series would allow you to condition your expected value calculations on how many points (or actually scoring possessions) remain. However, the values for the finite geometric series would only substantively differ from our infinite series calculations as you get near to the end of the game. By this point you’d be better thinking in terms of event outcomes rather than expected points. After all you don’t really care about expected points but chance of winning²¹. Event outcomes are a problem of probability rather than expected value and require a different type of analysis. In the future I may do a “Simulating pickup basketball outcomes” post²². However for my purposes in this post (identifying the rules that promote the most balanced games), infinite series work fine as an approximation for expected points. The calculations are also easy to describe as each statement of value does not need to be predicated on how many points remain in the game. From the Summary section on I stop saying “approximate expected points” and just say²³ “expected points.”

Table of imbalances

Positive values in the diff%_* columns represent imbalances in favor of interior shots whereas negative values indicate imbalances in favor of long range shots.

library(DT)

data_table <- data_chart_raw %>%
  select(type_combine, pct_long, pct_short, pct_imbalance) %>% 
  pivot_wider(names_from = type_combine, values_from = pct_imbalance) %>% 
  # mutate(across(everything(), scales::percent, accuracy = 1)) %>% 
  rename_with(~stringr::str_replace(., "pct", "fg%"), starts_with("pct_")) %>% 
  rename_with(~paste0("diff%_", .), contains(" ")) %>% 
  dplyr::relocate(contains("short")) %>% 
  mutate(across(everything(), ~round(.x, 2)))

dt_table <- datatable(data_table,
          options = list(dom = 'ltipr'),
          filter = "top",
          rownames = FALSE) %>%
  DT::formatPercentage(names(data_table), digits = 0)

htmltools::div(dt_table, id = "dt-table", style = 'overflow-x: auto')

Finite geometric series

Much of this section has limited relevance. I left it in mostly just as a check showing that the finite geometric series converge pretty quickly so that as long as you’re a few scores away from the end of the game, the infinite geometric series calculations that I use in the body of the post should work fine as a proxy for evaluating relative imbalances in expected values when playing make-it-take-it.

Calculating the finite geometric series allows us to consider how many points (/scoring rounds) are left in the game. Equation for calculating the value of a finite geometric series:

\[ \frac{\text{(start value)} \cdot (1 - \text{rate}^n)}{(1 - \text{rate})} \]

In our case, n is the number of remaining scoring possessions. However the number of remaining scoring possessions will depend on whether you’re shooting interior or long-range shots. Long range shots are worth more points so there would be fewer scoring possessions until the end of the game. For example, if you’re playing by 1s and 2s there will be ~half as many scoring possessions remaining if you’re sinking long-range compared to interior shots. Taking this into account, let’s look at the expected value of each shot (still using our 45% and 30% shooting percentages).

The usefulness of looking at these charts is limited for the reasons described in But you don’t play pickup basketball to infinity (namely that you should think in terms of event outcomes, particularly near the end of the game). I’ve also included some point values in the charts that are nonsensical for their scoring system. The purpose of the charts are mainly just to show the pretty quick rate at which the expected values converge – suggesting that, for most of the game, infinite series are a decent proxy to use when considering relative expected value by shot selection if playing make-it-take-it.

library(dplyr)
library(ggplot2)

finite_geometric_series <- function(start, rate, n){
  (start * (1 - rate^n)) / (1 - rate)
}

data_1s2s_finite <- bind_rows(
  tibble(shot_pts = 1,
         shot_pct = 0.45,
         points_remain = seq(2, 14, by = 1)) %>% 
    mutate(n_rounds = points_remain),
  tibble(shot_pts = 2,
         shot_pct = 0.30,
         points_remain = seq(2, 14, by = 1)) %>% 
    mutate(n_rounds = points_remain / 2)  
) %>% 
  mutate(exp_pts = finite_geometric_series(shot_pts*shot_pct, shot_pct, n_rounds)) %>% 
  mutate(shot_pts_title = ifelse(shot_pts == 1, "interior (1pts; 45%)", "long-range (2pts; 30%)")) 

data_1s2s_finite %>% 
  ggplot(aes(x = points_remain, y = exp_pts, colour = shot_pts_title))+
  geom_line()+
  scale_x_continuous(breaks = seq(2, 20, by = 2)) +  
  theme_bw()+
  ylim(0, NA) +
  labs(
    title = "Expected Value of Shots Based on Points Left in the Game",
    subtitle = "1s and 2s, make-it-take-it",
    x = "Points Left in the Game",
    y = "Expected Value",
    color = "Shot Type"
  ) +
  guides(color = guide_legend(title = NULL))

Finite sums for make-it-take-it 2s and 3s:

data_2s3s_finite <- bind_rows(
  tibble(shot_pts = 2,
         shot_pct = 0.45,
         points_remain = seq(3, 24, by = 1)) %>% 
    mutate(n_rounds = points_remain / 2),
  tibble(shot_pts = 3,
         shot_pct = 0.30,
         points_remain = seq(3, 24, by = 1)) %>% 
    mutate(n_rounds = points_remain / 3)  
) %>% 
  mutate(exp_pts = finite_geometric_series(shot_pts * shot_pct, shot_pct, n_rounds)) %>% 
  mutate(shot_pts_title = ifelse(shot_pts == 2, "interior (2pts; 45%)", "long-range (3pts; 30%)"))

data_2s3s_finite %>% 
  ggplot(aes(x = points_remain, y = exp_pts, colour = shot_pts_title))+
  geom_line()+
  # scale_x_continuous(breaks = seq(2, 20, by = 2)) +  
  theme_bw()+
  ylim(0, NA) +
  scale_x_continuous(breaks = seq(3, 24, by = 3)) +  
  labs(
    title = "Expected Value of Shots Based on Points Left in the Game",
    subtitle = "2s and 3s, make-it-take-it",
    x = "Points Left in the Game",
    y = "Expected Value",
    color = "Shot Type"
  ) +
  guides(color = guide_legend(title = NULL))

Scoring system in pickup basketball

In pickup basketball, two common scoring systems are used: “1s and 2s” and “2s and 3s.”

1s and 2s: In this system, shots made inside the three-point line are worth 1 point, and shots made beyond the three-point line are worth 2 points. This system puts incentives on long-range shooting because a successful three-point shot is worth twice as much as a shot made inside the arc.
2s and 3s: Here, shots made inside the three-point line are worth 2 points, and shots made beyond the three-point line are worth 3 points, mirroring the scoring system used in professional basketball.

Pickup basketball games also differ in how possession is handled after a score:

Make-it-Take-it: In this rule, the team that scores retains possession of the ball. This is common in half-court games.
Alternating Possession: In this rule, possession alternates between teams after each score. This is almost always the way possession is handled in full-court games.

It’s not that every player has to have a 30% long range shooting percentage and 45% interior shooting percentage. It could be that some of the players on the team are far worse at shooting 3s, but they just won’t take those shots.↩︎
This isn’t completely true when considering make-it-take-it but as long as things don’t depart too much from the shooting percentages I’m using, most of the takeaways stay the same.↩︎
I’m also ignoring free throws because playing with these are exceptionally rare in pickup basketball.↩︎
Article on symmetric measure for percentage difference.↩︎
Article surveying the most common rules of pickup.↩︎
and finds sympathy among those without a jump shot↩︎
and continuing to ignore turnovers, offensive rebounds, and other complicating factors↩︎
Proof on Khan Academy ↩︎
Maybe a slight edge for long range shots but pretty close (i.e. balanced).↩︎
It’s good then that 2s and 3s make-it-take-it is somewhat rare.↩︎
The other play formats lead to imbalances that favor either long range or interior shooting.↩︎
Also some adjustments may be difficult to make regardless of intention.↩︎
Being the ethical and upstanding individual you are, you should advocate for the most balanced system available. A more cynical person might say that you should try and skew the scoring system to whichever gives your team the greatest advantage… however I think it’s more fun to just try and keep things balanced.↩︎
You could push for 2s and 3s with alternating possession though which is a little more balanced and has the advantage of being more robust to variations in shooting percentage which I touch-on in a footnote in Other considerations.↩︎
Even under reasonably balanced rules you can still find opportunities where people don’t play in-line with expected value. One example, is in 1s and 2s make-it-take-it some players will be overly concerned about long range shots (and under appreciate the value of retaining possession) and defend in a way that opens themselves up to drives and high percentage interior shots. This is particularly the case if playing with less than 10 players (e.g. 3 vs 3 or 4 vs 4) where interior play is already more open and the shooting percentage on interior shots likely goes up by more than it does for long range shots.↩︎
There’s also the possibility that much of this post is rationalizing my own frustrations at growing up without a decent jump shot and having to always play 1s and 2s.↩︎
In the case of alternating possession, proportion imbalanced is linearly related to the ratio of {interior shooting %} : {long range shooting %}, e.g. in the case of 2s and 3s alternating possession, what matters is not the specific shooting percentages but how far they deviate from the ratio of 0.45 : 0.30. However when playing make-it-take-it the relationship is not linear (though within a small window of shooting percentages a linear relationship works OK as an approximation).↩︎
The “perfect balance” equations were found by setting the interior and long range expected points equations equal to each other and simplifying.↩︎
This is why the points on the graph for “standard shooting percentages” in the upper right and lower left charts are noticeably different distances from the “perfectly balanced” shooting percentage for that condition but similar in terms of proportion imbalanced.↩︎
You can see where the standard shooting percentages we investigate fall in these conditions. “2&3; alternating” is directly on the “perfect balance” line, “1&2; make-it-take-it” is close to it’s perfect balance curve. Whereas “1&2; alternating” is above it’s line, indicating an advantage for long range shooting and “2&3; make-it-take-it” is below it’s line indicating an advantage for interior shooting.↩︎
The expected points calculations for finite geometric series get weird near the end of the game because you can’t score more points after the game ends… but this doesn’t matter, you just care about getting to the end of the game.↩︎
E.g. 1s and 2s make-it-take to 11, you’re down 9 - 7 with the ball… what are your chances of winning if you…↩︎
This allows me to put both “alternating” and “make-it-take-it” calculations on the same figures without having to clarify that the former is expected points and the latter approximate expected points.↩︎

Untangling Nassim Taleb's Criticism of that Headline Grabbing Intermittent Fasting Study

Wed, 08 May 2024 00:00:00 +0000

TLDR: Parts of Taleb’s critique of an, albeit overhyped study on Intermittent Fasting, seem specious.

Veritasium recently put out a video on “The Problem With Science Communication” that details how news outlets and science journalism gravitate to big headlines that are often misleading, overstate the findings, or are outliers that go against the weight of evidence in the field. The video doesn’t just fault news outlets but also suggests labs and research institutions are incentivized to put out punchier sounding press releases that summarize their findings in ways that are more likely to draw media attention to their work.

In March a headline grabbing summary of a study came-out regarding an association between Intermittent Fasting and cardiovascular mortality. The American Heart Association’s release is largely representative of the media coverage¹:

These and other reports were based on a poster and abstract summarizing the study². The actual paper had not yet been released. The specific part of the poster that drew so much attention was this table of results that showed a greater risk of cardiovascular death in those practicing intermittent fasting (eating within a less than 8 hour window) compared to the reference group:

Nassim Taleb’s comments

Around this time I had just started reading Taleb’s book Antifragile. Early in the book he has an aside where he advocates for the health benefits of Intermittent Fasting³. Checking-out Taleb’s posts on X, I saw that he had a number of posts commenting on the recent study. He initially expressed surprise at the results⁴:

https://x.com/nntaleb/status/1770137480194396326?s=20

He goes on to make a few criticisms of the study and later states that “small probabilities REQUIRE increasingly larger samples”:

For the remainder of this post I will discuss his analysis regarding sample size in relation to the event rates observed in this study⁵.

There is indeed a need for a larger sample when there is a low event rate (414 participants in the intermittent fasting group is smaller than ideal)⁶. However the sample size in this case is not necessarily a justification for throwing-out the small p-value on a post-hoc basis. The p-value represents (taking into account the sample size and the underlying variability / event rate) how likely it is you would get the observed results by random⁷. The need for a large sample size for rare events often has to do with increasing the power of your study to accommodate for the difficulty of detecting rare events⁸ compared to the problem (that Taleb is alleging) of getting a false positive finding where an effect actually does not exist⁹ ¹⁰.

Still, again setting aside other concerns regarding design or the weaknesses inherent in this being an observational study, a researcher might question the validity of the p-value of 0.006 due to multiple tests being done within the study¹¹. Additionally, one might critique the use of frequentist statistics and instead suggest using Bayesian or nonparametric methods¹² which can be less susceptible to the distortions that can result when there are small subsamples¹³. (As a reminder, we’re just looking at a poster, the paper is not available, so we can’t review the details of the approach the researchers used.) However if you’ve read anything by Taleb you’ll know he rejects much of mainstream statistical inference¹⁴ and so his critiques tend to be more sweeping than just nitpicking the choice of statistical testing methodology.

“sigma change that doubles incidence”

While I found Taleb’s commentary overly harsh in the context of what we knew about the study, calling for a larger sample when studying rare events is generally a good idea; so my critique of his criticism at first felt pretty banal. However, after reading Taleb’s follow-up tweets explaining his position, I got the feeling his approach to thinking about small sample sizes in the context of this study goes a bit off-the-rails. Take a moment to read his tweets below for yourself and see if you can follow his reasoning and, in particular, how his calculations support it:

https://x.com/nntaleb/status/1770467069454073997

He packs a lot in here¹⁵ and after rereading it several times I think a lot of it falls somewhere from incomplete to uninformative to misleading. I will parse-out what he is saying¹⁶ and offer some commentary:

“Only 2% (tail prob) doing 8h”: There are 20,078 total individuals in the study, 414 of which were in the intermittent fasting group (eating within an 8 hour window), which represents about 2% of total participants ($\frac{414}{20,078}$). This 2% subsample is not particularly surprising as it just shows that most people in the survey the study is based on didn’t practice intermittent fasting¹⁷. Calling this proportion of total participants a “tail” seems strange as it’s not clear why you would characterize this group as existing in the “tail” compared to anywhere else along the population distribution¹⁸.

“(w/uncertainty on classification, from 2d recall)”: To be included in the study, surveyed participants had to have two days where they were able to recall their dietary intake, including the timing of meals. The average length of the eating windows in these recalls was used to group participants into “eating duration” groups (of either “<8h,” “8<10h,” “10<12h,” “12-16h,” or “>16 h”)¹⁹. Taleb is suggesting that only requiring 2 recall days (/surveys) to sort people into eating duration groups brings the risk of a high misclassification rate. While this is a fair concern, it’s not obvious why some people being put into the wrong “eating duration” group would increase the chance of getting a false positive result in the broader study²⁰. Uncertainty in subgroup classification often adds noise and reduces the power to find a positive result, yet in this case Taleb is accusing the study of falsely detecting an effect that doesn’t actually exist.
Screenshot of methods section from poster (that shows approach regarding dietary recall days):

“Confounders: >> smokers in IF & BMI.”: He is pointing to the “Baseline characteristics of study participants” in the poster where you can see that the intermittent fasting group tends to have a higher rate of smoking and a higher Body Mass Index (BMI) than other groups (I’m surprised he doesn’t point out the higher “Black, %” as well):

Confounding is difficult to speak to without the data or more information, however I’ll just note that if you look back at the methods section you’ll see that the model is supposedly accounting for these variables (as well as a bunch of others) when calculating the relative hazard.

That there are so many variables being controlled for could further reduce the power of the study which might actually make it more surprising to see a statistically significant result unless there is a strong underlying effect. Also, if there is multicolinearity between the variables (which he alludes to), this would tend to inflate the standard errors and cause instability in your parameter estimates. This can lead to weird effects, one common byproduct though is also a reduction in the ability to reach statistically significant results. However, as I mentioned previously, the effects of confounding²¹ can get complicated so I don’t want to focus too much on this as my interest is primarily in the calculations Taleb uses to support his skeptical stance concerning event rates.
“Of these, only 31 had an event .0015 vs .0008!”: 0.0015 comes from $\frac{31}{20,078}$ i.e. out of the ~20,000 people surveyed, ~0.15% of those individuals were intermittent fasting and also died of cardiovascular failure. The 0.0008 is where things get confusing, it comes from $\frac{\frac{31}{1.91}}{20,078}$ . I interpret this as essentially saying that if you took the intermittent fasting group’s subsample size and applied the reference level’s lower hazard rate ($\frac{31}{1.91}$) you would only have expected to get ~16 deaths, those 16 expected deaths then would only have represented ~0.08% of total data in the study ($\frac{16}{20,078}$). Restating this: if it’s actually the case that the intermittent fasting group doesn’t have a higher mortality rate, then you would have expected 16 deaths in that subsample which would represent ~0.08% of total data in the study²².

A problem here is he’s taking a series of somewhat irrelevant conjunctive steps to build up these tiny proportions. To a large extent, these proportions are small just as a function of intermittent fasting being comparatively uncommon in the national survey used by this study²³.
Next he displays some Mathematica code to show that “A tiny .2 sigma change doubles the incidence!”:

This code is taking the proportions mentioned previously and then putting them in terms of percentiles (tails) of data on a normal distribution²⁴. The 0.15 percentile would be at -2.96 standard deviations from the center and the 0.08 percentile would be at -3.15 standard deviations from the center – i.e. everything to the left of -2.96 represents 0.15% of the data and everything to the left of -3.15 contains 0.08%.
Visual representation:

He then states that a relatively small move to the right of just ~0.2 standard deviations (3.15 - 2.96) leads to a doubling of incidence²⁵:

“In other words, a tiny probability {such as} .0008 blows up with tiny changes in assumptions and requires a much, much larger sample size. A tiny .2 sigma change doubles the incidence! It is fragile to parameter change. I am not even discussing the flaws in the setup.”²⁶

However, this “small sigma change leads to doubling of the incidence” has little to do with the rare event rate or small sample size. Instead it’s mostly the arbitrary result of an inapplicable methodology that, even under conditions where an effect/association is obvious, could speciously support skeptical statements.

For example, let’s change the data to remove the “rare event rate” problem²⁷ and increase the death rates by 5x so the rate in the intermittent fasting group is now ~37% (and keep the same hazard ratio with the reference level). The “change in sigmas that ~doubles incidence” measure only goes up to ~0.23²⁸:

# This is R code for the same types of calculations Taleb had previously done in Mathematica
c(qnorm( (31 * 5) / 20000), qnorm( (31 * 5 / 1.91) / 20000)) |> round(2)

## [1] -2.42 -2.65

Now let’s also remove his “the intermittent fasting subsample represents just 2% of the participants in the total study” problem by also multiplying this by 5x (so this hypothetical intermittent fasting group would now represent ~10% of survey participants and would have a death rate of ~37%, again keeping the same relative hazard with the reference level). Our “difference in sigmas that ~doubles incidence” measure still only goes up to 0.28:

c(qnorm( (31 * 5 * 5) / 20000), qnorm( (31 * 5 * 5 / 1.91) / 20000)) |> round(2)

## [1] -1.77 -2.05

Even if you made it so half of the 20,000 participants were in the intermittent fasting group (and kept the 5x death rate – so 3,700 deaths in the intermittent fasting group) your “difference in sigmas that ~doubles incidence” would only go up to 0.4. Hence even with data that shows an obvious effect, Taleb’s calculations could superficially make it seem like the results are “fragile to parameter change.”

His approach also somewhat begs the question because what “difference in standard deviations from reference level that ~doubles incidence” would Taleb accept as being substantive²⁹…? but mostly this just isn’t a way to test things.

Wrapping-up

I am not trying to defend this study. I can appreciate the instinct to put down a (not even released) paper that has garnered an undue amount of attention and that may go against other results in the field³⁰. Observational as well as survey-based studies have notorious weaknesses. It’s also interesting that the study doesn’t find a significant difference in the “all cause mortality” rate for intermittent fasting³¹. Furthermore, I don’t think raising question marks related to sample size or event rate is a bad thing³². There are plenty of fair reasons to be skeptical³³. Where I think Taleb goes wrong is that rather than asking questions, or offering areas where he’d like to see further investigation, he rejects the results out-of-hand and then supports his position with what appears to be complicated nonsense³⁴.

There were also plenty of headlines and articles that spoke broadly to the field of study rather than focusing disproportionately just on the initial findings of this unreleased study.↩︎
Links of which can be found at the AMA release ↩︎
As part of a section where he describes the benefits of exposing the body to natural stressors.↩︎
He also noted that there didn’t seem to be any relevant financial conflicts of interest.↩︎
Which he suggests is all that is needed to dismiss the results, even if you ignore potential issues in setup: https://x.com/nntaleb/status/1770467069454073997 ↩︎
The concern about sample size often comes up during study design. (Note also that this study is using a database from a national survey, this isn’t an experiment or something where they are able to control sample size directly.) If the thing you are measuring is rare, you will need a much larger sample in order to detect it. This increases your “statistical power”: the chance you will reject the null hypothesis (i.e. get a small enough p-value) if indeed you should reject it. Given the underlying event rates and an alpha of 0.05 – a common default – a sample size of 414 would indeed likely be too small, but for the reason that this wouldn’t give you enough statistical power (a common default for statistical power is 0.80 and this sample size, given the other parameters, would put you short of that). (…at most points in this post I write a bit colloquially / loosely so as to avoid using terms like null hypothesis…)↩︎
p-value of 0.006 suggests these results are unlikely to have occurred by chance alone.↩︎
related to type 2 errors: ~failure to detect an effect when one truly exists – a false negative↩︎
Which relates to a type 1 error↩︎
The points in this paragraph are also articulated in my reply on his thread: ↩︎
and call for a corresponding adjustment of the p-value that takes this into account like a Bonferroni correction, an approach Taleb does mention in Antifragile.↩︎
It’s tough to know without actually trying it, my guess is p of 0.006 would have a good chance surviving adjustments in statistical approach though.↩︎
Are Orange Cars Really not Lemons walks through a change in methodology responding to such a distortion – in this case related to how small subsamples can disproportionately show extreme results.↩︎
Calling himself a “probablist.”↩︎
A lot of people on X will say something and leave the explanation and context unsaid, which is fine but can also lead to misunderstandings. Taleb seems particularly at risk of this as he has a lot of technical posts but most of the replies and quoted reposts seem to be primarily Taleb fans hyping him up rather than people meaningfully parsing his arguments. There’s also a risk of context collapse, that I wonder a bit if maybe I’ve fallen into here, where his posts are aimed at those who are highly versed in the methods / jargon he uses in his books and writing.↩︎
I’m mostly just talking about “1” as I’m mainly concerned with discussing his calculations concerning sample size and event rate in relation to his “sigma change that doubles incidence” metric that he cooks-up.↩︎
Taleb focuses a lot (here and in other tweets on this topic) on the relative size of the intermittent fasting group (2% of total study) over the raw number of samples in the group (414 individuals). The study is building a global model across participants and with many variables. Hence, there are circumstances where the relative proportions of different groups could theoretically influence the rates of different types of statistical errors (as opposed to just the raw number of samples) and contribute to some instability of estimates (/fragility). However it’s at least not obvious what the mechanisms would be for this leading to more false positive results. (I thought about setting-up a simulation to look at Type 1 error rates of coefficients in different situationsif you vary the group proportions to get a better look at this and under what conditions this “relative proportion size of each group in relation to total” may actually effect things but didn’t get to it… maybe in a future post…). I don’t find the mechanisms Taleb suggests, e.g. regarding misclassification of eating duration group, super compelling. However here’s a simple example where misclassification and relative subsample size would matter: imagine 10% of total participants just filled-out the information related to eating duration without paying attention (so these individuals will be roughly randomly distributed between groups) AND let’s say these individuals also have a much higher incidence of heart failure (for whatever reason). In this situation the small subsample (e.g. intermittent fasting) will have a higher rate of heart mortality simply as a function of it being a smaller subsample (because the high heart death misclassified people would make-up a large proportion of their subsample) compared to other groups. Given there are no solid controls on the eating duration grouping, observational survey based studies like this are open to various mechanisms such as this that can produce false results. Really though this is more about poor controls in the study design than it is about relative size of subsamples (as the same issue would be there even if the authors had excluded all data except for the Intermittent Fasting group and say 400 observations from the reference level).↩︎
My guess was that this “tails” terminology is in order to set-up Taleb’s ensuing calculations which look at the tails of normal distributions. However this construction feels strained and the approach inappropriate. “Tails” could also just be a linguistic way of emphasizing that 2% is a small proportion of the total data… or leaning into Taleb’s tendency to frame things in terms of “tails” (though this framing seems inappropriate here). It could be that I’m not following something.↩︎
For example if a participant ate everything within 5 hours on one of their recall days and then within 9 hours on another, the average of these would be 7 hours so they’d be grouped into the “<8 h” group (“<8 h” corresponds with Intermittent Fasting).↩︎
To increase the risk of detecting a false effect there’d generally have to be some kind of association between misclassification and the outcome, which is possible, but is a purely skeptical assertion. There is also the classic p-hacking problem where you just have a bunch of random results and only the “statistically significant” ones get picked-up for publication. This is a problem researchers face. However my interest with this post is in reviewing the explanations Taleb supports his argument with.↩︎
statnews article that suggests some ideas for how unmeasured confounding variables might associate with cardiovascular mortality.↩︎
Speaking loosely, this part comes off as having an almost false resemblance to what a p-value gets at, which is sort of: “how unlikely is it that you would have seen 31 deaths when you would have expected to see just 16 based on the reference level and underlying data/attributes.”↩︎
A reason the subsample’s size relative to the total study might seem relevant is that if there is improper classification of people into their eating duration groups, as he suggests (mentioned previously above), then the relative rates could matter as subsamples bleed into one another. However as I note in the prior bullet, that could reduce the chance of observing an effect as it may be washed out by noise, rather than raising the risk of a Type 1 error (as he suggests). While there are also mechanisms where a type 1 error may become more likely… without further details, it’s overly speculative to conclude how it would affect the results.↩︎
This seems like a strange thing to do for a number of reasons… but as mentioned previously, it’s not clear why, when grouping the subsamples into a population distribution, the intermittent fasting group should go at the tails rather than anywhere else in the distribution (where the density may differ).↩︎
but actually a 1.91x increase of incidence as that’s the hazard ratio which he divided things by to manufacture this example.↩︎
He leans into this point, asserting that he can dismiss the results using just this methodology without even considering other aspects the study. This confidence in a methodology that seems inappropriate is what made me decide to convert my thoughts into this blog post.↩︎
This study is looking at death rates of ~3.5 to 7.5% not the “black swan” events that Taleb writes a lot about but still somewhat rare.↩︎
The incidence rate is still roughly doubled (1.91x) which makes sense as the relative incidence rate here is just a product of dividing by the hazard ratio.↩︎
A common cut-point in traditional statistical analyses is to say if something is ~2 standard deviations from the reference level it is “statistically significant” (as this achieves a p value of <0.05 so is unlikely to be the result of chance). However it’s not clear what Taleb is looking for and his approach here isn’t a statistical test.↩︎
I don’t know anything about Intermittent Fasting or the relevant research so can’t speak to that↩︎
This may have to do with the small sample size of the group and the study being underpowered.↩︎
I’m somewhat hesitant to make a purely skeptical “well if you just move some of the observations around” hypothetical but it is indeed the case that if you move maybe somewhere between 7 and 14 of those cardiovascular deaths out of the “Intermittent Fasting” group and into another group you would likely no longer see a statistically significant result. (I don’t know exactly how many observations you’d need to move as I don’t have access to the model or the underlying data – that has things like censoring information and the attributes associated with each observation – which would be needed to do a sensitivity analysis and provide a precise review of how resilient that p value of 0.006 actually is.) While I’m skeptical that the “poor classification into eating duration groups” mechanism Taleb suggests is that likely to add to the risk of Type 1 errors… it is indeed the case that studies of rare events tend to be more sensitive and reminding people of that is often helpful.↩︎
Matthew Herper’s article and others describe many of the same potential problems in the study as Taleb. However Herper doesn’t support his points with inappropriate analytical methods.↩︎
Taleb has a story in Antifragile where he derides his business school professors by saying how he would essentially trick them into giving him better grades by adding a bunch of obfuscative pseudo-analytical jargon. I couldn’t shake the thought that maybe there is a little bit of that here. However, giving him the benefit of the doubt, I think it’s more likely that Taleb, being a prolific and highly opinionated poster, just didn’t think things through that much on this one. I also may be being a bit too harsh in this post… then another part of me wonders if I’m missing something and his “difference in sigmas that doubles incidence” methodology does actually support his point in some meaningful way that I don’t see.↩︎

Aggregating Measures of Uncertainty

Sun, 11 Feb 2024 00:00:00 +0000

There are many situations where you want to aggregate values, however if those values are on different scales or are related to measures of uncertainty, it’s typically more complicated than simply taking a simple mean or sum. You can’t take the aveage of p-values or standard deviations or statistical tests. You also can’t take the sum of confidence (or prediction) intervals to get to an interval for the aggregation of the parts. Similar to challenges with aggregating or combining items that are on different scales, you also can’t just simply aggregate items that in some way relate to measures of uncertainty¹.

As a practical example, say your firm sells a variety of different products. Each product has a different price and contracts are negotiated on a deal-by-deal basis so the final price has some level of variability that may differ across products. Sales teams are responsible for securing the final price for any sale but each team has a different product mix. You want a measure for evaluating sales teams to identify which are commanding higher or lower prices relative to their peers. Reliably characterizing a team’s performance as ‘good’ or ‘bad’ requires both an understanding of the value and variability of their specific product mix. You can’t evaluate their sales numbers in isolation (e.g. if their product mix is composed of items with high variability in price, what at first glance looks like good or bad performance may have more to do with luck / a greater level of variability in outcomes specific to their products). What you want is to be able to condense all of that information into some measure of how the sales team did based on their particular product mix that takes into account both the expected sales and also the amount of variability in their particular portfolio.

Another common example may be in forecasting. Perhaps you are producing forecasts for products at a county, state, and national level. In addition to point estimates though you are producing ranges (i.e. prediction intervals) for these forecasts. Preferably you want your prediction intervals at each level to be in some sensible way consistent with one another. However, aggregating lower and upper bounds from the lowest level forecasts up to higher levels requires more than just taking a sum².

In this post I’ll describe in broad terms a few of the general types of approaches an analyst may take when faced with the problem of aggregating measures that in some way rely on or reflect a measure of uncertainty or are on different scales (in future posts I may delve into the details of each approach in more detail). (These “type” distinctions are overlapping and more reflect distinctions I found convenient for articulation rather than concrete separations.)

Analytic approach

If each of the parts you want to aggregate follow a well-defined parametric distribution (or close), you may be able to aggregate the measures of uncertainty analytically. Say you have average sale prices across four separate products. If the sale price of each product follows a normal distribution, there are well established methods for figuring out what the variance is for the average sale price across products and you can use these measures to determine an appropriate bounded range for forecasts of aggregated sales. In the context of prediction intervals, if using a statistical forecasting approach within the fable package (e.g. ARIMA), a distribution object is saved. In hierarchical forecasting tasks, these distribution objects can be combined analytically when producing prediction intervals on aggregated forecasts at higher levels of the hierarchy.

Transformation to a common scale

Similar to what I called the “analytic approach” is the method of transforming each item onto a common scale at which point an aggregation can be done appropriately. This is commonly done to compare significance across of different variables. For example when applying simple filtering techniques on a prediction problem where you are investigating the relatedness of different variables with some target, you may be reviewing variables of different types (e.g. some categorical, others continuous). To evaluate each type of variable requires a different kind of statistical test. However, these tests each result in z-scores or p-values that provide a common scale for providing a general notion of relatedness with the target of interest. As mentioned previously you can’t simply take the average of p-values, however under some conditions there are other ways of combining p-values to get a common metric. For example by using [Fisher’s combined probably test]. A few years ago I wrote a toy package piececor for investigating piecewise correlations in a tidyverse friendly way that used Fisher’s method (or the related Stouferrer’s Z-score method) to get an overall p-value from a collection of tests for correlation³.

Simulate it

Often your data does not follow a parametric distribution or, even if it does, the math required in generating the joint distribution is incredibly complicated. In these cases you might take repeated samples of the underlying data and generate measures of uncertainty using simulation. The approach you take is defined by your problem and you need to ensure that the procedure of your simulation mirrors the type of uncertainty measure you are trying to estimate. Once you have your procedure well defined these methods often do a very good job of providing reliable estimates of uncertainty

Re-do the measure at each level

You may not care that the uncertainty measures of the components are consistent with the uncertainty measures of the aggregated whole. In these cases, you can simply estimate the values separately. In 2020, the M5 forecasting competition required participants to provide forecasts for each level of Walmart’s sales (across geographic levels and also across products). In addition to hosting a competition on the accuracy of forecasts, Kaggle also featured a competition on the quality of prediction intervals. Participants were evaluated based on the quality of prediction intervals across levels. If you look into the notebooks of some of the top performing participants, many of the participants did not worry about ensuring that the prediction intervals provided at each level of Walmart’s hierarchy were consistent with one another. They simply used various methods for identifying the typical quantiles at each level and used this investigation to create ranges independently. This type of approach may be combined with simulation based approaches where you simulate forecasts at each level and then use the distribution of the simulated errors at each level to provide a measure of uncertainty at each aggregation level.

Summary

Aggregating measures that encompass or reflect uncertainty requires consideration of the underlying distributions, and the context of the data. In future posts I may provide additional detail of common examples and approaches from each “type” of approach outlined here.

Doing so typically results in an over estimation of the level of uncertainty. E.g. summing the bounds of prediction intervals will result in too wide of a band.↩︎
or going from higher level bounds to lower levels is more than just dividing the minimum bound based on the proportion at the lower levels.↩︎
Again, these approaches are similar to the “analytic” based approaches in that they typically come with various distributional assumptions.↩︎

Odds Are You're Using Probabilities to Describe Event Outcomes

Fri, 03 Nov 2023 00:00:00 +0000

We grow up learning proportions, percentages, risks, probabilities. You encounter them when a teacher gives a grade on a test or a doctor describes the risk of an illness. On the other hand, we rarely interact with odds and when we do it’s often in contexts like:

So that in our modern cultural consciousness ‘odds’ are relegated to the shady domains of gambling, sports betting¹, and dystopian murder games.

However human understanding of odds predates our formal understanding of probability. You can find references to odds dating back to Shakespeare:

Knew that we ventured on such dangerous seas
That if we wrought out life ’twas ten to one;
- Shakespeare’s Henry IV, Part II, 1597

Yet, in most common settings, modern society has largely supplanted odds for probabilities. You can imagine if Shakespeare were writing today the line might end “’twas ten out of eleven².”

Beyond betting and dramatic playwriting, ‘odds’ remain a useful method for conceptualizing event distributions. My hope is you leave this post with some intuitions on when communicating with odds may be useful and an understanding of how the ‘odds way’ of framing a problem can be convenient for comparing event outcomes across circumstances³.

Note that if you are completely new to odds, you should first start here: Ratios and fractions of events or find a short primer online.

When odds are helpful

You make a glass of chocolate milk for your nephew and add 1 part chocolate for every 9 parts milk. He takes a sip and says, “Please make it three times as chocolatey.” The most obvious way to satisfy his request would be to triple the amount of chocolate in the cup⁴, i.e. the ratio of chocolate to milk should go from 1 : 9 to 3 : 9. To go from one part chocolate to three parts chocolate you simply add two more parts chocolate, bringing the beverage to the correct 3 to 9 chocolate to milk ratio (simplified: 1 to 3). If instead you took his comment as him wanting the cup to be composed of $\frac{3}{10}$ chocolate rather than the current $\frac{1}{10}$ chocolate, your calculations (without starting over on the drink) would be more complicated (in the original units it would come out to 2 and $\frac{6}{7}$ parts of chocolate needs to be added).

In a similar way that it can be easier to think of parts of recipes in terms of ratios rather than fractions out of a whole, it can sometimes be easier to think of event distributions in terms of odds rather than probabilities.

Odds are often helpful for thinking about discrete outcomes when:

the outcome of interest is essentially arbitrary or symmetric (e.g. you could frame the problem in terms of parts milk or parts chocolate)
you want to compare your ratio of outcomes across different contexts (e.g. how does the distribution of milk to chocolate in my cup relate to the distribution in some other cup?) or solve for relationships in a way that uses multiplication

Ratio of odds (odds ratio)

Let’s change examples to sports and substitute parts chocolate to milk for parts winning to losing when playing on one’s home court. Say in professional basketball the odds of the home team winning are 3 to 2, i.e. for every 3 home team wins the away team wins 2. Equivalently there is a $\frac{3}{5}$ chance (60% probability) of the home team winning. Maybe you want to compare this ‘home court advantage’ between professional and college basketball. Odds ratios are a useful method for describing such relationships as they are easy to calculate and manipulate and allow flexibility in problem formulation.

Perhaps you have a friend who says “College players get rattled easily. The impact of home court advantage in college is twice what it is at the professional level.” This type of comparison could be viewed as an odds ratio. You have two sets of odds each concerning winning to losing on home court, one for college and one for the pros, and you are taking the ratio of the odds to characterize the relationship of home court advantage depending on the league of play⁵. Pretend your friend next asks, “Given my statement above and what you know about home court advantage at the professional level, what would be the relationship between winning and losing at home in college⁶?”

Taking an odds approach, you might formalize the problem your friend gave you as:

Already known: Odds of winning at home in the pros is 3 to 2.
New information from friend: The ratio of home wins to losses in the pros vs college (i.e. the odds ratio) is 2 to 1.
Question friend is asking: What are the odds of winning at home in college?”

\[\frac{??}{3/2} = 2/1\]

To solve for the odds of winning at home in college you simply double the ratio of winning at home in the pros and you now have a meaningful answer for your friend: the odds of the home team winning at the college level are 3 to 1!

You could have framed this problem in terms of losses and come to an equivalent conclusion. E.g. let’s invert our example:

\[\frac{??}{2/3} = 1/2\]

Solving for this, our odds of losing at home in college simplifies to 1 to 3. This is the reciprocal of our 3 to 1 odds of winning at home. Whether we frame the problem in terms of ‘winning at home’ or ‘losing at home’, the odds are reciprocally related, demonstrating a consistent relationship.

It is less clear how you might formalize the problem in terms of probabilities; also the reciprocal nature of your solutions would not be preserved under multiplication. Doubling the probability of winning from $\frac{3}{5}$ would produce a meaningless answer of $\frac{6}{5}$. You could attempt to flip the problem and instead first frame it as ‘What is the chance of losing at home in college.’ Chance of losing at home is $\frac{2}{5}$, halved becomes $\frac{1}{5}$, entailing the chance of winning would be $\frac{4}{5}$. While you arrived at an answer for “chance of home team winning in college: $\frac{4}{5}$,” your solution was dependent on whether you started out from a “chance of winning” or a “chance of losing” perspective.

Another Example:

To reiterate this lack of symmetry when manipulating a probability and its complement, let’s say your friend instead says, “Home court advantage is only a third more important in college than in the pros.” We can think of ‘a third more’ as a relationship of 4 to 3.

For odds this becomes:

\[\frac{??}{3/2} = 4/3\]

This simplifies to a 2:1 odds of winning at home in college or, if you inverted the problem, you would calculate a 1:2 odds of losing – again, these solutions are consistent with one another.

For probability, if we start at a $\frac{3}{5}$ chance of winning at home in the pros and multiply this by $\frac{4}{3}$ we get an 80% chance of winning at home in college. However say you invert the problem to calculate it from the perspective of losing: $\frac{2}{5}*\frac{3}{4} = \frac{3}{10}$ this 30% home loss rate equates to a 70% home win rate. Hence, depending on whether you set-up the initial calculation in terms of home winning or home losing, you get a college home win rate of either 80% or 70% respectively. This lack of conformity of results when manipulating⁷ the relationship between event outcomes and their complements can make probabilities less wieldly compared to constructing problems as odds⁸.

Communicating odds

An advantage of probabilities over odds is that people are accustomed to probabilities. If you want to communicate odds to someone who is new to the concept, a helpful heuristic is to frame it in the template “for every ___ , there are ___.” E.g. “For every 3 home wins, there are 2 home losses”, or the odds of winning at home are 3 to 2. If you want to communicate a relationship between odds with an odds ratio you could tweak this structure slightly, e.g. “The ratio of home wins to home losses is twice as high in college compared to in the pros.” Keeping the context of the problem central to the explanation is central to being understood. The payoff of using odds is the ability to clearly articulate multiplicative relationships and comparisons between event outcomes which may be cumbersome or unclear when confined to the language of probability alone.

With a little help, we can stop acting like Han Solo.

Who, in the end, could not beat the odds.

Appendix

Ratios and fractions of events

Imagine a bag with 5 red marbles and 2 blue marbles that has been mixed thoroughly.

library(dplyr)
library(ggplot2)
library(DiagrammeR)

create_graph() %>%
  add_n_nodes(5, 
              label = "I am red!!",
              node_aes = node_aes(fillcolor = "red",
                                  fontsize = 5)) %>% 
  add_n_nodes(2, 
              label = "I am blue :-(",
              node_aes = node_aes(fillcolor = "blue",
                                  fontsize = 5)) %>% 
  render_graph()

Odds could be used to represent the ratio between red and blue marbles. Probability could represent the fraction of all marbles that are red⁹.

\[O(R) = \frac{R}{B}\]
\[P(R) = \frac{R}{B+R}\]

Ratio for odds¹⁰:

tibble(color = c(rep("red", 5), rep("blue", 2)),
       y = c(rep(.5, 5), rep(-.5, 2)), 
       x = c(1:5, 2.5, 3.5)) %>% 
  ggplot(aes(x = x, y = y, colour = color, size = 10))+
  geom_hline(yintercept = 0, size = 3, colour = "black")+
  geom_point()+
  theme_void()+
  guides(size = "none", colour = "none")+
  scale_colour_manual(values = c(blue = "blue", red = "red"))

Fraction for probability¹¹:

tibble(color = c(rep("red", 5), rep("red", 5), rep("blue", 2)),
       y = c(rep(.5, 5), rep(-.5, 7)), 
       x = c(1:5, seq(0, 6, length.out = 7))) %>% 
  ggplot(aes(x = x, y = y, colour = color, size = 10))+
  geom_hline(yintercept = 0, size = 3, colour = "black")+
  geom_point()+
  theme_void()+
  guides(size = "none", colour = "none")+
  scale_colour_manual(values = c(blue = "blue", red = "red"))

For odds (when there are only two possible outcomes¹²) each outcome could be thought of as existing either in the numerator or in the denominator. For probabilities, the class of interest (e.g. red marbles) influences both sides¹³.

You can switch between odds and probabilities using the function below:

\[O = \frac{P}{1-P}\]

Odds emphasizes the relationship between the frequencies of possible events whereas probabilities emphasizes the relative frequency of a particular event. Some problems lend themselves more naturally to probability and others to odds¹⁴.

Many variables

There are other factors that might affect the relationship between winning or losing at home other than college/professional level (e.g. ‘quality of coach’, ‘distance away team traveled’, etc.). Understanding odds and ratios of odds is important when using methods like Logistic Regression that can relate the association of multiple variables to an event outcome¹⁵.

Converting Between Currencies Using priceR

Thu, 16 Jun 2022 00:00:00 +0000

In this post I’ll walk through an example of how to convert between currencies. A challenge is that the conversion rate is constantly changing. If you have historical data you’ll want the conversion to be based on what the exchange rate was at the time. Hence the fields you need when doing currency conversion are:

Date of transaction
Start currency (what you’ll be converting from)
End currency (what you’ll be converting to)
Price (in units of starting currency)

For my example I’ll use the priceR package which provides an R interface to the exchangerate.host API. To limit the number of API hits required I first create a lookup table with all unique currency conversions and dates required and then use this table to convert between currencies.

Update

I’ve made some improvements and taken the key functionality in this post and put it into a function at the gist convert-currencies.R. I added an Appendix where I just use this function directly.

Simulate data

I’ll invent some data.

sale_date : date the transaction took place
local_currency : currency code that price is in
price : sale price in `local_currency

library(priceR)
library(dplyr)
library(tidyr)
library(purrr)
library(lubridate)

sim_count <- 10000

set.seed(123)
transactions <- tibble(
  sales_date = sample(
    seq(as.Date('2021/09/01'), 
        as.Date('2022/01/01'), 
        by = "day"), 
    replace = TRUE, sim_count) %>% 
    sort(),
  local_currencies = sample(
    c("CAD", "EUR", "JPY"), 
    replace = TRUE, sim_count),
  list_price = abs(rnorm(sim_count, 1000, 1000))
)

Note that I’m not worried here about keeping the sale prices consistent with one another – they’re all just random values hovering around 1000 units of the local currency. Also, for my first example, I’ll just convert everything to “USD.”

Create rates lookup table

data: dataframe of transactions of interest
currency_code: local currency code that you want to convert away from
date: date of transaction
to: string of currency code you want to convert to, default is “USD”¹
floor_unit: default is “day”. If is set to e.g. “month” it will lookup the conversion rate based on the day at the start of the month².

create_rates_lookup <- function(data, 
                                currency_code, 
                                date = lubridate::today(),
                                to = "USD", 
                                floor_unit = "day"){
  rates_start <- data %>% 
    count(currency_code = {{currency_code}}, 
          date = {{date}} %>% 
            as.Date() %>% 
            floor_date(floor_unit)
          ) 
  
  # When passing things to the priceR API it is MUCH faster to send over a range
  # of dates rather than doing this individually for each date. Doing such
  # reduces API calls.
  rates_end <- rates_start %>% 
    group_by(currency_code) %>% 
    summarise(date_range = list(range(date))) %>% 
    mutate(
      rates_lookup = map2(
        currency_code,
        date_range,
        ~ priceR::historical_exchange_rates(
          from = .x,
          to = to,
          start_date = .y[[1]],
          end_date = .y[[2]]
        ) %>%
          set_names("date_lookup", "rate")
      )
    ) %>% 
    select(-date_range) %>% 
    unnest(rates_lookup)
  
  rates <- rates_end %>% 
    semi_join(rates_start, c("date_lookup" = "date"))
  
  rates_lookup <- rates %>% 
    mutate(to = to) %>% 
    select(from = currency_code, to, date = date_lookup, rate)
  
  # this step makes it so could convert away from "to" currency --
  # i.e. so can both convert from "USD" and to "USD" from another currency.
  bind_rows(rates_lookup,
            rates_lookup %>%
              rename(from = to, to = from) %>%
              mutate(rate = 1 / rate)) %>% 
    distinct()
}

rates_lookup <- create_rates_lookup(transactions, 
                                    local_currencies, 
                                    sales_date)

rates_lookup

## # A tibble: 738 x 4
##    from  to    date        rate
##    <chr> <chr> <date>     <dbl>
##  1 CAD   USD   2021-09-01 0.793
##  2 CAD   USD   2021-09-02 0.796
##  3 CAD   USD   2021-09-03 0.799
##  4 CAD   USD   2021-09-04 0.798
##  5 CAD   USD   2021-09-05 0.798
##  6 CAD   USD   2021-09-06 0.798
##  7 CAD   USD   2021-09-07 0.790
##  8 CAD   USD   2021-09-08 0.788
##  9 CAD   USD   2021-09-09 0.790
## 10 CAD   USD   2021-09-10 0.788
## # ... with 728 more rows

Function to convert prices

This function is set-up to look-up the conversion rates based on the vector inputs³.

convert_currency <- function(price, 
                             date, 
                             from, 
                             to = "USD", 
                             currencies = rates_lookup){
  tibble(price = price, 
         from = from, 
         to = to, 
         date = date) %>% 
    left_join(currencies) %>% 
    mutate(output = price * rate) %>% 
    pull(output)
}

Convert Prices

Now let’s convert our original currencies to USD.

transactions_converted <- transactions %>%
  mutate(list_price_usd = 
           convert_currency(list_price,
                            sales_date,
                            from = local_currencies,
                            to = "USD"))

transactions_converted

## # A tibble: 10,000 x 4
##    sales_date local_currencies list_price list_price_usd
##    <date>     <chr>                 <dbl>          <dbl>
##  1 2021-09-01 CAD                  1002.         794.   
##  2 2021-09-01 CAD                   885.         701.   
##  3 2021-09-01 JPY                   284.           2.58 
##  4 2021-09-01 JPY                    83.6          0.760
##  5 2021-09-01 CAD                  2185.        1732.   
##  6 2021-09-01 EUR                   468.         554.   
##  7 2021-09-01 EUR                   668.         791.   
##  8 2021-09-01 EUR                  1064.        1260.   
##  9 2021-09-01 JPY                  1922.          17.5  
## 10 2021-09-01 JPY                  3334.          30.3  
## # ... with 9,990 more rows

Note that it is possible to then convert from “USD” to any currency type that is in the to field of our lookup table. Below I’ll convert list_price_usd to currencies other than USD⁴, list_price_converted will represent the value of list_price converted based on local_currencies and new_currencies (i.e. from and to respectively).

transactions_converted %>%
  mutate(new_currencies = sample(c("CAD", "EUR", "JPY"), replace = TRUE, sim_count)) %>%
  mutate(list_price_converted =
           convert_currency(list_price_usd,
                            sales_date,
                            from = "USD",
                            to = new_currencies))

## # A tibble: 10,000 x 6
##    sales_date local_currencies list_price list_price_usd new_currencies
##    <date>     <chr>                 <dbl>          <dbl> <chr>         
##  1 2021-09-01 CAD                  1002.         794.    JPY           
##  2 2021-09-01 CAD                   885.         701.    EUR           
##  3 2021-09-01 JPY                   284.           2.58  CAD           
##  4 2021-09-01 JPY                    83.6          0.760 EUR           
##  5 2021-09-01 CAD                  2185.        1732.    CAD           
##  6 2021-09-01 EUR                   468.         554.    CAD           
##  7 2021-09-01 EUR                   668.         791.    EUR           
##  8 2021-09-01 EUR                  1064.        1260.    JPY           
##  9 2021-09-01 JPY                  1922.          17.5   CAD           
## 10 2021-09-01 JPY                  3334.          30.3   EUR           
## # ... with 9,990 more rows, and 1 more variable: list_price_converted <dbl>

Appendix

For my initial “transactions” example dataset here, I’ll have from and to currencies that can vary line-by-line to showcase that the convert_currencies() function handles these fine.

set.seed(123)
transactions_complex <- tibble(
  sales_date = sample(
    seq(as.Date('2021/01/01'), 
        as.Date('2022/01/01'), 
        by = "day"), 
    replace = TRUE, sim_count) %>% 
    sort(),
  from_currency = sample(
    c("CAD", "EUR", "JPY", "USD"), 
    replace = TRUE, sim_count),
  to_currency = sample(
    c("CAD", "EUR", "JPY", "USD"), 
    replace = TRUE, sim_count),
  list_price_start = abs(rnorm(sim_count, 1000, 1000))
) %>% 
  filter(from_currency != to_currency)

# load function from gist
devtools::source_gist("https://gist.github.com/brshallo/650c1ad3f4bd9b74076592c6bc4ff8ae")

transactions_complex %>%
  mutate(
    list_price_converted = convert_currencies(
      price_from = list_price_start,
      from = from_currency,
      to = to_currency,
      date = sales_date
    )
  )

## # A tibble: 7,532 x 5
##    sales_date from_currency to_currency list_price_start list_price_converted
##    <date>     <chr>         <chr>                  <dbl>                <dbl>
##  1 2021-01-01 JPY           CAD                  1454.                17.9   
##  2 2021-01-01 CAD           USD                  3315.              2604.    
##  3 2021-01-01 CAD           USD                  1009.               793.    
##  4 2021-01-01 JPY           EUR                  2375.                18.9   
##  5 2021-01-01 USD           JPY                  2792.            288225.    
##  6 2021-01-01 CAD           USD                  2548.              2002.    
##  7 2021-01-01 JPY           CAD                  1017.                12.5   
##  8 2021-01-01 EUR           USD                   815.               992.    
##  9 2021-01-01 EUR           JPY                   548.             68870.    
## 10 2021-01-01 JPY           EUR                     7.58               0.0603
## # ... with 7,522 more rows

Can only be a single value when building the lookup function at this stage. However later when applying convert_currency() you can have it be any currency that is in the lookup table.↩︎
Highest granularity for API is day.↩︎
This could be converted to be an “all inclusive” function – i.e. no need to specify the rates_lookup in a separate step. However the advantage with the current set-up is you could use the rates_lookup table on multiple functions. It might also make sense to have create_rates_lookup() simply output a function that would be like convert_currency() but specific to the rates that were looked-up, i.e. just setting currencies = rates_lookup.↩︎
In this way you are not restricted to converting to a single to currency.↩︎

Pulling Twitter Engagements Using the v2 API as Well as rtweet

Mon, 11 Apr 2022 00:00:00 +0000

This is a follow-up to a short post I wrote on R Access to Twitter’s v2 API. In this post I’ll walk through a few more examples of pulling data from twitter using a mix of Twitter’s v2 API as well as the {rtweet} package¹.

I’ll pull all Twitter users that I (brshallo) have recently been engaged by (e.g. they like my tweet) or engaged with (e.g. I like their tweet). I’ll lean towards using {rtweet}² but will use {httr} in cases where it’s more convenient to use Twitter’s v2 API³.

For this post I’m not really worried about optimizing my queries, minimizing API hits, etc. E.g. when using {rtweet} I should authenticate through my project app which has higher rate limits (see Authentication options) but instead I just use the default {rtweet} user authentication. Note also that the default {rtweet} authentication only works when running scripts interactively⁴.

See prior post for links on authentication mechanisms. I’m assuming you have “TWITTER_BEARER”⁵ as well as “TWITTER_PAT”⁶ in your .Renviron file.

library(rjson)
require(httr)
require(jsonlite)
require(dplyr)
library(purrr)
library(lubridate)
library(rtweet)
library(tidyr)

# bearer_token only used when using httr and twitter v2 API
bearer_token <- Sys.getenv("TWITTER_BEARER")
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))

GETting all engagements

In each sub-section I’ll pull a different kind of engagement.

GET favorited users
GET all tweets from user – starting point for most of the following sections
From initial query GET references in those tweets
Filter to only tweets with likes, GET favoriters
Filter to only tweets with quotes, search URL’s to GET quoters
Filter to only tweets with retweets, GET retweeters
GET repliers and mentions

I’ll finish by Putting them together into a function. Note that not all queries are perfect at pulling all engagements⁷.

GET favorited users

It’s often easiest to just let {rtweet} do the work.

# Twitter id for brshallo
user_id <- "307012324"

favorites <- rtweet::get_favorites(user = user_id)

GET all tweets from user

Pulls up to 100 of the most recent tweets from a user⁸.

url_handle <- glue::glue("https://api.twitter.com/2/users/{user_id}/tweets?max_results=100", user_id = user_id)

params <- list(tweet.fields = "public_metrics,created_at,in_reply_to_user_id,referenced_tweets")

response <- httr::GET(url = url_handle,
                     httr::add_headers(.headers = headers),
                     query = params)

obj <- httr::content(response, as = "text")

json_data <- jsonlite::fromJSON(obj, flatten = TRUE)$data %>% 
  as_tibble()

GET references

statuses_referenced <- bind_rows(json_data$referenced_tweets) %>% 
  rename(status_id = id)

users_referenced <- rtweet::lookup_tweets(statuses_referenced$status_id)

GET favoriters

Filter initial query of tweets to only those with more than 0 likes.

liked_tweets <- json_data %>% 
  filter(public_metrics.like_count > 0)

Functionalize approach described in getting favoriters from prior post R Access to Twitter’s v2 API and map tweet-ids through.

tweet_ids <- liked_tweets$id

get_favoriters <- function(tweet_id){
  url_handle <- glue::glue("https://api.twitter.com/2/tweets/{status_id}/liking_users", status_id = tweet_id)
  
  response <- httr::GET(url = url_handle,
                       httr::add_headers(.headers = headers))
                       # query = params)
  
  obj <- httr::content(response, as = "text")
  x <- rjson::fromJSON(obj)
  
  x$data %>% 
    map_dfr(as_tibble)
}

tweet_favoriters <-
  map_dfr(tweet_ids, ~ bind_cols(tibble(liked_status_id = .x),
                                get_favoriters(.x))) %>%
  rename(user_id = id)

GET quoters

Filter to only posts with quotes.

tweet_ids_quoters <- json_data %>% 
  filter(public_metrics.quote_count > 0) %>%
  pull(id)

However I am not positive the approach below actually picks up all quotes⁹. I’d also reviewed some other approaches¹⁰.

search_tweets_urls <- function(tweet_id){
  rtweet::search_tweets(
    glue::glue("url:{tweet_id}", 
               tweet_id = tweet_id)
    )
} 

quoters <- map_dfr(tweet_ids_quoters, search_tweets_urls) %>% 
  filter(is_quote) %>% 
  as_tibble()

GET retweeters

Filter to only posts that were retweeted.

tweet_ids_rt <- json_data %>% 
  filter(public_metrics.retweet_count > 0) %>%
  select(status_id = id)

I use a slightly different approach in this section than in other similar sections¹¹.

retweeters <- tweet_ids_rt %>% 
  mutate(retweeters = map(status_id, get_retweeters)) %>% 
  unnest(retweeters)

GET repliers and mentions

Alternatively you might just use rtweet::get_mentions() but this only pulls mentions of the currently authenticated user. I also tried other approaches here¹².

get_mentions_v2 <- function(user_id){
  url_handle <- glue::glue("https://api.twitter.com/2/users/{user_id}/mentions", user_id = user_id)
  
  response <- httr::GET(url = url_handle,
                        httr::add_headers(.headers = headers))
  
  obj <- httr::content(response, as = "text")
  x <- rjson::fromJSON(obj)
  
  x$data %>% 
    map_dfr(as_tibble)
}

tweets_mentions <- get_mentions_v2(gorthon_id)

repliers_mentions <- lookup_tweets(mentions$id)

Putting them together into a function

The function at this gist returns the output from each of the above sections as a list.

# Twitter id for brshallo
user_id <- "307012324"

# load function get_engagements()
source("https://gist.githubusercontent.com/brshallo/119d6a1f858e0e5c20d77212dee8891a/raw/751d022c7bc2e2148292bb78a5178737d9914024/get-engagements.R")

brshallo_engagements <- get_engagements(user_id)

brshallo_engagements

## $favorites
## # A tibble: 10 x 91
##    user_id             status_id  created_at          screen_name text    source
##  * <chr>               <chr>      <dttm>              <chr>       <chr>   <chr> 
##  1 248350998           151302361~ 2022-04-10 05:18:34 BuildABarr  "Drop ~ Twitt~
##  2 368551889           151263551~ 2022-04-09 03:36:23 IsabellaGh~ "@elli~ Twitt~
##  3 1469531055736590337 151242047~ 2022-04-08 13:21:54 emkayco     "Have ~ Twitt~
##  4 35794978            151196918~ 2022-04-07 07:28:38 _lionelhen~ "@brsh~ Twitt~
##  5 29916355            151194957~ 2022-04-07 06:10:44 jimjam_slam "@brsh~ Twitt~
##  6 29916355            151195192~ 2022-04-07 06:20:03 jimjam_slam "@brsh~ Twitt~
##  7 29916355            151189984~ 2022-04-07 02:53:06 jimjam_slam "@mdne~ Twitt~
##  8 3089027769          151189179~ 2022-04-07 02:21:09 gyp_casino  "@mdne~ Twitt~
##  9 15772978            151132777~ 2022-04-05 12:59:55 jessicagar~ "@brsh~ Twitt~
## 10 144592995           151129000~ 2022-04-05 10:29:49 Rbloggers   "R Acc~ r-blo~
## # ... with 85 more variables: display_text_width <dbl>,
## #   reply_to_status_id <chr>, reply_to_user_id <chr>,
## #   reply_to_screen_name <chr>, is_quote <lgl>, is_retweet <lgl>,
## #   favorite_count <int>, retweet_count <int>, quote_count <int>,
## #   reply_count <int>, hashtags <list>, symbols <list>, urls_url <list>,
## #   urls_t.co <list>, urls_expanded_url <list>, media_url <list>,
## #   media_t.co <list>, media_expanded_url <list>, media_type <list>, ...
## 
## $favoriters
## # A tibble: 90 x 4
##    liked_status_id     user_id            name           username       
##    <chr>               <chr>              <chr>          <chr>          
##  1 1512295676004093955 117241741          Brett J. Gall  brettjgall     
##  2 1512295676004093955 2724597409         Peter Ellis    ellis2013nz    
##  3 1512294950905409543 274123666          Kristen Downs  KristenDDowns  
##  4 1512293864517750790 3656879234         <U+5F20><U+4EAE>           psychelzh      
##  5 1512293864517750790 703843771419484160 Ayush Patel    ayushbipinpatel
##  6 1512293864517750790 419185498          Kevin Gilds    Kevin_Gilds    
##  7 1512293864517750790 127357236          Juan LB        Juan_FLB       
##  8 1512293864517750790 49451947           Luis Remiro    LuisMRemiro    
##  9 1512293864517750790 253175044          Nicholas Viau  nicholasviau   
## 10 1512293864517750790 2202983986         Stefania Klayn Ettti_20       
## # ... with 80 more rows
## 
## $references
## # A tibble: 12 x 90
##    user_id            status_id  created_at          screen_name  text    source
##    <chr>              <chr>      <dttm>              <chr>        <chr>   <chr> 
##  1 307012324          151115943~ 2022-04-05 01:50:59 brshallo     "As an~ Twitt~
##  2 307012324          151229344~ 2022-04-08 04:57:09 brshallo     "@mdne~ Twitt~
##  3 307012324          150969487~ 2022-04-01 00:51:20 brshallo     "It al~ Twitt~
##  4 307012324          151229386~ 2022-04-08 04:58:49 brshallo     "@mdne~ Twitt~
##  5 307012324          147233714~ 2021-12-18 22:45:04 brshallo     "First~ Twitt~
##  6 29916355           151189984~ 2022-04-07 02:53:06 jimjam_slam  "@mdne~ Twitt~
##  7 29916355           151194957~ 2022-04-07 06:10:44 jimjam_slam  "@brsh~ Twitt~
##  8 144592995          151129000~ 2022-04-05 10:29:49 Rbloggers    "R Acc~ r-blo~
##  9 248350998          151302361~ 2022-04-10 05:18:34 BuildABarr   "Drop ~ Twitt~
## 10 3146735425         151226195~ 2022-04-08 02:52:00 mdneuzerling "Lovel~ Twitt~
## 11 983470194982088704 151182189~ 2022-04-06 21:43:22 R4DScommuni~ "The n~ Zapie~
## 12 2724597409         151226515~ 2022-04-08 03:04:44 ellis2013nz  "@mdne~ Twitt~
## # ... with 84 more variables: display_text_width <dbl>,
## #   reply_to_status_id <chr>, reply_to_user_id <chr>,
## #   reply_to_screen_name <chr>, is_quote <lgl>, is_retweet <lgl>,
## #   favorite_count <int>, retweet_count <int>, quote_count <int>,
## #   reply_count <int>, hashtags <list>, symbols <list>, urls_url <list>,
## #   urls_t.co <list>, urls_expanded_url <list>, media_url <list>,
## #   media_t.co <list>, media_expanded_url <list>, media_type <list>, ...
## 
## $quoters
## NULL
## 
## $retweeters
## # A tibble: 11 x 2
##    status_id           user_id            
##    <chr>               <chr>              
##  1 1512293864517750790 296222670          
##  2 1512293864517750790 307012324          
##  3 1511869112401596423 4034079677         
##  4 1511869112401596423 1306626901432324097
##  5 1511869112401596423 1011817655957893120
##  6 1511469730892156928 1011817655957893120
##  7 1511469730892156928 1306626901432324097
##  8 1511159434717761539 1448348827979747333
##  9 1511159434717761539 15772978           
## 10 1511159434717761539 1011817655957893120
## 11 1511159434717761539 1306626901432324097
## 
## $referencers
## # A tibble: 10 x 90
##    user_id             status_id  created_at          screen_name text    source
##    <chr>               <chr>      <dttm>              <chr>       <chr>   <chr> 
##  1 61542689            150992063~ 2022-04-01 15:48:26 twelvespot  "@brsh~ Twitt~
##  2 61542689            150994022~ 2022-04-01 17:06:17 twelvespot  "@brsh~ Twitt~
##  3 18433005            151007180~ 2022-04-02 01:49:09 rcrdleitao  "@brsh~ Twitt~
##  4 35794978            151196918~ 2022-04-07 07:28:38 _lionelhen~ "@brsh~ Twitt~
##  5 1346474633520824320 150985661~ 2022-04-01 11:34:03 markjrieke  "@brsh~ Twitt~
##  6 29916355            151195192~ 2022-04-07 06:20:03 jimjam_slam "@brsh~ Twitt~
##  7 29916355            151195162~ 2022-04-07 06:18:51 jimjam_slam "@brsh~ Twitt~
##  8 29916355            151194957~ 2022-04-07 06:10:44 jimjam_slam "@brsh~ Twitt~
##  9 15772978            151132777~ 2022-04-05 12:59:55 jessicagar~ "@brsh~ Twitt~
## 10 15772978            151117782~ 2022-04-05 03:04:04 jessicagar~ "@brsh~ Twitt~
## # ... with 84 more variables: display_text_width <dbl>,
## #   reply_to_status_id <chr>, reply_to_user_id <chr>,
## #   reply_to_screen_name <chr>, is_quote <lgl>, is_retweet <lgl>,
## #   favorite_count <int>, retweet_count <int>, quote_count <int>,
## #   reply_count <int>, hashtags <list>, symbols <list>, urls_url <list>,
## #   urls_t.co <list>, urls_expanded_url <list>, media_url <list>,
## #   media_t.co <list>, media_expanded_url <list>, media_type <list>, ...

Which as of this writing uses the 1.1 API.↩︎
As it takes less code.↩︎
Or in cases where the field isn’t available in {rtweet}. V2 is not yet supported by {rtweet} but is actively being worked on so this post may have a short shelf-life.↩︎
You’ll need to authenticate through a Twitter developer portal app keys if you want to run those sections automatically. You’ll notice that in creating this script I actually don’t evaluate most of the sections and then use some hidden code chunks to return output.↩︎
For the sections where I use {httr} in this post.↩︎
For the sections where I use {rtweet}. This should be set-up through the default {rtweet} set-up.↩︎
This seemed to particularly be the case when it came to seeing all quotes and mentions.↩︎
The reason I’m using {httr} and v2 instead of {rtweet} for this is that the 1.1 API (that {rtweet} currently uses) doesn’t pull quote count unless you have a premium or enterprise account rtweet#640.↩︎
Thread here seemed to suggest that just searching the url was the way to go.↩︎

This also seems to be way to see quoters: https://twittercommunity.com/t/how-we-can-get-list-of-replies-on-a-tweet-or-reply-to-a-tweet-in-twitter-api/144958/7

get_quoters <- function(tweet_id){
  url_handle <- glue::glue("https://api.twitter.com/2/tweets/search/recent?tweet.fields=author_id&query=url:{status_id}", status_id = tweet_id)

  response <- httr::GET(url = url_handle,
                       httr::add_headers(.headers = headers))
                       # query = params)

  obj <- httr::content(response, as = "text")
  x <- rjson::fromJSON(obj)

  x$data %>% 
    map_dfr(as_tibble)
}

quoters <- map(tweet_ids_quoters, get_quoters)

↩︎

rtweet::get_retweeters() has a lot fewer columns returned compared to that from rtweet::search_tweets(), which is why I use select() above and a different method than the section before and after this where I instead use pull() and then pass the ideas directly to purrr::map*() statements rather than wrapping them in a mutate() verb – which would have worked just as well. The structures of the manipulation are nearly the same… maybe should have stayed consistent here and written a function to make clear the pattern here is the same, c’est la vie.↩︎

Another simple approach would be to just try: rtweet::search_tweets("@brshallo") . I tried the approach below, but really didnt’ seem to work quite as expected…

tweet_ids_repliers <- json_data %>% 
  filter(public_metrics.reply_count > 0) %>%
  pull(id)

# pulled from here: https://twittercommunity.com/t/how-to-fetch-retweets-and-quote-tweets-from-the-twitter-v2-search-api/156573 but didn't really work as expected...
get_replies <- function(tweet_id){
url_handle <- glue::glue("https://api.twitter.com/2/tweets/search/recent?tweet.fields=author_id&query=conversation_id:{status_id}", status_id = tweet_id)

response <- httr::GET(url = url_handle,
                     httr::add_headers(.headers = headers))
                     # query = params)

obj <- httr::content(response, as = "text")
x <- rjson::fromJSON(obj)

x$data %>% 
  map_dfr(as_tibble)
}

repliers <- map(tweet_ids_repliers, get_replies)
filter(is_quote)

repliers <- bind_rows(repliers)

↩︎

R Access to Twitter's V2 API

Mon, 04 Apr 2022 00:00:00 +0000

The rtweet package is still the easiest way to GET and POST Twitter data from R. However its developers are currently working on adapting it to the new API. V2 comes with a variety of new features. The one I was interested in was being able to GET the users who liked a tweet.

academictwitteR is probably the most established package that provides a quickstart entry point to the V2 API. However it requires creating an academic account in twitter, i.e. the user must be affiliated with a university. I also stumbled onto RTwitterV2 and voson.tcn which both also provide quickstarts on the V2 API, but did not explore these.

Instead I followed the tutorial Getting started with R and v2 of the Twitter API by Twitter Developer Advocate Jessica Garson that uses {httr} to interact more directly with the API. I highly recommend reading her tutorial. The code below is mostly just copied from there but changed to provide an example of getting the usernames of those that liked a tweet¹.

GETting likers of tweet

Last year, twitter announced Likes lookup in API v2.

Below is a minimal example of how to find every user who liked any tweet, in my case I’ll check favoriters of my initial announcement of the pwiser package.

Compute all pairwise ratios, t-tests, distance metrics, or whatever using {pwiser} a new 📦 I'm writing with @CarloooMedina for applying arbitrary functions on pairwise combinations of columns using a dplyr::across() style syntax: https://t.co/DTXElWbuuX #rstats #tidyverse pic.twitter.com/wrMzX5mOYx
— Bryan Shalloway (@brshallo) May 16, 2021

First you need to:

Apply for a Twitter developer account.
Create a project
Create an app within that project²

Once you’ve done this, you’ll need to save your BEARER TOKEN somewhere secure, see Hadley’s write-up on Managing secrets and use your choice of approach. For the write-up below I’ll load the secret from the environment³.

library(rjson)
require(httr)
require(jsonlite)
require(dplyr)
library(purrr)

bearer_token <- Sys.getenv("TWITTER_BEARER")
headers <- c(`Authorization` = sprintf('Bearer %s', bearer_token))

tweet_id <- "1394072770661822464"
url_handle <- glue::glue("https://api.twitter.com/2/tweets/{status_id}/liking_users", status_id = tweet_id)

response <- httr::GET(url = url_handle,
                     httr::add_headers(.headers = headers))
                     # query = params)

obj <- httr::content(response, as = "text")
x <- rjson::fromJSON(obj)

And then let’s pull out the usernames of the people who liked the tweet:

x$data %>% 
  purrr::map_chr("username")

##  [1] "Rick_Scavetta"   "dikiprawisuda"   "technocrat"      "abuabara"       
##  [5] "kavenacca4"      "pheeree"         "mattansb"        "AndrewKostandy" 
##  [9] "1AliG"           "ncypris"         "jkregenstein"    "MarkDruffel"    
## [13] "pablofbaez"      "JobNmadu"        "LuisDVerde"      "dataleteo"      
## [17] "FredOnion"       "SebLammers"      "meier_flo"       "tomecicuta"     
## [21] "ameresv"         "ginanjar_utama"  "jmblanch"        "ThatBenFrost"   
## [25] "lan24hd"         "henda52"         "Md_Harris"       "EmilyRiederer"  
## [29] "pvallejomedina"  "thisisdaryn"     "ericarbailey"    "DrMeltemYucel"  
## [33] "c_welk"          "stewartli3"      "patilindrajeets" "rick_pack2"     
## [37] "AlainLesaffre"   "Carlos_Espeleta" "ellamkaye"       "pacoramon"      
## [41] "neslihanky"      "wang_minjie"     "PipingHotData"   "Jack36161714"   
## [45] "KenButler12"     "xeroluck"        "rstats4ds"       "francisco_yira"

One thing you may note is that I don’t use the query argument, I just pull the default parameters – again see Jessica’s posts for a more sophisticated example as well as descriptions of the steps above.

I will warn that it took me an embarrassing amount of time to catch a slight change in the Twitter API between now and when Jessica wrote her post see note on expansions.↩︎
Think you need the app within a project, not just a standalone app for access to V2. The Creating a Twitter app section of Authentication with rtweet is also a helpful tutorial for getting started here. However the documentation there does not indicate the need to create a project first, however {rtweet} uses V1, not V2.↩︎
Project has a local .Renviron file.↩︎

Network Visualizations of Code Collections (funspotr part 3)

Thu, 17 Mar 2022 00:00:00 +0000

Interactive network plots

In previous posts and threads I’ve alluded to the potential utility of visualizing the relationships between parsed functions/packages and files as a network plot.

It can be helpful to review the relationship between your #rstats code files by looking at a network graph of them by packages loaded.

Graph of 50+ of my gists (squares) and packages (circles) used.

That node at the center is {dplyr}. pic.twitter.com/XmNxOrgDtF
— Bryan Shalloway (@brshallo) March 16, 2022

I added the function network_plot() to funspotr. In this post I’ll simply output the network plots of the parsed-out packages from the code collections discussed in the prior two posts:

library(dplyr)
library(funspotr)

Interactive network plots

The network plots show files as squares and packages as circles, edges represent cases where a package is used in a given file¹.

Julia Silge Blog

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/jsilge-blog-funs-20220114.csv") %>% 
  # not including base R or any custom functions or packages I don't have installed
  filter(!is.na(pkgs), !(pkgs %in% c("base", "(unknown)"))) %>% 
  network_plot(to = pkgs)

tidymodels and tidyverse packages are both central to Julia’s posts. The cluster of tidymodels packages show-up (for the most part) just to the right of the cluster of core tidyverse packages.

David Robinson Tidy Tuesday

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/drob-tidy-tuesdays-funs-20220114.csv") %>% 
  filter(!is.na(pkgs), !(pkgs %in% c("base", "(unknown)"))) %>% 
  network_plot(to = pkgs)

Similar to Julia’s posts, tidyverse packages are central to David’s Tidy Tuesday files.. However the tidymodels packages are less central and can be seen in a cluster at the bottom of the plot.
In both plots we see {broom} not showing-up by the other tidymodels packages. This is unsurprising for while broom is in the tidymodels ecosystem it has many common uses outside of predictive modeling and has a longer legacy than most tidymodels packages.

R for Data Science Chapters

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/r4ds-chapter-files-funs-20220117.csv") %>% 
  filter(!is.na(pkgs), !(pkgs %in% c("base", "(unknown)"))) %>% 
  network_plot(to = pkgs)

My blog

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/brshallo-blog-funs-20220114.csv") %>% 
  filter(!is.na(pkgs), !(pkgs %in% c("base", "(unknown)"))) %>% 
  network_plot(to = pkgs)

My gists

readr::read_csv("https://raw.githubusercontent.com/brshallo/brshallo/master/content/post/2022-02-07-identifying-r-functions-packages-in-your-github-gists/data/brshallo-gists-20220314.csv") %>% 
  filter(!is.na(pkgs), !(pkgs %in% c("base", "(unknown)"))) %>% 
  network_plot(to = pkgs)

This figure is a bit different than the graph shown in my tweet above as it includes more of my gists and uses a different algorithm to construct the network.
dplyr, purrr, and tidyr are the three packages at the center

With all of these I think more time could go into tailoring the network plot. It would also be interesting to look into measures of network relatedness between the files… maybe in a future post…↩︎

Identifying R Functions & Packages in Github Gists (funspotr part 2)

Mon, 07 Feb 2022 00:00:00 +0000

This post is part two in a series of posts introducing funspotr. See also:

This post shows how funspotr can also be applied to parse gists:

By functions or packages used…?https://t.co/kbSLOpQZLF
— Bryan Shalloway (@brshallo) January 22, 2022

A problem I bumped into was that most of Chelsea’s gists don’t actually have .R or .Rmd extensions so my approach skipped most of her snippets. I wanted to parse my own gists but ran into a related problem that most of my github gist code snippets are saved as .md files¹ so knitr::purl() won’t work².

In this post I…

create a function to extract code chunks from simple .md files
parse the functions and packages in my code using funspotr ³.

This post was updated on 2023-10-11 to make it consistent with updated {funspotr} code. Tables were also updated to reflect brshallo gists at this time. The following post on network plots however was not updated.

Parsing code

First I used funspotr to get a table with all of my gists.

library(dplyr)
library(purrr)
library(stringr)
library(funspotr)

brshallo_gists <- funspotr::list_files_github_gists("brshallo", pattern = ".")

brshallo_gists

## # A tibble: 112 × 2
##    relative_paths                               absolute_paths                  
##    <chr>                                        <chr>                           
##  1 find_in_files.R                              https://gist.githubusercontent.…
##  2 permits-issued.md                            https://gist.githubusercontent.…
##  3 seattle-units-added-new-permits.md           https://gist.githubusercontent.…
##  4 rolling-mean-conditioned-date.R              https://gist.githubusercontent.…
##  5 rolling-mean-conditioned-on-iteration-date.R https://gist.githubusercontent.…
##  6 lag-multiple-across.md                       https://gist.githubusercontent.…
##  7 log-log-transform-example.md                 https://gist.githubusercontent.…
##  8 convert-currencies.R                         https://gist.githubusercontent.…
##  9 unique-set-speed-test.R                      https://gist.githubusercontent.…
## 10 unique-combinations.R                        https://gist.githubusercontent.…
## # ℹ 102 more rows

Parsing R files

funspotr is already set-up to parse all the unique functions and packages from R or Rmd files.

r_gists <- brshallo_gists %>% 
  filter(funspotr:::str_detect_r_docs(relative_paths))

r_gists_parsed <- funspotr::spot_funs_files(r_gists)

r_gists_unnested <- r_gists_parsed %>% 
  funspotr::unnest_results()

Hidden from this post but a warning message indicates a couple files which did not parse correctly. In this particular case those files were created using reprexes for .md output but I saved them as .R files – hence they failed parsing.

r_gists_unnested

## # A tibble: 701 × 4
##    funs      pkgs  relative_paths                  absolute_paths               
##    <chr>     <chr> <chr>                           <chr>                        
##  1 library   base  find_in_files.R                 https://gist.githubuserconte…
##  2 dir_ls    fs    find_in_files.R                 https://gist.githubuserconte…
##  3 map       purrr find_in_files.R                 https://gist.githubuserconte…
##  4 grep      base  find_in_files.R                 https://gist.githubuserconte…
##  5 readLines base  find_in_files.R                 https://gist.githubuserconte…
##  6 keep      purrr find_in_files.R                 https://gist.githubuserconte…
##  7 length    base  find_in_files.R                 https://gist.githubuserconte…
##  8 library   base  rolling-mean-conditioned-date.R https://gist.githubuserconte…
##  9 seq       base  rolling-mean-conditioned-date.R https://gist.githubuserconte…
## 10 map       purrr rolling-mean-conditioned-date.R https://gist.githubuserconte…
## # ℹ 691 more rows

Parsing markdown files

To parse my .md files, I wrote a function here extract_code_md() that…

reads in a file
extracts the text in code chunks⁴
saves it to a temporary file
returns the file path of the temporary file

subset_even <- function(x) x[!seq_along(x) %% 2]

extract_code_md <- function(file_path){
  
  lines <- readr::read_file(file_path) %>% 
    stringr::str_split("```.*", simplify = TRUE) %>%
    subset_even() %>% 
    stringr::str_flatten("\n## new chunk \n")
  
  file_output <- tempfile(fileext = ".R")
  writeLines(lines, file_output)
  file_output
}

I map extract_code_md() on all the .md gists and then parse the files using funspotr.

md_gists <- brshallo_gists %>% 
  filter(!funspotr:::str_detect_r_docs(relative_paths))

md_gists_local <- md_gists %>% 
  rename(urls = absolute_paths) %>% 
  # name absolute_paths because that's what funspotr::spot_funs_files() expects
  mutate(absolute_paths = map_chr(urls, extract_code_md))

md_gists_parsed <- funspotr::spot_funs_files(md_gists_local) %>% 
  mutate(absolute_paths = urls) %>% 
  select(-urls)
  
md_gists_unnested <- md_gists_parsed %>% 
  funspotr::unnest_results()

In this case also some files did not parse correctly though this is hidden due to warning = FALSE settings in the code chunks. These are essentially just not included in the unnested output.

md_gists_unnested

## # A tibble: 1,061 x 5
##    funs           pkgs    in_multiple_pkgs contents                 urls        
##    <chr>          <chr>   <lgl>            <chr>                    <chr>       
##  1 library        base    FALSE            grouped-nested-t-test.md "C:\\Users\~
##  2 require        base    FALSE            grouped-nested-t-test.md "C:\\Users\~
##  3 install_github remotes FALSE            grouped-nested-t-test.md "C:\\Users\~
##  4 na.omit        stats   FALSE            grouped-nested-t-test.md "C:\\Users\~
##  5 t.test         stats   FALSE            grouped-nested-t-test.md "C:\\Users\~
##  6 tidy           broom   FALSE            grouped-nested-t-test.md "C:\\Users\~
##  7 pull           dplyr   FALSE            grouped-nested-t-test.md "C:\\Users\~
##  8 group_by       dplyr   FALSE            grouped-nested-t-test.md "C:\\Users\~
##  9 summarise      dplyr   FALSE            grouped-nested-t-test.md "C:\\Users\~
## 10 list           base    FALSE            grouped-nested-t-test.md "C:\\Users\~
## # ... with 1,051 more rows

Note that I’m assuming all the code snippets are R code⁵.

Binding files together

I bind these files together and then arrange them based on the initial order in brshallo_gists⁶.

gists_unnested <- bind_rows(
  r_gists_unnested,
  md_gists_unnested
) %>% 
  # got this arranging by a vector trick from SO:
  # https://stackoverflow.com/questions/52216341/how-to-sort-rows-of-a-data-frame-based-on-a-vector-using-dplyr-pipe
  arrange(match(relative_paths, brshallo_gists$relative_paths)) %>% 
  # add back-in links to url's where files are rather than urls column being
  # local paths for .md snippets
  select(-absolute_paths) %>% 
  left_join(brshallo_gists, by = "relative_paths")

gists_unnested %>% 
  DT::datatable(rownames = FALSE,
            class = 'cell-border stripe',
            filter = 'top',
            escape = FALSE,
            options = list(pageLength = 20))

Organizing snippets

Perhaps I’ll do a follow-up and show some ways the relationships between the resulting parsed code snippets may be visualized in a network or organized in some other way.

Mentioned in the initial thread, Obsidian seems to be a product that does some things along these lines:

I’ve found this useful (https://t.co/OYzwfTltLG) I is a tool for writing, organizing, linking markdown files.
— John Lee (@Jdlee888) January 21, 2022

Appendix

Interactively save current gists to folder so can read from another file if want to

post_path <- fs::path_dir(rstudioapi::getSourceEditorContext()$path)

fs::dir_create(post_path, "data")

readr::write_csv(gists_unnested, fs::path(post_path, "data", paste0("brshallo-gists-", format(Sys.Date(), "%Y%m%d"), ".csv")))

As I often use this output type when creating a reprex.↩︎
knitr::purl() is used in functions within funspotr to parse R markdown files.↩︎
In the future I may do a follow-up that passes the parsed functions and packages through a network analysis or some other approach to better visualize the relationships between code snippets.↩︎
based on what exists between ticks. Kind of like a less reliable version of knitr::purl() but for .md files. Also posted function on SO question.↩︎
Otherwise the R code parsing steps in funspotr will fail.↩︎
Note that this will just return the unique functions in each file, if I want to see every time I used a function I would have passed in show_each_use = FALSE to github_spot_funs().↩︎

Identifying R Functions & Packages Used in GitHub Repos (funspotr part 1)

Tue, 18 Jan 2022 00:00:00 +0000

TLDR: funspotr provides helpers for spotting the functions and packages in R and Rmarkdown files and associated github repositories. See Examples for catalogues of the functions/packages used in posts by Julia Silge, David Robinson, and others. See follow-up posts for examples with github gists and how to pass parsed code collections into a network plot:

Documenting rstats posts

I was inspired by a few tweets I saw documenting the methods used in posts by popular #rstats communicators/developers:

To try and learn #tidymodels I made a list of examples of using each engine, etc, by @juliasilge, @topepos, and @Emil_Hvitfeldt. It's far from exhaustive, and in progress, but has helped a ton. #rstats https://t.co/yqecIo7CUS
— Jeff Rothschild (@EatsleepfitJeff) October 19, 2021

Anyone other #rstats people find @drob's #TidyTuesday screencasts useful?

I made a spreadsheet with timestamps for hundreds of specific tasks he does: https://t.co/HvJbLk1chd

Useful if, like me, you keep going back and ask, “Where in the video did he do [this thing] again?”
— Alex Cookson (@alexcookson) January 13, 2020

As a complement to these resources, I thought it’d be helpful to see precisely which functions and packages were used in each post. See Julia Silge Blog and David Robinson Tidy Tuesday for tables containing the unique functions and associated packages used in each of their posts. I also created tables for functions in the R for Data Science Chapters and Bryan Shalloway Blog.

These Examples were made using the funspotr package that provides helpers for identifying R functions & packages in self-contained R files and associated github repositories. See funspotr website for documentation on how to apply funspotr to #rstats file(s) or repos¹ as well as to see how the package works and current limitations. Message me if you use funspotr or feel free to open an issue if you’d be interested in adding additional funspotr-examples that I can link to!

Examples

The files in the github repos below were parsed and saved in funspotr-examples ² – specifically using R/save-examples.R to parse each file and record each instance of a function’s use to data/funs ³.

Below, I only return the first use of a function in each post⁴ – except for R for Data Science Chapters where I only show the first use of a function across the entire book⁵.

library(dplyr)
library(readr)
library(DT)

Julia Silge Blog

Also posted table to google drive here.

The following package dependencies were not installed locally during parsing. Any function from these packages (along with any custom functions) will most likely be assigned pkgs = "(unknown)":
Lingua; EN; silgelib; packagesurvey; drlib; sqlstackr; tvthemes; parttree

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/jsilge-blog-funs-20220114.csv") %>% 
  distinct() %>% 
  DT::datatable(rownames = FALSE,
              class = 'cell-border stripe',
              filter = 'top',
              escape = FALSE,
              options = list(pageLength = 10))

David Robinson Tidy Tuesday

Also posted table to google drive here.

The following package dependencies were not installed locally during parsing. Any function from these packages (along with any custom functions) will most likely be assigned pkgs = "(unknown)":
ggflags; ebbr; rKenyaCensus; tidymetrics; shinymetrics; drlib; shinybones; treesnip; StatsBombR

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/drob-tidy-tuesdays-funs-20220114.csv") %>% 
  distinct() %>% 
  DT::datatable(rownames = FALSE,
              class = 'cell-border stripe',
              filter = 'top',
              escape = FALSE,
              options = list(pageLength = 10))

R for Data Science Chapters

Also posted table to google drive here.

Pulling these files was slightly more complicated than the other examples (which were just simple calls to funspotr::github_spot_funs(repo, branch) ). In this case I first read in an index file so as to limit parsing to only those files that are used directly as chapters – see R/save-examples.R for details. Note too that R4DS is currently under an overhaul with version 2 coming-out, so the index I use doesn’t intuitively line-up with every chapter currently in the book.

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/r4ds-chapter-files-funs-20220117.csv") %>% 
  distinct(funs, .keep_all = TRUE) %>% 
  select(-in_multiple_pkgs) %>% 
  DT::datatable(rownames = FALSE,
              class = 'cell-border stripe',
              filter = 'top',
              escape = FALSE,
              options = list(pageLength = 10))

Bryan Shalloway Blog

Also posted table to google drive here.

Any custom functions will most likely be assigned pkgs = "(unknown)":

readr::read_csv("https://raw.githubusercontent.com/brshallo/funspotr-examples/main/data/funs/brshallo-blog-funs-20220114.csv") %>% 
  distinct() %>% 
  DT::datatable(rownames = FALSE,
              class = 'cell-border stripe',
              filter = 'top',
              escape = FALSE,
              options = list(pageLength = 10))

funspotr could also be used for R code analysis – e.g. review most frequent functions, changes in trends, etc.↩︎
Parsing all the files takes a few minutes, so was easier to split apart from blog and package.↩︎
Mostly just used funspotr::github_spot_funs()↩︎
Rather than each instance of them as is posted on funspotr-examples↩︎
Or at least across the files used for documenting the chapters. Showing unique functins across all files makes more sense in this case compared to the others because rather than each file being truly self-contained the files together make-up a collective.↩︎

Posts on Bryan Shalloway's Blog

Identifying the Owners of Unclaimed Property

Statistical Distributions of Shooting Drills

Scenario 1: difference of binomial distributions

Scenario 2: difference in negative binomial distributions

Which game should you play if you’re better?

Appendix

Hypothesis tests on observational data

How You Should Keep Score in Pickup Basketball

What if you’re playing make-it-take-it?

Summary

Taking action

Other considerations

Appendix

But you don’t play pickup basketball to infinity

Table of imbalances

Finite geometric series

Scoring system in pickup basketball

Untangling Nassim Taleb's Criticism of that Headline Grabbing Intermittent Fasting Study

Nassim Taleb’s comments

“sigma change that doubles incidence”

Wrapping-up

Aggregating Measures of Uncertainty

Odds Are You're Using Probabilities to Describe Event Outcomes

When odds are helpful

Ratio of odds (odds ratio)

Communicating odds

Appendix

Ratios and fractions of events

Many variables

Other problems

Converting Between Currencies Using priceR

Update

Simulate data

Create rates lookup table

Function to convert prices

Convert Prices

Appendix

Pulling Twitter Engagements Using the v2 API as Well as rtweet

GETting all engagements

GET favorited users

GET all tweets from user

GET references

GET favoriters

GET quoters

GET retweeters

GET repliers and mentions

Putting them together into a function

R Access to Twitter's V2 API

Network Visualizations of Code Collections (funspotr part 3)

Interactive network plots

Julia Silge Blog

David Robinson Tidy Tuesday

R for Data Science Chapters

My blog

My gists

Identifying R Functions & Packages in Github Gists (funspotr part 2)

Parsing code

Parsing R files

Parsing markdown files

Binding files together

Organizing snippets

Appendix

Identifying R Functions & Packages Used in GitHub Repos (funspotr part 1)

Documenting rstats posts

Examples

Julia Silge Blog

David Robinson Tidy Tuesday

R for Data Science Chapters

Bryan Shalloway Blog