This post is part two in a series of posts introducing funspotr. See also:
- Identifying R Functions & Packages Used in GitHub Repos (funspotr part 2)
- Network plots of code collections (funspotr part 3)
This post shows how funspotr can also be applied to parse gists:
By functions or packages used…?https://t.co/kbSLOpQZLF
— Bryan Shalloway (@brshallo) January 22, 2022
A problem I bumped into was that most of Chelsea’s gists don’t actually have .R or .Rmd extensions so my approach skipped most of her snippets. I wanted to parse my own gists but ran into a related problem that most of my github gist code snippets are saved as .md files1 so knitr::purl()
won’t work2.
In this post I…
- create a function to extract code chunks from simple .md files
- parse the functions and packages in my code using funspotr3.
Parsing code
First I used funspotr to get a table with all of my gists.
library(dplyr)
library(purrr)
library(stringr)
library(funspotr)
library(dplyr)
library(purrr)
library(stringr)
library(funspotr)
brshallo_gists <- funspotr::github_gists("brshallo")
brshallo_gists
## # A tibble: 97 x 2
## contents urls
## <chr> <chr>
## 1 funspotr-gists-cmparlettpelleriti-ex.R https://gist.githubusercontent.com~
## 2 custom-ggplot-and-labels.R https://gist.githubusercontent.com~
## 3 stratified-sampling-parameter-estimates.R https://gist.githubusercontent.com~
## 4 grouped-nested-t-test.md https://gist.githubusercontent.com~
## 5 benchmark-cdf-methods.md https://gist.githubusercontent.com~
## 6 split-group-nest-join.md https://gist.githubusercontent.com~
## 7 weighted-t-test-tidied.md https://gist.githubusercontent.com~
## 8 cdf_density.R https://gist.githubusercontent.com~
## 9 if_all-if_any-examples.R https://gist.githubusercontent.com~
## 10 weighted-grouped-bootstrap-simulation.md https://gist.githubusercontent.com~
## # ... with 87 more rows
Parsing R files
funspotr is already set-up to parse all the unique functions and packages from R or Rmd files.
r_gists <- brshallo_gists %>%
filter(funspotr:::str_detect_r_rmd(contents))
r_gists_parsed <- funspotr::github_spot_funs(custom_urls = r_gists)
r_gists_unnested <- r_gists_parsed %>%
funspotr::unnest_github_results()
r_gists_unnested
## # A tibble: 474 x 5
## funs pkgs in_multiple_pkgs contents urls
## <chr> <chr> <lgl> <chr> <chr>
## 1 library base FALSE funspotr-gis~ https://gist.~
## 2 github_gists funspotr FALSE funspotr-gis~ https://gist.~
## 3 filter dplyr TRUE funspotr-gis~ https://gist.~
## 4 str_detect_r_rmd (unknown) FALSE funspotr-gis~ https://gist.~
## 5 github_spot_funs funspotr FALSE funspotr-gis~ https://gist.~
## 6 unnest_github_results funspotr FALSE funspotr-gis~ https://gist.~
## 7 library base FALSE custom-ggplo~ https://gist.~
## 8 ggplot ggplot FALSE custom-ggplo~ https://gist.~
## 9 aes ggplot FALSE custom-ggplo~ https://gist.~
## 10 geom_point ggplot FALSE custom-ggplo~ https://gist.~
## # ... with 464 more rows
Parsing markdown files
To parse my .md files, I wrote a function here extract_code_md()
that…
- reads in a file
- extracts the text in code chunks4
- saves it to a temporary file
- returns the file path of the temporary file
subset_even <- function(x) x[!seq_along(x) %% 2]
extract_code_md <- function(file_path){
lines <- readr::read_file(file_path) %>%
stringr::str_split("```.*", simplify = TRUE) %>%
subset_even() %>%
stringr::str_flatten("\n## new chunk \n")
file_output <- tempfile(fileext = ".R")
writeLines(lines, file_output)
file_output
}
I map extract_code_md()
on all the .md gists and then parse the files using funspotr.
# display output was weird here so just copied code and set eval = FALSE
md_gists <- brshallo_gists %>%
filter(!funspotr:::str_detect_r_rmd(contents))
md_gists_local <- md_gists %>%
# name urls because that's what funspotr::githup_spot_funs() expects
mutate(urls = map_chr(urls, extract_code_md))
md_gists_parsed <- funspotr::github_spot_funs(custom_urls = md_gists_local)
md_gists_unnested <- md_gists_parsed %>%
funspotr::unnest_github_results()
md_gists_unnested
## # A tibble: 1,061 x 5
## funs pkgs in_multiple_pkgs contents urls
## <chr> <chr> <lgl> <chr> <chr>
## 1 library base FALSE grouped-nested-t-test.md "C:\\Users\~
## 2 require base FALSE grouped-nested-t-test.md "C:\\Users\~
## 3 install_github remotes FALSE grouped-nested-t-test.md "C:\\Users\~
## 4 na.omit stats FALSE grouped-nested-t-test.md "C:\\Users\~
## 5 t.test stats FALSE grouped-nested-t-test.md "C:\\Users\~
## 6 tidy broom FALSE grouped-nested-t-test.md "C:\\Users\~
## 7 pull dplyr FALSE grouped-nested-t-test.md "C:\\Users\~
## 8 group_by dplyr FALSE grouped-nested-t-test.md "C:\\Users\~
## 9 summarise dplyr FALSE grouped-nested-t-test.md "C:\\Users\~
## 10 list base FALSE grouped-nested-t-test.md "C:\\Users\~
## # ... with 1,051 more rows
Note that I’m assuming all the code snippets are R code5.
Binding files together
I bind these files together and then arrange them based on the initial order in brshallo_gists
6.
gists_unnested <- bind_rows(
r_gists_unnested,
md_gists_unnested
) %>%
# got this arranging by a vector trick from SO:
# https://stackoverflow.com/questions/52216341/how-to-sort-rows-of-a-data-frame-based-on-a-vector-using-dplyr-pipe
arrange(match(contents, brshallo_gists$contents)) %>%
# add back-in links to url's where files are rather than urls column being
# local paths for .md snippets
select(-urls) %>%
left_join(brshallo_gists, by = "contents")
gists_unnested %>%
DT::datatable(rownames = FALSE,
class = 'cell-border stripe',
filter = 'top',
escape = FALSE,
options = list(pageLength = 20))
Organizing snippets
Perhaps I’ll do a follow-up and show some ways the relationships between the resulting parsed code snippets may be visualized in a network or organized in some other way.
Mentioned in the initial thread, Obsidian seems to be a product that does some things along these lines:
I’ve found this useful (https://t.co/OYzwfTltLG) I is a tool for writing, organizing, linking markdown files.
— John Lee (@Jdlee888) January 21, 2022
Appendix
Interactively save current gists to folder so can read from another file if want to
post_path <- fs::path_dir(rstudioapi::getSourceEditorContext()$path)
fs::dir_create(post_path, "data")
readr::write_csv(gists_unnested, fs::path(post_path, "data", paste0("brshallo-gists-", format(Sys.Date(), "%Y%m%d"), ".csv")))
knitr::purl()
is used in functions within funspotr to parse R markdown files.↩︎In the future I may do a follow-up that passes the parsed functions and packages through a network analysis or some other approach to better visualize the relationships between code snippets.↩︎
based on what exists between ticks. Kind of like a less reliable version of
knitr::purl()
but for .md files. Also posted function on SO question.↩︎Otherwise the R code parsing steps in funspotr will fail.↩︎
Note that this will just return the unique functions in each file, if I want to see every time I used a function I would have passed in
show_each_use = FALSE
togithub_spot_funs()
.↩︎