~ jim

Six months of Bandcamp’s Jazz-ish Jazz Club

Jim Gardner — Wed, 04 Feb 2026 00:00:00 GMT

Last Autumn Bandcamp launched Clubs:

[Clubs are a] subscribe-to-own music discovery experience, curated by trusted experts and rooted in community. Inspired by old-school record clubs and online spaces where passionate fans come together to share and explore music, bringing together human curation with community connection.

They started with a handful of clubs, and I joined Jazz-ish Jazz, curated and hosted by the absolutely delightful DJ, broadcaster, and music journalist Tina Edwards. For the price of an album each month, I get a record hand picked by Tina, a listening party (with Tina and usually the artist/s or re-issuer), a written interview, and a community page of sorts where members can leave comments.

General praise for Bandcamp

These club albums are treated in exactly the same way as a normal purchase on Bandcamp. A “normal purchase” that will surprise many in the Spotify age, especially young millenials downward.

You get to download the music. You own it. No DRM, no time or download limits, no ads. You can also stream it if you like. Oh, and the artist receives something like 75% or more of the sales revenue.

If you care about the artist you are buying from, and can afford it, I don’t know why you wouldn’t use Bandcamp over Spotify or Apple Music. It is clearly the more ethical choice at the time I write this, especially as they have just banned AI-generated music in a bid to keep the platform a celebration of human artistry and connection. I’ve been using Bandcamp for nearly 15 years, and all I can say is, fuck yeah.

Why jazz

Anyway. I joined the jazz club to broaden my horizons in a genre that’s always tantalised me, but with which I’ve only ever flirted. There’s an enormous amount of variety within jazz, so for a record club it seemed like the perfect choice.

Reflections on the first six months

It’s been fantastic. A truly wonderful mixture of records. I’ll give each a mini-review below, but first want to talk about the listening parties.

Listening parties

I didn’t expect to enjoy them so much. It’s nice to see some of the same names each month, hang out with Tina and the artists, and generally be a bit silly. But every month is also a profound opportunity. These are incredible musicians… and they’re just there. Answering our questions. Enjoying the feedback. It’s so human, and not an opportunity you often get, let alone every month.

It can be touching, too. Recently I was away from home for a funeral, with extended family, and it was of course quite a sad and discombobulating few days. I’m autistic, so identifying my feelings and expressing them around something like a funeral can be challenging, to say the least. The evening before the funeral I went upstairs to the little guest room, put my headphones on, logged into the listening party, and had the most incredible experience across the 45 minutes.

The emotional fullness turned my body into what felt like an electrified pool of jelly. Partly because of the incredible music, and partly because of how I was experiencing it - the commensality. The artists were in a great mood, joking around and sharing their process, clearly appreciating our comments. Gratitude was flowing back and forth. It was a special experience.

They’re not always so intense. They have all been fun, though, and the focus it brings to your listening, the way people point out stuff you didn’t - or that you did! - consider, or enjoy, or find curiosity in, is very stimulating.

Album mini-reviews

In chronological order of receipt. Click the album title to go to the Bandcamp page.

I’ve put together a playlist on Bandcamp, made up of my favourite track from each album. Bandcamp account holders can listen to each track up to 3 times.

Orbits, by The Circling Sun (2025)

This is exactly the kind of album that I wouldn’t pick up of my own accord. It has a 70s vibe, despite being contemporary, and prominently features the cuíca (a Brazilian friction drum that sounds like a monkey), which I love. In some ways it’s a utopian album, full of samba charm. The opening track Constellation includes a choir singing, “Freedoooom, freedoooom!” over warm brass and synths. It evokes the optimism of a late spring train journey from Inverness to Kyle of Lochalsh, sparkling still water passing by the window. Good times ahead.

Favourite track: Flying. I can imagine the USA-1950s-retro-futurism-feminist music video for this in my mind’s eye. It takes place in a shopping mall. And involves a baseball bat.

The Prophet and The Madman, by Ami Taf Ra (2025)

Another that I wouldn’t have picked myself. It’s hard to describe! In her debut album Ami draws on her Arab North African roots, fusion jazz, and as the title suggests, lyrically and thematically from author Khalil Gibran. She sings mostly in English, but also classical Arabic, and the Moroccan dialect on Gnawa, which is a catchy, strutting-on-tiptoes affair during verses, sliding into almost-woozy group sung (chanted?) choruses. The following track, Gibran, is a haunting piano-led affair, mysterious, transporting you directly into a dark, comfortable, hazy jazz club. The solos on this album are all masterful and build the tracks up.

It’s a heady record, a lot to take in.

Favourite track: Love (feat. Ryan Porter). Almost sickly sweet, but it sends shivers down my legs every time. One to slow dance with your sweetheart:

Desire love to be like a running brook that sings to the night
To know the pain of too much tenderness
To be wounded by your own understanding
Willingly to bleed joyfully
To wake at dawn with a winged heart
To give thanks to another day of love
To return home with gratitude
To sleep with a prayer for your beloved
In your heart, a song of praise
To rest upon your lips
Speak to us of love

~~I’m not crying you’re crying~~

Jazz Report from the Hood, by Omasta (2025)

An album from a Krakow outfit, whose band name means “fat added to food for flavour”. I love that. It perfectly encapsulates the oily-thick feel of this hip-hop inspired jazz report. All the instruments used during recording were vintage, and the mixing and mastering was done on analogue equipment, so the sound is warm and deep as home made roasted chicken broth. The record is intended to be listened to as a whole piece, and it works. To me, it’s reminiscent of a particularly long and slightly scary Friday evening out drinking in my early 20s. It has confidence, cool, curiosity, danger, maybe some scraps and scrapes, before a slightly hyper return home as the sun comes up.

Special shout out to Antoni (soprano sax, flute), who had pneumonia while recording the album. Blimey.

Favourite track: Kazimierz. The closing track is just absolute cool sunglasses-wearing alternative hip-hop personified. The trumpet hook is sick, the guys deliver energetic solo after energetic solo, and the drum kit throughout is tightly yet expressively played, driving throughout. There’s a KILLER beat skip 35 seconds before the end of the track.

Sanctuary, by Levitation Orchestra (2025)

This is the album that drew such a strong emotional reaction out of me during the listening party. At times intensely warm and comforting, like a chocolatey dark roast cortado, elsewhere as electrifying and heady as the best jRPG soundtracks. All delivered with exceptionally tight playing and a cohesive feel throughout, despite slow burners being mixed in with livelier tracks.

For a while, I was telling all my friends to listen to Embrace, because it has some of the most incredible voice work (by the mononymous Plumm), almost reminiscent of Rajasthani folk music. Quite impressive. Then on the next track, Breathe It In, the piano absolutely dominates. It sounds like the Spirited Away OST meets the “Zelda’s Awakening” memory from Zelda: Breath of the Wild, with a dash of the saxophone from the final Zelda: Tears of the Kingdom trailer. Restrained drama, resolve in the face of darkness. Following that, Home acts as a gorgeous guided meditation, led by a male voice, “You know me / Within and without / In your embrace / I am safe to be”. The penultimate track, Premedetatio Malorum, sounds like it was made by three goblins wearing shades and trenchcoats, an assessment that went down well in the listening party: One band member said “Definitely goblin energy in the room when we came up with this one”, and another, “Where were you when we were naming these!?”.

Favourite track: Another Way. The closing track, which comes in at 9 minutes, has the best sax harmony I’ve ever heard. It’s also… scary. It illicts a strange fear in me, a fear that somehow thickens into defiance, before releasing into hopefulness.

Denga, by Robin Jones Quintet (1971)

A reissue (2022), this early 70s banger is best described by its Bandcamp page: “a scintillating fusion of Afro-Cuban and Afro-Brazillian rhythms laden with heavy Fender Rhodes sounds and no less than three Afro-Latin Percussionists”. The record still sounds fresh, with an absolutely superb mix. All the percussionists are spaced out such that you can hear them distinctively doing their thing. And what a thing they do! It makes you want to move. I joked in the listening party that El Sueno is a Latin James Bond theme, but I really do think there’s something to it! It wouldn’t be out of place in Series 2 of The Night Manager. Overall, this is probably the most crowd pleasing album of the club. I can’t imagine someone taking a dislike to it. Several of the tracks are a blast. Robin’s albums are hard to find, but luckily the reissuer has another, EL MAJA, also up on Bandcamp.

Favourite track: Goodbye Batucada. This one would pair really well with Flying, from Orbits, as both share the same retro-futurist shopping mall vibe. Incredibly upbeat, almost glassy-smiled.

Istanbull Sessions: Mahalle, by Ilhan Ersahin (2026)

I’ve only had a few days with this one, but wow. This album is a celebration of Istanbul’s diverse neighbourhoods, and is described as cinematic, which it certainly is. The record is quite rocky, and grungey, with powerful, almost forceful drumming at times, such as on the opener Yedi Tepe. I’m struck by Ersahin’s sax playing, which can be pretty wild, and has some seductive Turkish-sounding solos (like on Asmalı) that almost twist the ear into hearing a bass clarinet or such. And yet, the restraint shown on Galata, the soft playing, the dedication to the riff, it’s delightful, as is the entire song, which has an 80s flavour that arrived to me through early home video games, suddenly dropping into fully Nirvana-esque grunge at various points. Absolutely brilliant.

Favourite track: Galata. I just love that riff. The whole track sounds the way a vintage game cover looked, think Battletoads, Streets of Rage, night time, sunglasses.

CPD for July 2025

Jim Gardner — Thu, 10 Jul 2025 23:00:00 GMT

My team (Data Science) at Smart Data Foundry have started to spend an hour of CPD time together once a week. July is our first month doing it, and I figured I would use it as an excuse to blog a little bit.

2025-07-10

Advent of Code 2024 Day 1, Part 1

I did AoC Day 1 last year and challenged myself to only use base R. We figured this would be a nice little problem to start our sessions with.

This time, I wanted to try using Python. I’m kinda enjoying the language now, and between boot.dev and work am using it more than R. But! Instead of the standard library, I wanted to try using Polars and compare the solution to my base R one. Recently I’ve used Polars a fair bit for work but the operations I needed have been very simple. My spidey senses told me that if I used Polars then I’d end up learning something new.

I also used uv, the hot new Python project/package manager, to start the project off:

uv init
uv venv
source venv/bin/activate
uv add ruff
uv add polars

That creates a few files, including a uv.lock and pyproject.toml. These essentially contain metadata such as the Python and package versions. The files can be copied into another project, or a container image, to install the same version of Python and packages with a simple uv sync.

Solution

import polars as pl


df = (
    pl.read_csv("aoc_day1/input.txt", has_header=False)
    .with_columns(
        column_1=pl.col("column_1")
        .str.split_exact("   ", 1)
        .struct.rename_fields(["l", "r"])
    )
    .unnest("column_1")
)

df = df.with_columns(
    l=pl.col("l").cast(pl.Int32).sort(),
    r=pl.col("r").cast(pl.Int32).sort(),
)

res = df.select((pl.col("r") - pl.col("l")).abs().sum()).item()

print(res)

Comparing to the R version of the code:

df <- read.table("2024/01-input.txt", col.names = c("l", "r"))

df$l <- sort(df$l)
df$r <- sort(df$r)

sum(abs(df$r - df$l))

It’s more verbose. Most of that comes down to fundamental differences between base R and Polars. In this case - which is common - the R solution uses functions. We also benefit from data.frames being native data structures, and read.table which does two things for us: identifies the delimiter (three whitespace characters), and infers the types correctly (numeric).

The Polars solution is fiddlier. The csv parser can only take a single byte delimiter, meaning no regex, no multiple whitespace. I had to look this up, because it seemed frankly idiotic for a modern data processing package, and read the author saying they did this on purpose: It puts speed before anything else, and speed is at the heart of Polars. Doing anything else would go against that. Not so idiotic after all!!!

Anyway this leads to extra processing because the table gets read as a single string column. When it gets split, it becomes a Polars struct. I didn’t know what a struct actually was, though I saw the name appear in the user guide as I hacked on projects for work. Turns out they’re essentially a typed dictionary.

str.split_exact() returns “a struct of n+1 fields”. So the result of

split_exact("   ", 1)

on a row of data like 50123 10023 in a column called column_1 is

{"column_1": "50123", "column_2": "10023"}

Every row gets a dict like that, which taken together are the values in the struct.

Calling unnest() splits a struct into columns. The column names come from whatever the fields are called.

So, all in all, it’s a little bit like an R list-column situation.

Once the columns were unnested they were still string and have to be cast to the correct integer type. And then finally we get to do the calculation, which also looks way more verbose to me. One benefit of chaining methods is that you get to see what happens sequentially. That’s not the case with nested function calls, like in the R code. I don’t like the amount of noise in having to refer to columns with pl.col("l"), but it’s not that bad in this case to be honest.

2025-07-17

Advent of Code 2024 Day 1, Part 2

Same deal as last week, but part two of the problem.

Solution

# df was defined in the week 1 section

counts = df.select("r").to_series().value_counts()

joined = df.join(counts, how="inner", left_on="l", right_on="r")

joined = joined.with_columns(multiplied=pl.col("count") * pl.col("l"))

summed = joined.select("multiplied").sum().item()

print(f"Part 2 result: {summed}")

Comparing to the R version:

# df already defined in the week 1 section

freq <- as.data.frame(table(df$r[df$r %in% df$l]))
names(freq) <- c("l", "f")

df <- merge(df, freq, by = "l", all.x = TRUE, sort = FALSE)

sum(df$l * df$f, na.rm = TRUE)

It’s a similar approach: Make a frequency table of the values in the right-hand column, join those values onto the original table, multiply the left-hand values against the frequency values and sum up the results.

I couldn’t remember what I did in the R version, but my mind went to almost exactly the same places. In both versions I started by making a frequency table, which was trivial. However, in the R version I first filtered the values in the right-hand column by those of the left-hand column, so that the frequency table would be much smaller (there were only 37 shared values between the columns in my dataset). This seemed more efficient for further steps because with a smaller dataframe there will be fewer comparisons to make.

This was what I wanted to do in the Polars version, too, but it was fiddly. There might be a better way of doing this, but my code ended up looking like:

mask = df.select("r").to_series().is_in(df.select("l").to_series())
counts = df.select("r").to_series().filter(mask).value_counts()

That’s not terrible, but to me the R way of subsetting a vector looks far more elegant. Once again Polars is more verbose; I get lost trying to read the repetition of select and to_series and the brackets. There is a df.to_series() method, so the code could be streamlined slightly, but it takes a column’s position as it’s arg, rather than a name. I stand resolute and refuse to do df.to_series(0). That… is opaque bullshit!

Instead, I did an inner join between the two DataFrames. An inner join is a “filtering join”, where the left-hand df gets filtered by the contents of the right-hand df. And in the R version I did the same thing using merge().

At this juncture of writing the blog post something quite nice happened. I realise that the R code was doing some redundant things. Because merge() does an inner join by default, there was no need to specify all.x = TRUE. In fact, it causes the opposite behaviour of what is needed: It’s like saying “Yes, make sure you keep all the rows from the left-hand df, and if you don’t mind, add NAs whenever there was no matching row in the right-hand df!”. That’s why the subsequent sum requires na.rm = TRUE.

In my opinion the more elegant approach is the Polars version: Don’t bother filtering the frequency table first, simply allow the inner join to do its job and filter for you.

With those changes made, the R code is even slicker (isn’t it satisfying to delete things??):

freq <- as.data.frame(table(df$r))
names(freq) <- c("l", "f")

df <- merge(df, freq, by = "l", sort = FALSE)

sum(df$l * df$f)

All in all:

Humbling to look back at simple problems and try to solve them with a new tool
Satisfying to revisit concepts and end up streamlining old code
Interesting to observe what feels nicer to read as I look at the code. Never underestimate how important feel is to craft

Accessing R HTML help pages from outside a container

Jim Gardner — Thu, 08 May 2025 23:00:00 GMT

The radian R terminal running from inside a container, showing a custom startup message. To the side is a browser with the R HTML help page loaded, which is being served from the container.

Another challenge!

This is a follow-on about my attempt to make a container that can be used as an interactive Neovim development environment with R and Quarto.

Pete, who started me dancing down the devil’s road in the first place, had a follow-up request:

[…] is there a way to run R’s HTML help server in the container and view it in a browser on the host?

Sounds like that should be easy to do. After all, I was able to get the Quarto HTML preview and hdg graphics viewer for R sorted. I bet it’ll take 30 minutes or so!

How naive of me.

Problem / solution

For those who don’t know - and this included me before this week - you can launch a website containing all of the help pages for both R’s standard library (base, tools, utils, Matrix, etc.) and user installed packages. You can launch this website by using the help.start() function that comes with R. The more you know.

This is useful if, for example, you’re using R in a terminal setting, which as we established last time is what Pete does.

I quickly found out that the root cause of the problem is within the help.start() function body. Specifically, both help.start() and a function it calls, tools::startDynamicHelp(), have the IP address 127.0.0.1 hard coded into their definitions. This is a problem because HTTP servers running inside containers using 127.0.0.1 can’t be accessed from outside the container. Instead, you have to use 0.0.0.0. Thankfully, it is at least possible to tell R that you always want to use the same port when running the server.

A few hours later I hit a brick wall and posted about it on Mastodon. R legend Yihui Xie jumped in and replied, kindly telling us exactly what we needed to do. This was very much appreciated – I had clearly gone down a dead end. It gave me hope!

What Yihui turned us onto was a command line tool called socat. You can use it to create proxies and bind ports. In our case, the magic command looked like this:

socat TCP-LISTEN:20633,bind=0.0.0.0,fork TCP:127.0.0.1:20000 &

This means: Accept traffic (listen) on 0.0.0.0:20633, i.e. receive requests for web pages as we browse the site. Then pass the requests to 127.0.0.1:20000. Fork allows us to handle multiple requests. And the final ampersand tells bash to run the socat process in the background, otherwise socat will make the terminal busy.

To put this into practice I added some more R code to ~/.Rprofile during container build time. This is an important file because whenever R starts it runs the code in this file. That means we can use it for performing setup, such as getting socat to forward our traffic. The code looks like this:

#| eval: false
if (interactive()) {
  options('help.ports' = 20000)
  system('socat TCP-LISTEN:20633,bind=0.0.0.0,fork TCP:127.0.0.1:20000 &')

  cli::cli_h1('HTML help pages')
  cli::cli_alert_info('Starting httpd help server...')

  resp <- suppressMessages(tools::startDynamicHelp())

  if (resp != 20000) {
    cli::cli_alert_danger('Failed to start server. Try running {.fn tools::startDynamicHelp} manually to see what happens.')
    cli::cli_text('')
  } else {
    cli::cli_alert_info('If container is running with {.code -p 20633:20633}')
    cli::cli_alert_info('{.url http://localhost:20633/doc/html/index.html}')
    cli::cli_text('')
  }

  rm(resp)

  .Last <- function() {
    # Attempt to clean up after ourselves when R closes.
    # When R closes the help pages are no longer served, so we
    # should also stop socat.

    # A bit of a blunt instrument as it will terminal ALL instances
    # of socat. But I'm assuming for this container that R help will
    # be the sole user of the command, so it's safe to do
    x <- suppressWarnings(system('pgrep socat', intern = TRUE))
    if (length(x) >= 1L) {
      system('pgrep socat | xargs kill -9')
    }
  }
}

When starting R with the radian terminal, we get this lovely startup message:

The radian terminal immediately following boot up. Brief messages tell the user that the http server has been started for the HTML help pages.

As my comments say in the code block, this is a rather inelegant solution. There is the possibility of using the bash command $! in order to get the PID of the socat process ($! gets the PID of the most recent background process) and do a targeted kill. But I found that socat was a bit erratic, sometimes spawning a few processes. Unfortunately the second and third processes weren’t children of the first, otherwise we could have captured the value of $!, say as panret_pid and used pkill -P parent_pid. On the plus side I think this is a very clean solution, insofar as socat will always be started and killed in tandem with R.

How it’s going

Well, I wrote this blog post using Neovim from the container in question! That includes previewing the post :)

The Containerfile now exists in a GitHub repo. At time of writing there’s no documentation, but I will get to it soon. One thing I’ve noticed is that the container image size is rather large, about 2.7GB. Maybe that’s ok - there is a lot of stuff being installed into it, after all. It’s also very tiresome having the plugins install every time the container is restarted and Neovim gets invoked for the first time. I think it might be possible to simply copy everything from .local/share/nvim/lazy into the GitHub repo and copy it into the image at build time instead, but that requires some experimentation.

A containerised Neovim dev environment for R and Quarto

Jim Gardner — Wed, 30 Apr 2025 23:00:00 GMT

Neovim running from inside a container showing a Quarto document and radian REPL. A browser is viewing the live HTML preview being served by Quarto

The question

Over on Mastodon Pete Jones asked the following:

[…] I’ll jump straight to my noob questions which I’m hoping the #Rstats community can help me answer:

Does anyone have any good recommendations for primers/video tutorials etc that are particularly geared towards (social science) research uses of containers?

I can see how containers make sense for sharing your research environment at the end of a project for reproducibility, but how sensible is it to be doing all your iterating and exploratory stuff inside a container during the research? Do folks just do all that on their host system and then containerise it when they’re done?

What’s the advantage of using, say, Apptainer versus podman versus something like rix? There seems to be a lot of overlap so it’s hard to know which one to dive into. For context, my coding work is almost entirely in the form of interactive sessions throwing stuff at the REPL and making data visualisations etc, and rarely a scripts-as-jobs type of workflow. In other words, I want to be sat inside a command line environment writing and evaluating code on the fly.

My response was basically that it’s straightforward-ish to use an image with RStudio Server, or VS Code. That’s how I work these days, barely touching the system installation of R.

It turns out that Pete uses Neovim as his IDE, so those images aren’t that useful. Thankfully, he recently wrote a nice blog post specifically about using Neovim for interactive R and Quarto work. Reading over that post again, and looking at his nvim config files, the desire to prove it was possible took me over.

A couple of days later I’m happy to say it is possible. The rest of this post is dedicated to explaining how.

Terminology

Containers aren’t tied to Docker, like most people think. There’s an open source alternative called Podman, and that’s what I used to build and run the container in question. Podman is essentially a drop-in replacement for Docker.

I also refer to the recipe for the container image not as a Dockerfile, but a Containerfile.

The Containerfile explanation

Here it is in full.

One note to mention is that I haven’t consistently pinned versions throughout. That would definitely be worthwhile, but it takes a bit of time and I don’t have it right now.

From Pete’s config, I knew he would need Python (because the radian R REPL is written in it), R, Quarto, and of course, Neovim. You can see that installing and configuring these different tools happen in short blocks, most of which are heredocs. It’s fine to have small scripts that get copied in and run instead, but I used heredocs so that everything is together in the same file.

FROM docker.io/library/python:3.13-bookworm

# Python setup. Makes a venv and sets it as the default Python interpreter
ENV VIRTUAL_ENV=/root/.venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install radian, a Julia-inspired R REPL written in Python
RUN pip install -U radian

ENV R_VERSION="4.5.0"
ENV QUARTO_VERSION="1.7.29"

RUN <
# Install R
# Taken from https://github.com/r-lib/rig?tab=readme-ov-file#installing-rig-on-linux-

# Get their certificate
curl -L https://rig.r-pkg.org/deb/rig.gpg -o /etc/apt/trusted.gpg.d/rig.gpg
sh -c 'echo "deb http://rig.r-pkg.org/deb rig main" > /etc/apt/sources.list.d/rig.list'
apt-get update

# Now rig is available to install
apt-get install r-rig

# Install this version of R
rig add ${R_VERSION}

echo "options(repos = c(CRAN = 'https://packagemanager.posit.co/cran/__linux__/bookworm/latest'))\n\n" >> ~/.Rprofile
Rscript -e "options(warn = 2); pak::pak(c('knitr', 'rmarkdown', 'languageserver', 'nx10/httpgd'))"
EOF

RUN <
# Set up the R graphics driver
echo "if (interactive()) {\n
  if ('httpgd' %in% .packages(all.available = TRUE)) {\n
    options(device = function(...) {\n
      httpgd::hgd(host = '0.0.0.0', port = 9001, token = FALSE)\n
    })\n
  }\n
}\n\n" >> ~/.Rprofile
EOF

RUN <
# Install Quarto.
# Slightly modified from 
# https://docs.posit.co/resources/install-quarto.html

mkdir -p /opt/quarto/${QUARTO_VERSION}

curl -o quarto.tar.gz -L \
    "https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.tar.gz"

tar -zxvf quarto.tar.gz \
    -C "/opt/quarto/${QUARTO_VERSION}" \
    --strip-components=1

rm quarto.tar.gz

# Symlink so that `quarto` can be used as a command
ln -s /opt/quarto/${QUARTO_VERSION}/bin/quarto /usr/local/bin/quarto

# Verify successful installation
quarto check
EOF

# deno is required for one of the plugins
COPY --from=denoland/deno:bin /deno /usr/local/bin/deno

RUN <
# Neovim plugin system deps

apt-get update
apt-get install -y --no-install-recommends ripgrep fd-find

# Instructions for fd from: https://github.com/sharkdp/fd?tab=readme-ov-file#on-debian
mkdir -p ~/.local/bin/fd
ln -s $(which fdfind) ~/.local/bin/fd
echo 'export PATH="$PATH:/$HOME/.local/bin"' >> ~/.bashrc
EOF

RUN <
# Get Neovim config

mkdir -p ~/.config/nvim
curl -o config.tar.gz -L "https://codeberg.org/pjphd/neovim_config/archive/main.tar.gz"
tar -zxvf config.tar.gz -C ~/.config/nvim --strip-components=1

rm config.tar.gz
EOF

RUN <
# Install Neovim
# Taken from https://github.com/neovim/neovim/blob/master/INSTALL.md#pre-built-archives-2

curl -LO https://github.com/neovim/neovim/releases/latest/download/nvim-linux-x86_64.tar.gz
rm -rf /opt/nvim
tar -C /opt -xzf nvim-linux-x86_64.tar.gz
echo 'export PATH="$PATH:/opt/nvim-linux-x86_64/bin"' >> ~/.bashrc

rm nvim-linux-x86_64.tar.gz
EOF

# Tidy up apt
RUN apt clean && rm -rf /var/lib/apt/lists/*

CMD /bin/bash

Python

FROM docker.io/library/python:3.13-bookworm

# Python setup. Makes a venv and sets it as the default Python interpreter
ENV VIRTUAL_ENV=/root/.venv
RUN python -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# Install radian, a Julia-inspired R REPL written in Python
RUN pip install -U radian

I used an “official” Python base image with version 3.13, and Debian Bookworm as the Linux distro. At time of writing bookworm is the latest release of Debian.

Python no longer allows you to install packages into the system version of Python. I think that’s fine - virtual environments are great (I wish R’s ecosystem was better in this regard). So here, I make a venv and add that version of Python to the container’s PATH. This means when you invoke python or pip it uses the venv. If you don’t take the step of adding it to the PATH your subsequent RUN commands, e.g. pip installing requirements, will use the system Python installation. Not a lot of people know this so I thought it was worth pointing out. Handily, this also means you never need to source /venv/bin/activate.

R

RUN <
# Install R
# Taken from https://github.com/r-lib/rig?tab=readme-ov-file#installing-rig-on-linux-

# Get their certificate
curl -L https://rig.r-pkg.org/deb/rig.gpg -o /etc/apt/trusted.gpg.d/rig.gpg
sh -c 'echo "deb http://rig.r-pkg.org/deb rig main" > /etc/apt/sources.list.d/rig.list'
apt-get update

# Now rig is available to install
apt-get install r-rig

# Install this version of R
rig add ${R_VERSION}

echo "options(repos = c(CRAN = 'https://packagemanager.posit.co/cran/__linux__/bookworm/latest'))\n\n" >> ~/.Rprofile
Rscript -e "options(warn = 2); pak::pak(c('knitr', 'rmarkdown', 'languageserver', 'nx10/httpgd'))"
EOF

RUN <
# Set up the R graphics driver
echo "if (interactive()) {\n
  if ('httpgd' %in% .packages(all.available = TRUE)) {\n
    options(device = function(...) {\n
      httpgd::hgd(host = '0.0.0.0', port = 9001, token = FALSE)\n
    })\n
  }\n
}\n\n" >> ~/.Rprofile
EOF

I use rig to install R. Then, I set the CRAN repo to the Posit Public Package Manager, specifically the version with binaries for Bookworm. This means the packages don’t need to build from source so the installation process is nice and fast.

Rig comes with pak, a package installation manager for R. This fantastic package can find and install Linux system dependencies for R packages, which is a real blessing when making your own container. Here I just install the packages I know are required for Quarto (knitr, rmarkdown) and Neovim (languageserver, httpgd).

Speaking of httpgd, I add some configuration to the Rprofile for this. The purpose of this package is to offer a graphics device, for viewing plots and HTML. Basically, the browser on your computer can be used to view images provided by httpgd while it runs inside the container.

Quarto

RUN <
# Install Quarto.
# Slightly modified from 
# https://docs.posit.co/resources/install-quarto.html

mkdir -p /opt/quarto/${QUARTO_VERSION}

curl -o quarto.tar.gz -L \
    "https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.tar.gz"

tar -zxvf quarto.tar.gz \
    -C "/opt/quarto/${QUARTO_VERSION}" \
    --strip-components=1

rm quarto.tar.gz

# Symlink so that `quarto` can be used as a command
ln -s /opt/quarto/${QUARTO_VERSION}/bin/quarto /usr/local/bin/quarto

# Verify successful installation
quarto check
EOF

There’s nothing particularly special going on here. I basically just followed Posit’s instructions.

Neovim dependencies

# deno is required for one of the plugins
COPY --from=denoland/deno:bin /deno /usr/local/bin/deno

RUN <
# Neovim plugin system deps

apt-get update
apt-get install -y --no-install-recommends ripgrep fd-find

# Instructions for fd from: https://github.com/sharkdp/fd?tab=readme-ov-file#on-debian
mkdir -p ~/.local/bin/fd
ln -s $(which fdfind) ~/.local/bin/fd
echo 'export PATH="$PATH:/$HOME/.local/bin"' >> ~/.bashrc
EOF

deno is a system dep of peek.nvim, and thankfully they had straightforward instructions that allowed me to copy the binary from one of their own container images.

Pretty much everyone using Neovim will use a plugin called Telescope, which requires fd, and I think ripgrep too.

Neovim config

RUN <
# Get Neovim config

mkdir -p ~/.config/nvim
curl -o config.tar.gz -L "https://codeberg.org/pjphd/neovim_config/archive/main.tar.gz"
tar -zxvf config.tar.gz -C ~/.config/nvim --strip-components=1

rm config.tar.gz
EOF

This grabs Pete’s config files from Codeberg and puts them into the expected place in the file system.

Neovim

RUN <
# Install Neovim
# Taken from https://github.com/neovim/neovim/blob/master/INSTALL.md#pre-built-archives-2

curl -LO https://github.com/neovim/neovim/releases/latest/download/nvim-linux-x86_64.tar.gz
rm -rf /opt/nvim
tar -C /opt -xzf nvim-linux-x86_64.tar.gz
echo 'export PATH="$PATH:/opt/nvim-linux-x86_64/bin"' >> ~/.bashrc

rm nvim-linux-x86_64.tar.gz
EOF

Again, pretty straightforward - followed instructions provided by the Neovim maintainers.

Tidy and cmd

# Tidy up apt
RUN apt clean && rm -rf /var/lib/apt/lists/*

CMD /bin/bash

The lists of available packages are just bloat at this point, so I get rid of them. And I add a new cmd, which is what gets run when the container is booted up. In this case it’s just bash, so that the user gets a flexible launch/base point to work from.

Build the container

This bit is really simple. In a terminal cd to the directory where the Containerfile lives. Then run:

podman build -t pete . --arch=linux/amd64

It will take a minute or two depending on your internet and processing power.

pete is the name I’ve given the image. The . just means ‘look in the current directory for the Containerfile’. The --arch=linux/amd64 is to “set the architecture of the image to be built, and that of the base image to be pulled, if the build uses one, to the provided value instead of using the architecture of the build host”. I’m on an arm64 Macbook, which is the build host, but most of the things we’re installing into the image are explicitly the amd64 versions.

One thing to bear in mind is the “build context” when you build the image. All the files in the directory containing the Containerfile will get copied into the build context, so that they’ve available for the build. Podman will look like it’s frozen if you have large files in the build context. In reality, it’s copying them. You can either add them to a .Containerignore (aka .Dockerignore) or do the build in a different directory.

Run the container

Before running, I made a tiny Quarto file in the same directory as the Containerfile, called index.qmd:

---
title: "test"
---

## Quarto

Quarto enables you to weave together content and executable code into a finished document. To learn more about Quarto see .

```{r}
mtcars
```

```{r}
plot(mtcars)
```

This is when the magic happens.

In the terminal (from the directory with index.qmd) run:

podman run --rm -it -p 9000:9000 -p 9001:9001 -v ./:/test_project pete

The container will start. We’ve set it up to run with an interactive terminal, forwarding ports 9000 and 9001 from within the container to the same ports on the host machine, and also mounted the current directory to /test_project inside the container.

The ports will be used to preview the qmd file and for showing plots made during an interactive R session. The volume mounting means that we can edit the qmd file inside the container and the changes will persist on the host machine - in fact we can also make entirely new files in /test_project and they will appear on the host file system!

With the container running, start neovim:

nvim

It will take a moment to install all of the plugins and LSPs. When it’s done follow any prompts that come up.

Once finished, do :tabnew, then :terminal. Go into Insert mode and run:

quarto preview test_project/index.qmd --host 0.0.0.0 --port 9000

In your web browser, go to localhost:9000. You should see a live preview of the Quarto document.

Now, in the other nvim tab, let’s :e test_project/index.qmd. Go ahead and change something in the file, and :w. Look at the browser and the doc should update right in front of you.

For the cherry on top, let’s start a REPL at the same time and view a plot. With the config at current time of writing localleader is set to Space. Do rS, this should open a radian terminal. Now highlight the line of R code plot(mtcars) with Shift + V, then do r followed by Return. Radian should say that it’s running a httpgd server. Go to your browser and in a new tab/window go to localhost:9001/live. There’s your plot!

This shows how Neovim with a plot viewer and Quarto live preview, plus a ‘live link’ between the container and host file systems is possible.

Is it worth it?

The short answer is, I’m not sure.

R itself is quite mature and so are many of its most well-used packages, so developing on your system and then putting the finished project into a container is fine. Creating a dev environment with a lot of dependencies, like in this case, is fairly involved. It’s also annoying that the Neovim packages have to install every time you start the container, though there may be a solution to that I’ve yet to find.

At the same time, once you have a couple of Containerfiles, like this dev one, and another that encapsulates the final project, you’ve got most of what you need for future projects. It’s also amazing how quickly we can adapt to slightly different workflows like this. I think it’s really cool that Neovim can offer the REPL and native terminal. I’m also a believer that the ephemeral nature of containers forces us to be a lot better about capturing code, and making sure it runs from start to end. Too often people rely heavily on the REPL and are scared to restart their IDE in case their analysis no longer works, for some reason.

Two extra learnings

Two extras things I learned during this process:

Ever wondered about what has gone into a base image that you are using? As in, what did the Containerfile / Dockerfile have in it? That’s possible to get. First run podman image ls and note down the image id of the image you’re interested in. Then use podman history --no-trunc image_id, where image_id is (you guessed it) the image id. The output isn’t clean like a Containerfile - it contains timestamps and suchlike - and it is in reverse order. But it’s still a good one to know, particularly if you find yourself in a situation like I did while making this post: The Containerfile disappeared after a forced laptop restart. Ten minutes of tedious reformatting later, all was repaired.
How to close buffers. I used to open files in Neovim sometimes using the :e (alias of :edit) command. I knew that these buffers had to be closed somehow but never figured it out. It’s actually very simple. You can list all buffers with :buffers. Then you can do db #, where # is the number of the buffer to be closed.

Learning Neovim is Hard

Jim Gardner — Mon, 24 Mar 2025 00:00:00 GMT

Warning

This is NOT a place for you to learn anything. It is NOT a guide. It’s a “traditional” blog post where I waffle on and indulge myself. There may be some accidental nuggets of wisdom… but I cannot be held liable for any strain you experience trying to find them.

R script in Neovim running in Wezterm

Wouldn’t it be nice?

I wonder if this resonates with you? Ever read people’s LinkedIn profiles, or their blogs/websites/CVs, and think:

Wow. This person is absolutely incredible. They have achieved/know so much.

Wouldn’t it be nice if that person posted on their LinkedIn about the shitty fucking week they had trying to learn about containers for the first time? Or how that one time they had to work a weekend because they re-ran their Jupyter notebook from top to bottom and the results were completely different to what they showed their boss?

How about if they tried something and spent many days – weeks, months! – only to realise they just couldn’t understand it, and kinda like, gave up?

Well that’s me with Neovim.

I’m a mid-level programmer/Data Scientist. I’m now way beyond a total beginner, and there are actually people who look up to me as a professional. It feels unbelievable at times, but, that’s what happens when you work hard, learn, and care about the craft (and other people).

Read on to hear the struggle.

Obliterated

When I started programming in March 2020 I was working in evolutionary biology. I don’t know how it is now, but back then we were using a mix of proprietary GUI-based programs and command line tools written and published by academics. That meant my introduction to programming was the command line (specifically using bash), then Python, and then R. If I needed to write a script I’d come up with a little toy dataset, open the terminal, invoke Python/R, and write every line of code myself using the vanilla REPL.

That’s right. There was no

Syntax highlighting
Linting
Hovers
Keyboard shortcuts
Formatting

There wasn’t even ‘bracketed paste’, which as I understand it just means ‘multi-line code pasting into the terminal’. It was a literal line-by-line workflow where even a for-loop had to be input and sent to the interpreter in parts.

At first I did it because I really didn’t know any better. As I became more aware of the options, I inevitably switched to using Spyder for Python and RStudio for R, and eventually RStudio for Python for a while (yes, really o_O).

I’ve always been grateful for starting in the terminal though. I’ve never been afraid to fire up an R/Python REPL and do quick checks on language behaviour.

In fact, at times I’ve felt a little smug: I’ve had colleagues who don’t know the difference between R and RStudio. Meanwhile, I “graduated” last year to using VS Code as my main IDE. Yeehaw, I’m a cowboy riding out there in the wild west of Microsoft’s Extension Marketplace. I dazzled colleagues by rummaging around in the settings.json, pointing to an R binary for the Radian terminal. I even exposed addin_reshape() from codegrip and contributed to the docs..! Ok, I admit to being genuinely proud of that one, even if it was easy.

Then came the vim extension for VS Code. Wow. So cool. Even though the productivity gains from vim motions are marginal at best, and the time spent trying to adjust will never be won back. But you know, the important thing at the time was it kept me engaged with programming. It also exposed me to more computer sciencey developers and low-level language practitioners. Which in turn, of course, lead me to Neovim.

And Neovim obliterated any sense of smugness that I had.

It’s good to get in over your head every once in a while. It’s humbling.

Neovim

If you don’t know, Neovim (A.K.A nvim) is a text editor that can be used as an IDE. It’s a bit like VS Code in that it’s configurable, but way harder. In the nvim world people juice up the editor with extra functionality by installing plugins, most of which are written in Lua.

Apparently Lua is the easiest language ever to learn and literal children use it to make Roblox games. If you use nvim you’ll also be coding in Lua: it’s the language used for your config file/s.

I’m not going to say that I didn’t understand anything in the config files. But I didn’t find Lua super easy to get a flavour of. I just wanted my config to work most of the time, so I would mostly look at other people’s configs – limiting my exposure to Lua as a general language. This meant whenever the config would break or I’d get unexpected behaviour I didn’t really know what to do. It turns out that knowing words is important¹ when you’re learning programming.

I also found the documentation for most plugins threadbare. Developers: Don’t assume too much of your potential users. Many of us are, frankly, morons, and many more will have a TikTok user’s attention span. You do actually need to tell us what your software does and how it can be used.

Clearly I needed more help, so I read and watched a lot of tutorials. I found typecraft, who I would like to hug. I found the mononymous TJ, nvim contributor and creator of treesitter. There is of course fellow sober addict and programming craftsman ThePrimeagen (“Primeagen mentioned!”, “Let’s go boys!!”, etc.). I watched Josean Martinez’s and Henry Misc’s guides. Probably the best series of them all was by Vhyrro, who gave examples and explained the ‘why’ of things² for different facets of the editor – but who maddeningly never finished the video series!

They all helped, but in the end, it was still a huge amount of work. My knowledge wasn’t coming together. I couldn’t touch type, which is kind of a vim pre-requisite, so I tried to learn that via TypingClub. That definitely helped, but didn’t stick because it was too slow for work. Then a young 20-something on Mastodon tooted about how they were able to touch type faster than their old speed after just a month of TypingClub, and it completely unlocked nvim for them, and… I died a bit inside. Touch typing for me is honestly just super hard – even after several months.

I’d estimate that all told I probably spent a few weeks dedicated to learning about Neovim as a tool, lots of vim motions and commands, editing and using different configs, the language server protocol, linters, formatters, stuff like that. I must have used it as an IDE for a good 3 months. There were periods that I really enjoyed it.

I’d like to try again one day. There’s just something alluring about it. It reminds me of watching my older brother on the computer in the early 90s. The vibe appeals.

Not all wasted

So now I’m mostly using VS Code and the vim extension. I’m maybe faster in some ways than pre-nvim. One thing I like to do is open nvim in the terminal VS Code’s integrated terminal. That way I can qickly edit a file without having to open and close a new tab in the main editor.

It’s also given me gems like the vim ‘find and replace within highlighted text’ command: :s/find this phrase/replace with this/g.

Maybe even better is that it started to teach me about the differences between formatters, linters, and LSPs. In reality there’s overlap between the three categories, with some LSPs using linters, or formatters that use linters, and whatever ruff does. But at least it’s not a complete mystery. Case in point, I was recently able to help someone on Mastodon with an R linting bugbear in the context of nvim – even though it’s been 6 months since I used it in earnest!

Learning is a journey and even when it is hard, or doesn’t work out, we are bound to learn some things along the way.

Footnotes

See “How to teach programming (and other things)?” by Felienne Hermans↩︎
Again, see Felienne’s talk. It’s important!↩︎

VS Code for R on macOS

Jim Gardner — Wed, 20 Dec 2023 00:00:00 GMT

I never thought I’d say this, but… I’m enjoying Visual Studio Code (VS Code) as an IDE for R. I’d even recommend it as an alternative to RStudio for the average R user. This is particularly true if you - like me - are getting interested in languages other than R, such as python.

Setting it all up isn’t super hard. The essential thing to know is that VS Code is primarily a customisable IDE that relies on extensions to add in support for most languages. For R, there’s a nice, simple guide on the VS Code website specifically for us. Install VS Code, follow the linked guide, and you should be ready to code. There’s also a wiki for the extension that is worth looking at.

What’s more esoteric are the bits and bobs of context you need to start getting the most out of VS Code, particularly if you come from the RStudio world - like I do! So this post contains some of the extra context that I think is worth knowing during/after setup.

Warning

This post is specifically for mac users
I haven’t done any package development in VS Code yet
I don’t claim to be an advanced R or VS Code user
Use any of my suggestions/instructions at your own risk!

Rproj vs Folder/Directory/Workspace

You’re likely familiar with the .Rproj file generated by RStudio when you create a new project. We like the Rproj because: double-clicking it opens the project in its own RStudio session, with the working directory already set. That’s good.

With VS Code you just open the project folder/directory (also known as a workspace). All terminals, including the R one, will have the working directory set. No Rproj required.

Like RStudio you can have multiple windows open, each for a separate project. It’s worth knowing that if you open another folder with File > Open Folder…, it will close the currently open folder. One window, one folder. If you want to have multiple windows each with their own folder, you first have to open a window with Cmd + Shift + N, and then open a folder.

In the radian section you will learn about the settings json, and in the custom keybindings section about keybindings.json. It’s worth knowing that these can be set at the workspace level and will override any user-level settings when anyone works in the folder.

Radian

While it is recommended to install and use radian as the R console for VS Code, a word of warning. This became the biggest pain point during my setup. I would try to run R code and get errors about not being able to find R and/or radian. If the same happens to you, try the following.

First, install radian however the site suggests. Then open VS Code’s settings.json. To do that, hit Cmd + Shift + P (this will open the command palette) and start typing in “open user settings (json)”. When the option is visible to you, click it and settings.json will open.

The settings json is a file of customisable settings for controlling the behaviour of VS Code. In this case, we want to tell it where to look for R and radian. Copy the following into your settings file and then replace the file paths for your machine.

{
  "r.bracketedPaste": true,
  "r.rpath.mac": "/usr/local/bin/R",
  "r.rterm.mac": "/Library/Frameworks/Python.framework/Versions/3.11/bin/radian",
  "r.rterm.option": [
    "--no-save",
    "--no-restore",
    "--r-binary=/usr/local/bin/R"
  ]
}

r.rpath.mac and --r-binary point to your R. To find out where that is, run which R in the terminal.

r.rterm.mac points to your radian. You can find that with which radian.

Save the settings file. Now if you open an R file and attempt to run some code it should open a terminal and just “do the thing”.

Interactive viewer pane

You should have installed the R package httpgd during the initial setup. This package is a pre-requisite for using the interactive viewer pane. In VS Code the viewer works much the same way as in RStudio, but it has a couple of benefits like opening a new tab each time you view something, and better interactive filtering for tables. When running through this setup with a few people I’ve noticed that the interactive viewer is often a place where we encounter errors.

The solution every time is to add the code block below to ~/.Rprofile.

First of all, check if you have an ~/.Rprofile. If you don’t, make one. If you do, add the contents of this code block.

if (interactive() && Sys.getenv("RSTUDIO") == "") {
  source(file.path(Sys.getenv(if (.Platform$OS.type == "windows") "USERPROFILE" else "HOME"), ".vscode-R", "init.R"))
}

if (interactive() && Sys.getenv("TERM_PROGRAM") == "vscode") {
  if ("httpgd" %in% .packages(all.available = TRUE)) {
    options(vsc.plot = FALSE)
    options(device = function(...) {
      httpgd::hgd(silent = TRUE)
      .vsc.browser(httpgd::hgd_url(history = FALSE), viewer = "Beside")
    })
  }
}

options(vsc.rstudioapi = TRUE)

Strictly speaking the options(vsc.rstudioapi = TRUE) line is not needed for this step. But it is a prerequisite for using RStudio addins, including awesome ones like codegrip, so you may as well include it.

Format on save

Did you know that there’s such a thing as “format on save”? It sounds terrible, right? Someone else’s idea of what looks good absolutely destroying your inspiring, elegant, unique code?

Well for me… It’s actually been kind of liberating. I would never have tried this, but recently at work I’ve been collaborating with colleagues on a python codebase. For consistency’s sake we all have the same settings for python, including use of a formatter with format on save enable. Whenever we save files the code gets formatted.

Yeah, it means your personal style disappears somewhat. But the truth is that most of us code similarly anyways. At work it has been great, because I don’t know all of the python idioms and apparently my code stinks of R.

Once we switched to the formatter? No more stinky typey. Idioms are largely upheld and we get to focus on solving the actual problems, not fretting over formatting and indentation. I’ve spent quite a bit of time with R in VS Code over the past month, and I have found the same sense of freedom.

There is a linter for R (lintr) and it is automatically set up when you install the R extension¹. VS Code can pick this linter up and use it to format your code on save. You’re going to get shouted at a lot by the linter anyway, so if you’re curious about trying format on save, this is all you need to do:

In settings.json add the following items

[
  "[r]": {
    "editor.formatOnSave": true,
    "editor.tabSize": 2,
    "editor.detectIndentation": false
  }
]

The tab size and detect indentation items are there to stop lintr acting goofy with format on save. If you’re going to change lintr’s indentation settings (e.g. you prefer 4-space indentation), remove the tabsize one (or change it to 4).

With RStudio people use lintr in combination with styler, and sometimes GitHub actions, to apply style to their code. That’s fine. But it’s not for me, I prefer format on save. I literally just press Cmd + S and the formatting is taken care of, no matter what the issue is. It’s simple and immediate.

Note

If you hate the idea of format on save, or even a linter, and see them as an affront to the craft of programming: It’s ok. I get it. Everything you have to say is valid - you do you!

Extensions

These are the notable extensions that I have been using. You can find them by searching the extension marketplace with their name - they’re all rather distinct.

None of them are inherently needed if you’re working with plain R scripts.

Rainbow CSV

I work with a lot of csv and dat files for work, and they require regular inspection. If I needed to look at them outside of R, usually I’d open the file in TextEdit or TextMate if small/simple, or Excel if not.

But that’s been massively reduced thanks to this extension, which is very fast and simple. It works on csv, tsv, and dat, with good instructions in the readme for customising behaviour.

Yay! That’s nice, right? 🌈

There’s a few non-obvious tools available as well. First, when hovering over values it tells you the column name.

Second, in the toolbar at the bottom of VS Code, you can see “Query”, and “Align”. These are awesome.

Query will open a table-viewer and access to RBQL (rainbow query language), which is basically SQL.

Align will align all of the colums by inserting whitespace, and you can then click the same button to shrink the table back. If you are going to do this the docs (and I!) recommend that you put the following in your settings json: It will create a temp file where the alignment and shrinking happens, leaving the original file untouched.

{
  "rainbow_csv.align_in_scratch_file": true
}

Note

If you work with AWS there’s an extension for that, too. You can interact with all your usual services, including S3. Unlike the GUI, you don’t have to download (smaller) files from S3 just to view them. So if you have some csv that needs checking, you can literally just use the file explorer, click the filename, it will open in preview here in VS Code, and it will get the rainbow csv treatment! I think this is very cool!!!

GitLens

If you work with git this is nice. When you click on a line of code in the editor you get to see the git blame (ugh why is it called that) and commit message. There’s also a GitLens tab with history and graph in the bottom panel.

Quarto

Pretty self-explanatory. There’s great support for Quarto in VS Code. That’s how I’m writing this blog post :)

Peacock

Peacock adds colour to various parts of the VS Code window.

The real use for it is in colouring different workspaces. The settings get added to the workspace-specific settings json so they persist over time. If you have a couple of projects that you tend to flick between, the colour can help to orient you by differentiating between the windows.

R Debugger

To be honest I’ve not played with this much yet, or used it at all in earnest. I usually use the native debugger in the REPL. But it does have one enticing feature, which is a variable viewer.

Keyboard shortcuts

General

These are the ones that I find notable.

Show/hide
Cmd + J	Show/hide the terminal
Cmd + B	Show/hide the sidebar
Editor
Ctrl + 1	Focus on editor
Cmd + Shift + [ / ]	Cycle left / right through editor windows
Cmd + W	Close current editor window
Option + ↑ / ↓	Move current line (or highlighted block) of text up/down
Cmd + Option + ↑ / ↓	Insert cursor above or below current position
Cmd + Shift + V	Preview markdown file
Explorer
Cmd + Shift + E	Focus on explorer
Cmd + ↓	Open highlighted file from explorer
Ctrl + Enter	Open highlighted file from explorer (to the side)
Cmd + ←	Collapse all folders
Terminal
Cmd + Ctrl + ↑ / ↓	Grow/shrink the terminal pane
Cmd + Shift + [ / ]	Cycle left / right through terminals
Workspace
Cmd + Shift + N	Open a new window
Cmd + O	Open a file or folder

Note

Shrinking the terminal will cause the macOS system beep to kick in, which is annoying as hell. I think this shortcut is so useful that it’s actually worth deactivating the beep. It’s simple to do. First check if you have a file at ~/Library/KeyBindings/DefaultKeyBinding.dict

If you do, add the line from the code block below. If you don’t:

Create the directory ~/Library/KeyBindings
Create the file DefaultKeyBinding.dict in the new directory
Add the below to the file²

{
  "^@\UF701" = "noop:";
}

Alternatively, you can make custom keybindings in VS Code (see below for examples of how to do that).

R specific

These should all be familiar if you’re used to RStudio. The Quarto ones come with the Quarto extension.

Run code
Cmd + Enter	Run the current line or highlighted code
Cmd + Shift + Enter	Run the entire script
Quarto
Cmd + Shift + K	Render and show preview
Cmd + Shift + I	Insert code chunk

Custom keybindings

Like the settings file, there’s a keybindings.json for any bespoke keyboard shortcuts that you want to make. Open the command palette as before, but this time you’re looking for “Open Keyboard Shortcuts (JSON)”.

It is super useful to have something that can toggle between the terminal and editor. You can choose whatever keystrokes you like but Ctrl + ` is a common choice.

[
  // toggle between the terminal and editor
  {
    "key": "ctrl+`",
    "command": "workbench.action.terminal.focus"
  },
  {
    "key": "ctrl+`",
    "command": "workbench.action.focusActiveEditorGroup",
    "when": "terminalFocus"
  }
]

For R, most of us have used RStudio extensively and are familiar with three shortcuts, namely

Insert pipe (Cmd + Shift + M)
Insert assignment operator (Option + -)
Restart R session (Cmd + Shift + 0)

The keybindings below will add all of these to VS Code. It might look like there’s some duplication here for the pipe and assignment operator, and there kind of is. It’s just that you need to make the keybindings available to the editor and terminal separately (I tried to combine them but it didn’t work - if anyone knows how please get in touch!).

[
  // keybindings for r/rmd/qmd
  {
    "key": "Cmd+Shift+m",
    "command": "type",
    "args": {
      "text": " |>"
    },
    "when": "editorTextFocus && editorLangId =~ /r|rmd|quarto/"
  },
  {
    "key": "Alt+-",
    "command": "type",
    "args": {
      "text": " <- "
    },
    "when": "editorTextFocus && editorLangId =~ /r|rmd|quarto/"
  },
  // keybindings for R terminal (radian included)
  {
    "key": "Cmd+Shift+m",
    "command": "workbench.action.terminal.sendSequence",
    "args": {
      "text": " |>"
    },
    "when": "terminalFocus"
  },
  {
    "key": "Alt+-",
    "command": "workbench.action.terminal.sendSequence",
    "args": {
      "text": " <- "
    },
    "when": "terminalFocus"
  },
  // restart R session
  {
    "key": "Cmd+Shift+0",
    "command": "runCommands",
    "args": {
      "commands": [
        "workbench.action.terminal.kill",
        "r.createRTerm"
      ]
    }
  }
]

Footnotes

If you want to change default lintr settings see https://github.com/REditorSupport/vscode-R/wiki/Code-linting↩︎
If you’re on a version of macOS Sonoma < 14.4, the line should read "^@\UF701" = "noop";↩︎

Day 3 - Advent of Code 2023

Jim Gardner — Fri, 15 Dec 2023 00:00:00 GMT

AoC day 3 using base R. I got caught in the matrix 🥲

Spoilers ahead.

Solution without commentary

Code

Day 2 - Advent of Code 2023

Jim Gardner — Sun, 10 Dec 2023 00:00:00 GMT

AoC day 2 using base R.

I spent multiple evenings working on this and trying to understand three other programmer’s solutions.

If you’re interested in the comparative piece - which you should be because it’s the best part - go to the last section of this post.

Spoilers ahead.

Solution without commentary

Code

Day 1 - Advent of Code 2023

Jim Gardner — Mon, 04 Dec 2023 00:00:00 GMT

I did AoC day 1 using base R.

Spoilers ahead. Also, while this code works, I don’t know how elegant it is or if there are better ways to do parts of it in base. If you have any tips or pointers I’d love to hear them!

Part 1

The problem

— Day 1: Trebuchet?! —

Something is wrong with global snow production, and you’ve been selected to take a look. The Elves have even given you a map; on it, they’ve used stars to mark the top fifty locations that are likely to be having problems.

You’ve been doing this long enough to know that to restore snow operations, you need to check all fifty stars by December 25th.

Collect stars by solving puzzles. Two puzzles will be made available on each day in the Advent calendar; the second puzzle is unlocked when you complete the first. Each puzzle grants one star. Good luck!

You try to ask why they can’t just use a weather machine (“not powerful enough”) and where they’re even sending you (“the sky”) and why your map looks mostly blank (“you sure ask a lot of questions”) and hang on did you just say the sky (“of course, where do you think snow comes from”) when you realize that the Elves are already loading you into a trebuchet (“please hold still, we need to strap you in”).

As they’re making the final adjustments, they discover that their calibration document (your puzzle input) has been amended by a very young Elf who was apparently just excited to show off her art skills. Consequently, the Elves are having trouble reading the values on the document.

The newly-improved calibration document consists of lines of text; each line originally contained a specific calibration value that the Elves now need to recover. On each line, the calibration value can be found by combining the first digit and the last digit (in that order) to form a single two-digit number.

For example:

1abc2
pqr3stu8vwx
a1b2c3d4e5f
treb7uchet

In this example, the calibration values of these four lines are 12, 38, 15, and 77. Adding these together produces 142.

Consider your entire calibration document. What is the sum of all of the calibration values?

Solution

lines <- readLines("input.txt")

lines[1:3]

[1] "mxmkjvgsdzfhseightonetwoeight7" "3five4s84four9rtbzllggz"       
[3] "75sevenzdrpkv1onetwo"

matches <- gregexpr(pattern = "\\d", text = lines)
digits <- regmatches(lines, m = matches)

digits[1:3]

[[1]]
[1] "7"

[[2]]
[1] "3" "4" "8" "4" "9"

[[3]]
[1] "7" "5" "1"

concatenated <- lapply(digits, \(x) {
  number_of_digits <- length(x)

  # if only one number in a line it gets counted as both the
  # first AND last number
  ifelse(
    number_of_digits == 1,
    paste0(x, x),
    paste0(x[1], x[number_of_digits])
  )
})

concatenated[1:3]

[[1]]
[1] "77"

[[2]]
[1] "39"

[[3]]
[1] "71"

sum(as.integer(unlist(concatenated)))

[1] 55621

Part 2

The problem

— Part Two —

Your calculation isn’t quite right. It looks like some of the digits are actually spelled out with letters: one, two, three, four, five, six, seven, eight, and nine also count as valid “digits”.

Equipped with this new information, you now need to find the real first and last digit on each line. For example:

two1nine
eightwothree
abcone2threexyz
xtwone3four
4nineeightseven2
zoneight234
7pqrstsixteen

In this example, the calibration values are 29, 83, 13, 24, 42, 14, and 76. Adding these together produces 281.

What is the sum of all of the calibration values?

Solution

This is the point at which I cursed my commitment to doing this in base R. After a few hours I had to look up a hint, and in the process also checked for other solutions in R. Everyone else is just using stringr, dplyr, and/or tidyr. As a result they have reasonably simple regex and are mutating tibbles. I’m out here using functions I don’t fully understand, trying to understand lookarounds, lapplying my way over lists 🥲

Many of us who got stuck encountered the same issue: overlapping digits in strings. For example, the first two digits from zqtwonethreekcz3seven2 are two and one. In this case it wouldn’t really matter, but without getting overlapping strings separated like this, it could make me count a string like mwnineight as just containing nine, and then I’d end up with the answer as 99.

And unfortunately the example doesn’t mention this at all.

I managed to get a working regex on regex101.com but of course it didn’t work with gregexpr. Honestly, at this point I’m out of my depth, so just systematically worked through the small number of functions available to me seeing what looked like it worked.

And…

# Expand the regex to include word versions and a positive lookahead assertion, to accommodate for overlapping matches
pattern <- "(?=(\\d)|(one)|(two)|(three)|(four)|(five)|(six)|(seven)|(eight)|(nine))"
matches <- gregexec(pattern = pattern, text = lines, perl = TRUE)
res <- regmatches(lines, m = matches)

res[[985]]

      [,1]  [,2]  [,3]    [,4] [,5]    [,6]
 [1,] ""    ""    ""      ""   ""      ""  
 [2,] ""    ""    ""      "3"  ""      "2" 
 [3,] ""    "one" ""      ""   ""      ""  
 [4,] "two" ""    ""      ""   ""      ""  
 [5,] ""    ""    "three" ""   ""      ""  
 [6,] ""    ""    ""      ""   ""      ""  
 [7,] ""    ""    ""      ""   ""      ""  
 [8,] ""    ""    ""      ""   ""      ""  
 [9,] ""    ""    ""      ""   "seven" ""  
[10,] ""    ""    ""      ""   ""      ""  
[11,] ""    ""    ""      ""   ""      ""

The ffff??? a MATRIX?

Like most people who use R to work with data but never do statistics, I’ve never. Ever. Interacted with a matrix. I have a vague idea about how to index them.

But it turns out they’re easy to collapse and I also got to use one of my fave base functions, nzchar. It returns TRUE for a non-zero length ‘scalar’ character vector, which makes it excellent for logical subsetting.

# Collapse the results into just the matches
digits <- lapply(res, \(x) x[nzchar(x)])

digits[[985]]

[1] "two"   "one"   "three" "3"     "seven" "2"

From here it’s fairly straightforward.

mapping <- stats::setNames(
  1:9,
  c("one", "two", "three", "four", "five", "six", "seven", "eight", "nine")
)

all_numeric <- lapply(digits, \(x) {
  mapped <- mapping[x]

  # Backfill the NAs with the original numeric values
  numeric_indices <- which(is.na(mapped))
  mapped[numeric_indices] <- x[numeric_indices]

  mapped
})

length(all_numeric[lengths(all_numeric) == 1])

[1] 62

There are still a small number of lines where there is only a single number, so I’ll stick with the same code from before… yes I am going to copy and paste considering this is so trivial.

concatenated <- lapply(all_numeric, \(x) {
  number_of_digits <- length(x)

  ifelse(
    number_of_digits == 1,
    paste0(x, x),
    paste0(x[1], x[number_of_digits])
  )
})

concatenated[1:3]

[[1]]
[1] "87"

[[2]]
[1] "39"

[[3]]
[1] "72"

sum(as.integer(unlist(concatenated)))

[1] 53592

Making the terminal zsh-tosh on macOS

Jim Gardner — Wed, 11 Oct 2023 23:00:00 GMT

1 Overview

In this post I’m going to share a few tools that have made my zsh terminal experience on macOS a little bit nicer.

There’s two separate sets of tools. One is to Oh My Zsh with a couple of plugins. The other are so-called modern versions of a couple of commonly used command line tools. The purpose of all these tools is to make the terminal colourful and easier to use.

All of the changes here will also apply to terminals in VS Code. If you’re an R programmer using RStudio, I’m ~~sorry, that’s not a nice thing people like us get to have :(~~ a fool! Of course this is possible - though not the default. See Section 4 to see how to make it so.

2 Installation quick ref

Assuming you are putting installs in default locations

2.1 zsh stuff

Install zsh (if you need to)

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Install zsh-autosuggestions

git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

Install zsh-syntax-highlighting

git clone https://github.com/zsh-users/zsh-syntax-highlighting.git
echo "source ${(q-)PWD}/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh" >> ${ZDOTDIR:-$HOME}/.zshrc

Update zshrc

# Add the autosuggestions and highliting to the plugins variable like so

plugins=(
   git
   zsh-autosuggestions
   zsh-syntax-highlighting
)

Source

source ~/.zshrc

2.2 cli tools

Require homebrew

Install eza

brew install eza

Install bat

brew install bat

exit your terminal session and start a new one.

3 Context

3.1 zsh

zsh has come as the default shell for mac’s terminal since October 2019, with the release of macOS Catalina.

The standard experience is perfectly fine, just like bash, but is rather bare bones. There’s no syntax highlighting even for the native shell scripting. In fact, everything is black-and-white by default. I’m not down on zsh or bash as they are. I just prefer some colour to help visually parse wtf I’m looking at.

If you’re unsure you have it, run the following to check. You should see something similar to my output.

zsh --version

zsh 5.9 (x86_64-apple-darwin23.0)

Also run this to check that it’s your default shell. You should see the same output as I have here.

echo $SHELL

/bin/zsh

If you get command not found or the default shell is something else (like bash), follow the instructions here to install zsh and make it your default.

3.2 Oh My Zsh

Oh My Zsh makes the terminal a bit more colourful and can show you things like which git branch you’re on. I’ve found it quite useful for that alone, and it is really easy to install, with a great default theme.

zsh has a config file, typically found at ~/.zshrc. What Oh My Zsh does is essentially modify your .zshrc, putting some defaults in and setting things up to further configure as you see fit.

Once installed, you will see something like this

It doesn’t look like a lot, true.

But once you actually start doing something…

You can see the shortened current directory, which git branch I’m on (“zsh”, that I’m working in for this blog post), and some colour when calling ls -l.

Oh My Zsh also has loads of plugins to enhance functionality. Most are for working with common tools, languages and frameworks for developers. The plugins I’ve enabled are for syntax highlighting and autosuggestions. The syntax highlighting is nice because if I’m writing a little shell script, or piping a bunch of commands together, it helps me to more quickly visually parse the content. The autosuggestions are a bit like tab completion, but they use your execution history to autosuggest what might come next. This can be useful if you’re doing a repetitive workflow like making regular git commits, or moving back and forth between a small number of directories.

After adding the plugins to my zshrc using the nano editor, you can see there’s now syntax highlighting in this for loop.

Here’s an example of an autosuggestion. If I hit the right arrow key it would accept the suggestion.

3.3 bat and eza

Both bat and eza are “modern” versions of the classic command line tools cat and ls, respectively. The former is for printing the contents of files, and the latter is for listing files and directories.

bat is particularly nice for several reasons. Let’s look at the contents of my zshrc with cat first

zshrc printed with cat

The output is single-colour, printed via stdout, in full. It’s literally just a dump of the file contents to the screen. Compare that to bat

zshrc printed with bat

That’s more like it. We’ve got syntax highlighting, and the print is paginated a la less. There’s more though - if you have git it shows modifications in the sidebar, it can show all non-printable characters, concatenate files together, and when used in pipes will behave as a drop-in replacement for cat. So all around pretty awesome.

eza I like for two reasons.

First, yes, it’s colour again. You can see in the image below that the return values for eza -l are colourised in comparison to ls -l. And I always use the -l flag, so that matters to me.

Second is the tree flag -T. Sometimes a simple tree is all you need to see a project’s structure. The really nice thing about -T is you can combine it with -L (level), and specify a depth. For example if I only wanted to show stuff in the root of my current directory, I can do eza -TL=1. You may also have noticed that the trees are colourised 👀

Examples of using eza

4 RStudio settings

It’s really simple to make zsh the default terminal in RStudio.

In the Terminal tab, click the down arrow next to Terminal n. Then click Terminal Options

New terminals open with: zsh. Then click Apply, and OK

Start a new terminal and there you go!

5 Conclusion

Go get some colour in your terminal ;)

Comfy keyboard shortcuts for RStudio

Jim Gardner — Thu, 03 Aug 2023 23:00:00 GMT

Intro

Found, by Helvetica Blanc

I know quite a few keyboard shortcuts in RStudio these days.

That said, such knowledge has been hard won. My background is not in programming, so a lot of the time I wasn’t aware that certain shortcuts were possible. Or if I did, I may not know how to describe them/their names.

Much of my learning has been done by bumbling around, giving it a go, and pairing with colleagues. That’s all good, but it is nice to have a practical list for reference.

So, with help from some good peeps over at mastodon, I’m happy to share just such a reference list. The aim is to help get you feeling confident, comfy, and fancy-free when coding in RStudio. It’s nice to feel at home in your IDE of choice ☺️

I’m a Mac user currently running RStudio 2023.06.1. Please let me know if there’s anything wrong with the Windows/Linux defaults listed.

Keyboard shortcut quick reference

: Option + Shift + K

/ : Alt + Shift + K

Brings up the keyboard shortcut quick ref.

However, I can never remember the layout of the panel and get mentally overloaded seeing so many symbols and columns on screen at once. I prefer the web version instead (though it is a bit out of date).

Restart R session

: Command + Shift + 0

/ : Ctrl + Shift + F10

Generally, it’s a good habit to restart the R session regularly. If your global environment is a graveyard of variables and functions - and you can’t tell which are living or dead - this is what you need.

Executing code

: Command + Enter

/ : Ctrl + Enter

Executes the line, or lines, your cursor is on.

Do you highlight an entire series of piped functions with your mouse and then click the Run button?

What if I told you that instead, you just put your cursor wherever the hell you like in a pipe and press Command + Enter? And it just runs the whole thing?

Behold!

You can even work your way through a script by hitting Command + Enter on repeat. Don’t trust me?

Try this for size.

That’s right: Even when the focus is in the console (which you can tell because the cursor is blinking there) you can continue hitting Command + Enter to run subsequent lines of your R script.

Switch focus between Source/Console

/ / : Your + choice!

Toggle the keyboard focus between your script and the console.

This is an unbound keyboard shortcut by default. While there are shortcuts to focus individually on the source and console panels, I like this one because it acts like a toggle. Easier to remember than two separate shortcuts. Also my brain small… like cute cat :)

Go to Tools, Modify Keyboard Shortcuts... and search for “focus”.

Click into the empty middle column for Switch Focus Between Source/Console. Press whatever shortcut keys you want to use, then hit enter. I used Control + 1, but this was already assigned to Move Focus to Source so I had to delete it there too.

Indenting code

: Command + I

/ : Ctrl + I

Standardises indentation for highlighted code.

You’re typing, copying from the web, moving stuff about. I know it gets messy. Forget tab/space/backspace tedium and use this shortcut.

Commenting crap

: Command + Shift + C

/ : Ctrl + Shift + C

Comment/uncomment the line your cursor is on, or highlighted lines. Trust me that this one is worth memorising. You’re going to be using it a lot.

Finding crap

: Command + F

/ : Ctrl + F

Brings up the find/replace menu for the script currently open. The In selection check box is very useful.

: Command + Shift + F

/ : Ctrl + Shift + F

Brings up the Find in Files dialogue box. I use this when it’s been a while since I worked on a project. For example, I might want to know which files a specific function gets used in.

Inserting operators

: Option + -

/ : Alt + -

Drops in the assignment operator <-

: Command + Shift + M

/ : Ctrl + Shift + M

Drops in the pipe operator.

If you’ve had RStudio installed for a while the default might be the magrittr pipe %>%. These days, the base R pipe |> has got you covered for 99% of use cases, and doesn’t add yet another dependency or library call to your projects, so it is my preferred pipe.

If you want to make it the default, go to Tools, Global Options, and check the box as in the image below.

Using multiple carets/cursors

These are for when you want to do stuff like ‘find and replace’, but the text you’re finding/adding/removing is heterogeneous in some way. You might be tidying up some lines of text after pasting them from a web page, for example.

: Control + Option + Up/Down

/ : Ctrl + Alt + Up/Down

Adds carets above/below your current position.

: Option + Click and drag

/ : Alt + Click and drag

Highlight text or place carets in contiguous chunks.

Click/move your cursor into the first place you want a caret
Hold down Option
Click and drag the cursor

In this example I add carets to multiple contiguous lines of text.

: Control + Option + Click

/ : Control + Alt + Click

This is for when things have started to get a little bit messier and you can’t easily highlight text in contiguous chunks.

Click/move your cursor into the first place you want a caret
Then hold down Control + Option
Each mouse click will place another caret wherever your mouse is hovering

In this example I delete several lines across a few blocks of text.

Note

In my experience the terms “caret” and “cursor” are are synonymous in programming IDEs like RStudio. However, some IDEs will use “caret” and others “cursor”. One to be aware of.

Moving the cursor

: Command + Up/Down/Left/Right

/ : Ctrl + Up/Down/Left/Right

The up/down keys get you to the top/bottom of a document.

The left/right keys get you to the start/end of the current line.

: Option + Left/Right

/ : Ctrl + Left/Right

Skip over whole words. Whee!

Highlighting and deleting text

: Option + Shift + Left/Right

/ : Ctrl + Shift + Left/Right

Highlights whole words. Oooh 😮

: Option + Backspace

/ : Ctrl + Backspace

Deletes the word preceding the cursor position. Cya 🙋

Summon the command palette 👹

: Control + Shift + P

/ : Ctrl + Shift + P

Bringing up the command palette lets you search for commands (and settings) by name. Here I bring up the palette because I can never remember the shortcut to turn some text into a function.

Enclosing

Highlight some text and enclose it with e.g. parentheses () or quotes "".

If you highlight some text, you only need to give the opening character of the enclosing pair. RStudio will add the closing character.

Using just ", (, and [ to enclose some text.

Publishing content in Quarto using conditionals and if/else code flow

Jim Gardner — Sun, 23 Apr 2023 23:00:00 GMT

Damsel the chimera. Copyright Andy Stacey.

In this post I will describe some of the ways I’ve found to conditionally include content in a Quarto-generate HTML document.

I’m using RStudio and the knitr rendering engine. If you know of Jupyter solutions please let me know! I will happily link to them for the disappointed Pythonistas who have ended up on my blog ;)

Quarto

The {{< include >}} shortcode

This isn’t conditional, but it is really useful.

If you have company marketing boilerplate, or headers that start all of your reports, a neat solution is to keep these in their own qmd file and bring them into any other qmd using the include shortcode.

In your report you would use the code below

{{< include _content.qmd >}}

This is akin to manually copying and pasting in the contents of content.qmd prior to the report getting rendered.

We’ll come back to the shortcode later.

Conditional content

To my mind this one unfairly hogs the “conditional” title.

This is for when you have parts of a document that you only want rendered (or not) to a specific format, such as HTML, PDF, etc.

If you’re interested in this, here’s the Quarto guide.

Project profiles

In this scenario you will have content to be included (or excluded) based on the profile/s specified. The div below would only be included if the “advanced” profile is in use

::: {.content-visible when-profile="advanced"}
This content will only appear in the advanced version.
:::

You can also specify configurations in a profile-specific YAML that can be fully, or selectively used, to override those in the default profile. Here are the docs.

Code flow

Most of the following examples follow the usual if/else structure.

I’ll be referring to params$my_boolean throughout this section so let me take a moment to explain.

In the YAML header of a Quarto document you can pass in parameters. These get stored in a list params that can be called throughout the document, just like any variables you make and keep in the global environment.

---
params:
  my_boolean: FALSE
---

Reports made this way are known as parameterised reports.

If you’re using a Quarto project it is also possible to include variables in a root directory file _variables.yml. I’m not doing that here, but if it sounds appropriate for you, here are the docs.

Inline code

You can conditionally execute inline code. If my_boolean is TRUE the example below results in a level 2 heading that says “Plots for mtcars”.

## Plots `r if (params$my_boolean) "for mtcars" else "for iris"`

In functions

Sometimes you just need a bit of perspective. Rather than something convoluted, do you just need to make a function with flow control of some kind?

plotting_func <- function(dat, col1, col2) {
  
  if (params$my_boolean) {
    
    dat |>
      ggplot2::ggplot() +
      ggplot2::geom_point(ggplot2::aes({{ col1 }}, {{ col2 }}))
    
  } else {
    
    dat |>
      ggplot2::ggplot() +
      ggplot2::geom_boxplot(ggplot2::aes({{ col1 }}, {{ col2 }}))
    
  }
  
}

Or if the content is amenable to it, something like this

plotting_func <- function(dat, col1, col2) {
  
  dat |>
    ggplot2::ggplot() +
    if (params$my_boolean) {
      ggplot2::geom_point(ggplot2::aes({{ col1 }}, {{ col2 }}))
    } else {
      ggplot2::geom_boxplot(ggplot2::aes({{ col1 }}, {{ col2 }}))
    }
  
}

You could put the function into a code block. Or you could hide it away in a folder R/ and a script functions.R. That’s useful if you write long ugly functions like me (lol), but also if you have multiple functions. In this case, you could just source the functions silently along with library calls at the start of the Quarto doc.

```{r}
#| include: false
library(dplyr)
library(downloadthis)
library(reactable)
source(file.path("R", "functions.R"))
```

If your content is too complicated for flow control inside of a function, then just put the content into separate functions in functions.R, source them as above, and use flow control in the Quarto doc

```{r}
if (params$my_boolean) {
  fancy_model_a(mtcars, wt, mpg) 
} else {
  fancy_model_b(iris, Species, Sepal.Length)
}
```

knitr::knit_child

Saving the best for last, because this one is really cool!

Remember how I started with the {{< include >}} shortcode and said that it wasn’t conditional?

{{< include _content.qmd >}}

If anyone knows how to (simply) use the shortcode in a conditional manner I’d love to hear about it. In the meantime I consider the conditional version of this is using knitr::knit_child.

Let’s say I make interactive plots of my data using Observable code blocks and keep this code in a separate Quarto doc ojs-cells.qmd. The code can be added to the main Quarto doc under certain conditions, as dictated by a boolean param ojs_include in the YAML header

```{r}
#| output: asis
if (params$ojs_include) { 
  
  cat(sep = "\n",
      knitr::knit_child(quiet = TRUE,
                        text = readr::read_file("ojs-cells.qmd")
      )
  )
  
}
```

What’s happening here is

readr::read_file brings in ojs-cells.qmd as a string
cat restores the line-by-line formatting and ‘prints’ the content
The output: asis chunk option means Quarto treats the content as raw markdown output

Worth a mention - though not about conditionality - is this demo by Mickaël Canouil. It shows how the knit_child method can be called in a for loop to create n code chunks, and therefore n outputs (like tables and plots). This is great when you don’t know how many plots there will be or even what they are called.

Summary

Damsel the chimera. Copyright me!

I hope these approaches give you a practical set of tools for including conditional content in Quarto-published HTML documents.

You can use

The shortcode {{< include >}} to bring in a sub-document
Publish content dependent on output format using “conditional content”
if/else statements either inline or in code chunks. This can be combined with
1. Functional programming by putting code into functions - either in code blocks in the document, or in a separate sourced file
2. With knitr’s knit_child to bring in entire sub-documents as required

Thank you to Bruno Rodrigues and John MacKintosh for proofreading a draft of this post.