Mike Mahoney https://mm218.dev/blog.html Technology and the Environment quarto-1.8.26 Fri, 19 Dec 2025 00:00:00 GMT Ten short thoughts about AGU25 Mike Mahoney https://mm218.dev/posts/2025-12-19-agu/ This was my first time at AGU in New Orleans – and my first time in New Orleans in general. My review is that I really enjoy New Orleans, and I really enjoy AGU, and I really hate the New Orleans convention center.1 Mostly, I hate that the convention center is about a mile long from end to end, and talks were in rooms on either side of a single hallway spanning the entire mile – meaning I often had to pass all 36,000 attendees in that single mile-long hall as I went from Global Change (at ~0.2 miles deep) to Informatics sessions (at ~1.0).

I found a third wave coffee place (Fourth Wall) via Reddit and went there each morning. The line at 7:45am got notably longer every single day – including Friday, when I’d assume a decent chunk of folks had already gone home. I wonder if the human geographers have insights into how conferences, with their odd, massive groups of tourists with correlated commutes and schedules, interact with their host cities over the span of the event. Feels like it’d be a great AGU talk.

I’m always surprised at how much better the “science and society” talks are than the average session. Then I’m surprised that I’m surprised professional science communicators give good talks. We should fund more science communication. I’m pretty sure I don’t just think this because I work for our “Web Communications Branch”.

Why were only random doors unlocked? It seemed like doors marked “Enter Here” had a 50% chance of being open, while the rest of the doors had a 10% chance.

We all agree ten minute talks are bad, right? Maybe we don’t need ten talks per session?

I’ve always been a skeptic when it comes to automated data harmonization/interoperability. The idea is nice – augment your data with some magic set of metadata fields and boom it can automatically be rbind()’d with any other data that does the same – but real data is usually too messy for this to be practical. My favorite example is that, in 2021, a court in New York State idiotically changed the definition of a “tree” for some purposes from anything above 3 inches diameter to anything above 1 inch. So if you’re measuring trees, now you need to add whether you’re using the standard from before this case or after it – or perhaps the federal standard, which starts at 5 inches. This is before we even get to the fact that “tree” isn’t a well-defined category; rhododendrons usually aren’t included in the category, for instance, even though they can get to 33 feet!

All that to say, resolving methodology differences down to a level where data can be automatically harmonized (or rejected) seems to me a tall task for metadata alone, and I don’t see the AI era solving this problem any time soon.

The coffee stations this year all used composable coffee cups, despite the conference center not having a single compost waste bin. That has to be higher impact than the alternative, right? Don’t get me wrong – I flew to Louisiana, this isn’t a meaningful source of my2 emissions for the week. It’s just a little odd seeing this sort of green-washing at a conference like this.

For a conference of geospatial scientists, not a lot of spatial awareness in the crowd.

I think we’d all be better off if we were more honest about the role of money in shaping studies. I saw several talks where the sampling region seemed to be defined as roughly “where I could get on half a tank of gas”, and depending what you’re doing and what conclusions you’re drawing, that’s basically fine! We don’t need to pretend you chose the state your university is in because it’s a unique understudied ecotype that the rest of the researchers at your university have somehow overlooked.

It seems like a problem that we basically can’t run meaningful and low-stakes experiments with data sharing and discoverability anymore.3 I was in a few sessions where NASA ESDIS were discussing their project of moving 190 petabytes of imagery into S3, and the solutions they’ve built out to deal with it. And I think the tech looks great – at least, I love STAC and I hear nice enough things about CMR, as someone who’s never touched it. But it seems like a shame that it’s basically impossible for non-NASA people to try building other access patterns, to see if there are places for improvement that we’ve missed. I mean, where would you even start? You don’t have 190 petabytes of data to test with – and you certainly don’t have the money to pay for it, if you did.

Of course, it’s not like folks were able to build open-sourced versions of the Apollo missions, either. Maybe it’s not a bad thing to have smart, dedicated people collaborating together on a single solution for a hard problem. I like the tech they’ve got so far!

Footnotes

  1. Currently, I’m annoyed enough that I’m tempted to skip the conference the next time we’re in NOLA, but still grab a hotel nearby and do my usual handshakes and meetings. Realistically, I know myself well enough to know I’ll have completely forgotten by January.↩︎

  2. Exercise left for the reader: how many coffee cups would you need to go through before it was meaningful? And could you do it before the staff stopped you?↩︎

  3. Emphasis on meaningful – I specifically mean that independent researchers basically can’t contribute to modern data repository infrastructure, at least not for repositories at operational scales.↩︎

]]> AGU https://mm218.dev/posts/2025-12-19-agu/ Fri, 19 Dec 2025 00:00:00 GMT Converting New York’s Forest Carbon Assessment to Tidymodels Mike Mahoney https://mm218.dev/posts/2024-07-19-tidymodels/ We’ve been working this summer on version 2.0 of the New York Forest Carbon Assessment, incorporating changes in how the US estimates its forest carbon resources and building on our past work using satellite imagery and LiDAR patchworks to create wall-to-wall estimates of forest carbon stocks across New York for the last 30 years.

One of the big themes of version 2.0 is operationalization. We’re working to automate as much of our modeling and mapping pipelines as possible, to streamline generating annual maps as time goes on and to make it easier to experiment with new models and data sources in the future. In practice, this means I’ve been spending a lot of time merging individual scripts into targets pipelines, and generally looking at ways to streamline our processes.

The biggest change to come from this rewriting is that, while version 1.0 of our modeling pipeline used an internal library for tuning and creating stacked ensembles, version 2.0 shifts these tasks over to the tidymodels framework.1 Where we had previously only been using the rsample and yardstick packages from tidymodels, we’ve now adopted the framework more-or-less wholesale in order to take advantage of a few advanced features.2

In this post, I want to walk through what that workflow looks like for us – how we get from our field measurements to prepped data for models, then through hyperparameter tuning, into our final ensemble model construction. And then I want to talk about a few rough edges we ran into through the process, and why we’re still using tidymodels despite those stubbed toes.

So, without further ado, let’s start walking through how we’re building our ensemble models using tidymodels.

Building ensemble models using tidymodels

What are we doing here?

My intent with this section is to give a pretty straightforward, if fast-paced, walkthrough of how we’re using tidymodels to build our ensemble models. In order to do that, I need to load quite a few packages:3

# Used to download our predictor data
library(rsi)
# Contains our "field data", used for cross-validation
library(spatialsample)

# Used to wrangle our predictor data
library(sf)
Linking to GEOS 3.12.1, GDAL 3.8.4, PROJ 9.3.1; sf_use_s2() is TRUE
library(terra)
terra 1.7.71
library(exactextractr)

# Tidymodels libraries used to actually fit and tune models
library(tidymodels)
── Attaching packages ────────────────────────────────────── tidymodels 1.2.0 ──
✔ broom        1.0.5      ✔ recipes      1.0.10
✔ dials        1.2.1      ✔ rsample      1.2.1 
✔ dplyr        1.1.4      ✔ tibble       3.2.1 
✔ ggplot2      3.5.1      ✔ tidyr        1.3.1 
✔ infer        1.0.7      ✔ tune         1.2.1 
✔ modeldata    1.3.0      ✔ workflows    1.1.4 
✔ parsnip      1.2.1      ✔ workflowsets 1.1.0 
✔ purrr        1.0.2      ✔ yardstick    1.3.1 
── Conflicts ───────────────────────────────────────── tidymodels_conflicts() ──
✖ purrr::discard()  masks scales::discard()
✖ tidyr::extract()  masks terra::extract()
✖ dplyr::filter()   masks stats::filter()
✖ dplyr::lag()      masks stats::lag()
✖ recipes::step()   masks stats::step()
✖ recipes::update() masks terra::update(), stats::update()
• Use tidymodels_prefer() to resolve common conflicts.
library(hardhat)
library(bonsai)
library(finetune)
library(stacks)

In order to fit a stacked ensemble model, we first need to fit a number of “component” models. For our purposes today, we’ll fit a MARS model using the earth package and a gradient boosting model using LightGBM. While we don’t need functions from these packages, we do need them to be installed so that tidymodels packages can find them:

rlang::check_installed("earth")
rlang::check_installed("lightgbm")

We also need some data worth modeling. While the actual field data we use for our models is non-public,4 we can demonstrate the workflow using pretty much any example data. For our purposes, we’ll build our models to predict the amount of tree canopy in Boston in 2019, using data shipped with the spatialsample package:

library(ggplot2)
ggplot(spatialsample::boston_canopy, aes(fill = canopy_percentage_2019)) + 
    geom_sf() + 
    scale_fill_distiller(palette = "Greens", direction = 1)

This will be our outcome variable for our models.

Data preprocessing

In order to actually fit the models, we’re going to need predictor data too! We’ll derive our predictors from Landsat imagery taken in the summer of 2019, when all the trees in Boston were at their greenest.

To actually download this data, we’ll use the rsi package. In order to make the download go faster, we’ll first use future to register a parallel backend, letting us download multiple pieces of data at the same time:

library(future)
plan("multisession")

And we can then actually download our Landsat imagery using rsi::get_landsat_imagery():

landsat_imagery <- rsi::get_landsat_imagery(
    spatialsample::boston_canopy,
    "2019-06-01",
    "2019-09-01",
    platforms = c("landsat-8", "landsat-9"),
    output_filename = tempfile(fileext = ".tif")
)

terra::plot(terra::rast(landsat_imagery))

This winds up generating a median-composite image, made up of all of the “good” data available for this time period.

We normally don’t fit our models using these raw reflectance values, but rather use them to calculate a number of spectral indices that we then use as predictors. We can automate this process using rsi as well; for instance, we can use this code to calculate every spectral index possible using the bands in the raster we downloaded:

landsat_indices <- rsi::calculate_indices(
    landsat_imagery,
    rsi::filter_bands(bands = names(terra::rast(landsat_imagery))),
    output_filename = tempfile(fileext = ".tif")
)

|---------|---------|---------|---------|
=========================================
                                          
terra::plot(terra::rast(landsat_indices))

Now we’ve got a raster with a ton of spectral indices that we can use as predictors. We now need to associate those with our canopy percentage measurements from spatialsample. Generally we accomplish this using exactextractr to calculate the mean of all of these indices within our sample locations:

cell_values <- exactextractr::exact_extract(
    terra::rast(landsat_indices),
    spatialsample::boston_canopy,
    "mean",
    progress = FALSE,
    colname_fun = \(values, ...) values
)

cell_values <- cbind(
    spatialsample::boston_canopy["canopy_percentage_2019"], 
    cell_values
    )

head(cell_values)[1:6]
Simple feature collection with 6 features and 6 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 749383.6 ymin: 2913059 xmax: 801174.4 ymax: 2965741
Projected CRS: NAD83 / Massachusetts Mainland (ftUS)
  canopy_percentage_2019  AFRI1600  AFRI2100      ANDWI       AVI     AWEInsh
1              8.6786929 0.2429339 0.4686573 -0.2066887 0.2127504  0.09513274
2             32.9273863 0.4255704 0.6931654 -0.5248111 0.3976697 -0.20890425
3             12.2586142 0.4492964 0.7004566 -0.5057031 0.4135522 -0.17314616
4             15.5646636 0.2985486 0.5540271 -0.3153332 0.2875552 -0.03299006
5              0.4253207 0.1876189 0.3655822 -0.1460433 0.1548586  0.20539446
6             43.3838631 0.4718267 0.7542459 -0.5869194 0.4660867 -0.31844634
                        geometry
1 MULTIPOLYGON (((781922.7 29...
2 MULTIPOLYGON (((752945.6 29...
3 MULTIPOLYGON (((800842.8 29...
4 MULTIPOLYGON (((751419.1 29...
5 MULTIPOLYGON (((772252.2 29...
6 MULTIPOLYGON (((763631.7 29...

We’ve now got index values at each of our sampled polygons!

Before we can start using these to predict canopy coverage, we need to do a bit more pre-processing. For instance, for some reason a few of our polygons are missing values in at least one spectral index. While in the real world we’d probably want to debug exactly why this is happening, for our purposes today it’s a lot easier to just drop those rows from our data frame:

cell_values <- cell_values[
    complete.cases(sf::st_drop_geometry(cell_values)), 
]

Similarly, a few indices have non-finite values in at least one row, and we’ll want to go ahead and drop those as well:

infinite_cols <- vapply(
    sf::st_drop_geometry(cell_values), 
    \(x) any(is.infinite(x)),
    logical(1)
)
cell_values[infinite_cols] <- NULL

Model fitting

We’ve now got our cell_values object fully ready for modeling workflows. We’re now able to start the actual model-fitting process.

Our first step is to define the formula we’re using to fit our models. Because I’ve got the cell_values dataframe structured with our outcome column in front, followed by all the predictor columns, I can create this formula using the neat DF2formula() utility:

formula <- DF2formula(sf::st_drop_geometry(cell_values))

Did you know that’s a function in base R? What a wild language.

Anyway, with our formula built, we’re ready to start actually using tidymodels packages. Our first step is to create a recipe that uses our formula, and will pass that information on to the other tidymodels packages:

recipe <- recipes::recipe(formula = formula, data = cell_values)

Up next, we’ll create the model specifications that define what models we want to fit, and pass hardhat::tune() to every hyperparameter that we want to be tuned.5 We’ll also set the “engine” that we want to use (the package for fitting the models), and the mode of model:

lightgbm_spec <- parsnip::boost_tree(
    tree_depth = hardhat::tune(),
    trees = hardhat::tune(),
    learn_rate = hardhat::tune(),
    mtry = hardhat::tune(),
    min_n = hardhat::tune(),
    loss_reduction = hardhat::tune()
  ) |>
    parsnip::set_engine(engine = "lightgbm") |>
    parsnip::set_mode(mode = "regression")

mars_spec <- parsnip::mars(
    num_terms = hardhat::tune(),
    prod_degree = hardhat::tune()
) |> 
    parsnip::set_engine("earth") |> 
    parsnip::set_mode("regression")

We’ve now got our recipe and our model specifications created, meaning we’ve defined all the preprocessing and modeling we’re asking tidymodels to perform.

workflow_set <- workflowsets::workflow_set(
    list(recipe),
    list(
        lightgbm = lightgbm_spec,
        mars = mars_spec
    )
)
workflow_set
# A workflow set/tibble: 2 × 4
  wflow_id        info             option    result    
  <chr>           <list>           <list>    <list>    
1 recipe_lightgbm <tibble [1 × 4]> <opts[0]> <list [0]>
2 recipe_mars     <tibble [1 × 4]> <opts[0]> <list [0]>

I’ll talk about this a bit more later, but I think it’s worth noting how complex the thing we just did is. The workflow_set() function just created a “workflowset”, which is a tibble that contains multiple “workflows”6 which each contain our “recipe” and “model specification” which has some arguments “marked for tuning”7. The code we’ve written is pretty minimal – something like 20 lines of code to create the workflowset itself – but it winds up producing quite a number of abstractions that we need to reason around to actually understand what we’re doing.

But with our workflowset created, we’re now able to actually tune our component models! Before we do so, we need to set our future strategy back to sequential processing. The LightGBM implementation in bonsai uses parallel processing by default, and our tuning code will also try and run in parallel if we have a future backend registered, resulting in thread contention and making tuning never finish. As such, we’ll run plan("sequential") before tuning:

plan("sequential")

And then we’ll perform our tuning. We need to tune each row of our workflowset – each unique recipe/model combination – separately.

Luckily, this is pretty easy using workflowsets::workflow_map(), which can apply a function to each workflow separately. Here, we’ll use this to use finetune::tune_race_anova() to evaluate our models against 10 different sets of hyperparameters8 – making a point to save the predictions and workflows from each set, as we’ll need those to build our ensemble model later on.

resamples <- spatialsample::spatial_clustering_cv(cell_values, 10)
tuning <- workflowsets::workflow_map(
    workflow_set,
    "tune_race_anova",
    resamples = resamples,
    grid = 10,
    metrics = yardstick::metric_set(yardstick::rmse),
    control = finetune::control_race(save_pred = TRUE, save_workflow = TRUE)
)

Our tuning object now contains the outputs from the tuning process. We could use this to, for example, find the best evaluated set of hyperparameters for each of our models:

tuning$result |> 
    lapply(tune::select_best, metric = "rmse")
[[1]]
# A tibble: 1 × 7
   mtry trees min_n tree_depth learn_rate loss_reduction .config              
  <int> <int> <int>      <int>      <dbl>          <dbl> <chr>                
1   114   562    12          8    0.00372           18.6 Preprocessor1_Model09

[[2]]
# A tibble: 1 × 3
  num_terms prod_degree .config             
      <int>       <int> <chr>               
1         3           2 Preprocessor1_Model5

But we aren’t actually all that interested in just-the-best. Because we’re going to be combining these models into ensembles, we can actually use all the models fit using good enough hyperparameter sets. That process can happen more or less automatically thanks to stacks, which will fit a lasso regression to determine which of the candidate models are better than useless and combine them into a final prediction:

ensemble <- stacks::stacks() |>
  stacks::add_candidates(tuning) |>
  stacks::blend_predictions() |>
  stacks::fit_members()

ensemble
── A stacked ensemble model ─────────────────────────────────────


Out of 8 possible candidate members, the ensemble retained 4.

Penalty: 0.1.

Mixture: 1.


The 4 highest weighted members are:
# A tibble: 4 × 3
  member               type       weight
  <chr>                <chr>       <dbl>
1 recipe_lightgbm_1_09 boost_tree  0.418
2 recipe_mars_1_4      mars        0.315
3 recipe_lightgbm_1_05 boost_tree  0.169
4 recipe_mars_1_6      mars        0.132

Now, the best thing about stacks9 is that predicting with this model is dead simple. Even though predicting with our ensemble requires predicting with each of the selected earth and LightGBM models, and then predicting with the final lasso regression, from our perspective as users we can just call a single, typical predict():

predict(ensemble, cell_values) |> head()
# A tibble: 6 × 1
  .pred
  <dbl>
1  7.81
2 33.6 
3 27.2 
4 16.6 
5  1.25
6 43.8

For our carbon mapping workflows, however, we wind up generating predictions using the same 30 meter raster we grabbed our original predictor values from. Ensembles from stacks also work pretty well with this workflow:

terra::predict(
    terra::rast(landsat_indices),
    ensemble
) |>
    terra::plot()

|---------|---------|---------|---------|
=========================================

We’ve also been using bundle to help us serialize these ensemble models and load them in new R sessions. The flexibility of bundle is another great plus of leaning in with tidymodels.

Some complaints

That said, as I mentioned at the start, I’ve certainly had a few unpleasant experiences through this transition, and I want to talk about them now.

I want to be clear at the start – I’m not intending for this to be some sort of “tidymodels skeptic” post. We’re still using tidymodels, and I think our codebase is more maintainable and – with some caveats – understandable following this ground-up rewrite. These are just the things that bit us during that rewriting process.

Another disclaimer here is that I interned with the tidymodels team back in 202210 and deeply enjoyed my time with the team. As a result, my experience with this rewrite was pretty nonstandard; I was already pretty familiar with some corners of the ecosystem (especially the parts touching rsample), am pretty willing to push past frustration with other packages based on my trust in the people developing them, and could DM a team member if I was ever particularly stuck. So this isn’t the most objective outsider experience you could ever imagine; it’s just the one I had.

Learning one thing requires learning everything

A recurring challenge I have with the tidymodels framework is that often, to understand one part of tidymodels, it can feel like you need to know all of it.

For example: if we want to automatically tune a hyperparameter, we need to set it to tune() inside of our model specification. But what does tune() actually do – how does it determine the hyperparameter values worth searching?

Well, the tune() function itself comes from the hardhat package, which describes itself as a “developer focused package designed to ease the creation of new modeling packages”. That means that if I’m trying to figure out what this tune() function actually does, running ?tune() winds up getting me the help page of a developer-focused package, and I start wondering how much of that developer-focused documentation I need to read and learn in order to tune my models.

For its part, the tune() help page actually points you to tune::tune_grid() and tune::tune_bayes() for more information on how these parameter values are being set. I don’t see any section here that explicitly says how parameter ranges are set when using tune() but the “Parameter Grids” section of tune_grid() and “Parameter Ranges and Values” section of tune_bayes do both discuss tuning parameter ranges. These sections don’t mention tune(),11 but do mention that dials::finalize() “can be used to derive the data-dependent parameters”.

We don’t call dials::finalize() ourselves – we don’t actually load dials at all – but if we go to the dials website, we could read the Getting Started vignette12 which starts to explain how parameter ranges are estimated. The “Unknown Values” section sort of explains how some ranges – like the max value for mtry and so on – are calculated, but I’m still a bit lost on how our hardhat::tune() gets turned into a dials function (if it does) and when these parameter ranges are set.

So I haven’t entirely figured out how these grids are made, but I do have five tabs open from three different package websites. At this point, I’m starting to get a little bit buried by the different layers going on here – and I don’t know how much of hardhat, dials, or tune I need to understand before things start making a bit more sense. I also don’t have an obvious first point of attack to answer this question, as the packages all link back to each other as places to get further information.13

A bit of speculation: I think the issue I run into here is that the boundaries between the packages – where the logical breaks between concepts are for developers – don’t line up with where the logical breaks between concepts are for me, as a user. There’s no one package that owns the concept of “tuning a hyperparameter” – tune owns the grid searching, hardhat owns the infrastructure, dials owns the grid construction. As a result, each package’s documentation explains the piece of tuning that package covers, but none of them explain the concept as a whole.

Hidden options can cause delayed failures

This particular issue is more likely my fault – the behavior I’m about to complain about is documented, and is a valid design for an interface to have, but I got badly bitten by it and want to whinge.

The default arguments to tuning functions – specifically, those in control_grid() – are set (I think) to create small, efficient objects. This is a sensible goal to have. The problem is that it also makes these objects less useful, in a way that you can’t fix after the fact. You’ll only learn that your pipeline is broken after already spending a lot of time running computationally intensive tuning code.

To be more specific: say that you tried to tune the same workflow that we used above, but didn’t know that you needed to specify that you wanted to save the workflow:

tuning <- workflowsets::workflow_map(
    workflow_set,
    "tune_race_anova",
    resamples = resamples,
    grid = 10,
    metrics = yardstick::metric_set(yardstick::rmse)
)

If we then want to fit the best model, the obvious choice would be to use tune::fit_best(). But if we call it on our tuning results:

try(
    tuning$result |> 
        lapply(tune::fit_best, metric = "rmse")
)
Error in FUN(X[[i]], ...) : 
  ✖ The control option `save_workflow = TRUE` should be used when tuning.

We get an error. Now, if we had read the documentation for fit_best(), we’d see that the documentation for the x argument ends by saying “The control option save_workflow = TRUE should have been used” – so this is documented. But if you missed that, or if you’re writing your pipeline as you go,14 you’re just told – in past tense – that you did the wrong thing and need to try again, throwing out the computational time you spent tuning the first time around.15

Now, the version of this that actually bit me came when I tried to add my tuned models to an ensemble using add_candidate(). This function has a similar pattern to fit_best(), except it wants you to have saved both your workflow and your predictions:

try(
    stacks::stacks() |>
        stacks::add_candidates(tuning)
)
Error in .f(out, .x[[x_i]], .y[[y_i]], ...) : 
  The inputted `candidates` argument was not generated with the
appropriate control settings. Please see `control_stack()`
(`?stacks::control_stack()`).

And again, this is documented, for instance in the “define candidate ensemble members” section of one vignette16 and in one of the argument definitions for add_candidates(). But I didn’t see it when I skimmed the stacks documentation initially, and as a result I wound up wasting about ~26 hours spent tuning without saving workflows or predictions. The defaults here are good for making your objects smaller, but mean that tidymodels workflows fail extremely slowly.

Part of the issue with this particular example is that the way these options get set is pretty indirect, and I get a bit lost in the documentation. My tuning here is controlled by workflowsets::workflow_map(), which has five documented arguments: the workflow set, the tuning function to run, a flag for verbosity and a random seed, and then ..., documented as “Options to pass to the modeling function”. The word “control” doesn’t appear on this page.

A minor nit is that I don’t think of tune_* as being modeling functions, but rather tuning functions, so ... doesn’t jump out at me as a place to set my save_pred and save_workflow arguments. But I think a bigger problem I had is that if I do click over into the documentation for my tuning function, in this case finetune::tune_race_anova, I still don’t see obvious places to set these arguments.

Instead, the tune_* functions have a control argument which itself takes the output of another function. There are a surprising number of these control functions – at least control_race() and control_sim_anneal() from finetune, plus control_bayes() and control_grid() from tune17 – which I think are paired 1:1 with tuning functions.

I’ll be honest, I don’t actually know what these functions do at a fundamental level,18 or if they’re sharing some plumbing under the hood which makes this indirection much more efficient for the developers. All I know is that I need to go through a decent number of man pages and a bit of guesswork to land on this final chunk of code:

workflowsets::workflow_map(
    workflow_set,
    "tune_race_anova",
    resamples = resamples,
    grid = grid,
    metrics = yardstick::metric_set(yardstick::rmse),
    control = control_race(verbose = TRUE, save_workflow = TRUE, save_pred = TRUE)
  )

So, in order to override the default arguments so that I can fit my models or create an ensemble after tuning, I need to pass the new values to control_race(), and then pass the output from control_race() to tune_race_anova by providing it as a named argument metrics to .... That’s a lot of deep magic to learn – and a lot of concepts to be thinking about at the same time in order to use this function.

In some sense, this is another flavor of “learning one thing requires learning everything”. Because a lot of functions that you’d use later in a workflow require specific non-default arguments in earlier steps, you need to know every function you intend on using when you start writing a pipeline. And as understanding the arguments to these functions generally requires understanding multiple other functions in multiple other packages, it becomes difficult to learn the framework as you go, or to think about modeling pipelines in smaller chunks.

We’re still using tidymodels

With those complaints now vented, I wanted to close by saying that the pipeline I was building – the one that originally motivated this post – is still using tidymodels, and is going to keep using the framework for the forseeable future. There’s a few really nice features of the framework that motivated the switch:

  • First, finetune is really cool. We’ve been using finetune to conduct a much more thorough grid search than would be possible if we needed to see every set of hyperparameters through to the end, and to do so faster than we could do the smaller searches we had been using.
  • Second, stacks is also great. A generic interface to fit ensemble models (something I’ve written about before), which easily accepts new models and writes its own predict methods? Sign me up. Wanting to use these two packages then motivated us to use tune, and deal with the pain points I mentioned above.19
  • Third, even before we adopted the full tidymodels framework, we were already using packages like rsample and yardstick (not to mention my extensions for these packages, spatialsample and waywiser20) to assess our models. We’ve always found a number of individual tidymodels packages to be extremely useful; most of our pain points (and most of the complaints above) come when we need to hand things off between two packages.
  • And finally, I do think our final pipeline here is more readable, and will likely be more maintainable moving forward. Only time will tell, of course, but I’d rather be reading code that included a lot of tune_race_anova() and boost_tree() than the model-specific interfaces and nested lapply() calls that it replaced.

So given all that, we’re still using the tidymodels framework. By my estimation, the benefits have been worth the growing pains so far, and with our modeling pipelines now operational we’re hoping that we’re mostly finished with the worst of the surprises. We’re just making extra sure to highlight the bandaids we’ve had to add to the pipeline, wherever we got caught on a rough edge – and hopefully this post helps flag a few of them for you as well!

Footnotes

  1. A slightly less relevant change is that I wrote most of the pipeline while beta testing Posit’s new Positron IDE, which I mention mostly as a humblebrag. I generally really like Positron – this post was written using it – though I have a really hard time specifying why. It just feels nice to use.↩︎

  2. By the way – if you work on a package I mention in this post, and it would be useful for you to say “Used in New York State’s Forest Carbon Assessment” somewhere, drop me a line. We’re part of New York’s Climate Act Scoping Plan and partner with the state DEC on our mapping, and your work is helping make that possible.↩︎

  3. I’m throwing this in a footnote because it doesn’t really fit anywhere else: There’s a trendy sort of post where people talk about how the tidyverse or tidymodels ecosystem being made up of many packages is itself inherently bad. I think this is a bad opinion. We’re asking these ecosystems to do a lot of complicated things for us, which typically means we’re asking them to include a lot of functions and code. It’s natural – and good practice – for these functions to get broken up into logical units, each providing an interface for users and the rest of the ecosystem to build upon. As such, I think that measuring supply-chain risk based on a pure count of dependencies is the wrong approach. If you’re worried about dependencies changing their interfaces or getting archived and breaking your workflows, I personally think you should be more concerned about multitool-packages developed by non-tenured academics than about the ecosystem maintained by a small team of professional software engineers.↩︎

  4. I’m realizing now, four-ish years into this job, that I don’t actually know the right phrasing for FIA data. It’s not “confidential”, as no one in our group has a security clearance, but we have access to plot locations under an MOU with the Forest Service. I guess “non-public” is the best I’ve got at the moment.↩︎

  5. There’s a note in the tune documentation that some models – specifically earth models, but presumbably LightGBM could do the same thing – can evalute more models than are fit, speeding up tuning times. I don’t actually see any information on how to combine this trick with tune itself, though, so we’re just going to pretend I didn’t see that part of the docs.↩︎

  6. Which come from a package, workflows, that we won’t ever call explicitly here.↩︎

  7. With parameter values determined by the dials package which, again, we won’t call explicitly here↩︎

  8. Which is a small number chosen to run quickly for this blog post; in practice we tune with a much, much larger grid.↩︎

  9. I mean, in my opinion.↩︎

  10. Go star spatialsample on GitHub↩︎

  11. In the context of grid creation – tune_grid() mentions tune() in the context of user-provided grids.↩︎

  12. Not linked from the finalize() page, but in the top navigation bar.↩︎

  13. For what it’s worth, a full explanation of how tuning works is in the tidymodels book, but I didn’t see that linked anywhere in this chain.↩︎

  14. Meaning you didn’t look up the code you needed post-tuning until you were done tuning.↩︎

  15. If you’re tuning a workflow, I think there is a workaround where you can use select_best(), finalize_workflow(), fit(), and extract_workflow() to get your fitted workflow without needing to retune everything here. I’m not entirely sure– I’m guessing based off of the “Details” and “See also” sections of the fit_best() documentation, and this doesn’t seem to work with workflowsets.↩︎

  16. Though isn’t spelled out in either other vignette, or the README↩︎

  17. I think it’s most fair to not include control_resamples() or control_last_fit() here, because I think they’re not used for tuning, but I don’t actually know. A particular oddity is that control_grid() is listed in the tune reference as a developer function, but is used throughout the documentation, while control_last_fit() is listed as a “function for tuning” and isn’t used in any examples.↩︎

  18. Creating a list with a finite set of named elements, with some default values? The print method just says <x> control object.↩︎

  19. We had been hand-rolling our own ensemble code, which was mostly painless. The hardest part initially was writing a predict method that would generate predictions from all of our component models, and then combine those into a final ensemble prediction. The function we wrote to handle this could easily extend to use multiple models from the same package – if we wanted to use add another LightGBM to the mix, for instance – but adding new packages required adding new complicated chunks to the predict method. We were interested in making future extensions easier, so stacks seemed like an obvious choice – but our home-rolled grid search code was a lot more straightforward than tune, so we were initially resistant to making the jump.↩︎

  20. The waywiser package was originally developed explicitly for the NYS carbon mapping project, as it happens; we use just about every function in waywiser as part of our assessment process.↩︎

]]> tidymodels R Tutorials AGB Data science Spatial geospatial data machine learning https://mm218.dev/posts/2024-07-19-tidymodels/ Fri, 19 Jul 2024 00:00:00 GMT New allometric models for the USA create a step-change in forest carbon estimation, modeling, and mapping Mike Mahoney https://mm218.dev/posts/2024-05-08-nsvb/ Late last year, the United States Department of Agriculture’s Forest Inventory and Analysis Program (FIA) quietly announced that they were changing how they calculated vegetation structural parameters – specifically the volume, biomass, and carbon content of trees across the USA – to use a brand-new set of allometric models. These models are a great step forward. They do a much better job of accounting for the non-merchantable parts of a tree and provide a nationally consistent method for how we calculate these values, replacing the old system which had sharp deviations across regional (not ecological) boundaries.

But fundamentally, this change means that going forward FIA is providing a different set of estimates relative to what they have provided for years. Anyone who’s building models from FIA data is going to need to update their workflows to adapt. In a new preprint out today, we – a team led by Lucas Johnson and with Grant Domke and Colin Beier – start exploring what that might mean for model-based carbon estimates and maps.

What we find is that, generally, the shift to these new allometrics is complicated! The new values aren’t pure rescalings of old estimates – while you can get close by using a linear model to update old estimates to the new allometrics, it’s not a perfect match, and some of the remaining variation is likely due to environmental factors and species composition across the landscape. And while model-based estimation approaches are still effective at expanding from field measurements to point-in-time carbon estimates across the landscape, we find that the new allometrics attribute a lot of forest growth to forests with full crowns, where passive remote sensing based models tend to saturate. That means that these models, even though they’re good at making point-in-time estimates, are worse at tracking growth over time in mature forested landscapes.

Three maps of New York State, showing estimates of carbon stock changes between 2005 and 2019 under old allometrics and new allometrics, as well as the difference between these two maps.
Figure 1: Stock-change map comparison. a) 15-year NSVB stock-change map ( NSVB; 2019 AGB - 2005 AGB). b) 15-year CRM stock-change map ( CRM; 2019 AGB - 2005 AGB). c) Stock-change difference map computed as NSVB minus CRM. Mapped values capped at +/- 40 .

There’s plenty of more information about these challenges, alongside other explorations of what the new allometrics will mean for model-based estimation workflows, in the preprint.

This was a really fun paper to write – and as is tradition, we originally thought it would be a “quick hit” sort of project when we started on it back in Novemeber, when the new allometrics were released. It’s also (I think!) my first time actually “co-writing” a paper, where I wrote the Introduction and Methods section and then handed that shell to Lucas so he could actually do the real work involved. I enjoyed that process and think it produced a good result here; I’m really happy to see this preprint go out!

]]> ecology papers remote sensing machine learning earth science AGB https://mm218.dev/posts/2024-05-08-nsvb/ Wed, 08 May 2024 00:00:00 GMT Test warnings faster Mike Mahoney https://mm218.dev/posts/2024-04-12-testing-expensive-functions/ Here’s another small little note from package development corner (see also using classed errors in rlang, executing untrusted code in minimal environments, and not pre-allocating vectors isn’t as bad as it used to be). Say you’ve got some function in your package that takes a long time to execute:

func <- function(x) {
  Sys.sleep(3)
  x * 2L
}

Maybe the function is downloading data over a network connection, maybe it’s doing a ton of computations, maybe it’s not written super efficiently but you’ve got other priorities right now – the point is, this function takes a long time to execute, and that’s not going to change.

But you still want to properly check user inputs and throw warnings/errors as appropriate. For instance, a clear issue with this function is that it will overflow to NA when given a large integer x:

func(.Machine$integer.max)
Warning in x * 2L: NAs produced by integer overflow
[1] NA

So maybe we add some code to give a friendly warning about this situation, to hopefully make the specific issue clearer for our users:

func <- function(x) {
  if (x > (.Machine$integer.max / 2L)) {
    rlang::warn(
      "`x` is too large, so this function will return NA",
      class = "big_x"
    )
  }
  
  Sys.sleep(3)
  x * 2L
}

And because we’re diligent package developers, we want to test to make sure that this warning fires when we’d expect. Since we’re using a classed error, we can write a test to make sure that specifically our big_x warning fires when we pass an x that’s too big:1

library(testthat)
suppressMessages(testthat::local_edition(3))

test_that("large integers get a custom warning", {
  expect_warning(
    func(.Machine$integer.max),
    class = "big_x"
  )
})
Test passed

This is all good practice!2 But it has one big downside: whenever we run this function, we need to wait for the entire function to finish before our test passes. Which means for expensive functions, these can be pretty expensive tests:

tictoc::tic()
test_that("large integers get a custom warning", {
  expect_warning(
    func(.Machine$integer.max),
    class = "big_x"
  )
})
Test passed 
tictoc::toc()
3.056 sec elapsed

What we can do instead is use tryCatch() to promote this specific warning into an error, aborting the function (and not triggering any of the expensive code). By giving that new error its own class, and using expect_error() to check for an error of that class, we’re able to make sure that our warning has fired (and no other errors happened) without needing to wait:

tictoc::tic()
test_that("large integers get a custom warning", {
  expect_error(
    tryCatch(
      func(.Machine$integer.max),
      big_x = rlang::abort("the warning fired", class = "success")
    ),
    class = "success"
  )
})
Test passed 
tictoc::toc()
0.028 sec elapsed

Now, an obvious downside is that we’re no longer testing to make sure the function works after the warning gets fired. In this case, where we’re expecting that triggering this warning means this function will return NA, we should probably be testing to make sure that this function actually does return NA after the warning fires. But in plenty of other situations this can be a useful way to speed up your test suites while still making sure that you’re giving your users as much feedback as possible, when you’re expecting to give it.

Footnotes

  1. I’ve been bitten so many times by tests that expect a warning, rather than a specific warning. Giving functions malformed input often triggers multiple warnings, so if you aren’t checking for your specific warning message or class, you might be surprised that your custom warning never actually fires!↩︎

  2. Well, the classed warnings and testing specifically for that warning. The function is a mess.↩︎

]]> R Tutorials Package development https://mm218.dev/posts/2024-04-12-testing-expensive-functions/ Fri, 12 Apr 2024 00:00:00 GMT rsi 0.2.0 is now on CRAN! Mike Mahoney https://mm218.dev/posts/2024-03-29-rsi-020/ I’m deligted to announce that version 0.2.0 of rsi, my package for handling common spatial ML data pre-processing tasks, is now officially on CRAN. rsi aims to handle downloading, masking, rescaling, and compositing data from STAC endpoints, computing spectral indices from that same data, amd wrangling the outputs into bricks ready for modeling workflows – and to do so in a user-friendly and extensible way. This release adds wrappers for more data sources, makes it easier to download high-quality water data from Landsat, and fixes some bugs while simplifying the internals of the package.

You can install rsi from CRAN via:

install.packages("rsi")

This post will walk through a few of the most user-visible changes, starting with…

Downloading water data

In older versions of rsi, the default landsat_mask_function() would mask your data so that your final files (and composites) only contained the highest quality observations over land. That meant that waterbodies (like the large area in the top left of this image) would always be empty:

library(rsi)
future::plan("multisession")

aoi <- sf::st_point(c(-76.1376841, 43.0351335))
aoi <- sf::st_set_crs(sf::st_sfc(aoi), 4326)
aoi <- sf::st_buffer(sf::st_transform(aoi, 5070), 10000)

landsat <- get_landsat_imagery(
  aoi,
  start_date = "2021-06-01",
  end_date = "2021-08-31",
  output_filename = tempfile(fileext = ".tif")
)

terra::plot(terra::rast(landsat))

This works great if you only care about land observations, but has an obvious flaw otherwise. Thanks to @mateuszrydzik, landsat_mask_function() starting in version 0.2.0 now has an argument, include, which you can use to also include high quality observations over water in your final outputs:

landsat <- get_landsat_imagery(
  aoi,
  start_date = "2021-06-01",
  end_date = "2021-08-31",
  mask_function = \(r) landsat_mask_function(r, include = "both"),
  output_filename = tempfile(fileext = ".tif")
)

terra::plot(terra::rast(landsat))

You can also set include = "water" to only include data over waterbodies, and exclude all data over land.

Downloading even more data

Two new functions in this release provide friendly wrappers around get_stac_data() to help you access specific data sources.

First, thanks to @h-a-graham, the new get_alos_palsar_imagery() function provides a wrapper for accessing data from ALOS PALSAR:

alos <- get_alos_palsar_imagery(
  aoi,
  "2021-06-01",
  "2021-09-01"
)

terra::plot(terra::rast(alos))

And separately, the new get_naip_imagery() function provides access to data from the National Agricultural Imagery Program across the United States:

naip <- get_naip_imagery(
  aoi,
  "2018-01-01",
  "2021-01-01",
  pixel_x_size = 30,
  pixel_y_size = 30
)

terra::plotRGB(terra::rast(naip))

New vignette

One of the last significant user-facing changes is the addition of a new vignette, called “How can I…?”. This vignette is meant to collect common use-cases into a single document, providing users with a “cookbook” containing methods they might use to approach their current problems. If you’ve got a use-case that took you a moment to figure out, or a problem that you think rsi should be able to solve, let me know through an issue on GitHub so I can incorporate it into this vignette!

The other improvements in this release focus mostly on bug squashing – including a nasty bug where downloading multiple tiles using composite_function = NULL could fail – and simplifying the internals of get_stac_data() to make it more maintainable and extensible into the future.

Acknowledgments

As always, huge thanks to the folks who have been involved in testing and improving this package since our last release: @agronomofiorentini, @h-a-graham, and @mateuszrydzik. It’s extremely appreciated.

]]> R Spatial geospatial data R packages https://mm218.dev/posts/2024-03-29-rsi-020/ Fri, 29 Mar 2024 00:00:00 GMT rsi is now on CRAN! Mike Mahoney https://mm218.dev/posts/2024-01-10-rsi-cran/ I’m thrilled to announce that version 0.1.0 of rsi, a new package for handling common spatial ML data pre-processing tasks, is now officially on CRAN. rsi helps you download1 data from STAC APIs, calculate spectral indices from that data (with an interface to the Awesome Spectral Indices project), and efficiently stack rasters together to help build predictor bricks.

You can install it from CRAN via:

install.packages("rsi")

rsi is motivated by my own work doing landscape-level natural resources mapping, both with Permian Global Research (who have been incredibly supportive of me open-sourcing this project that originated from my contract work) and with CAFRI (see for instance our newest Landsat-based forest carbon maps). Most of these projects have a pretty consistent data-processing workflow: I need to go get data from somewhere, calculate indices and other predictors against it, and then glue all those derived products together into a predictor brick I can actually extract my predictor values from. rsi handles every step of that process and does so efficiently, letting you spend less time on your pre-processing and more time on your actual models.

I’ve written about rsi a few times before, including about how rsi does sandboxing when running code from untrusted sources and how to make the most of the get_stac_data() family of functions. This post is going to provide a more holistic introduction to rsi,2 focusing on the main functions in the package and how you might use them as part of your pre-processing workflows.

Downloading data from STAC APIs

Perhaps the most useful piece of rsi3 is the get_stac_data() family of functions, which help you download data from STAC APIs. As an example, let’s say that we’ve used sf to define some area of interest that we want to download data for:

aoi <- sf::st_point(c(-76.1376841, 43.0351335))
aoi <- sf::st_set_crs(sf::st_sfc(aoi), 4326)
aoi <- sf::st_buffer(sf::st_transform(aoi, 5070), 10000)

If we wanted to get land cover data for this area from 2021, using the USGS LCMAP product from Planetary Computer, we could use the get_stac_data() function from rsi like so:

library(rsi)
future::plan("multisession")

lcpri <- get_stac_data(
  aoi,
  start_date = "2021-01-01", # Making sure we only grab data from 2021
  end_date = "2021-12-31",
  asset_names = "lcpri", # The name of the primary land cover product in LCMAP
  stac_source = "https://planetarycomputer.microsoft.com/api/stac/v1/",
  collection = "usgs-lcmap-conus-v13", # The name of LCMAP on Planetary Computer
  output_filename = tempfile(fileext = ".tif"),
)

terra::plot(terra::rast(lcpri))

If our data is split across multiple tiles, get_stac_data() will automatically merge them into a single composite output. There’s an article on the rsi website that gives a brief overview of the STAC family of standards, and how you can use the arguments to get_stac_data() to flexibly control exactly what you’re downloading and how.

In addition to get_stac_data(), rsi also provides a number of higher-level functions for interacting with popular satellite imagery data sources – specifically, Landsat, Sentinel-2, and Sentinel-1 (including the Sentinel-1 RTC product). In addition to downloading and merging tiles, these functions will also handle creating composite rasters from multiple separate images, masking out clouds and other low-quality data, and (where possible) automatically rescaling your data using offsets and scaling factors defined in the STAC metadata

landsat <- get_landsat_imagery(
  aoi,
  start_date = "2021-06-01",
  end_date = "2021-08-31",
  output_filename = tempfile(fileext = ".tif")
)
terra::plot(terra::rast(landsat))

Note that the inputs to these functions are normal R objects, and the outputs are the file paths your data was saved to. A key difference of rsi from other packages for downloading data from STAC endpoints is that rsi doesn’t introduce a new data model,4 and instead is focused on getting data onto your local filesystem as quickly as possible. Ideally, you’ll be able to let rsi handle all the “cloud-native geospatial” stuff for you, and then use the outputs with whatever tools you’re most comfortable with.5

For more information on the get_stac_data() family of functions, check out the corresponding article on the rsi website!

Calculate spectral indices

Most other functions in rsi also use the approach of accepting normal R objects as function inputs, and returning file paths to raster outputs. For instance, rsi provides a function, calculate_indices(), which can be used to calculate spectral indices from the band values of an input raster.

By default, this function is designed to work with spectral indices from the (awesome) Awesome Spectral Indices project. In fact, the spectral_indices() function in rsi will give you a processed tibble containing the full list of indices from that project:

spectral_indices()
# A tibble: 231 × 9
   application_domain bands     contributor   date_of_addition formula long_name
   <chr>              <list>    <chr>         <chr>            <chr>   <chr>    
 1 vegetation         <chr [2]> https://gith… 2021-11-17       (N - 0… Aerosol …
 2 vegetation         <chr [2]> https://gith… 2021-11-17       (N - 0… Aerosol …
 3 water              <chr [6]> https://gith… 2022-09-22       (B + G… Augmente…
 4 vegetation         <chr [2]> https://gith… 2021-09-20       (1 / G… Anthocya…
 5 vegetation         <chr [3]> https://gith… 2022-04-08       N * ((… Anthocya…
 6 vegetation         <chr [4]> https://gith… 2021-05-11       (N - (… Atmosphe…
 7 vegetation         <chr [4]> https://gith… 2021-05-14       sla * … Adjusted…
 8 vegetation         <chr [2]> https://gith… 2022-04-08       (N * (… Advanced…
 9 water              <chr [4]> https://gith… 2021-09-18       4.0 * … Automate…
10 water              <chr [5]> https://gith… 2021-09-18       B + 2.… Automate…
# ℹ 221 more rows
# ℹ 3 more variables: platforms <list>, reference <chr>, short_name <chr>

The first time you run spectral_indices() will attempt to download the current set of spectral indices from GitHub, and then will store that download in a cache file. Future calls to spectral_indices() will then be a bit faster, as rsi will only try to update its downloaded indices if the cache is more than a day old.

There are also two functions, filter_bands() and filter_platforms(), which make it easy to filter the tibble of indices based on what bands or platforms you have available in your data. For instance, if we wanted to get the full set of indices that we could calculate with our downloaded Landsat data, we could pass the names of those bands to the second argument in filter_bands():

spectral_indices() |> 
  filter_bands(names(terra::rast(landsat)))
# A tibble: 128 × 9
   application_domain bands     contributor   date_of_addition formula long_name
   <chr>              <list>    <chr>         <chr>            <chr>   <chr>    
 1 vegetation         <chr [2]> https://gith… 2021-11-17       (N - 0… Aerosol …
 2 vegetation         <chr [2]> https://gith… 2021-11-17       (N - 0… Aerosol …
 3 water              <chr [6]> https://gith… 2022-09-22       (B + G… Augmente…
 4 vegetation         <chr [2]> https://gith… 2022-04-08       (N * (… Advanced…
 5 water              <chr [4]> https://gith… 2021-09-18       4.0 * … Automate…
 6 water              <chr [5]> https://gith… 2021-09-18       B + 2.… Automate…
 7 burn               <chr [2]> https://gith… 2021-04-07       1.0 / … Burned A…
 8 burn               <chr [2]> https://gith… 2022-04-20       1.0/((… Burned A…
 9 vegetation         <chr [3]> https://gith… 2022-01-17       B / (R… Blue Chr…
10 soil               <chr [4]> https://gith… 2022-04-08       ((S1 +… Bare Soi…
# ℹ 118 more rows
# ℹ 3 more variables: platforms <list>, reference <chr>, short_name <chr>

And then we can pass this tibble directly to calculate_indices() to calculate all 128 of these indices against our downloaded Landsat image:

indices <- calculate_indices(
  landsat,
  filter_bands(spectral_indices(), names(terra::rast(landsat))),
  tempfile(fileext = ".tif")
)

|---------|---------|---------|---------|
=========================================
                                          
terra::plot(terra::rast(indices))

You should always skim the formulas you’re going to run before calling calculate_indices() – these are downloaded from a live GitHub URL and can change over time – but the actual execution of downloaded code happens in a sandboxed environment, which should make it harder for any untrustworthy code to damage your system. If you’re interested, I wrote post a few months ago with a bit more information about this sandboxing.

Glue it all together

And last, but certainly not least, rsi provides a function called stack_rasters() which helps you bind multiple rasters into a single predictor brick. Similar to calculate_indices(), the first argument is a (vector of) file paths to the rasters you want to stack together, and the output is a new file path to the raster that was created:

stack_rasters(c(lcpri, indices), tempfile(fileext = ".tif")) |> 
  terra::rast() |> 
  terra::plot()
Warning in CPL_gdalwarp(source, destination, options, oo, doo, config_options,
: GDAL Error 6: /tmp/RtmpvJTqGP/file302c19635cbe42.tif, band 1: SetColorTable()
not supported for multi-sample TIFF files.
Warning in CPL_gdalwarp(source, destination, options, oo, doo, config_options,
: GDAL Message 1: /tmp/RtmpvJTqGP/file302c19635cbe42.tif, band 2: Setting
nodata to nan on band 2, but band 1 has nodata at 0. The TIFFTAG_GDAL_NODATA
only support one value per dataset. This value of nan will be used for all
bands on re-opening

If your output file ends in vrt, you can even do this without copying any data thanks to GDAL’s virtual raster format:

stack_rasters(c(lcpri, indices), tempfile(fileext = ".vrt")) |> 
  terra::rast() |> 
  terra::plot()
Warning in x@cpp$sampleRegularRaster(size): GDAL Message 6: Resampling method
not supported on paletted band. Falling back to nearest neighbour

This feels like such a small thing when you say it out loud, but I think is my favorite part of the package. This was a surprisingly hard thing to do with existing tools. If your rasters are different extents or resolutions, stack_rasters() will automatically set the extent of your output and your target resolution based on the first raster in your input vector – or you can override that default behavior using the extent and resolution argument.

Crucially, stack_rasters() relies almost entirely on GDAL’s warper and VRT format to do this, meaning it’s able to efficiently stack together much larger than memory data sets. And (particularly when using VRT outputs), this means that stack_rasters() can be a lot faster than approaches that involve reading the raster into R and then writing it back out.

Acknowledgments

And that’s rsi version 0.1.0! Hopefully this package can be as useful for others as it’s already been for me at simplifying my data pre-processing workflows. A huge, huge, huge thank you to the folks who have been involved in testing and improving the alpha release, and helping me reshape it into this first release: @agronomofiorentini, @h-a-graham, and @mateuszrydzik. It would be a worse package without you!

Footnotes

  1. (and mask, and rescale, and composite)↩︎

  2. Updating the first post introducing the first alpha development version.↩︎

  3. And most involved; cloc tells me there are 1300 lines of code in the R directory of this package, with 718 of them in get_stac_data.R.↩︎

  4. Kinda; the band mapping objects are – as discussed in the article linked above – a relatively complex data structure. But hopefully users mostly don’t need to think about those.↩︎

  5. Carl Boettiger at AGU this year: “Cloud native just means using HTTP range requests. We’ve been doing cloud native since the 90s”.↩︎

]]> R Tutorials Spatial geospatial data R packages https://mm218.dev/posts/2024-01-10-rsi-cran/ Wed, 10 Jan 2024 00:00:00 GMT Why is View() capitalized, anyway? Mike Mahoney https://mm218.dev/posts/2023-12-07-View/ Over on BlueSky, David John Baker asks:

Post from David John Baker on BlueSky: Why is does the `View()` function in R start with a capital V? I have no idea why. Does this have to do with functions that are not just dealing with some sort of standard output? #rstats

My first thought on reading this was that View() in RStudio is built to mask the View() function from utils, and so capitalizes the V because utils capitalized the V. Things are the way they are because they are the way they are, yet again.

But of course, that’s not the actual question – not only did David specify he’s talking about the function in R, not in RStudio, but of course someone decided that utils should capitalize the View() function as well. And this is weird! There are not that many functions in base R that start with a capital letter:

getOption("defaultPackages") |> 
  setdiff("datasets") |> 
  vapply(\(x) paste0("package:", x), character(1)) |> 
  ls() |> 
  strsplit("") |> 
  Filter(f = \(x) x[[1]] %in% LETTERS) |> 
  vapply(paste0, character(1), collapse = "")
 [1] "Rprof"               "Rprofmem"            "RShowDoc"           
 [4] "RSiteSearch"         "Rtangle"             "RtangleFinish"      
 [7] "RtangleRuncode"      "RtangleSetup"        "RtangleWritedoc"    
[10] "RweaveChunkPrefix"   "RweaveEvalWithOpt"   "RweaveLatex"        
[13] "RweaveLatexFinish"   "RweaveLatexOptions"  "RweaveLatexSetup"   
[16] "RweaveLatexWritedoc" "RweaveTryStop"       "Stangle"            
[19] "Sweave"              "SweaveHooks"         "SweaveSyntaxLatex"  
[22] "SweaveSyntaxNoweb"   "SweaveSyntConv"      "URLdecode"          
[25] "URLencode"           "View"

These functions all follow one of three patterns:

  • Functions that start with R or S, capitalized because it’s the name of a programming language;
  • Functions that start with URL;
  • View() by itself!

Update:

A few minutes after publishing, Konrad Rudolph points out on Mastodon that there are plenty of functions in the base package itself that are capitalized and don’t match these rules:

ls("package:base") |> 
  strsplit("") |> 
  Filter(f = \(x) x[[1]] %in% LETTERS) |> 
  vapply(paste0, character(1), collapse = "")
 [1] "Arg"                     "Conj"                   
 [3] "Cstack_info"             "Encoding"               
 [5] "Encoding<-"              "F"                      
 [7] "Filter"                  "Find"                   
 [9] "I"                       "Im"                     
[11] "ISOdate"                 "ISOdatetime"            
[13] "La_library"              "La_version"             
[15] "La.svd"                  "LETTERS"                
[17] "Map"                     "Math.data.frame"        
[19] "Math.Date"               "Math.difftime"          
[21] "Math.factor"             "Math.POSIXt"            
[23] "Mod"                     "NCOL"                   
[25] "Negate"                  "NextMethod"             
[27] "NROW"                    "OlsonNames"             
[29] "Ops.data.frame"          "Ops.Date"               
[31] "Ops.difftime"            "Ops.factor"             
[33] "Ops.numeric_version"     "Ops.ordered"            
[35] "Ops.POSIXt"              "Position"               
[37] "R_compiled_by"           "R_system_version"       
[39] "R.home"                  "R.version"              
[41] "R.Version"               "R.version.string"       
[43] "Re"                      "Recall"                 
[45] "Reduce"                  "RNGkind"                
[47] "RNGversion"              "Summary.data.frame"     
[49] "Summary.Date"            "Summary.difftime"       
[51] "Summary.factor"          "Summary.numeric_version"
[53] "Summary.ordered"         "Summary.POSIXct"        
[55] "Summary.POSIXlt"         "Sys.chmod"              
[57] "Sys.Date"                "Sys.getenv"             
[59] "Sys.getlocale"           "Sys.getpid"             
[61] "Sys.glob"                "Sys.info"               
[63] "Sys.localeconv"          "Sys.readlink"           
[65] "Sys.setenv"              "Sys.setFileTime"        
[67] "Sys.setLanguage"         "Sys.setlocale"          
[69] "Sys.sleep"               "Sys.time"               
[71] "Sys.timezone"            "Sys.umask"              
[73] "Sys.unsetenv"            "Sys.which"              
[75] "T"                       "UseMethod"              
[77] "Vectorize"

One of those is Filter(), which I literally used in the original code chunk and somehow didn’t notice started with a capital letter.

These functions fall into a number of other groups – functions for working with complex numbers, for functional programming and recursion, for setting and getting system information and variables, group generic functions, and a few proper nouns and acronyms like “C” and “RNG”. It’s still not common – 77 out of 1268 objects in the base namespace start with a capital letter – but it’s more common than in recommended packages.

< / update >

But even with that said, View() is the only function in recommended packages that starts with a capital letter that’s not named after a proper noun.1 And it’s been weird for a long time. The original version of the function was also capitalized when it was added by Professor Brian D Ripley way back in 2007:

Commit by Prof. Brian D Ripley on Feb 19 2007, adding the capitalized `View()` function.

Now here’s where we enter the world of wild speculation. Because if we look at the documentation for View(), we can see that this function is meant to “Invoke a Data Viewer”, with “Data Viewer” capitalized like a proper noun in both the function title and description:

The rd documentation file for `View()`. Notably, the Description field capitalizes the phrase Data Viewer.

This isn’t a universal style. If we look for instance at the documentation for edit(), which pre-dates View(),2 we can see that “text editor” is capitalized in the title but not in the description:

The rd documentation file for `edit()`. Notably, the Description field _does not_ capitalize the phrase text editor.

So this makes me suspect that Data Viewer here is being used as a proper noun. This wouldn’t be a unique usage; for instance, the RStudio user guide also describes its viewer as a Data Viewer, capitalized, and a search for “Data Viewer” suggests the phrase is capitalized something like 2/3 of the time on public websites. I have absolutely no knowledge of the tech jargon of 2007, or of the proper styling of “data viewer”,3 but it does seem like it’s sometimes a proper noun.

So, to wildly speculate just a little further: is View() capitalized because “Viewer” was capitalized, at least back in 2007?

Now, here’s a decent piece of evidence against this conjecture: the details section of the View() documentation, added at the same time as the title and description,4 doesn’t capitalize the phrase “data viewer”:

The details section of the `View()` documentation. Data viewer is not capitalized.

That said, this is also not unique. That same RStudio user guide also sometimes writes “data viewer” in lowercase, though in most of the guide it’s treated as a proper noun.

So there’s one guess: maybe View() is capitalized because, like the functions starting with R and S and URL, it’s named after a proper noun. Things are the way they are because they are the way they are, yet again.

If anyone knows more, though – or feels like asking Professor Ripley – please drop me a line. I’d love to know the actual answer.

Footnotes

  1. Merriam-Webster calls “URL” a noun, even if it originated as an acronym, get out of my comments.↩︎

  2. Though I’m not sure who originally wrote it; I get a bit lost in the VCS logs from before base functions were split into sub-packages.↩︎

  3. And I assume these two things are related!↩︎

  4. And the function itself; all of this documentation is also still in the official help file.↩︎

]]> R https://mm218.dev/posts/2023-12-07-View/ Thu, 07 Dec 2023 00:00:00 GMT Helping R find the right methods for unserialized complex objects. Mike Mahoney https://mm218.dev/posts/2023-11-27-objects-loading-namespaces/ So here’s a problem you may have encountered. Say you’ve serialized some complicated R object using a function like saveRDS(), which saves your objects as a binary file. For instance, we can take the boston_canopy sf object from spatialsample, which looks like this:

spatialsample::boston_canopy |>
  head()
Simple feature collection with 6 features and 18 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 749383.6 ymin: 2913059 xmax: 801174.4 ymax: 2965741
Projected CRS: NAD83 / Massachusetts Mainland (ftUS)
# A tibble: 6 × 19
  grid_id land_area canopy_gain canopy_loss canopy_no_change canopy_area_2014
  <chr>       <dbl>       <dbl>       <dbl>            <dbl>            <dbl>
1 AB-4      795045.      15323.       3126.           53676.           56802.
2 I-33      265813.       8849.      11795.           78677.           90472.
3 AO-9      270153        6187.       1184.           26930.           28114.
4 H-10     2691490.      73098.      80362.          345823.          426185.
5 V-7       107890.        219.       3612.             240.            3852.
6 Q-22     2648089.     122211.     154236.         1026632.         1180868.
# ℹ 13 more variables: canopy_area_2019 <dbl>, change_canopy_area <dbl>,
#   change_canopy_percentage <dbl>, canopy_percentage_2014 <dbl>,
#   canopy_percentage_2019 <dbl>, change_canopy_absolute <dbl>,
#   mean_temp_morning <dbl>, mean_temp_evening <dbl>, mean_temp <dbl>,
#   mean_heat_index_morning <dbl>, mean_heat_index_evening <dbl>,
#   mean_heat_index <dbl>, geometry <MULTIPOLYGON [US_survey_foot]>

And save it out as an RDS file:

bos_rds <- tempfile(fileext = ".rds")
saveRDS(spatialsample::boston_canopy, bos_rds)

When you unserialize that file, R won’t automatically be able to find all the methods associated with your complicated object. For instance, in a new session, our boston_canopy data doesn’t print nearly as nicely:1

console_output <- tempfile()
callr::r(
  \(bos_rds) {
    readRDS(bos_rds) |>
      head(1) |>
      print()
    NULL
  },
  args = list(bos_rds = bos_rds),
  stdout = console_output
) |>
  invisible()
readLines(console_output)
 [1] "  grid_id land_area canopy_gain canopy_loss canopy_no_change canopy_area_2014"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
 [2] "1    AB-4  795044.8    15323.45    3126.004         53676.05         56802.05"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
 [3] "  canopy_area_2019 change_canopy_area change_canopy_percentage"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
 [4] "1          68999.5           12197.44                  21.4736"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
 [5] "  canopy_percentage_2014 canopy_percentage_2019 change_canopy_absolute"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
 [6] "1                7.14451               8.678693               1.534183"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
 [7] "  mean_temp_morning mean_temp_evening mean_temp mean_heat_index_morning"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
 [8] "1          75.72113          86.04341  91.51711                76.97335"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
 [9] "  mean_heat_index_evening mean_heat_index"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
[10] "1                90.91202         96.9585"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
[11] "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                geometry"
[12] "1 781922.7, 781441.9, 780424.1, 780361.1, 780386.0, 780385.0, 780385.7, 780386.0, 780386.7, 780388.0, 780389.3, 780390.9, 780392.8, 780394.5, 780394.8, 780394.8, 780394.1, 780393.5, 780391.9, 780391.5, 780393.2, 780393.8, 780394.1, 780395.1, 780394.5, 780414.6, 780415.6, 780416.9, 780429.8, 780430.5, 780430.5, 780519.8, 780522.4, 780522.7, 780523.0, 780523.4, 780523.7, 780524.0, 780524.3, 780524.3, 780524.7, 780525.0, 780525.6, 780526.3, 780527.3, 780528.6, 780529.9, 780531.2, 780532.8, 780534.1, 780535.7, 780537.6, 780539.6, 780541.5, 780543.8, 780546.1, 780548.4, 780550.6, 780552.9, 780555.2, 780557.5, 780559.7, 780562.0, 780564.3, 780566.5, 780568.2, 780570.1, 780571.7, 780573.4, 780575.0, 780576.6, 780577.6, 780578.9, 780579.9, 780580.5, 780581.5, 780582.1, 780582.8, 780584.1, 780585.4, 780586.0, 780587.0, 780587.0, 780586.4, 780586.0, 780585.7, 780585.4, 780584.7, 780584.4, 780584.4, 780583.1, 780582.5, 780582.1, 780582.1, 780582.1, 780582.8, 780583.4, 780584.1, 780584.4, 780585.4, 780586.0, 780586.7, 780587.6, 780588.6, 780589.6, 780590.9, 780591.9, 780594.5, 780596.7, 780599.3, 780601.3, 780602.3, 780664.3, 780665.3, 780665.3, 780669.1, 780676.3, 780683.1, 780705.5, 780713.0, 780713.0, 780712.0, 780751.0, 780750.6, 780750.3, 780764.0, 780803.6, 780809.1, 780809.1, 780809.1, 780810.1, 780811.0, 780812.7, 780814.3, 780816.6, 780817.9, 780819.5, 780821.4, 780822.7, 780824.0, 780825.3, 780827.0, 780828.6, 780829.9, 780831.5, 780842.2, 780846.1, 780858.1, 780856.2, 780855.9, 780856.2, 780856.2, 780856.2, 780856.5, 780856.5, 780856.8, 780857.2, 780857.5, 780857.8, 780858.5, 780858.8, 780859.4, 780859.7, 780860.7, 780861.4, 780862.0, 780863.0, 780864.0, 780864.6, 780865.6, 780866.6, 780867.2, 780868.2, 780869.2, 780870.1, 780871.1, 780871.8, 780872.7, 780873.7, 780874.7, 780875.7, 780876.6, 780877.6, 780878.6, 780879.6, 780880.2, 780881.2, 780881.8, 780882.8, 780883.5, 780884.4, 780885.1, 780885.4, 780886.1, 780886.4, 780887.3, 780888.0, 780890.3, 780892.9, 780894.5, 780895.5, 780944.5, 780945.1, 780980.5, 781028.3, 781076.6, 781123.7, 781155.6, 781169.2, 781167.6, 781163.7, 781166.3, 781165.6, 781186.7, 781186.7, 781210.8, 781211.1, 781306.2, 781340.6, 781341.9, 781373.4, 781399.7, 781431.2, 781449.7, 781463.4, 781483.8, 781483.8, 781485.8, 781488.7, 781490.6, 781491.9, 781492.9, 781493.9, 781494.2, 781495.8, 781496.2, 781495.8, 781496.2, 781496.2, 781497.1, 781497.8, 781498.8, 781499.7, 781500.4, 781500.7, 781501.4, 781501.7, 781502.0, 781502.7, 781503.0, 781504.0, 781505.3, 781506.2, 781507.2, 781509.8, 781510.8, 781512.1, 781512.7, 781513.4, 781514.0, 781514.3, 781514.7, 781514.7, 781515.3, 781515.6, 781516.3, 781516.6, 781517.3, 781517.9, 781519.2, 781520.5, 781521.5, 781522.1, 781523.1, 781523.8, 781524.7, 781525.7, 781526.0, 781526.4, 781527.0, 781527.3, 781529.3, 781534.2, 781539.0, 781543.2, 781544.2, 781545.2, 781545.8, 781546.5, 781547.1, 781547.8, 781548.8, 781549.7, 781551.0, 781552.3, 781553.6, 781554.9, 781555.9, 781557.2, 781557.5, 781558.2, 781558.8, 781559.2, 781559.5, 781560.1, 781560.1, 781560.8, 781564.0, 781566.0, 781567.9, 781570.5, 781571.8, 781572.5, 781572.8, 781573.4, 781574.1, 781574.7, 781575.4, 781576.0, 781576.4, 781577.0, 781577.7, 781579.9, 781582.9, 781585.5, 781588.1, 781591.0, 781593.2, 781595.5, 781597.8, 781598.8, 781599.7, 781602.3, 781608.2, 781614.0, 781617.3, 781620.5, 781626.4, 781629.3, 781632.2, 781636.1, 781639.4, 781642.3, 781644.6, 781646.5, 781648.4, 781650.4, 781651.4, 781652.3, 781653.3, 781654.3, 781656.6, 781658.8, 781664.7, 781670.5, 781677.7, 781682.5, 781684.8, 781687.1, 781689.4, 781690.7, 781691.6, 781692.6, 781693.6, 781695.2, 781696.8, 781698.5, 781700.4, 781702.3, 781703.3, 781704.0, 781704.9, 781705.6, 781706.6, 781708.2, 781710.5, 781712.7, 781714.7, 781715.3, 781716.3, 781717.6, 781718.3, 781718.9, 781719.2, 781719.2, 781717.9, 781717.3, 781717.0, 781716.6, 781716.6, 781716.3, 781716.6, 781717.0, 781717.3, 781718.9, 781719.2, 781719.2, 781719.6, 781720.2, 781720.2, 781719.9, 781719.6, 781719.2, 781718.6, 781717.6, 781717.3, 781717.0, 781717.0, 781717.3, 781717.6, 781717.9, 781718.3, 781718.9, 781720.2, 781720.9, 781720.9, 781720.9, 781721.2, 781721.2, 781721.8, 781722.5, 781723.5, 781724.1, 781724.8, 781725.1, 781725.7, 781726.7, 781728.3, 781729.0, 781729.9, 781731.2, 781732.5, 781736.8, 781743.6, 781743.6, 781743.3, 781780.0, 781831.3, 781904.3, 781922.7, 2965534.6, 2964701.8, 2964701.8, 2964810.9, 2964811.0, 2964821.4, 2964824.0, 2964826.3, 2964828.6, 2964835.7, 2964847.4, 2964859.4, 2964869.8, 2964880.2, 2964885.4, 2964890.6, 2964894.5, 2964898.4, 2964906.5, 2964966.6, 2965059.1, 2965076.0, 2965081.2, 2965132.8, 2965190.0, 2965190.0, 2965144.8, 2965108.1, 2965107.2, 2965094.5, 2965079.6, 2965081.2, 2965079.2, 2965078.6, 2965077.3, 2965076.3, 2965075.0, 2965074.4, 2965073.4, 2965072.1, 2965070.5, 2965068.8, 2965066.9, 2965065.3, 2965063.6, 2965061.7, 2965060.4, 2965058.8, 2965056.8, 2965055.9, 2965054.6, 2965053.6, 2965052.3, 2965051.3, 2965050.3, 2965049.4, 2965049.0, 2965048.4, 2965048.1, 2965047.7, 2965047.7, 2965048.1, 2965048.1, 2965048.4, 2965049.0, 2965049.7, 2965050.0, 2965050.7, 2965051.3, 2965052.0, 2965053.3, 2965054.2, 2965055.5, 2965056.8, 2965058.1, 2965059.4, 2965061.0, 2965062.0, 2965066.9, 2965071.1, 2965075.7, 2965092.2, 2965108.8, 2965133.8, 2965146.1, 2965159.1, 2965164.6, 2965208.8, 2965222.7, 2965236.7, 2965246.8, 2965252.0, 2965256.8, 2965261.1, 2965265.0, 2965269.5, 2965273.1, 2965274.7, 2965276.0, 2965277.0, 2965278.3, 2965279.6, 2965280.5, 2965281.5, 2965282.5, 2965283.5, 2965284.4, 2965285.1, 2965285.7, 2965286.1, 2965286.1, 2965286.1, 2965287.4, 2965287.4, 2965279.6, 2965279.9, 2965279.6, 2965279.9, 2965281.2, 2965327.0, 2965385.4, 2965445.5, 2965445.5, 2965391.3, 2965325.4, 2965324.7, 2965321.8, 2965359.8, 2965363.7, 2965367.9, 2965373.4, 2965378.9, 2965384.5, 2965390.3, 2965395.2, 2965399.1, 2965402.3, 2965405.9, 2965406.5, 2965407.2, 2965408.2, 2965408.5, 2965409.1, 2965409.5, 2965409.8, 2965410.1, 2965409.8, 2965408.8, 2965448.1, 2965449.7, 2965451.0, 2965451.3, 2965452.0, 2965453.0, 2965453.9, 2965454.3, 2965454.9, 2965455.6, 2965456.2, 2965456.9, 2965457.5, 2965458.2, 2965458.5, 2965459.1, 2965459.8, 2965460.1, 2965460.4, 2965460.8, 2965461.1, 2965461.4, 2965461.4, 2965461.4, 2965461.7, 2965461.7, 2965461.4, 2965461.4, 2965461.1, 2965461.1, 2965460.8, 2965460.4, 2965460.1, 2965459.5, 2965459.1, 2965458.5, 2965457.8, 2965456.9, 2965456.2, 2965455.9, 2965454.3, 2965452.3, 2965451.3, 2965450.0, 2965448.4, 2965447.1, 2965445.5, 2965436.4, 2965419.8, 2965392.9, 2965363.7, 2965340.0, 2965318.9, 2965318.5, 2965300.4, 2965301.7, 2965302.3, 2965302.6, 2965303.6, 2965304.6, 2965303.6, 2965376.0, 2965416.6, 2965441.6, 2965741.3, 2965738.1, 2965442.2, 2965442.9, 2965359.1, 2965359.1, 2965488.7, 2965493.2, 2965490.6, 2965487.4, 2965485.1, 2965484.5, 2965483.2, 2965481.9, 2965465.0, 2965463.4, 2965460.8, 2965456.5, 2965452.6, 2965449.1, 2965445.8, 2965443.5, 2965430.9, 2965426.3, 2965424.7, 2965423.1, 2965421.8, 2965419.5, 2965417.2, 2965415.3, 2965413.0, 2965412.7, 2965412.1, 2965411.4, 2965411.1, 2965410.8, 2965410.4, 2965410.4, 2965409.5, 2965409.1, 2965408.8, 2965408.8, 2965408.5, 2965408.5, 2965408.2, 2965407.8, 2965407.8, 2965407.5, 2965407.2, 2965407.2, 2965406.9, 2965406.9, 2965406.2, 2965405.9, 2965405.6, 2965405.2, 2965404.9, 2965404.6, 2965403.9, 2965403.6, 2965403.0, 2965402.0, 2965401.0, 2965399.7, 2965398.4, 2965397.4, 2965396.8, 2965396.1, 2965395.5, 2965393.9, 2965390.6, 2965388.3, 2965386.4, 2965386.1, 2965385.7, 2965385.7, 2965385.7, 2965385.7, 2965385.7, 2965385.4, 2965385.1, 2965384.5, 2965383.8, 2965383.2, 2965382.2, 2965381.2, 2965380.2, 2965379.9, 2965379.3, 2965378.6, 2965377.6, 2965376.3, 2965375.0, 2965373.1, 2965369.8, 2965358.5, 2965352.3, 2965346.1, 2965340.3, 2965338.3, 2965337.7, 2965337.0, 2965336.7, 2965336.1, 2965335.7, 2965335.1, 2965334.8, 2965334.8, 2965334.4, 2965334.1, 2965333.8, 2965334.1, 2965334.1, 2965334.4, 2965334.4, 2965334.1, 2965333.5, 2965332.8, 2965332.5, 2965332.2, 2965331.8, 2965331.8, 2965331.8, 2965331.8, 2965331.5, 2965331.5, 2965331.5, 2965331.5, 2965330.9, 2965330.2, 2965329.3, 2965328.6, 2965327.6, 2965326.0, 2965325.0, 2965324.1, 2965323.4, 2965322.8, 2965322.4, 2965321.8, 2965321.1, 2965320.8, 2965320.5, 2965320.5, 2965321.1, 2965321.5, 2965322.1, 2965322.8, 2965323.1, 2965323.7, 2965324.1, 2965325.0, 2965326.7, 2965328.3, 2965330.2, 2965333.1, 2965335.4, 2965337.4, 2965339.3, 2965341.9, 2965344.8, 2965348.4, 2965353.6, 2965358.8, 2965364.0, 2965369.8, 2965374.4, 2965379.3, 2965384.8, 2965387.4, 2965390.3, 2965394.8, 2965401.3, 2965430.6, 2965435.1, 2965436.1, 2965437.1, 2965438.4, 2965439.7, 2965440.3, 2965441.3, 2965443.5, 2965459.1, 2965462.7, 2965466.3, 2965470.2, 2965473.7, 2965477.3, 2965481.2, 2965484.1, 2965487.7, 2965491.0, 2965494.2, 2965498.1, 2965501.7, 2965505.9, 2965509.8, 2965511.7, 2965513.0, 2965514.0, 2965515.0, 2965518.6, 2965520.8, 2965521.2, 2965521.8, 2965522.8, 2965523.1, 2965523.7, 2965524.7, 2965525.7, 2965526.0, 2965526.7, 2965527.0, 2965527.6, 2965529.3, 2965532.2, 2965533.8, 2965535.4, 2965536.7, 2965539.0, 2965546.8, 2965554.6, 2965566.9, 2965589.3, 2965578.3, 2965563.0, 2965540.3, 2965534.6"

What on Earth is up with that geometry column? That’s unworkable.

Before we dig into this further, let’s write a quick wrapper function that will handle the console_output and readLines() dance we just did for the rest of this post:

run_isolated <- function(..., libpath = .libPaths()) {
  console_output <- tempfile()
  callr::r(
    ...,
    libpath = libpath,
    stdout = console_output
  )
  readLines(console_output)
}

I’ve also added a libpath argument, which will let me control what packages these new R sessions are able to access. By default, R sessions run via run_isolated() will have access to all the libraries installed on my machine:

run_isolated(
  \() print(requireNamespace("sf"))
)
[1] "[1] TRUE"

But because I’ve got all my packages installed into a user library (not the system library), I can use that new libpath argument to make it so these R sessions are entirely isolated, without access to any non-base packages at all:2

run_isolated(
  \() print(requireNamespace("sf")),
  libpath = "/dev/null"
)
[1] "[1] FALSE"

Anyway, back to the point. Complicated objects print ugly, and don’t have any of their other methods available, when you unserialize them in a new session.

The reason for this is pretty straightforward: the print (and other) methods are inside the namespace of the package that created the objects,3 if that namespace isn’t loaded then R can’t find the methods when they’re needed, and unserializing the object doesn’t automatically load the relevant namespace. So the reason that boston_canopy printed nicely in our current session is that spatialsample sneakily loaded sf in the background when we called the boston_canopy object:4

is_sf_loaded <- function() {
  sessionInfo()[c("otherPkgs", "loadedOnly")] |>
    lapply(names) |>
    grepl(pattern = "sf") |>
    any()
}
is_sf_loaded()
[1] TRUE

But when we unserialize the object this doesn’t happen:

callr::r(
  \(bos_rds, is_sf_loaded) {
    readRDS(bos_rds)
    is_sf_loaded()
  },
  args = list(bos_rds = bos_rds, is_sf_loaded = is_sf_loaded)
)
[1] FALSE

Because print.sf() is defined in sf, and sf isn’t loaded, we fall back to the default, ugly print method.

This has broader-reaching impacts. For instance, a number of dplyr functions fail for seemingly nonsensical reasons if they’re used on a unserialized object:

callr::r(
  \(bos_rds) {
    readRDS(bos_rds) |>
      dplyr::arrange() |>
      try()
  },
  args = list(bos_rds = bos_rds)
)
[1] "Error in vec_size() : \n  `x` must be a vector, not a <sfc_MULTIPOLYGON/sfc> object.\n"
attr(,"class")
[1] "try-error"
attr(,"condition")
<error/vctrs_error_scalar_type>
Error in `vec_size()`:
! `x` must be a vector, not a <sfc_MULTIPOLYGON/sfc> object.
---
Backtrace:
     ▆
  1. ├─base::tryCatch(...)
  2. │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  3. │   ├─base (local) tryCatchOne(...)
  4. │   │ └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  5. │   └─base (local) tryCatchList(expr, names[-nh], parentenv, handlers[-nh])
  6. │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  7. │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  8. ├─base::withCallingHandlers(...)
  9. ├─base::saveRDS(...)
 10. ├─base::do.call(...)
 11. ├─base (local) `<fn>`(...)
 12. ├─global `<fn>`(bos_rds = base::quote("/tmp/Rtmpb70291/file7571358e23a4.rds"))
 13. │ ├─base::try(dplyr::arrange(readRDS(bos_rds)))
 14. │ │ └─base::tryCatch(...)
 15. │ │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 16. │ │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 17. │ │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 18. │ ├─dplyr::arrange(readRDS(bos_rds))
 19. │ └─dplyr:::arrange.data.frame(readRDS(bos_rds))
 20. │   ├─dplyr::dplyr_row_slice(.data, loc)
 21. │   └─dplyr:::dplyr_row_slice.data.frame(.data, loc)
 22. │     ├─dplyr::dplyr_reconstruct(vec_slice(data, i), data)
 23. │     │ └─dplyr:::dplyr_new_data_frame(data)
 24. │     │   ├─row.names %||% .row_names_info(x, type = 0L)
 25. │     │   └─base::.row_names_info(x, type = 0L)
 26. │     └─vctrs::vec_slice(data, i)
 27. └─vctrs:::stop_scalar_type(`<fn>`(`<s_MULTIP>`), "x", `<fn>`(vec_size()))
 28.   └─vctrs:::stop_vctrs(...)

Just like with printing, the root cause here is that R can’t find the arrange.sf() method when sf isn’t loaded, and winds up using the basic data frame method instead (which then errors out).5

sf:::arrange.sf
function (.data, ..., .dots) 
{
    sf_column_name = attr(.data, "sf_column")
    class(.data) = setdiff(class(.data), "sf")
    st_as_sf(NextMethod(), sf_column_name = sf_column_name)
}
<bytecode: 0x558e252883b0>
<environment: namespace:sf>

The fix is pretty straightforward: load sf (or whatever package has the methods you’re looking for). That’ll let R find your methods, arrange your data, print all pretty, and do everything else you want:6

run_isolated(
  \(bos_rds) {
    library(sf)
    readRDS(bos_rds) |>
      dplyr::arrange() |>
      head(1) |>
      print()
  },
  args = list(bos_rds = bos_rds)
)
 [1] "Simple feature collection with 1 feature and 18 fields"                       
 [2] "Geometry type: MULTIPOLYGON"                                                  
 [3] "Dimension:     XY"                                                            
 [4] "Bounding box:  xmin: 780361.1 ymin: 2964702 xmax: 781922.7 ymax: 2965741"     
 [5] "Projected CRS: NAD83 / Massachusetts Mainland (ftUS)"                         
 [6] "# A tibble: 1 × 19"                                                           
 [7] "  grid_id land_area canopy_gain canopy_loss canopy_no_change canopy_area_2014"
 [8] "  <chr>       <dbl>       <dbl>       <dbl>            <dbl>            <dbl>"
 [9] "1 AB-4      795045.      15323.       3126.           53676.           56802."
[10] "# ℹ 13 more variables: canopy_area_2019 <dbl>, change_canopy_area <dbl>,"     
[11] "#   change_canopy_percentage <dbl>, canopy_percentage_2014 <dbl>,"            
[12] "#   canopy_percentage_2019 <dbl>, change_canopy_absolute <dbl>,"              
[13] "#   mean_temp_morning <dbl>, mean_temp_evening <dbl>, mean_temp <dbl>,"       
[14] "#   mean_heat_index_morning <dbl>, mean_heat_index_evening <dbl>,"            
[15] "#   mean_heat_index <dbl>, geometry <MULTIPOLYGON [US_survey_foot]>"

This all might sound pretty familiar; I wrote a post about more or less about this a year ago. So why bring it up again?

Well, because I saw a cool trick in patchwork that is currently in the development version of sf that I like a lot better than the one I wrote about last year. It’s possible for these unserialized objects to load their relevant packages themselves, making all their methods available as soon as they exist in your R session. For this example to work on your machine, you’re going to need the development version (at time of writing) of sf:

packageVersion("sf") > "1.0-14"
[1] TRUE

Let’s create a new RDS file, this time using an object that was created by the development version of sf:

nc_rds <- tempfile(fileext = ".rds")
system.file("shape/nc.shp", package = "sf") |>
  sf::st_read() |>
  saveRDS(nc_rds)
Reading layer `nc' from data source 
  `/home/mikemahoney218/R/x86_64-pc-linux-gnu-library/4.3/sf/shape/nc.shp' 
  using driver `ESRI Shapefile'
Simple feature collection with 100 features and 14 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -84.32385 ymin: 33.88199 xmax: -75.45698 ymax: 36.58965
Geodetic CRS:  NAD27

Just like before, let’s unserialize this RDS file and print it to our console:

run_isolated(
  \(nc_rds) {
    readRDS(nc_rds) |>
      head(1) |>
      print()
    invisible(NULL)
  },
  args = list(nc_rds = nc_rds)
)
[1] "Simple feature collection with 1 feature and 14 fields"                        
[2] "Geometry type: MULTIPOLYGON"                                                   
[3] "Dimension:     XY"                                                             
[4] "Bounding box:  xmin: -81.74107 ymin: 36.23436 xmax: -81.23989 ymax: 36.58965"  
[5] "Geodetic CRS:  NAD27"                                                          
[6] "   AREA PERIMETER CNTY_ CNTY_ID NAME  FIPS FIPSNO CRESS_ID BIR74 SID74 NWBIR74"
[7] "1 0.114     1.442  1825    1825 Ashe 37009  37009        5  1091     1      10"
[8] "  BIR79 SID79 NWBIR79                       geometry"                          
[9] "1  1364     0      19 MULTIPOLYGON (((-81.47276 3..."

Somehow, magically, R found the right print method!

R can now also find the right methods for other functions, like arrange():

callr::r(
  \(nc_rds) {
    readRDS(nc_rds) |>
      dplyr::arrange() |>
      try()
  },
  args = list(nc_rds = nc_rds)
) |>
  head(1)
Simple feature collection with 1 feature and 14 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: -81.74107 ymin: 36.23436 xmax: -81.23989 ymax: 36.58965
Geodetic CRS:  NAD27
   AREA PERIMETER CNTY_ CNTY_ID NAME  FIPS FIPSNO CRESS_ID BIR74 SID74 NWBIR74
1 0.114     1.442  1825    1825 Ashe 37009  37009        5  1091     1      10
  BIR79 SID79 NWBIR79                       geometry
1  1364     0      19 MULTIPOLYGON (((-81.47276 3...

So what’s the trick? Well, under the hood these new sf objects are quietly loading sf when they get unserialized:

callr::r(
  \(nc_rds, is_sf_loaded) {
    readRDS(nc_rds)
    is_sf_loaded()
  },
  args = list(nc_rds = nc_rds, is_sf_loaded = is_sf_loaded)
)
[1] TRUE

The magic here is that sf objects now have an attribute, .sf_namespace, that’s a simple stub function defined in the sf namespace:

nc_from_rds <- readRDS(nc_rds)
attr(nc_from_rds, ".sf_namespace")
function () 
NULL
<bytecode: 0x558e2d2322d8>
<environment: namespace:sf>

That attribute – which takes up nearly no RAM or disk space – is enough to cause R to automatically load the sf namespace when these objects are unserialized. You now magically have access to all the methods for your complex objects right out of the box.

What if sf isn’t installed? Then it doesn’t get loaded. But that means objects fall back to their default methods, which isn’t great but seems fine:

run_isolated(
  \(nc_rds) {
    readRDS(nc_rds) |>
      head(1) |>
      print()
    invisible(NULL)
  },
  args = list(nc_rds = nc_rds),
  libpath = "/dev/null"
)
[1] "   AREA PERIMETER CNTY_ CNTY_ID NAME  FIPS FIPSNO CRESS_ID BIR74 SID74 NWBIR74"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
[2] "1 0.114     1.442  1825    1825 Ashe 37009  37009        5  1091     1      10"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
[3] "  BIR79 SID79 NWBIR79"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
[4] "1  1364     0      19"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
[5] "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               geometry"
[6] "1 -81.47276, -81.54084, -81.56198, -81.63306, -81.74107, -81.69828, -81.70280, -81.67000, -81.34530, -81.34754, -81.32478, -81.31332, -81.26624, -81.26284, -81.24069, -81.23989, -81.26424, -81.32899, -81.36137, -81.36569, -81.35413, -81.36745, -81.40639, -81.41233, -81.43104, -81.45289, -81.47276, 36.23436, 36.27251, 36.27359, 36.34069, 36.39178, 36.47178, 36.51934, 36.58965, 36.57286, 36.53791, 36.51368, 36.48070, 36.43721, 36.40504, 36.37942, 36.36536, 36.35241, 36.36350, 36.35316, 36.33905, 36.29972, 36.27870, 36.28505, 36.26729, 36.26072, 36.23959, 36.23436"

I think this is really cool! It feels like a user-friendly way to make unserialized complex objects work like you’d expect them to, and prevents confusing error chains like the ones above.

Footnotes

  1. Note that we need to use the stdout argument to capture how this object actually prints to R’s console, and to then use readLines() to print the contents of that file into our current session.↩︎

  2. Thanks to Gabor on Mastodon for teaching me that R will always have access to the system library.↩︎

  3. I mean, mostly. Some packages like broom provide a bunch of methods for objects that don’t come from that package. But those packages aren’t automatically loaded on unserialization either (and can you imagine how chaotic a world that would be!), so the basic point stands.↩︎

  4. More on the trick in this old blog post.↩︎

  5. Well, the root root cause is that vctrs can’t calculate the length of sf geometry columns, because they don’t subclass the list class.↩︎

  6. I’m trying to be very precise with my wording here, so let me highlight that this call to library() is actually attaching sf, when all that’s necessary is loading it. For this post this is a distinction without a difference, so I’m not spending a lot of time on it. But only loading the namespace is necessary, you don’t need to attach it to the search path.↩︎

]]> R Tutorials Spatial geospatial data R packages https://mm218.dev/posts/2023-11-27-objects-loading-namespaces/ Mon, 27 Nov 2023 00:00:00 GMT Downloading STAC data using rsi when you’ve got a geographic CRS or don’t want a composite. Mike Mahoney https://mm218.dev/posts/2023-11-21-rsi-null/ A quick post today, inspired by a GitHub issue.

I’ve been working recently on the new rsi package which helps you download, reproject, resample, mask, rescale, and composite data from STAC APIs.1 The standard function interface does all of these steps: it grabs all the relevant files from your STAC source, reprojects them to match your AOI and desired resolution, masks and rescales the component files, and then merges them into a composite:

library(rsi)
future::plan("multisession")

aoi <- sf::st_point(c(-74.912131, 44.080410)) |> 
  sf::st_sfc() |> 
  sf::st_set_crs(4326) |> 
  sf::st_transform(3857) |> 
  sf::st_buffer(1000)

start_date <- "2022-06-01"
end_date <- "2022-07-01"

get_landsat_imagery(
  aoi = aoi,
  start_date = start_date,
  end_date = end_date,
  output_filename = tempfile(fileext = ".tif")
) |> 
  terra::rast() |> 
  terra::plot()

What if you want to skip some of these steps? For instance, if you try to call get_landsat_imagery() with an AOI in geographic coordinates, you’ll get a warning (likely followed by an error) saying that you’re asking to resample the data to 30 degree pixels, which is probably not what you wanted:

try(
  get_landsat_imagery(
    aoi = sf::st_transform(aoi, 4326),
    start_date = start_date,
    end_date = end_date,
    output_filename = tempfile(fileext = ".tif")
  )
)
Warning: The default pixel size arguments are intended for use with projected AOIs, but `aoi` appears to be in geographic coordinates.
ℹ Pixel X size: 30. Pixel Y size: 30.
ℹ These dimensions will be interpreted in the same units as `aoi` (likely degrees), which may cause errors.
Warning in CPL_gdalwarp(source, destination, options, oo, doo, config_options,
: GDAL Error 1: Attempt to create 0x0 dataset is illegal,sizes must be larger
than zero.
Warning: Failed to download LC08_L2SP_015029_20220617_02_T1 from
2022-06-17T15:45:03.055481Z
Warning in CPL_gdalwarp(source, destination, options, oo, doo, config_options,
: GDAL Error 1: Attempt to create 0x0 dataset is illegal,sizes must be larger
than zero.
Warning: Failed to download LC09_L2SP_015029_20220609_02_T2 from
2022-06-09T15:44:23.649712Z
Warning in CPL_gdalwarp(source, destination, options, oo, doo, config_options,
: GDAL Error 1: Attempt to create 0x0 dataset is illegal,sizes must be larger
than zero.
Warning: Failed to download LC08_L2SP_015029_20220601_02_T1 from
2022-06-01T15:44:51.569374Z
Warning in new_CppObject_xp(fields$.module, fields$.pointer, ...): GDAL Error
4: /tmp/Rtmp7E1urK/filedddb8dfac1e.tif: No such file or directory
Error : [rast] file does not exist: /tmp/Rtmp7E1urK/filedddb8dfac1e.tif

That’s coming from the resampling step of the function’s workflow. Can we just skip that?

Short answer: yes! If we pass NULL to the pixel_*_size arguments, we’ll skip the resampling stage and instead download our data in its native resolution:

get_landsat_imagery(
  aoi = sf::st_transform(aoi, 4326),
  start_date = start_date,
  end_date = end_date,
  pixel_x_size = NULL,
  pixel_y_size = NULL,
  output_filename = tempfile(fileext = ".tif")
) |> 
  terra::rast() |> 
  terra::plot()

This is a pattern throughout the rsi API design: if you want to skip something, pass NULL to the relevant argument. For instance (and this is where the GitHub issue comes in), if you want to not composite and instead download all the images within your spatiotemporal area of interest, we can pass NULL to the composite_function argument to skip compositing. I’ll also skip masking by passing NULL to the mask_function argument, because otherwise a handful of these images are entirely masked out:

get_landsat_imagery(
  aoi = aoi,
  start_date = start_date,
  end_date = end_date,
  output_filename = tempfile(fileext = ".tif"),
  composite_function = NULL,
  mask_function = NULL # otherwise half of these images are blank
) |> 
  lapply(terra::rast) |> 
  lapply(terra::plot)

[[1]]
NULL

[[2]]
NULL

[[3]]
NULL

Hopefully this helps people use rsi to only perform the data wrangling steps they want!

Footnotes

  1. And calculate spectral indices from these data, and wrangle multiple rasters into a multi-band VRT – it’s a pretty neat package if I do say so myself.↩︎

]]> R Tutorials Spatial geospatial data R packages https://mm218.dev/posts/2023-11-21-rsi-null/ Tue, 21 Nov 2023 00:00:00 GMT Classed conditions from rlang functions Mike Mahoney https://mm218.dev/posts/2023-11-07-classed-errors/ I’m a huge fan of the condition functions from rlang – rlang::inform() for sending messages, rlang::warn() for warnings, and rlang::abort() for errors. Compared to their base equivalents (message(), warning(), and stop(), respectively) these functions are extremely flexible and make it easy to specify which user-facing function actually caused the condition. And recently I’ve become a huge fan of how these functions let you easily set the class of your conditions, which makes it a lot easier to implement logic to handle these conditions.

For instance, let’s say we’ve got some function that sends up a warning if you give it an unexpected input:

f1 <- function(x) {
  if (!is.numeric(x)) {
    rlang::warn(
      "`x` wasn't numeric. Was this expected?"
    )
  }
  mean(x)
}
f1(TRUE)
Warning: `x` wasn't numeric. Was this expected?
[1] 1

If we know that we’re going to be passing unexpected inputs to this function, we might consider using suppressWarnings() to hide this warning. I do this every so often in package code, where I know my inputs to another function are going to trigger a condition that I don’t need the user to see:1

suppressWarnings(f1(TRUE))
[1] 1

The challenge with this is that suppressWarnings(), used this way, is a blunt tool that hides all warnings sent up by this function. For instance, if we passed a character vector as input to this function, we’d also trigger a warning from mean() that it’s going to return NA:

f1("a")
Warning: `x` wasn't numeric. Was this expected?
Warning in mean.default(x): argument is not numeric or logical: returning NA
[1] NA

And that useful warning also gets hidden by the suppressWarnings() call:

suppressWarnings(f1("a"))
[1] NA

Adding a subclass to our warning helps solve this. By specifying the class argument in any of the rlang condition functions, we’re able to easily subclass our warning. This doesn’t change how the warning displays during standard usage:

f2 <- function(x) {
  if (!is.numeric(x)) {
    rlang::warn(
      "`x` wasn't numeric. Was this expected?",
      class = "non_numeric_x"
    )
  }
  mean(x)
}
f2(TRUE)
Warning: `x` wasn't numeric. Was this expected?
[1] 1

But it does mean that we can now use the classes argument to suppressWarnings() to only supress the warnings we care about, without accidentally hiding other unexpected warnings we might trigger:

suppressWarnings(f2("a"), classes = "non_numeric_x")
Warning in mean.default(x): argument is not numeric or logical: returning NA
[1] NA

This is great, and makes it a lot easier to incorporate conditions into your program’s control flow. For instance, we can use these classed warnings with tryCatch() or rlang::try_fetch() to “catch” conditions, perhaps running a cleanup script or fallback method in the event that a specific classed warning is returned:2

rlang::try_fetch(
  f2("a"),
  non_numeric_x = function(...) "We're running a completely different function now!"
)
[1] "We're running a completely different function now!"

Last but not least, classed errors help in package testing. A huge number of my tests are designed to make sure that conditions fire when they’re supposed to – bad inputs trigger errors, concerning outputs trigger warnings and so on. Using classed errors can help me make sure I’m triggering the error or warning that I want to, not just any random error or warning that might be lurking in my code.

If you’re using testthat’s 3rd edition, the expect_condition() set of functions (including expect_message(), expect_warning(), expect_error()) all share a class argument which will make sure the warning or error you’re triggering is actually the one you expect:

testthat::local_edition(3)
testthat::expect_warning(f2(TRUE), class = "non_numeric_x")

If our condition class doesn’t match the expected class, these tests will fail:

try(testthat::expect_warning(f2(TRUE), class = "wrong_class"))
Warning: `x` wasn't numeric. Was this expected?
Error : `f2(TRUE)` did not throw the expected warning.

I’m a late adopter of classed conditions, only really systematically adopting them for the new rsi package, but I’ve found them super useful so far and am planning to slowly use them more and more in the rest of my packages over time!

Footnotes

  1. For instance, the way that autoplot() in spatialsample adds grids to spatial_block_cv() plots always triggers the same message, which is expected and not worth worrying about. I hide that message so my users don’t need to be concerned.↩︎

  2. I don’t currently, but I should do this in terrainr, where I currently assume that any error during merge_rasters() can be fixed by the fallback method.↩︎

]]> R Tutorials Package development https://mm218.dev/posts/2023-11-07-classed-errors/ Tue, 07 Nov 2023 00:00:00 GMT spatialsample 0.5.0 is now on CRAN Mike Mahoney https://mm218.dev/posts/2023-11-03-spatialsample/ The newest version of spatialsample, the tidymodels package I maintain for spatial cross-validation, just landed on CRAN, with binaries for Windows and Mac coming in the next few days.

This release mostly fixes a few bugs in spatial_block_cv() and spatial_nndm_cv(). The only new feature is that get_rsplit() is now reexported from rsample, providing a nicer interface for extracting individual rsplit objects from an rset:

library(spatialsample)

folds <- spatial_clustering_cv(boston_canopy, 2)

all.equal(
  get_rsplit(folds, 1),
  folds$splits[[1]]
)
[1] TRUE

More pressing are two sets of breaking changes. The first of these is that spatial_block_cv() now creates slightly different grids, covering a very slightly larger area, which may change what fold any given observation is assigned into. This is to address a problem reported on StackOverflow where, if data fell exactly on grid lines (which was somewhat common with regularly-spaced grids of data), it would be assigned to both of the polygons on either side of the line.

The amount of grid expansion performed can be controlled using the new expand_bbox argument to spatial_block_cv(). If observations are still assigned to multiple folds, the function will now throw an error:

drought_sf <- sf::st_as_sf(
  expand.grid(
    x = seq(995494, 1018714, 430),
    y = seq(1019422, by = 430, length.out = 55)
  ),
  coords = c("x", "y"),
  crs = 7760
)

try(spatial_block_cv(drought_sf, expand_bbox = 0))
Error in generate_folds_from_blocks(data, centroids, grid_blocks, v, n,  : 
  Some observations fell exactly on block boundaries, meaning they were assigned to multiple assessment sets unexpectedly.
ℹ Try setting a different `expand_bbox` value, an `offset`, or use a different number of folds.

But hopefully the expansion will make this error relatively uncommon:

folds <- spatial_block_cv(drought_sf)

autoplot(folds)

This is a breaking change for data in projected coordinate reference systems. Data in geographic coordinates was actually already using these slightly larger grids, due to issues with straight-line grids in non-planar CRS, so this change just makes the amount of grid expansion controllable there.

The other breaking change/bug-fix is in spatial_nndm_cv(). The prediction_sites argument to spatial_nndm_cv() lets you specify the actual sites you were going to generate predictions at. In older versions of spatialsample, if any of the data in prediction_sites weren’t points, then this function would instead randomly sample points from inside the bounding box of the entire prediction_sites object.

Starting with spatialsample 0.5.0, passing a single polygon to prediction_sites will cause spatial_nndm_cv() to instead sample points from inside that polygon, allowing you fine-grained control over the boundaries for this sampling stage. This feels like a more intuitive interface, and you can always revert to previous behaviors by passing sf::st_as_sf(sf::st_as_sfc(sf::st_bbox(prediction_sites))) if you’d rather sample from the bounding box instead.

]]> R Spatial geospatial data spatialsample R packages https://mm218.dev/posts/2023-11-03-spatialsample/ Fri, 03 Nov 2023 00:00:00 GMT Adding context to maps made with ggplot2 Mike Mahoney https://mm218.dev/posts/2023-10-31-map-context/ A colleague asked me today how they could best add a larger data set for context to a map of a (spatially) smaller data set, without the map expanding to incorporate the whole of the larger data set. I didn’t have a great answer off the top of my head, so this blog post is here to record what we tried, and what wound up working for us!

Say we’ve got some spatial data set that covers a broad area; for instance, the nc data set from sf that contains the counties of North Carolina:

suppressPackageStartupMessages(library(sf))
library(ggplot2)
nc <- system.file("shape/nc.shp", package = "sf") |> 
  read_sf()

ggplot(nc) + 
  geom_sf()

We also need a comparatively smaller data set that we’re interested in visualizing. For this blog, simulate some number of observations inside one of the more central counties:

johnston <- nc[which(nc$NAME == "Johnston"), ]
johnston_obs <- st_sample(johnston, 500)

ggplot() + 
  geom_sf(data = johnston) + 
  geom_sf(data = johnston_obs, color = "red")

Our goal is to add a bit more context to this map by drawing the borders of surrounding counties. The challenge is that ggplot will, by default, expand our visualization to contain the largest layer that we add:

ggplot() + 
  geom_sf(data = nc) + 
  geom_sf(data = johnston_obs, color = "red")

We could control this using expansion() inside of scale_*_continuous() functions, in order to restrict the range of our visualization:

ggplot() + 
  geom_sf(data = nc) + 
  geom_sf(data = johnston_obs, color = "red") + 
  scale_x_continuous(expand = expansion(c(-0.63, -0.29))) + 
  scale_y_continuous(expand = expansion(c(-0.5, -0.28)))

But these expansion() calls are relative to the scale of our larger data set, which makes them a bit difficult to reason about. We aren’t specifying our extents in terms of the data that we actually care about visualizing, we’re forced to specify them relative to the larger context that we don’t care as much about.

We could make this a bit easier by filtering our larger data set to only observations that are near (or in this case, touching) the area we’re actually trying to visualize:

neighbors <- nc[st_touches(johnston, nc)[[1]], ]
ggplot() + 
  geom_sf(data = neighbors) + 
  geom_sf(data = johnston_obs, color = "red") + 
  scale_x_continuous(expand = expansion(c(-0.3, -0.25))) + 
  scale_y_continuous(expand = expansion(c(-0.4, -0.25)))

But while this reduces the problem, moving the center of the larger layer closer to the center of the layer we care about, it still has the same issue as when we used the entire nc object.

So what we wound up doing was embracing a little bit of jank and reaching into the ggplot2 internals. We started off by making a “base plot” object that was zoomed out to the level of detail that we wanted:

base_plot <- ggplot(johnston_obs) + 
  geom_sf() + 
  scale_x_continuous(expand = expansion(0.1)) + 
  scale_y_continuous(expand = expansion(0.1))

base_plot

Defining our boundaries here is a bit easier to reason about – the expansion() calls are centered on our data of interest and are expanding the scales relative to this focal data set. Once we’ve got our level of zoom where we want it, we can build our plot using ggplot_build() and then extract the ranges of our x and y scales from the constructed plot object:

base_plot <- ggplot_build(base_plot)
xlim <- base_plot$layout$panel_params[[1]]$x_range
ylim <- base_plot$layout$panel_params[[1]]$y_range

We can then use those ranges as limits, to force our final plot to have the same level of zoom as our simple map. That means we can add whatever layers we want to add context to our map, and not need to worry about fiddling with our scales in term of the largest layer we’ve added:

ggplot() + 
  geom_sf(data = nc) + 
  geom_sf(data = johnston_obs, color = "red") + 
  scale_x_continuous(limits = xlim) + 
  scale_y_continuous(limits = ylim)

I have no idea how stable this approach will be – we’re decidedly not playing on the same team as ggplot2 with this approach – but it works as of ggplot version 3.4.3, and it’s made making maps a bit easier for us at the moment!

]]> R Tutorials Spatial geospatial data https://mm218.dev/posts/2023-10-31-map-context/ Tue, 31 Oct 2023 00:00:00 GMT Executing R code from untrusted sources in minimal environments Mike Mahoney https://mm218.dev/posts/2023-10-27-minimal-environments/ Yesterday we released rsi, an R package that (among other things) makes it easy to retrieve spectral indices from the Awesome Spectral Indices project and calculate them against any images you have on hand.

The actual code required to do these calculations is mostly just a bit of glue. The ASI project provides the formulas you’d use to calculate any spectral indices you might be interested in, which as relatively simple arithmetic are easily transformed into R code via str2lang().

Once we’ve done that transformation, we can evaluate that R code against our images through a slightly off-label use of terra::predict(). This function loads chunks of our raster into R as a data frame with column names corresponding to our band labels, meaning that if we’re careful to ensure that our band names align with the standardized band names used by ASI1 we can compute indices by evaluating their formulas “with” the data frame.

This sounds complex, but doesn’t actually require that much code to put together:

calculate_indices <- function(raster,
                              indices,
                              output_filename) {
  if (!inherits(raster, "SpatRaster")) raster <- terra::rast(raster)
  formulas <- lapply(indices[["formula"]], str2lang)
  terra::predict(
    raster,
    list(),
    fun = function(model, newdata) {
      out <- lapply(formulas, \(calc) with(newdata, eval(calc)))
      names(out) <- indices[["short_name"]]
      out
    },
    filename = output_filename
  )
  output_filename
}

Turn our formulas into calls, load our raster into R chunk by chunk, evaluate those calls in the context of our raster, lather rinse repeat. The conceptual complexity here is a lot higher, in my view, than the code complexity.

Because this code is pretty straightforward, and relies on the fantastic, highly-optimized terra package to actually do these computations, we’re able to calculate these indices fast:

example_index <- rsi::spectral_indices()
example_index <- example_index[example_index$short_name == "DPDD", ]

system.time({
  out <- calculate_indices(
    system.file("rasters/example_sentinel1.tif", package = "rsi"),
    example_index,
    tempfile(fileext = ".tif")
  )
})
   user  system elapsed 
  1.754   0.040   1.794 
terra::plot(terra::rast(out))

This is pretty nifty!

That said, we should be careful about what text we’re willing to turn into R code and execute. In particular, rsi is designed to integrate nicely with the Awesome Spectral Indices project, and to retrieve and compute the ASI set of indices – which, phrased differently, means we’re downloading code from the internet and running it on our computers. If someone were to mess with our indices – either by corrupting the GitHub repository or by editing the cached file on your machine – this could wind up giving them access to system() or other scary commands:

evil_index <- example_index
evil_index$formula <- "system('echo oh no > /tmp/example.txt')"

try(
  calculate_indices(
    system.file("rasters/example_sentinel1.tif", package = "rsi"),
    evil_index,
    tempfile(fileext = ".tif")
  ),
  silent = TRUE
)

readLines("/tmp/example.txt")
[1] "oh no"

So, how can we make this safer?

One way is by taking away the number of toys any malicious code has available to play with. We can do this by running the code in a locked-down environment, where it won’t have access to functions that might let code mess with our machine.

One way of creating a locked down environment is rlang::new_environment(). By default, this function creates a new environment with nothing in it – no built-in functions or objects:

ls(envir = rlang::new_environment())
character(0)

This environment is also going to have the empty environment as its parent, meaning that code executed in this scope won’t be able to use functions or objects from the global environment2:

# inheriting from the global environment
local(
  2 + 2,
  new.env()
)
[1] 4
# inheriting from the empty environment
try(
  local(
    2 + 2,
    rlang::new_environment()
  )
)
Error in 2 + 2 : could not find function "+"

That means that any code we run inside of this new environment will only have access to whatever functions and variables we purposefully include in the environment. The data argument to rlang::new_environment() makes it relatively easy to define whatever objects we’re looking to make available in this new environment:

local(2 + 2, rlang::new_environment(list(`+` = `+`)))
[1] 4

That means that, if we create a minimal environment containing only the functions and variables essential for calculating our indices, we should hopefully be able to reduce the potential blast radius of malicious code – or at least make it a lot harder for malicious code to impact anything we care about. In rsi, that means we wind up calculating indices inside a minimal environment that looks like this:

calculate_indices <- function(raster,
                              indices,
                              output_filename) {
  if (!inherits(raster, "SpatRaster")) raster <- terra::rast(raster)
  formulas <- lapply(indices[["formula"]], str2lang)
  
  exec_env <- rlang::new_environment(
    list(
      `::` = `::`,
      `-` = `-`,
      `(` = `(`,
      `*` = `*`,
      `/` = `/`,
      `^` = `^`,
      `+` = `+`,
      `<-` = `<-`,
      `{` = `{`,
      `names<-` = `names<-`,
      `function` = `function`,
      list = list,
      lapply = lapply,
      with = with,
      eval = eval,
      formulas = formulas,
      short_names = indices[["short_name"]],
      raster = raster,
      output_filename = output_filename
    )
  )
  
  local(
    {
      terra::predict(
        raster,
        list(),
        fun = function(model, newdata) {
          out <- lapply(
            formulas,
            function(calc) {
              with(newdata, eval(calc))
            }
          )
          names(out) <- short_names
          out
        },
        filename = output_filename
      )
    },
    envir = exec_env
  )
  
  output_filename
}

This shouldn’t impact anything from the user’s perspective when calculating well-behaved formulas:

system.time({
  out <- calculate_indices(
    system.file("rasters/example_sentinel1.tif", package = "rsi"),
    example_index,
    tempfile(fileext = ".tif")
  )
})
   user  system elapsed 
  0.016   0.000   0.015 
terra::plot(terra::rast(out))

But it makes the most obvious malicious code fail:

try(
  calculate_indices(
    system.file("rasters/example_sentinel1.tif", package = "rsi"),
    evil_index,
    tempfile(fileext = ".tif")
  )
)
Error in system("echo oh no > /tmp/example.txt") : 
  could not find function "system"

Update 2023-10-27: However, we actually need to go one step further. Because we’ve included :: in our minimal environment, we’ve left in an “escape hatch” that malicious code can use to access any functions it wants:

evil_index$formula <- "base::system('echo oh no > /tmp/example2.txt')"
try(
  calculate_indices(
    system.file("rasters/example_sentinel1.tif", package = "rsi"),
    evil_index,
    tempfile(fileext = ".tif")
  )
)
Error : [predict] the number of values returned by 'fun' (model predict function) does not match the input. Try na.rm=TRUE?
readLines("/tmp/example2.txt")
[1] "oh no"

Instead of calling terra::predict() via :: inside our environment, we’ll need to include that function in the environment directly, in order to remove this escape hatch:

calculate_indices <- function(raster,
                              indices,
                              output_filename) {
  if (!inherits(raster, "SpatRaster")) raster <- terra::rast(raster)
  formulas <- lapply(indices[["formula"]], str2lang)
  
  exec_env <- rlang::new_environment(
    list(
      `-` = `-`,
      `(` = `(`,
      `*` = `*`,
      `/` = `/`,
      `^` = `^`,
      `+` = `+`,
      `<-` = `<-`,
      `{` = `{`,
      `names<-` = `names<-`,
      `function` = `function`,
      list = list,
      lapply = lapply,
      with = with,
      eval = eval,
      formulas = formulas,
      short_names = indices[["short_name"]],
      raster = raster,
      output_filename = output_filename,
      predict = terra::predict
    )
  )
  
  local(
    {
      predict(
        raster,
        list(),
        fun = function(model, newdata) {
          out <- lapply(
            formulas,
            function(calc) {
              with(newdata, eval(calc))
            }
          )
          names(out) <- short_names
          out
        },
        filename = output_filename
      )
    },
    envir = exec_env
  )
  
  output_filename
}
try(
  calculate_indices(
    system.file("rasters/example_sentinel1.tif", package = "rsi"),
    evil_index,
    tempfile(fileext = ".tif")
  )
)
Error in base::system : could not find function "::"

Thanks Gábor Csárdi for the catch!

This still isn’t a perfect fix – and rsi also checks to make sure that all of your formulas are only using symbols that match the band names of your rasters. Even with these checks, you should investigate the formulas you’re going to run before you actually run them – or save a copy of the trusted indices you’re going to calculate and provide those to calculate_indices(), rather than using spectral_indices() directly. But this hopefully makes this function a pinch safer.

I wrote this piece of rsi back in August and then more or less didn’t think of it again until yesterday, when we released rsi on GitHub – and at the same time, started using GitHub Actions for CI for the package. All of a sudden, I started seeing a lot of CI runs that looked like this:

A screenshot of GitHub Actions, where all workflows -- including R CMD check -- are successful, but test coverage is failing.

R CMD check succeeding and test coverage failing made no sense to me, as theoretically they’re both running a full check and reporting the results. In classic developer tradition, I spent a few hours flailing around before finally giving up and resorting to the final option I had available: reading the error messages.

And it turned out that each of those failed test coverage runs had the same error message:

Expected `... <- NULL` to run without any errors. Actually got a <simpleError> with text: could not find function :::

could not find function ":::".

My test coverage workflow is using the (fantastic) covr package to measure line coverage. Behind the scenes, covr is doing a lot more than just running R CMD check, like my other workflows – covr is actually changing what R runs when it runs your code, in order to measure how many times any given line of code gets called. This is a really neat workflow, but it doesn’t play nicely with our minimal environment here; the new code added inside of our local() statement depends upon functions that we didn’t (and aren’t going to) provide to the minimal environment, such as :::. Adding a new environment variable to the test coverage workflow, and skipping tests that ran the local() call when that environment variable was defined, wound up solving the issue.

Footnotes

  1. Which rsi will automatically enforce when using get_*_imagery() functions.↩︎

  2. Or any other environments this one inherits from.↩︎

]]> R Tutorials Package development https://mm218.dev/posts/2023-10-27-minimal-environments/ Fri, 27 Oct 2023 00:00:00 GMT Introducing rsi Mike Mahoney https://mm218.dev/posts/2023-10-26-rsi/ I am so, so excited to share that rsi,1 a new R package for handling common spatial data wrangling tasks, is now available on GitHub! Specifically, rsi handles:

Most of my work on the package happened while I was at Permian Global, helping them automate their MMRV pipelines used to make sure their carbon credit projects are actually preserving carbon sinks and have additive benefits over time.2 I’m really excited and grateful that Permian has agreed to open-source this work.

Let’s take a whirlwind tour through the features in this package! For the purposes of this blog post, let’s download and process imagery for Middlesex county in Massachusetts, USA.3 We’ll use the tigris package to get the borders for this county, and then reproject it into the Massachusetts State Plane:

ma_counties <- tigris::counties("MA", progress_bar = FALSE)
middlesex <- ma_counties[ma_counties$NAME == "Middlesex", ]
# Reprojecting into the MA state plane, a planar CRS:
middlesex <- sf::st_transform(middlesex, 26986)
plot(sf::st_geometry(middlesex))

This is the area we’re going to download and process data for.

Specifically, let’s start off by downloading Landsat imagery from Microsoft’s Planetary Computer STAC API! This is pretty straightforward using rsi: use the get_landsat_imagery() function with an area of interest and a timeframe, and you’ll automatically get a cloud-masked composite image of all acquisitions for that spatiotemporal window. Let’s grab all the imagery from September 2023:

library(rsi)
future::plan("multisession")

middlesex_imagery <- get_landsat_imagery(
  middlesex,
  "2023-09-01",
  "2023-09-30"
)
middlesex_imagery
[1] "shell_liquid_doctor.tif"

Note that I’ve used future::plan() here to specify a parallelization methodology, as the data retrieval functions in rsi are all compatible with future4 to speed up downloads by using multiple threads. These functions also use progressr to let users specify progress reporting methods, if they want them, by calling progressr::handlers().

By default, get_landsat_imagery() will download a composite of all bands available in Landsat 8 and 9 imagery for our timeframe:

terra::rast(middlesex_imagery) |> 
  terra::plot()

Notice that this composite has been cloud-masked (using the QA pixel band) and rescaled5 (using the scale and offset specified in metadata provided by the STAC endpoint) automatically. You can control these behaviors via function arguments.

We’re able to download more than just imagery via rsi functions – for instance, we could also grab a DEM for this area from Planetary Computer, using the get_dem() function:

middlesex_dem <- get_dem(middlesex)
terra::rast(middlesex_dem) |> 
  terra::plot()

Under the hood, both of these functions (and their friends, get_sentinel2_data() and get_sentinel1_data()) are powered by a lower-level get_stac_data() function, which should theoretically work with any imagery provided by any STAC API, anywhere. These functions simply provide user-friendly defaults to make it faster to get the data you care about.

In addition to these STAC-focused data-downloading functions, rsi also has an interface to the Awesome Spectral Indices project, via the spectral_indices() function:

spectral_indices() |> 
  head()
# A tibble: 6 × 9
  application_domain bands     contributor    date_of_addition formula long_name
  <chr>              <list>    <chr>          <chr>            <chr>   <chr>    
1 vegetation         <chr [2]> https://githu… 2021-11-17       (N - 0… Aerosol …
2 vegetation         <chr [2]> https://githu… 2021-11-17       (N - 0… Aerosol …
3 water              <chr [6]> https://githu… 2022-09-22       (B + G… Augmente…
4 vegetation         <chr [2]> https://githu… 2021-09-20       (1 / G… Anthocya…
5 vegetation         <chr [3]> https://githu… 2022-04-08       N * ((… Anthocya…
6 vegetation         <chr [4]> https://githu… 2021-05-11       (N - (… Atmosphe…
# ℹ 3 more variables: platforms <list>, reference <chr>, short_name <chr>

This function attempts to grab the newest version of the spectral indices JSON file from the ASI repo, and then stores that data in a cache folder on your computer. If the downloading fails, the package will fall back (with a warning) to use your possibly outdated cache instead; if you don’t have a cache and can’t download the files, the package will instead (with a different warning) resort to using a packaged version of the indices file. This ensures that you’re always getting the latest and greatest version of the ASI list possible, but that the package can still be used without an internet connection.

There are also functions in rsi to sort through the ASI list of indices. For instance, the filter_platforms() function can be used to, well, filter the list to only indices that can be calculated from a given platform. For instance, to filter to only indices that can be calculated using data from Landsat’s Operational Land Imager:

filter_platforms(platforms = "Landsat-OLI") |> 
  head()
# A tibble: 6 × 9
  application_domain bands     contributor    date_of_addition formula long_name
  <chr>              <list>    <chr>          <chr>            <chr>   <chr>    
1 vegetation         <chr [2]> https://githu… 2021-11-17       (N - 0… Aerosol …
2 vegetation         <chr [2]> https://githu… 2021-11-17       (N - 0… Aerosol …
3 water              <chr [6]> https://githu… 2022-09-22       (B + G… Augmente…
4 vegetation         <chr [4]> https://githu… 2021-05-11       (N - (… Atmosphe…
5 vegetation         <chr [4]> https://githu… 2021-05-14       sla * … Adjusted…
6 vegetation         <chr [2]> https://githu… 2022-04-08       (N * (… Advanced…
# ℹ 3 more variables: platforms <list>, reference <chr>, short_name <chr>

There’s an equivalent function to filter indices based upon the bands that require. For instance, we can filter the list to only indices that use the red and blue band of images:

filter_bands(bands = c("R", "B")) |> 
  head()
# A tibble: 2 × 9
  application_domain bands     contributor    date_of_addition formula long_name
  <chr>              <list>    <chr>          <chr>            <chr>   <chr>    
1 vegetation         <chr [2]> https://githu… 2022-04-08       (R - B… Kawashim…
2 vegetation         <chr [2]> https://githu… 2022-04-08       (R^2.0… Modified…
# ℹ 3 more variables: platforms <list>, reference <chr>, short_name <chr>

Arguments to these functions let you control whether you’re looking for indices that match all platforms and bands you’ve specified, or any of them.

But rsi doesn’t simply make these formulas available in R, it also helps you compute these indices from imagery, via the calculate_indices() function. This function takes your imagery and a subset of spectral_indices() as arguments, and creates a raster containing all of those indices as an output. We can use filter_bands() to quickly get a list of the indices we can compute from our Landsat imagery, and then calculate_indices() to compute all 128 of those indices from our images:

middlesex_indices <- calculate_indices(
  middlesex_imagery,
  filter_bands(bands = names(terra::rast(middlesex_imagery))),
  "middlesex_indices.tif"
)

|---------|---------|---------|---------|
=========================================
                                          
terra::rast(middlesex_indices) |> 
  terra::plot()

Note that calculate_indices() is evaluating the formulas in the formula column of the spectral indices data frame as if they were code.6 These formulas are evaluated inside of a very limited environment, which doesn’t have access to the global environment or most R fixtures, which does reduce the amount of harm malicious code could do:

evil_index <- spectral_indices()[1, ]
evil_index$formula <- "system('echo OHNO')"
try(
  calculate_indices(
    middlesex_imagery,
    evil_index,
    "test.tif"
  )
)
Error in system("echo OHNO") : could not find function "system"

But it’s worth scanning your formulas before running calculate_indices(), just to make sure you aren’t going to be accidentally running something surprising!

Last but not least, rsi also provides a way to combine disparate data sets covering the same geographic region into a single VRT, quickly creating a file that you can treat as a single raster without taking up much additional storage space. This is a great way to create predictor bricks from your indices and downloaded data7 which you can then use for model fitting and prediction:

combined_layers <- stack_rasters(
  c(middlesex_imagery, middlesex_dem, middlesex_indices),
  "middlesex.vrt"
)

terra::rast(combined_layers) |> 
  terra::plot()

I’m extremely excited for this package to be out in the open, and for people to start using it. If you find the package useful or interesting, drop us a star on GitHub – and if you have any questions/comments/concerns about how it does things, please open an issue or a PR! We’re planning a CRAN release in the not-too-distant future, and would love to incorporate any feedback we get into that first released version.

Footnotes

  1. The name is a nonsense acronym. Initially the package was going to be “rsi: An R interface to the Rsome Spectral Indices project”, but the scope very quickly outgrew that; now it’s a convenient short package name that’s not taken on CRAN, and I’m backronyming as many additional meanings into those three letters as I can make fit.↩︎

  2. I am not currently employed by Permian, and nothing I say here or elsewhere reflects their opinions! That said, Permian is a copyright holder and funder to rsi, just as Posit is to waywiser, given their support of the initial development of the package.↩︎

  3. I was originally going to use Boston’s Suffolk county, but Suffolk county’s borders are ludicrous, because they include not just Boston’s mainland but also Boston’s islands, meaning the county is a normal-enough shape with a huge rectangle extending into the Atlantic off to the East.↩︎

  4. Via future.apply, to be specific.↩︎

  5. Explaining this blog post from a few months ago.↩︎

  6. Explaining this blog post from two days ago.↩︎

  7. Not that you should be regressing using raw imagery bands, but this lets you combine a DEM and other computed metrics with calculated indices.↩︎

]]> R Tutorials Spatial geospatial data R packages https://mm218.dev/posts/2023-10-26-rsi/ Thu, 26 Oct 2023 00:00:00 GMT Three fun R functions Mike Mahoney https://mm218.dev/posts/2023-10-24-fun-r-funcs/ Inspired by Maëlle who was inspired by Yihui who was inspired by Maëlle(who has a whole series about this), I wanted to share three useful base R functions that I think maybe don’t get enough love. And inspired by Maëlle again, my list here is actually four functions.

sweep()

If you ever need to do math with matrices, then sweep() is going to be your best friend. Say for instance we want to center and scale each column in a matrix. This is a pretty straightforward operation – we need to calculate the mean and standard deviations for each column, subtract the column mean from each observation, and then divide those by the corresponding standard deviation.

We can use apply to get our means and standard deviations:

# Generate some fake data in a 10x10 matrix:
x <- matrix(data = rnorm(100), nrow = 10)
# Calculate one mean and sd for each column of our matrix:
col_means <- apply(x, 2, mean)
col_sds <- apply(x, 2, sd)

The subtraction and division are a bit less straightforward. R’s base math operators will attempt to do element-wise operations, treating our vector as a one-column array and replicating as needed. That’s not what we want:

all.equal(
  (x - col_means) / col_sds,
  scale(x)
)
[1] "Attributes: < Length mismatch: comparison on first 1 components >"
[2] "Mean relative difference: 0.360556"

We could replicate our vector ourself, in order to take advantage of these element-wise operations:

all.equal(
  ((x - matrix(rep(col_means, 10), 10, byrow = TRUE)) / 
    matrix(rep(col_sds, 10), 10, byrow = TRUE)) |> as.vector(),
  scale(x) |> as.vector()
)
[1] TRUE

But that’s silly, especially if we were working with more observations.

Better instead is to use sweep() to perform some operation between each element of our vector and each column of the matrix:

# Take every value in our matrix, and subtract its corresponding column mean:
centered <- sweep(
  x = x, 
  MARGIN = 2, # just like in apply()
  STATS = col_means, 
  FUN = "-" # "-" is the default argument -- we don't NEED to provide it here
)

And we can similarly use sweep() to divide each column by its corresponding standard deviation, finishing up our centering and scaling:

# Divide each value by its corresponding column sd:
centered_and_scaled <- sweep(centered, 2, col_sds, "/")

# Works out identically to the built-in scale function:
all.equal(
  as.vector(centered_and_scaled),
  as.vector(scale(x))
)
[1] TRUE

This is the main way I use sweep(), but there’s no requirement you use it for math – it works just as well with non-mathematical functions or non-numeric matrices:

letter_mat <- matrix(rep(letters[1:5], 5), 5)
letter_mat
     [,1] [,2] [,3] [,4] [,5]
[1,] "a"  "a"  "a"  "a"  "a" 
[2,] "b"  "b"  "b"  "b"  "b" 
[3,] "c"  "c"  "c"  "c"  "c" 
[4,] "d"  "d"  "d"  "d"  "d" 
[5,] "e"  "e"  "e"  "e"  "e" 
sweep(letter_mat, 2, LETTERS[1:5], paste0)
 [1] "aA" "bA" "cA" "dA" "eA" "aB" "bB" "cB" "dB" "eB" "aC" "bC" "cC" "dC" "eC"
[16] "aD" "bD" "cD" "dD" "eD" "aE" "bE" "cE" "dE" "eE"

reformulate() and DF2formula()

The reformulate() function is a lifesaver if you’re trying to write long or complicated formulas, or multiple formulas generated by some other logic in your code.

The function is pretty straightforward. If you’re trying to make a formula y ~ x + z, provide your predictors as the first argument and your outcome as the second:

reformulate(c("x", "z"), "y")
y ~ x + z

The nice thing is that reformulate accepts vectors as inputs, making it easy to construct a vector of predictors and automatically turn them into a formula:

reformulate(letters, "outcome")
outcome ~ a + b + c + d + e + f + g + h + i + j + k + l + m + 
    n + o + p + q + r + s + t + u + v + w + x + y + z
reformulate(names(Orange), "age")
age ~ Tree + age + circumference

And in particular, this is an excellent alternative to dropping a few columns in order to use outcome ~ . – instead, you can use setdiff() to exclude those columns from your formula:

outcome_variable <- "age"
reformulate(setdiff(names(Orange), outcome_variable), outcome_variable)
age ~ Tree + circumference

Relatedly, the function DF2formula() will automatically turn the column names from a data frame into a formula. The first column will become the outcome variable, and the rest will be used as predictors:

DF2formula(Orange)
Tree ~ age + circumference

To change what column is used as the outcome variable, reorder the columns in your data frame:

DF2formula(Orange[3:1])
circumference ~ age + Tree

str2lang()

Shockingly enough, str2lang() function turns a string into a language object:

growth_rate <- "circumference / age"
str2lang(growth_rate)
circumference/age
class(str2lang(growth_rate))
[1] "call"

Wooooo!

I think that, to most people, this does not sound immediately useful.1 But the idea that your code can turn plain text into code at runtime is pretty powerful, and some of the most R-esque nonsense that R has to offer.

For instance, we can use eval() to actually execute the call created by str2lang() in our global environment:

eval(str2lang("2 + 2"))
[1] 4

And that string can do anything that regular R code can do – assign variables, manage connections, any procedure that normal R code can do:

eval(str2lang("x <- 3"))
x
[1] 3

We can also use this with with() or local() to execute our code inside of other environments. For instance, if we want to calculate our growth_rate from earlier, we can run that code with the Orange data frame:

with(Orange, eval(str2lang(growth_rate)))
 [1] 0.25423729 0.11983471 0.13102410 0.11454183 0.09748172 0.10349854
 [7] 0.09165613 0.27966102 0.14256198 0.16716867 0.15537849 0.13972380
[13] 0.14795918 0.12831858 0.25423729 0.10537190 0.11295181 0.10756972
[19] 0.09341998 0.10131195 0.08849558 0.27118644 0.12809917 0.16867470
[25] 0.16633466 0.14541024 0.15233236 0.13527181 0.25423729 0.10123967
[31] 0.12198795 0.12450199 0.11535337 0.12682216 0.11188369

This can be a powerful way to “import” code from other sources, for instance if you have a CSV of equations you want to run against a data frame. You want to be careful when using this with untrusted inputs, of course – if your input includes a call to system(), it might wind up wrecking your computer!

Footnotes

  1. I think, to most people, this barely sounds like English.↩︎

]]> R Tutorials https://mm218.dev/posts/2023-10-24-fun-r-funcs/ Tue, 24 Oct 2023 00:00:00 GMT Cloud-Native Geospatial If You Don’t Speak Snake Mike Mahoney https://mm218.dev/posts/2023-09-20-cngf/ I’m over in the Cloud-Native Geospatial Foundation’s blog today with a post called “Cloud-Native Geospatial If You Don’t Speak Snake”. This post comes from an UnConference session at the 2023 ESIP July Meeting, organized by Alexey Shiklomanov, focusing on what tools and what gaps exist in the cloud-native geospatial workflow for non-Python users. It was an absolutely fantastic UnConference session, and I’m hoping that my summary of our takeaways is as useful for others as it was for me.

One challenge in the session – and one that I don’t get into in the post – is that “cloud-native” means a lot of different things to different folks! The CNGF themselves have a pretty straightforward definition:

Cloud-Native data formats are structured to be efficiently retrived from cloud object storage services which are designed to serve large volumes of data using generic RESTful / HTTP data transfer protocols.

But our UnConference session was interested in a much broader definition, one that was a bit closer to the CNCF definition of the term:

Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

My post focuses a bit more on tools and gaps relating to the CNGF definition – it’s on their blog, after all! – but our working group was interested in API specifications, serverless runtimes, and all sorts of other topics that I think haven’t seen the same amount of standardization as the data format and retrieval side of geospatial workflows.1 There’s a lot of activity around geospatial workflows right now, and I’m excited to see what the future has in store!

Footnotes

  1. Most of my data storage and retrieval work boils down to “use COGs, use GDAL’s virtual filesystem interface”. I wish my analysis, runtime, and codebase setups could be boiled down to that level.↩︎

]]> R Spatial geospatial data https://mm218.dev/posts/2023-09-20-cngf/ Wed, 20 Sep 2023 00:00:00 GMT Pre-allocating vectors is for nerds Mike Mahoney https://mm218.dev/posts/2023-08-29-allocations/ The second circle of R hell, in Patrick Burns’ seminal book The R Inferno, is titled “Growing Objects”. This refers to a common antipattern for R users, usually among the first things taught when dealing with iteration: it is extremely inefficient to grow a vector using c(), like so:

vector_c <- function(n) {
  out <- c()
  for (i in 1:n) {
    out <- c(out, i)
  }
  out
}

Instead, Burns says, it is better to pre-allocate our vector out, and assign our function’s output to a specific position in out using either [ or [[:

vector_prealloc_one_bracket <- function(n) {
  out <- vector("numeric", n)
  for (i in 1:n) {
    out[i] <- i
  }
  out
}

vector_prealloc_two_bracket <- function(n) {
  out <- vector("numeric", n)
  for (i in 1:n) {
    out[[i]] <- i
  }
  out
}

Of course, it would be better yet to avoid our loop entirely, and simply create our final object using the colon operator:

colon_operator <- function(n) {
  1:n
}

But that’s beside the point right now.

This advice was originally written in 2011, but is even more important today. In Burns’ book, subsetting is roughly 7 times faster when n is 10,000; on my computer today, subsetting is roughly 200 times faster:

n <- 10000
bench::mark(
  c = vector_c(n),
  one_bracket = vector_prealloc_one_bracket(n),
  two_brackets = vector_prealloc_two_bracket(n),
  colon = colon_operator(n),
  filter_gc = FALSE
)[c("expression", "median", "itr/sec", "mem_alloc")]
# A tibble: 4 × 4
  expression     median `itr/sec` mem_alloc
  <bch:expr>   <bch:tm>     <dbl> <bch:byt>
1 c                51ms      19.3   191.2MB
2 one_bracket     277µs    3548.     99.1KB
3 two_brackets    276µs    3538.     96.7KB
4 colon           361ns 2124339.         0B

But what if n is unknowable? Well, to quote Burns:

Often a reasonable upper bound on the size of the final object is known. If so, then create the object with that size and then remove the extra values at the end. If the final size is a mystery, then you can still follow the same scheme, but allow for periodic growth of the object.

This is still probably a decent approach: over-allocate and trim down, or allocate in chunks and only grow when those chunks are exhausted.

Or… perhaps we might try growing a vector with [ or [[, rather than with c()? To anyone raised on R traditions, this might seem like a code smell:

vector_unalloc_one_bracket <- function(n) {
  out <- c()
  for (i in 1:n) {
    out[i] <- i
  }
  out
}

vector_unalloc_two_bracket <- function(n) {
  out <- c()
  for (i in 1:n) {
    out[[i]] <- i
  }
  unlist(out)
}

But if we test it out:1

bench::mark(
  c = vector_c(n),
  prealloc_one_bracket = vector_prealloc_one_bracket(n),
  unalloc_one_bracket = vector_unalloc_one_bracket(n),
  unalloc_two_brackets = vector_unalloc_two_bracket(n),
  filter_gc = FALSE
)[c("expression", "median", "itr/sec", "mem_alloc")]
# A tibble: 4 × 4
  expression             median `itr/sec` mem_alloc
  <bch:expr>           <bch:tm>     <dbl> <bch:byt>
1 c                     54.02ms      16.6  191.23MB
2 prealloc_one_bracket 285.52µs    3428.    78.17KB
3 unalloc_one_bracket    1.24ms     710.   871.73KB
4 unalloc_two_brackets   2.76ms     337.     1.72MB

Growing a vector via [ is still notably slower than assigning values to a pre-allocated vector; it looks like it’s roughly ~5 times slower. But that still means it’s ~50 times faster than growing a vector via c(), and allocates ~200 times less memory to do so. Growing a vector via [[ isn’t quite as efficient – taking roughly twice the time and memory as [ here – but still blows c() out of the water.

That’s not too shabby, for a code smell. How does a method like vapply() compare?

vapply_lambda <- function(n) {
  vapply(1:n, \(i) i, numeric(1))
}

bench::mark(
  c = vector_c(n),
  prealloc_one_bracket = vector_prealloc_one_bracket(n),
  unalloc_one_bracket = vector_unalloc_one_bracket(n),
  unalloc_two_brackets = vector_unalloc_two_bracket(n),
  vapply = vapply_lambda(n),
  filter_gc = FALSE
)[c("expression", "median", "itr/sec", "mem_alloc")]
# A tibble: 5 × 4
  expression             median `itr/sec` mem_alloc
  <bch:expr>           <bch:tm>     <dbl> <bch:byt>
1 c                     50.87ms      19.5   191.2MB
2 prealloc_one_bracket 279.79µs    3501.     78.2KB
3 unalloc_one_bracket    1.18ms     649.      853KB
4 unalloc_two_brackets   2.69ms     345.      1.7MB
5 vapply                 3.41ms     272.     78.2KB

vapply() uses as little memory as our pre-allocation approaches, but is slower than either of our un-allocated methods.2

It’s worth emphasizing that the differences between these methods are microscopic compared to the difference between them and c() for growing vectors:

benchmarks <- bench::press(
  bench::mark(
    c = vector_c(n),
    prealloc_one_bracket = vector_prealloc_one_bracket(n),
    unalloc_one_bracket = vector_unalloc_one_bracket(n),
    unalloc_two_brackets = vector_unalloc_two_bracket(n),
    vapply = vapply_lambda(n),
    filter_gc = FALSE
  ),
  n = c(10, 100, 1000, 10000, 100000)
)

library(ggplot2)
ggplot(benchmarks, aes(n, median, color = as.character(expression))) + 
  geom_line() + 
  theme_minimal() + 
  labs(y = "Median execution time (s)")

But as far as execution speed goes, well, maybe growing objects in general isn’t worthy of its own circle of hell anymore:

benchmarks[as.character(benchmarks$expression) != "c", ] |> 
  ggplot(aes(n, median, color = as.character(expression))) + 
  geom_line() + 
  theme_minimal() + 
  labs(y = "Median execution time (s)")

Though of course, vapply() and the pre-allocated methods still win out in terms of memory allocation:3

benchmarks[as.character(benchmarks$expression) != "c", ] |> 
  ggplot(aes(n, mem_alloc, color = as.character(expression))) + 
  geom_line() + 
  theme_minimal() + 
  labs(y = "Memory allocation (bytes)")

So: pre-allocate your vectors when you’re able. But maybe it’s fine to grow an object every once in a while, as a treat. It probably won’t get you sent to hell.

I have no idea when things changed to make growing vectors via [ so much more efficient now than in 2011 – and please let me know in the comments/Mastodon/BlueSky if you know any more details here.

Footnotes

  1. I dropped prealloc_two_brackets from the benchmarks because it was performing ~the same as the one-bracket alternative.↩︎

  2. Usual disclaimer that this is probably not a type of slowness that matters for your code, that you should look into moving computation to C++/Rust if you care about a few milliseconds execution time, and that the real benefits of *apply functions come from readability and their potential for parallelization, not speed.↩︎

  3. The pre-allocated line is hidden by the vapply() line; they’re practically identical, and possibly also literally identical.↩︎

]]> R Tutorials Package development https://mm218.dev/posts/2023-08-29-allocations/ Tue, 29 Aug 2023 00:00:00 GMT Yes, you can rescale Landsat images after compositing them Mike Mahoney https://mm218.dev/posts/2023-08-24-landsat-scaling/ Landsat data are distributed as unsigned 16-bit images, which need to be rescaled to get raw band values. The rescaling formulas are dependent upon the band type and collection used, but for current Collection 2 data boil down to two equations:

  • For surface reflectance data, the formula is (where is the scaled band value)
  • For surface temperature data, the formula is .

For my use-cases, I’m only rarely looking to download (and rescale) a single Landsat image. More often, I want to take all the Landsat images from a given timeframe (for instance, the growing season in the area I care about) and combine them into a composite image, taking the mean or median pixel value for that time period.

And every single time I go to do this, I need to figure out for the umpteenth time whether I can composite the images and then rescale them, or whether I need to rescale each individual image before making my composite. This is analytically solvable, and I think is pretty straightforward to solve – and, to spoil the rest of this post, the answer is that it doesn’t matter when you rescale. But I can never remember that, and I never trust my algebra when I try to prove that you can rescale before or after compositing either.

But this is an easy thing to simulate – just make a bunch of replications of compositing some number of 16-bit values, rescaling either before or after making the composite, and test for equality. That’s do-able in a few lines of R:

landsat_rescale <- function(x) x * 0.0000275 - 0.2

vapply(
  c(mean, median),
  \(f) {
    replicate(1000, {
      values <- replicate(100, sample.int(65455, 1))
      all.equal(
        f(landsat_rescale(values)),
        landsat_rescale(f(values))
      )
    }) |> 
      all()
  },
  logical(1)
) |>
  all()
[1] TRUE

It’s fine! You can rescale before or after compositing, whether you’re using a mean or a median composite. Do whatever is easiest for your workflow. Go in peace.

]]> R geospatial data Tutorials https://mm218.dev/posts/2023-08-24-landsat-scaling/ Thu, 24 Aug 2023 00:00:00 GMT A long digression about the word ‘vector’ Mike Mahoney https://mm218.dev/posts/2023-08-07-vector/ In linguistics, there’s this concept called “semantic overload” that refers to when a word has more than one distinct meaning, and the appropriate meaning needs to be inferred from context. The classic example is when someone says they’re “running to the store”: we can guess from context that the speaker isn’t going for a jog, but we’re forced to guess.

Software engineering loves semantic overload. An “agile team” might be a vague way to say that you’re very responsive, or it might mean you work in tightly-defined two week sprints. A “transaction” might be a customer ordering from a website, or a database writing a new row. When you’re developing software that’s tackling new types of problems, there’s often not existing language that describes exactly what your tool is trying to do, and so instead programmers use existing terms and rely upon metaphors and analogies to adapt them for a new purpose. But because the meaning of these terms changes depending on the context they’re used in, this overload can be a real barrier to learning for new users who don’t yet have the context to understand the overloaded term. For example, I’ve worked with a number of new programmers who were afraid of opening “Issues” on GitHub projects, because in other contexts announcing an issue you have with someone’s work is an aggressive action.1 Lacking the shared context makes it hard to decode what these terms mean.

And so, the term “vector”. For a bit of context, my undergraduate degree was in ecology – forest ecosystem science specifically, a specialization chosen in order to not need to take courses in Calculus 2 or Organic Chemistry. Which meant that, coming out of my degree, I had one course in physics, a handful in GIS and spatial data, and none in computer science. So when I learned that the base unit of R data, the result of running code like:

c(1, 2)
[1] 1 2

Was called a “vector”, I was frustrated. At this point, I had been told that:

  • In physics, a “vector” was any line with a magnitude (“length”) and a direction.

  • In GIS, a “vector” was pretty much any type of data; points, lines, polygons, whatever.2

  • In R and apparently computer science, a “vector” was quite literally any data whatsoever.

This felt to me like another case of pointless complexity, of the word “vector” being overloaded beyond the point of usefulness – and I didn’t feel like the word “vector” was particularly useful in the first place.

It turns out, though, that I was just missing the context that linked these three meanings together. I wouldn’t get that context until I was in grad school, procrastinating by watching 3Blue1Brown’s excellent videos on the fundamentals of linear algebra. I’m not nearly qualified to teach anything about linear algebra, and I don’t have the space nor the inclination to try to do so here – but I’m going to try and share the thing that gave me an “aha!” moment. To do that, I need to start off by getting really abstract.

First off, let’s say we’ve got some 2-dimensional plane, that looks like this:

library(ggplot2)
df <- data.frame(x = runif(100) * 100, y = runif(100) * 100)
ggplot(df, aes(x, y)) +
  scale_x_continuous(expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous(expand = c(0, 0), limits = c(0, 100)) + 
  coord_fixed()

We’ve got our X axis and our Y axis here, and both meet at the origin – the place where the X coordinate is 0, and the Y coordinate is 0. We’d say that the coordinates at that point are (0, 0).

Now let’s go back to our physics definition of a vector – any line with a known length and direction. We could draw one of those in this coordinate plane – say our line is a bit longer than 70 “units”, going from (0, 0) all the way to (50, 50):

ggplot(df, aes(x, y)) +
  scale_x_continuous(expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous(expand = c(0, 0), limits = c(0, 100)) + 
  annotate("segment", 0, 0, xend = 50, yend = 50) + 
  annotate("point", 50, 50, shape = 25, size = 3, fill = "black") + 
  coord_fixed()

This is how I was taught to think about vectors in those physics classes – arrows on some abstract plane. For instance, we could turn this into an acceleration vector by labeling these axes, so that our line is now charting speed over time:

ggplot(df, aes(x, y)) +
  scale_x_continuous("Time", expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous("Speed", expand = c(0, 0), limits = c(0, 100)) + 
  annotate("segment", 0, 0, xend = 50, yend = 50) + 
  annotate("point", 50, 50, shape = 25, size = 3, fill = "black") + 
  coord_fixed()

If you change the axis labels to distance over time, you have a velocity vector instead:

ggplot(df, aes(x, y)) +
  scale_x_continuous("Time", expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous("Distance", expand = c(0, 0), limits = c(0, 100)) + 
  annotate("segment", 0, 0, xend = 50, yend = 50) + 
  annotate("point", 50, 50, shape = 25, size = 3, fill = "black") + 
  coord_fixed()

The meaning of the line is defined by the axis labels – by the actual coordinate plane your vector is in. Looking at our velocity vector, we can tell how far we’ve gone (position on the Y axis) for any given time (position on the X axis).

So if we use a different coordinate plane and relabel our axes to show “distance away from the origin”:

ggplot(df, aes(x, y)) +
  scale_x_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  annotate("segment", 0, 0, xend = 50, yend = 50) + 
  coord_fixed()

We get vectors in a spatial sense. Rather than showing distance at a given time, we’re now showing the position of our data – in this case, a linestring – in one direction when it’s at a given position in the other direction. Just like in physics, the actual meaning of this line depends on the coordinate plane – on the coordinate reference system of the data. The CRS of your data is a standardized way to define where your origin is, what units your distances are measured in, and which direction away from the origin you’re going.

And we can use spatial vector data to replace our physics vector. We just need to define a matrix containing the beginning and end coordinates of our line:

our_matrix <- matrix(c(0, 0, 50, 50), nrow = 2, byrow = TRUE)
our_matrix
     [,1] [,2]
[1,]    0    0
[2,]   50   50

And then we can tell sf that it should understand that matrix as being a line:

our_line <- sf::st_linestring(our_matrix)
our_line
LINESTRING (0 0, 50 50)

And voila, we have a spatial vector:

ggplot() +
  scale_x_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  geom_sf(
    data = our_line
  )

At each point along this line on our X axis, our line is a single, known position on the Y axis. If we only have one measurement of XY position – say, a single GPS measurement – then our line would be of length 0. We’d have a point instead:

ggplot() +
  scale_x_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  geom_sf(
    data = sf::st_point(c(50, 50))
  )

That point represents a single position, which we’d understand through our coordinate reference system as being a certain distance away from a reference point.

Similarly, this is what vectors in R are abstracting. Imagine that, instead of using sf to make this a spatial vector, we turned our matrix into a data frame and used that with ggplot2 instead:

our_matrix |>
  as.data.frame() |>
  setNames(c("x", "y")) |>
  ggplot(aes(x, y)) +
  geom_line() + 
  scale_x_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  scale_y_continuous("Distance away from the origin in this direction", expand = c(0, 0), limits = c(0, 100)) + 
  coord_fixed()

Just like with sf, we’re representing vectors by the places they start and end. And because we’ve plotted this as a line, we’re able to tell the position of our data at each distance along either the X or Y axis, within this coordinate reference system.

Or take for instance the age vector inside the Orange data frame:

Orange$age |> head()
[1]  118  484  664 1004 1231 1372

According to ?Orange, this vector represents the age of the tree, in units of days since 1968-12-31. Similarly, the circumference vector is the circumference of each tree in millimeters. We can plot those vectors just as easily as our physics and our spatial vectors:

ggplot(Orange, aes(age, circumference, color = Tree)) + 
  scale_x_continuous(limits = c(0, 1700), expand = c(0, 0)) +
  scale_y_continuous(limits = c(0, 225), expand = c(0, 0)) +
  geom_point()

This graph is using an abstract coordinate system – instead of “meters away from the origin”, one axis is “distance in time from a reference date”, and the other is “distance in length from not existing at all”. But just like our physics vectors, each of these points represents a magnitude in some direction. Our age vector is a set of magnitudes along a time axis; our circumference vector a set of magnitudes along a length axis.

Because I didn’t have a ton of formal math education, I never made the connection across these three types of vectors, and never entirely understood that they were all different ways of understanding and representing position along a coordinate plane, under some coordinate reference system. Recognizing that these different versions of “vectors” are all sharing an underlying meaning made it a lot easier for me to understand what “vector data” actually meant, and to understand the semantic difference between vector and raster representations of the same data. Hopefully this digression makes things a bit clearer for someone else, as well.

Footnotes

  1. This has been partially addressed by the newer Discussions feature.↩︎

  2. Don’t worry, Mastodon commenter, I’m aware that rasters exist. The point is that my education here was incomplete.↩︎

]]> R Tutorials https://mm218.dev/posts/2023-08-07-vector/ Mon, 07 Aug 2023 00:00:00 GMT From the inbox: How can I get fold assignments from spatialsample? Mike Mahoney https://mm218.dev/posts/2023-06-06-spatialsample_splits/ In my inbox,1 someone asks:2

I’m using spatial_clustering_cv() from spatialsample to do cross-validation. How can I get separate data frames with each split created by this function?

I think this question is decently common, because a lot of the spatialsample documentation is written assuming that you’re familiar with rsample already, which is often not the case for people working with spatial data. The functions to do this sort of thing live in rsample, and aren’t (currently) re-exported by spatialsample, so it can be hard to find the right function.

First and foremost, let’s assume that you’ve got some object called my_folds created by spatial_clustering_cv():

library(spatialsample)
my_folds <- spatial_clustering_cv(boston_canopy, v = 2)
my_folds
#  2-fold spatial cross-validation 
# A tibble: 2 × 2
  splits            id   
  <list>            <chr>
1 <split [277/405]> Fold1
2 <split [405/277]> Fold2

The “my_folds” object that gets created should have a “splits” column, which is a list. Each element of that list contains your analysis and assessment sets.3 To get a single split, use rsample::get_rsplit():

rsample::get_rsplit(my_folds, 1)
<Analysis/Assess/Total>
<277/405/682>

To get just the analysis data for that fold, use rsample::analysis():

rsample::get_rsplit(my_folds, 1) |>
  rsample::analysis()
Simple feature collection with 277 features and 18 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 755424.9 ymin: 2935616 xmax: 812069.7 ymax: 2970073
Projected CRS: NAD83 / Massachusetts Mainland (ftUS)
# A tibble: 277 × 19
   grid_id land_area canopy_gain canopy_loss canopy_no_change canopy_area_2014
   <chr>       <dbl>       <dbl>       <dbl>            <dbl>            <dbl>
 1 AB-4      795045.      15323.       3126.           53676.           56802.
 2 AO-9      270153        6187.       1184.           26930.           28114.
 3 V-7       107890.        219.       3612.             240.            3852.
 4 X-4       848558.       8275.       1760.            6872.            8632.
 5 AC-4     2069814.      82201.      50944.          240161.          291104.
 6 AC-15    1175032.      24517.      24010.          111148.          135158.
 7 U-14     2690727.      69780.      51404.          263796.          315201.
 8 AQ-15     453368.      13971.       3401.          343677.          347077.
 9 Q-10      156688.       9237.       3094.           57327.           60421.
10 T-10      215340.      13984.       3947.           59539.           63487.
# ℹ 267 more rows
# ℹ 13 more variables: canopy_area_2019 <dbl>, change_canopy_area <dbl>,
#   change_canopy_percentage <dbl>, canopy_percentage_2014 <dbl>,
#   canopy_percentage_2019 <dbl>, change_canopy_absolute <dbl>,
#   mean_temp_morning <dbl>, mean_temp_evening <dbl>, mean_temp <dbl>,
#   mean_heat_index_morning <dbl>, mean_heat_index_evening <dbl>,
#   mean_heat_index <dbl>, geometry <MULTIPOLYGON [US_survey_foot]>

Similarly, to get just the assessment data for that fold, use rsample::assessment():

rsample::get_rsplit(my_folds, 1) |>
  rsample::assessment()
Simple feature collection with 405 features and 18 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 739826.9 ymin: 2908294 xmax: 781347.5 ymax: 2959751
Projected CRS: NAD83 / Massachusetts Mainland (ftUS)
# A tibble: 405 × 19
   grid_id land_area canopy_gain canopy_loss canopy_no_change canopy_area_2014
   <chr>       <dbl>       <dbl>       <dbl>            <dbl>            <dbl>
 1 I-33      265813.       8849.      11795.           78677.           90472.
 2 H-10     2691490.      73098.      80362.          345823.          426185.
 3 Q-22     2648089.     122211.     154236.         1026632.         1180868.
 4 P-18     2690726.     110928.     113146.          915137.         1028283.
 5 J-29     2574479.      38069.      15530.         2388638.         2404168.
 6 G-28     2641525.      87024.      39246.         1202528.         1241774.
 7 M-23     2690727.      87621.     124032.          748742.          872774.
 8 M-9      2690727.      52443.      53467.          304239.          357706.
 9 S-15     2690728.      93787.     162118.          478257.          640375.
10 Q-21     2690727.      54712.     101816.         1359305.         1461121.
# ℹ 395 more rows
# ℹ 13 more variables: canopy_area_2019 <dbl>, change_canopy_area <dbl>,
#   change_canopy_percentage <dbl>, canopy_percentage_2014 <dbl>,
#   canopy_percentage_2019 <dbl>, change_canopy_absolute <dbl>,
#   mean_temp_morning <dbl>, mean_temp_evening <dbl>, mean_temp <dbl>,
#   mean_heat_index_morning <dbl>, mean_heat_index_evening <dbl>,
#   mean_heat_index <dbl>, geometry <MULTIPOLYGON [US_survey_foot]>

If you’re trying to get your original data, with a column indicating which fold each row belongs to, there’s not a provided function for that. Instead, what you can do is take the assessment set from each split (which is “what fold data is assigned to”), add a new column to it with the fold name, and then combine those assessment sets into a single data frame. I do this via the function:4

purrr::map2(
  my_folds$splits, 
  my_folds$id, 
  \(split, id) cbind(rsample::assessment(split), fold_name = id)
) |> 
  dplyr::bind_rows()
Simple feature collection with 682 features and 19 fields
Geometry type: MULTIPOLYGON
Dimension:     XY
Bounding box:  xmin: 739826.9 ymin: 2908294 xmax: 812069.7 ymax: 2970073
Projected CRS: NAD83 / Massachusetts Mainland (ftUS)
First 10 features:
   grid_id land_area canopy_gain canopy_loss canopy_no_change canopy_area_2014
1     I-33  265813.3    8848.818    11795.11         78676.56         90471.67
2     H-10 2691489.9   73098.168    80361.85        345823.19        426185.04
3     Q-22 2648088.6  122211.269   154236.43       1026631.85       1180868.27
4     P-18 2690726.1  110927.833   113145.85        915137.00       1028282.85
5     J-29 2574478.7   38068.676    15529.73       2388638.19       2404167.92
6     G-28 2641525.3   87024.318    39246.15       1202527.94       1241774.09
7     M-23 2690727.2   87620.730   124031.79        748742.13        872773.92
8      M-9 2690726.6   52443.164    53466.56        304239.49        357706.04
9     S-15 2690727.8   93786.589   162118.16        478257.33        640375.48
10    Q-21 2690727.2   54711.827   101815.82       1359305.11       1461120.93
   canopy_area_2019 change_canopy_area change_canopy_percentage
1          87525.38          -2946.293               -3.2565923
2         418921.35          -7263.685               -1.7043502
3        1148843.12         -32025.158               -2.7120009
4        1026064.83          -2218.014               -0.2157008
5        2426706.87          22538.944                0.9374946
6        1289552.26          47778.164                3.8475730
7         836362.86         -36411.060               -4.1718776
8         356682.65          -1023.393               -0.2860988
9         572043.92         -68331.566              -10.6705469
10       1414016.94         -47103.991               -3.2238256
   canopy_percentage_2014 canopy_percentage_2019 change_canopy_absolute
1                34.03579               32.92739            -1.10840701
2                15.83454               15.56466            -0.26987600
3                44.59323               43.38386            -1.20936883
4                38.21581               38.13338            -0.08243181
5                93.38465               94.26013             0.87547604
6                47.00974               48.81847             1.80873391
7                32.43636               31.08315            -1.35320518
8                13.29403               13.25600            -0.03803406
9                23.79934               21.25982            -2.53951988
10               54.30208               52.55148            -1.75060448
   mean_temp_morning mean_temp_evening mean_temp mean_heat_index_morning
1           74.26247          83.87540  90.85933                75.63458
2           74.64432          84.96917  91.71625                75.86767
3           73.19889          82.29358  89.70302                74.47757
4           73.77269          84.29003  91.26480                75.03802
5           72.26419          79.77278  88.70229                73.65608
6           73.60919          82.80297  90.33156                74.96955
7           74.24167          83.34713  90.41143                75.66013
8           76.74740          84.69933  91.96502                77.91048
9           75.18260          84.85431  92.00132                76.39949
10          73.37669          82.38064  90.59503                74.63029
   mean_heat_index_evening mean_heat_index fold_name
1                 89.71880        96.70939     Fold1
2                 89.88733        96.19667     Fold1
3                 87.34062        95.53811     Fold1
4                 88.93811        96.43569     Fold1
5                 81.32060        95.56059     Fold1
6                 88.47864        96.82653     Fold1
7                 89.23434        96.05418     Fold1
8                 90.02009        96.14348     Fold1
9                 89.91342        96.92160     Fold1
10                86.90021        96.23439     Fold1
                         geometry
1  MULTIPOLYGON (((752945.6 29...
2  MULTIPOLYGON (((751419.1 29...
3  MULTIPOLYGON (((763631.7 29...
4  MULTIPOLYGON (((763122.9 29...
5  MULTIPOLYGON (((753963.4 29...
6  MULTIPOLYGON (((749383.6 29...
7  MULTIPOLYGON (((758543.1 29...
8  MULTIPOLYGON (((758543.1 29...
9  MULTIPOLYGON (((767702.6 29...
10 MULTIPOLYGON (((764649.4 29...

I think it would make sense for get_rsplit(), analysis(), and assessment() to get ported over to spatialsample, to make it a bit easier for the folks whose first point-of-entry into tidymodels work is via spatialsample. I’ve got a GitHub issue to remind me to look into that before the package’s next release.

Footnotes

  1. I want to mention that I include a link to Yihui Xie’s excellent blog post in replies to help questions sent via email. I love seeing people use my packages, and I love helping people use them, but I don’t always have the time to give 1:1 help via email. If you post a question somewhere publicly, then other people might give an even better answer; if no one answers in a day or two, then email me the link, so I can answer it publicly and have a link to send the next person with the same question as you. That’s also why I turned this into a blog post – so that I can send others with the same question a pre-written answer!↩︎

  2. Anonymized and heavily paraphrased.↩︎

  3. Sometimes called training and testing, respectively – rsample uses the analysis/assessment terminology to make it clear that all of this data should be in your training set, and doesn’t touch your final held-out test set.↩︎

  4. This is how spatialsample’s autoplot() methods do it, for instance.↩︎

]]> R spatialsample tidymodels R packages geospatial data Tutorials https://mm218.dev/posts/2023-06-06-spatialsample_splits/ Tue, 06 Jun 2023 00:00:00 GMT