rplatypus rebuilt: deep learning for medical images in R

In October 2020 I wrote about platypus, an R package for object detection and image segmentation: YOLOv3 and U-Net in a few lines of R, on top of the keras and tensorflow packages. People used it. It collected 134 stars and a queue of issues - and then it quietly stopped working, and I stopped answering them.

This year I went back to it. Computer vision is the part of data science I enjoy most, and this package has always been my passion project - the place where I get to learn things my day job does not ask of me. What came out is not an update but a different package, with a new name, a different backend and a narrower purpose: rplatypus, deep learning for medical images in R. It has been submitted to CRAN and is waiting for review; until it is accepted, install it from GitHub:

# install.packages("remotes")
remotes::install_github("maju116/rplatypus")

This post is about why it had to be rewritten rather than fixed, what is in it now, and where the series of posts that follows is going.

Why the old one broke

R’s keras and tensorflow packages are not bindings to a C++ library. They are bridges, through reticulate, into a Python installation of TensorFlow - and in 2020 that was the Python installation, the one shared with everything else on the machine. Upgrade numpy for an unrelated project and an R session using platypus could stop loading. The failure never appeared in my package’s code, which made it miserable to support: the bug reports were about other people’s Python.

The second problem is the one that actually made me stop. platypus sat on two layers I did not control: the R keras and tensorflow packages on top of TensorFlow itself. When TensorFlow changed in Python, the R packages had to catch up before platypus could, and that often took a very long time. Until they did, my package was stuck, with issues I could not fix from my side. After enough of that, I lost the will to keep working on it.

Two things have changed since, and together they remove both problems at the root rather than patching around them.

reticulate learned to isolate. Since version 1.41, a package declares its Python requirements with py_require(), and reticulate builds a private environment for them with uv, cached and shared with nobody. There is no common Python left for two packages to fight over. rplatypus never touches a Python anyone else uses - and the person installing it never chooses an interpreter, activates anything or matches a version. The first call that needs the engine builds the environment; every call after that reuses it. And there is no R wrapper in between any more: rplatypus calls its own engine, which I release myself, so nothing waits for anybody else to catch up.

The medical imaging world settled on PyTorch. MONAI, nnU-Net, TorchIO, timm, the pretrained encoders and the foundation models - all of it is PyTorch. TensorFlow for medical images is effectively over, and the TensorFlow 2.9 the old engine was pinned to does not even install on Python 3.12. Staying would have meant rewriting anyway, onto a backend with no future in the field.

A third thing changed too. It does not fix the old failure, but it is the reason the rewrite happened at all instead of staying a good idea: writing a package got much cheaper. In 2020 I wrote every line of platypus myself, and the parts that make a package trustworthy - tests, documentation, error messages that say what went wrong - were the parts I never had time left for. This time I worked with an AI coding agent, Claude Code.

I want to be precise about what that means, because “AI wrote my package” is not it. The decisions were mine: what the package is for, which datasets to measure it on, what it should refuse, and what a result has to show before it goes into the documentation - every number in this series was measured, not generated. What the agent changed was the cost of everything around those decisions: writing the tests, keeping two packages in two languages in step, chasing a failing check through reticulate, uv and CUDA. That is exactly the work that used to end side projects like this one. The rewrite took around a month.

So the computation moved to pyplatypus, a Python engine on PyTorch, and rplatypus is the R package on top of it. The split is deliberate:

The engine is in Python, because that is where the models, the weights and the medical libraries already are. One codebase to maintain, and everything PyTorch has is within reach.
The product is in R, because that is where the gap is. Reading and combining masks, plotting, comparing models, the analysis around the network - that part is plain R and needs no Python at all.

I did consider native R torch, which binds libtorch directly and needs no Python. For one person maintaining this in their evenings it would have meant writing every architecture, loss and data pipeline twice, and giving up every pretrained weight in the Python ecosystem.

pyplatypus: the same engine, from Python

The engine is not hidden plumbing. pyplatypus is a package on PyPI in its own right - it too started life in 2022 on TensorFlow and was rewritten alongside - and if you work in Python you can use it directly:

pip install "pyplatypus[hub]"

The example below runs from both languages, out of one file.

The two are developed together, and in a fixed order. A feature lands in the engine first, the engine is published to PyPI, and only then does rplatypus move to it - each release of the R package pins one exact version of pyplatypus, so it cannot break because the engine changed underneath it. They also share one configuration format: the same YAML file describes the same experiment whichever language reads it, so an R user and a Python user can hand each other a run.

That is also how this series is written. The examples are in R, because that is where the gap is, and each post shows what the same step looks like from Python.

The gap

Here is the part that convinced me this was worth doing. R has excellent tools for reading medical images - RNifti, oro.dicom, divest - and a healthy deep learning stack. Almost nothing joins them. When I checked in September, the CRAN Task View for Medical Image Analysis (last updated August 2025) listed no deep learning packages at all.

So rplatypus is meant to be that missing piece: not a wrapper around a Python library you then have to learn, but an R package that happens to compute in Python.

A mask, without training anything

The quickest way to see whether a package does anything useful is to borrow a model someone already trained. rplatypus publishes weights by name - here a U-Net trained on the 2018 Data Science Bowl, the same nuclei dataset I used on this blog in 2020.

The folder nuclei/ holds four microscopy images in the dataset’s own layout - one directory per image, with images/ and masks/ inside.

library(rplatypus)

spec <- platypus_spec(
  data = segmentation_data("nuclei", "nuclei", colormap = binary_colormap),
  models = list(u_net("nuclei", input_shape = c(256, 256), blocks = 4, filters = 16,
                      weights = "dsbowl-unet", fit = FALSE))
)
fit <- platypus_fit(spec, verbose = FALSE)
masks <- predict(fit, split = "validation")
dim(masks)
#> [1]   4 256 256

Four images, 256 by 256 pixels, one class label per pixel. This ran on the processor, with no GPU; platypus_device() tells you whether yours will use one, and why not if it can’t.

images <- list.files("nuclei", pattern = "[.]png$", recursive = TRUE, full.names = TRUE)
images <- read_images(grep("/images/", images, value = TRUE), size = c(256, 256))
plot_masks(images, prediction = masks, colormap = binary_colormap)

Figure 1: Nuclei found by the published `dsbowl-unet` weights, on four images of different kinds.

The first three are fluorescence images, and the nuclei come out as clean blobs. The fourth is a brightfield image of stained cells, and the model finds specks - the small dark nuclei inside the cells. That is the hard kind of image in this dataset, and the next post looks at it closely.

There is no score here, and that is on purpose. These four images come from the same collection the weights were trained on, so any number measured on them would flatter the model. The numbers that mean something were measured on a held-out split when the weights were published, and available_weights() says what they are and what the model cannot do - for instance, that it is a semantic model, so touching nuclei come back as one region and a count taken from them would be wrong.

The same thing from a file

The same specification can live in a file. This one started as save_spec(spec, "nuclei.yaml"), with the lists tidied by hand:

nuclei.yaml

task: semantic_segmentation
data:
  train_path: nuclei
  validation_path: nuclei
  colormap: [[0, 0, 0], [255, 255, 255]]
models:
- name: nuclei
  architecture: u_net
  input_shape: [256, 256]
  blocks: 4
  filters: 16
  weights: dsbowl-unet
  fit: false

Reading it back gives exactly the object built from arguments above:

identical(as.list(platypus_spec("nuclei.yaml")), as.list(spec))
#> [1] TRUE

So platypus_fit(platypus_spec("nuclei.yaml")) does everything the R code did, and nothing downstream can tell which way the specification was written. The Python engine reads the very same file:

from pyplatypus import build_engine, from_yaml

engine = build_engine(from_yaml("nuclei.yaml"))
engine.fit()  # with `fit: false` this loads the published weights instead of training
masks = engine.predict("nuclei", split="validation")

A configuration file and an R script describe the same experiment, and either can be handed to a colleague - whichever language they work in.

What is in it

Briefly, because most of these get their own post:

Segmentation, 2D and 3D: U-Net, U-Net++, Res-U-Net and LinkNet, eleven losses, five metrics that can leave the background out - which matters when the background is most of the image.
Classification - one label for a whole image, scored by AUC rather than accuracy, because a classifier that answers “no finding” where 3% have one scores 0.97 and has learnt nothing.
Object detection with YOLOv3, with anchors fitted to your boxes rather than borrowed from COCO.
Instance segmentation - a count of objects instead of an area.
Volumes: NIfTI and DICOM in, with voxel spacing handled, a split by patient rather than by slice, and results in millilitres rather than voxels.
A score per case, not one per dataset, so the worst image has a name.
Refusals before a wasted day: a colormap that matches none of your masks, channels in an order you did not intend, a DICOM series with a missing slice - each stopped before training, by name.
One specification - from R arguments or from a YAML file, read the same way by the Python engine.

The full reference and seven vignettes, six of them precomputed from real runs, are at maju116.github.io/rplatypus.

This is where the package starts, not where it ends. The U-Net family, YOLOv3 and the classifier are the first models in it, not the only ones that will be: more architectures are coming, along with what is still missing - multi-label classification, ensembling, boxes in three dimensions. Which of those comes first depends partly on what people actually need, which is one more reason for the question at the end of this post.

Development is moving fast right now, and CRAN asks for new versions no more often than every month or two, so once the package is on CRAN, install.packages("rplatypus") will usually be a step behind. The GitHub version above will always be the newest.

What comes next

This is the first post of a series. Each of the next ones takes one dataset and one thing that measurement on it taught me:

Nuclei, five years later - a model with a mean Dice of 0.855 that scores 0.006 on three images, and why the mean could not tell you.
Classification - a classifier with 0.97 accuracy that has learnt nothing, and why balanced accuracy does not rescue it.
Blood cells and YOLOv3 - anchors, and why a point of mAP across five seeds is noise.
CT volumes - a Dice of 0.88 and volumes 12-23% too large, which is not a contradiction.
Retinal vessels - why you cannot resize a vessel nine pixels wide.
Counting touching nuclei - 245 objects or 180, from the same forward pass.
Pretrained encoders - measured on three modalities, where a 1.9M-parameter encoder beats ResNet-34 on two of them.

One question for you

Everything above rests on an assumption I have not been able to test: that people do medical image analysis in R and would rather stay in R to do deep learning on it. If that is you - or if it is your lab, and you went to Python because there was nothing here - I would really like to hear about it. Open an issue, write to me at michalmaj116@gmail.com, or find me on LinkedIn.

And one more thing, since this is a good place to say it: I am looking for my next role. If your team works on computer vision, medical imaging, or any problem where a model has to be measured honestly before anyone trusts it, I would be glad to talk - the same email works.