Research on Paco's Log

Multimodal AI foundation models that combine digital pathology + transcriptomics: what has been done and what is coming

Sat, 02 May 2026 00:00:00 +0000

Digital pathology data—specifically tissue stained with Hematoxylin and Eosin (H&E)—is widely utilized in both clinical settings and preclinical research. Its cost-effectiveness, high availability, and suitability for computational analysis have established it as one of the most prevalent modalities in the field. Following a core tissue or needle biopsy, samples may undergo processing to sequence the patient’s transcriptomic profile, resulting in bulk RNA-Seq data. Although more advanced techniques have emerged recently—such as single-cell sequencing and spatial transcriptomics—H&E slides and RNA-Seq remain two of the most common and foundational biomedical modalities used today.

The future of AI for biology needs to be multimodal, and that means we need to get our data right.

Sat, 13 Sep 2025 00:00:00 +0000

The use of AI is becoming more and more widespread in biology, with significant progress and a noticeable increase in research papers in specific areas. For example, digital pathology has been completely transformed by AI models. One of the first papers I read in this field, published in Nature Medicine back in 2018 by Coudray et. al., was a landmark achievement that showed what AI was capable of at the time. While it wasn’t the very first paper on AI in digital pathology (Ertosun et al.. published a paper in 2015, for instance), it was a sensation because it demonstrated that a tumor’s mutational status could be predicted from inexpensive and widely available digital pathology slides. This was during the beginning of my PhD, and it was a revelation: data plus computing power equals powerful models.

Med-CrossViT: A WSI-RNA-Seq fusion model

Fri, 02 May 2025 00:00:00 +0000

I while back I had the idea of using cross-attention for implementing an early fusion model of my two modalities of choice: digital pathology and transcriptomics. I decided to go for it, since it was not available, and called it Med-CrossViT. Instead of using two augmentations of the same image as input, this model uses two different modalities: bags digital pathology tiles features and RNA-Seq.

An example of the original architecture can be seen below: Image of the original architecture