Interpretable, reusable concept directions in biological foundation model embedding spaces

Ahmad Shamail1, Claire D. McWhite2

Ahmad Shamail: 0009-0009-8090-9698

Claire D. McWhite: 0000-0001-7346-3047

1Department of Molecular and Cellular Biology
The University of Arizona
Tucson, AZ, USA

Correspondence: clairemcwhite@arizona.edu

ABSTRACT: Biological foundation models encode rich information about protein structure, function, and cellular state in high-dimensional embeddings, but it remains difficult to isolate specific biological signals within these representations. Here, we show that diverse biological properties can be mapped to interpretable, reusable directions in embedding space. In the Concept Activation Vector (CAV) approach, examples representing a concept are distinguished from a background set using a linear model, and the resulting weight vector defines the CAV for that concept. In protein embeddings, CAVs recover Gene Ontology and Enzyme Commission annotations, localize Pfam domains, reveal novel structural relationships, and capture DNA-binding and protein-interaction interface specificity. In single-cell embeddings, CAVs isolate cancer-associated expression transitions, and gene variation along these axes substantially agrees with conventional differential-expression despite being a distinct computational approach. Together, these results establish concept directions as a common representation, enabling diverse biological features to be compared, localized, and manipulated.

INTRODUCTION

Biological foundation models learn high-dimensional representations that encode substantial information about biological structure, function, and state. Protein language models, for example, learn contextual relationships among amino acids that reflect biochemical and structural patterns, while single-cell foundation models learn representations of gene-expression programs and cellular states. These embeddings provide general-purpose representations that support diverse downstream applications. For proteins, these include structure prediction and design(Madani et al. 2023; Lin et al. 2023), homology search(Kaminski et al. 2023; Liu et al. 2024), coevolutionary analysis(Z. Zhang et al. 2024), and a broad range of functional prediction tasks (Leclercq and Droit 2026; Chen et al. 2025). For single cells, foundation models have been applied to cell-type annotation, data integration, gene-network inference, and perturbation-response prediction (Cui et al. 2024; Hao et al. 2024; Baek et al. 2025). In other words, the representations are increasingly general, but the biological questions used to interrogate them remain largely task-specific. Specific biological properties are therefore typically accessed through task-specific models with predefined outputs, rather than represented directly within the embedding space.

This creates a mismatch between the generality of foundation-model representations and the specificity of many biological questions. Researchers may wish to study properties that fall outside existing annotation vocabularies, such as a newly identified structural motif, a shared interaction interface, or a transcriptional program associated with a particular cellular transition. In each case, the relevant biological signal may already be encoded in the pretrained representation, but existing approaches provide limited flexibility for turning that signal into a reusable representation of the biological concept. More broadly, biological properties such as protein function, enzyme activity, domain identity, interaction specificity, and cellular state are typically represented by separate prediction systems with different outputs and scoring schemes, making their predictions difficult to compare directly across concepts or biological scales.

Figure 1. Concept Activation Vectors represent biological concepts as directions in embedding space. (A) A concept such as “stripes” can be represented as a direction in embedding space, with higher alignment to an image of a zebra than to a horse. (B) A zinc-finger CAV is learned from protein language model embeddings by separating concept examples from a background set. (C) The same framework can be applied to single-cell embeddings, illustrated with a macrophage concept. (D) CAVs learned in a common embedding space can be directly compared. (E) New concepts can be added independently without retraining existing CAVs. (F) Protein or cell embeddings can be scored for alignment with each concept by dot product. (G) The same operation can be applied to residue embeddings to localize concept-associated signal within a protein. (H) Contrastive CAVs can distinguish closely related concepts. (I) Projection can remove variation associated with a covarying or confounding concept. (J) Differences between concept directions can define transitions between biological states.

Concept Activation Vectors (CAVs), originally developed for interpreting neural networks (Kim et al. 2018), represent human-interpretable concepts as directions within a model’s internal representation. For example, a CAV can define a direction corresponding to “striped,” and the associated TCAV framework asks whether a model’s prediction of “zebra” is sensitive to movement along that direction (Figure 1A). Here, we instead use the concept direction itself as the representation of interest, directly measuring how strongly biological embeddings align with it. Given examples such as proteins belonging to a Pfam family (Figure 1B) or cells sharing a transcriptional state (Figure 1C), we train a linear classifier to distinguish those examples from a background distribution. We then discard the classifier and retain only its weight vector as the corresponding concept direction. New proteins, sequence regions, residues, or cells can then be scored by their alignment with this vector.

The resulting CAV is a reusable representation that can be compared, manipulated, and applied independently. CAVs learned against a common background can be directly compared and clustered (Figure 1D), and new concepts can be added independently without retraining existing ones (Figure 1E). Protein and cell embeddings can be rapidly scored for alignment with a concept by dot product (Figure 1F) and the same alignment operation can be applied to amino acid embeddings or sequence-window embeddings to localize concept-associated signal within a protein (Figure 1G), while contrastive CAVs can distinguish closely related biological properties (Figure 1H). Because the concepts are represented as vectors, standard geometric operations can also be used to remove known sources of variation (Figure 1I) or define transitions between biological states (Figure 1J).

We demonstrate these capabilities across several biological settings. We first ask whether CAVs can recover established protein functional annotations and organize Gene Ontology and Enzyme Commission concepts within a shared space. We then use concept directions to localize protein domains and uncover structural relationships not captured by existing annotations, and show that contrastive CAVs can resolve molecular interaction specificity that is difficult to distinguish using sequence or embedding similarity alone. Finally, we extend the same framework to single-cell embeddings, where vector projection separates tissue- and cell-type-associated variation from cancer-associated expression changes. Together, these analyses test whether biological concepts can be represented as interpretable, reusable directions in pretrained embedding space.

RESULTS

CAVs unify and organize protein functional annotations

The function of most proteins is inferred computationally, as relatively few proteins have been experimentally characterized (The UniProt Consortium 2023; Škunca et al. 2017; de Crécy-Lagard et al. 2022). While homology-based annotation remains highly effective, sequence divergence can obscure relationships among functionally conserved proteins, while functional divergence among close homologs can make annotation transfer unreliable (Schnoes et al. 2009; Rembeza and Engqvist 2021). We therefore asked whether CAVs learned from pretrained protein embeddings could directly recover established functional annotations. Unless otherwise noted, all protein CAVs were trained against the same random protein background, allowing the resulting directions to be compared directly.

We evaluated CAVs across two hierarchical annotation systems: Gene Ontology (GO), which describes molecular functions, biological processes, and cellular components (The Gene Ontology Consortium 2026), and Enzyme Commission (EC) numbers, which classify enzymatic reactions (Tipton and Boyce 2000). Using the proFAB benchmark (Özdilek et al. 2023), comprising 6,678 GO terms and 978 EC terms, we compared CAV-based prediction with two sequence-only annotation models: DeepGO-SE for GO (Kulmanov et al. 2023) and CLEAN-SupCon for EC (Yu et al. 2023). Both use pretrained protein embeddings in supervised models trained to predict a fixed set of functional classes.

Figure 2. CAVs recover newly annotated Gene Ontology functions. (A) For each GO term, a CAV was trained using pre-2021 annotated proteins against a common background of random proteins. CAV scores were standardized to z-scores using term-specific GO negatives not used for training, and evaluated on proteins receiving the corresponding annotation after 2021. (B) Distribution of CAV z-scores for newly annotated positives and GO-term negatives. (C) Rank of the true GO annotation among all candidate terms for CAVs and DeepGO-SE, shown separately for Molecular Function (MF), Biological Process (BP), and Cellular Component (CC). CAV scores below z = 1 were treated as no prediction. (D–E) Distribution of term-wise AUROC (D) and AUPR (E) for CAVs and DeepGO-SE across the three GO branches.

For each GO term, we learned a CAV separating annotated proteins from the common background. We then evaluated these CAVs on proteins that received the corresponding GO annotation after the 2021 proFAB dataset cutoff. Each protein was scored by its projection onto the GO-term CAV, and scores were converted to z-scores using the corresponding proFAB negative set, which was not used during training (Figure 2A). Newly annotated positive proteins were shifted toward higher CAV scores relative to negatives, indicating that individual CAVs captured term-specific functional signals (Figure 2B).

We next tested whether the correct GO annotation could be identified among competing terms for each protein. Candidate terms were ranked by CAV z-score, with z < 1 treated as no prediction. Across Molecular Function, Biological Process, and Cellular Component, CAVs placed the correct annotation among the highest-ranked terms more often than DeepGO-SE, particularly at rank 1 and ranks 2–3, while also producing fewer no-prediction cases (Figure 2C). Term-wise evaluation showed the same pattern: CAVs achieved consistently high AUROC (Figure 2D) and AUPR (Figure 2E) across all three GO branches.

Figure 3. CAVs recover enzyme classifications and relationships between functional ontologies. (A) Recovery of EC annotations by CAVs and CLEAN-SupCon for 209 validation protein–EC pairs at EC levels 4 (complete EC number) and 3. CAV predictions are shown by rank of the correct annotation. (B) CAV projection scores for representative proteins across successive levels of the EC hierarchy, from broad reaction class to complete EC number. (C) UMAP visualization of GO and EC CAVs in their shared embedding space, colored by ontology or GO branch. (D) Cosine similarity between GO and EC CAVs for curated EC2GO annotation pairs compared with unrelated GO–EC pairs. (E) Rank of the curated EC partner for 239 GO terms among 952 trained EC CAVs, based on GO–EC CAV cosine similarity.

We then turned to EC classification, where the hierarchy progresses from broad reaction classes to increasingly specific reaction types. CAVs recovered the correct annotation particularly well at the complete four-digit level 4 EC assignments and also performed strongly at level 3, outperforming CLEAN-SupCon in both settings (Figure 3A). Notably, CAV performance was strongest for the more specific EC categories (Figure 3B). This contrasts with many multiclass settings, where broader classes are often easier to predict. For CAVs, broad EC classes may be harder to represent with a single direction because they group mechanistically diverse enzymes. More specific classes are functionally more coherent, producing a clearer shared signal in the protein embedding space.

Because all CAVs are defined in the same embedding space, annotations from otherwise separate ontologies can also be compared directly. GO and EC CAVs occupied a shared functional space (Figure 3C), and GO–EC pairs linked by the GO Consortium’s curated EC2GO mapping showed substantially higher cosine similarity than unrelated pairs (Figure 3D). For 88% of the 239 GO terms represented in both EC2GO and our dataset, the curated EC partner was the nearest of 952 EC CAVs, with most remaining partners also ranked near the top (Figure 3E). Thus, the CAV space recovers known relationships between GO and EC annotations without explicitly encoding the mapping between them.

CAVs distinguish and localize structural motifs

Protein motifs and domains are modular units that encode catalytic, binding, regulatory, and structural functions. Existing resources such as Pfam, InterPro, and PROSITE identify these features largely through sequence profiles, alignments, and curated patterns (Blum et al. 2025; Finn et al. 2014; Sigrist et al. 2026). These approaches are highly effective, but can become less sensitive as sequence similarity declines.

Protein language models provide an alternative representation in which structurally or functionally related regions can remain similar despite substantial sequence divergence (Liu et al. 2024; Pantolini et al. 2024). Recent methods have extended motif and domain annotation using sparse autoencoders such as MotifAE, structure-based domain discovery as in TED, and transformer-based sequence segmentation as in InterPro-N (Blum et al. 2025; Hou et al. 2025; Lau et al. 2024). Here, we instead ask whether structural motifs can be represented directly as simple directions in pretrained protein embedding space, allowing CAVs to both distinguish motifs and localize them within proteins.

Because CAVs trained against a common background are directly comparable, we hierarchically clustered all Pfam CAVs and cut the resulting tree at multiple levels to generate a nested map of Pfam families (Figure 4A). The complete clustering is provided in Supplemental File 1 and as an interactive visualization at mcwhitelab.org/pfam-circlepack. This organization provides a global view of relationships among Pfam families and a framework for guilt-by-association analysis of poorly characterized domains.

We next examined how motif signal varies across model layers. CAV scores were strongest and most sharply localized at intermediate ESM-C layers, particularly layers 21–31, while early and late layers showed weaker signal (Figure 4B), consistent with intermediate layers capturing transferable higher-order features (Zeiler and Fergus 2013; Yosinski et al. 2014). Layer 26 was therefore used for subsequent analyses. Using layer 26, we illustrate CAV-based motif localization in the large multidomain protein human Plexin-A1 (Figure 4C). CAVs corresponding to different domains produce distinct peaks that closely coincide with their annotated boundaries. For clarity, we show raw CAV scores, which in this example delineate domains at scores above approximately 5; these scores can also be calibrated as z-scores against negative examples.

Figure 4. Pfam CAVs reveal global relationships, domain localization, and candidate structural reinterpretations.
(A) Hierarchical clustering of all Pfam CAVs, visualized as a nested circlepack of Pfam families. Selected clusters are highlighted. (B) CAV scores across ESM-C layers for representative domains in VAV_DROME, showing strongest and most localized signal at intermediate layers. (C) Domain localization in human Plexin-A1 using layer 26 embeddings; distinct Pfam CAVs peak at their corresponding annotated domains. (D) Structural alignment of MOM1-like/PF25029 with a cyclin-like domain. (E) Structural alignment of ATP-grasp_6 /PF18419 with the pre-ATP-grasp domain of GshB rather than the ATP-grasp domain. (F) Structural comparison identifying a cryptic GIY-YIG-like endonuclease fold within DUF2797/PF10977, with inset highlighting corresponding conserved residues and motif features. (G) Candidate Zn-binding site in DUF4187/PF13821 at the C terminus of GPATCH11. (H) Two candidate C4 Zn-binding sites in DUF2039/C9orf85/PF10217, including a nested cysteine arrangement. (I) Candidate Zn-binding architecture in DUF6349/PF19876, in which a discontinuous C3H site wraps around a compact C4 site.

We hypothesized that the structure of the Pfam CAV map could expose relationships not captured by existing domain annotations. Among the many potentially informative relationships in the map, we highlight several examples in which the inferred domain homology is absent from existing annotations and, to our knowledge, has not previously been proposed in the literature or online domain resources. As a simple example of structural inference, we find that PF25029, a domain within the large chromatin-associated silencing protein MOM1, adopts a cyclin-like fold (Figure 4D). PF25029 is currently defined only by its association with MOM1 and has no assigned molecular function. We also identify potentially misleading Pfam annotations. PF18419, currently annotated as “ATP-Grasp_6/ATP-grasp-like domain,” clusters strongly with pre-ATP-grasp domains. Structural comparison confirms that PF18419 matches the pre-ATP-grasp domain of GshB rather than its ATP-grasp domain (Figure 4E).

We next highlight DUF2797 as a striking example of a cryptic domain embedded within a larger, otherwise uncharacterized domain. Our approach identifies a GIY-YIG-like endonuclease fold hidden within the larger DUF2797 architecture (Figure 4F). This relationship is not recovered by Foldseek, demonstrating that CAVs can recover biologically meaningful structural signals that are not readily detectable by conventional structural homology searches. This finding provides both a structural reinterpretation of DUF2797 and a specific hypothesis for endonuclease activity.

A particularly prominent signal in the Pfam CAV map was a large set of previously unannotated candidate zinc-binding motifs within domains. We identified 82 Pfam families that clustered with known zinc-binding domains and contained structurally plausible candidate Zn-binding sites (Supplemental File 2). We highlight several examples spanning distinct architectures. We first demonstrate a likely CCCH Zn-binding module in the uncharacterized domain PF13821/DUF4187 (Figure 4G). This domain frequently occurs at the C-terminus of GPATCH11, a gene recently established to cause retinal and neurodevelopmental disease in humans (Zanetti et al. 2024).

We next highlight two candidate zinc-binding domains with less canonical architectures. In DUF2039 (PF10217), the sole annotated domain of the uncharacterized human protein C9orf85, we identify two candidate C4 zinc-binding sites (Figure 4H). These sites are organized in a nested AA-BB-BB-AA cysteine arrangement, conceptually similar to the nested RING0 domain of Parkin. Interestingly, C9orf85 was recently found to be strongly physically associated with the BRISC complex (Fischer et al. 2025). As a second example, DUF6349, an uncharacterized domain found largely in proteobacteria, contains a striking discontinuous C3H site that wraps around a compact C4 site (Figure 4I).

Together, these examples show that CAV-based motif clustering can recover structural relationships that extend beyond existing annotations and generate specific hypotheses for previously uncharacterized domains.

CAVs can capture binding-interface specificity

As CAVs can be created from arbitrary sequence selections, we next tested whether they could capture binding specificity at molecular interaction interfaces. As a first test, we embedded a single example of the DNA-binding region of human MYC and trained a CAV against our standard background set of proteins (Figure 5A). Surprisingly, even this single example produced a direction that recovered CACGTG-binding proteins when scanned across the human proteome (Figure 5B). To recover this specificity, it was necessary to pool embeddings over local amino-acid regions rather than entire proteins, as length differences appeared to dilute the binding-site signal in whole-sequence embeddings.

Figure 5. CAVs capture binding-interface specificity from local sequence regions. (A) A single-example CAV was constructed from the CACGTG-binding region of human MYC using pooled amino acid embeddings from residues 366–420. (B) Top human proteins aligned to this single-example CACGTG-binding CAV; all are known CACGTG-binding transcription factors. (C) A contrastive CAV for LRR-RK co-receptor specificity was trained from membrane-proximal co-receptor-binding regions, using SERK-binding interfaces as positives and CIK/CORYNE/ANX/non-SERK-binding interfaces as negatives. (D) Sequence similarity among representative LRR-RK co-receptor-binding regions, showing that SERK-binding specificity is not fully explained by primary-sequence similarity. (E) Separation of SERK- and non-SERK-binding receptors by nearest-neighbor cosine margin in the underlying embeddings (left) and by the resulting contrastive CAV score (right).

We then tested the more complex problem of leucine-rich repeat receptor kinase (LRR-RK) co-receptor specificity (Figure 5C). Despite the widespread use of shared co-receptors by plant LRR-RKs, determining which receptors preferentially pair with which co-receptors remains a major challenge (Smakowska-Luzan et al. 2018; Xi et al. 2019). For each receptor, we restricted both sequence and embedding comparisons to the membrane-proximal co-receptor-binding region, spanning residues −170 to −20 relative to the transmembrane helix. Although many co-receptor pairs can be inferred from sequence similarity, this is not always reliable (Figure 5D). For example, the SERK-binding BRI1 is more similar in sequence to the non-SERK-associated receptor BAM1 than to several known SERK-binding receptors.

Nearest-neighbor comparisons using the underlying embeddings were already highly predictive of SERK-binding specificity, indicating that the relevant information is present in the pretrained representation, although the separation occurred over a relatively narrow similarity margin (Figure 5E). A CAV trained on SERK-binding interfaces against the common random protein background failed to sharpen this distinction, likely because it captured a more generic co-receptor interface signal. We therefore trained a contrastive CAV using 17 SERK-binding Arabidopsis LRR-RK interfaces as positives and 18 non-SERK-binding interfaces as negatives. This CAV perfectly separated SERK- from non-SERK-binding receptors in a test set composed entirely of 36 non-Arabidopsis proteins (AUROC = 1.00; Figure 5F). Thus, the contrastive CAV isolates a binding-specificity signal already present in the embedding space and represents it as a more clearly separated, transferable direction.

CAVs organize single-cell gene expression

Thus far, we have used CAVs to identify concepts encoded in protein sequence embeddings. However, the framework is not specific to protein representations and can be applied to other biological feature spaces. We therefore asked whether CAVs could also identify concepts in single-cell gene-expression embeddings, where representations capture cellular expression profiles rather than protein sequences.

Figure 6. CAVs organize single-cell expression variation into cancer-associated transition axes. (A) Schematic of the projection procedure used to isolate a cancer-associated direction for each tissue–cell-type combination. Tissue-level normal CAVs define a first axis (L0), normal cell-type CAVs within a tissue define a second axis after removal of the tissue component (L1), and the cancer-associated transition direction (L2) is obtained by removing both tissue- and cell-type-associated components from the corresponding cancer CAV. (B–D) Comparison of gene-level differential expression between normal and cancer cells (x-axis) with correlation between gene expression and position along the L2 transition axis (y-axis) for neutrophils in breast (B), epithelial cells in lung (C), and fibroblasts in colorectum (D). Selected genes are labeled. Lower panels show the top five positively and negatively correlated genes across cells ordered by L2 score, with normal and cancer cells indicated above each strip. Spearman correlations between the CAV-based and differential-expression analyses are shown for each case.

We applied CAVs to a multi-tissue single-cell tumor microenvironment atlas containing cells from normal and cancer samples across multiple tissues (Guimarães et al. 2024). We hypothesized that cancer-associated expression signals could be isolated by removing variation associated with tissue and cell identity. Because CAVs are represented as vectors in a common embedding space, standard vector operations can be used to separate overlapping biological signals. We trained CAVs for each tissue–cell-type combination against a common background of 10,000 randomly sampled cells, then averaged CAVs within tissues and cell types and removed these components by Gram–Schmidt orthogonalization. The resulting direction captures the cancer-associated expression shift (Figure 6A). Projecting individual cells onto this direction defines a cancer-associated transition axis, with each cell assigned a score according to its position along the axis.

We examined three representative transitions spanning distinct cellular contexts: neutrophils in breast cancer (Figure 6B), epithelial cells in lung cancer (Figure 6C), and fibroblasts in colorectal cancer (Figure 6D). We next asked which genes varied along each cancer-associated transition axis, and whether these changes agreed with conventional differential expression. For each case study, we correlated the expression of each gene across individual cells with their position along the transition axis and compared these correlations with normal-versus-cancer differential expression. Because cells were drawn from a small number of donors, differential expression was estimated using a per-gene linear mixed model with donor as a random effect. Both analyses were restricted to donors represented in both normal and cancer samples.

The two approaches showed substantial agreement for neutrophils in breast cancer (Spearman r = 0.60; 10,681 genes) and epithelial cells in lung cancer (r = 0.73; 11,140 genes). Strong concordant signals included MHC class II genes (HLA-DRA, HLA-DPA1, and CD74) in neutrophils, consistent with the HLA-DR⁺CD74⁺ antigen-presenting neutrophil state described across human cancers (Wu et al. 2024), and an epithelial keratin program (KRT7, KRT18, and KRT19) in lung, consistent with their enrichment in tumor epithelial cells in NSCLC (Xu et al. 2024). In colorectal fibroblasts, the CAV axis and normal-versus-cancer differential expression showed weaker agreement (r = 0.27). Along the CAV axis, a fibroblast program marked by MGP, DCN, OGN, C3, and CCDC80 decreased, while both CXCR4 and ADAMDEC1 increased. Notably, OGN⁺ fibroblast populations are strongly depleted in colorectal cancer tissue, with a corresponding loss of DCN expression (Li et al. 2022). CXCR4 and ADAMDEC1 were not strongly co-expressed with one another, indicating that they represent distinct transcriptional programs that vary along the same CAV-defined transition (Supplemental Figure 1). This pattern is consistent with the continuous cell-state and compositional remodeling that accompanies colorectal cancer progression(Becker et al. 2022), and suggests that such changes may be better captured by a cellular transition axis than by a binary normal-versus-cancer comparison.

Comparison across the successive projection steps further supported the importance of removing tissue- and cell-type-associated variation: agreement with differential expression increased substantially in breast neutrophils and colorectal fibroblasts after these components were removed, while remaining high in lung epithelial cells (Supplemental Figure 2). These results show that CAVs can organize single-cell expression variation into interpretable biological axes, providing a complementary view to conventional differential expression.

DISCUSSION

Representing biological concepts as directions in embedding space provides a simple framework for connecting accumulated biological knowledge with the representational power of biological foundation models. Concept Activation Vectors (CAVs) are rapid to train, reusable, and can be used to search for proteins exhibiting a learned property. Because CAVs trained within the same embedding space share a common coordinate system, they can also be compared directly, allowing relationships to be examined within and across annotation systems. In some cases, useful concepts can be defined from as little as a single example, while representing concepts as numeric vectors makes standard vector operations available for biological analysis. The same concept directions can be projected onto local sequence representations to identify where a property is encoded, while contrastive CAVs can isolate specificity between closely related biological properties. CAV-based comparison also generated structural hypotheses for previously uncharacterized domains, including 82 candidate zinc-binding domains, suggesting a route toward connecting poorly annotated sequence space to known structural and functional classes.

In single-cell embeddings, projection along a CAV axis recovered gene-level changes that substantially agreed with conventional differential-expression analysis despite being obtained through a distinct computational approach. Because a CAV defines a continuous axis rather than a binary comparison, it can also capture graded expression changes, including those involving genes with relatively low expression.

Despite the simplicity of a linear separator, directions in embedding space can capture protein and gene-expression properties that are often modeled using substantially more complex approaches. This likely reflects the representational work already performed during pretraining: complex sequence or expression relationships are reorganized into a high-dimensional space in which a shared biological property can appear as a coordinated shift across many dimensions. A CAV therefore does not learn these features from scratch, but instead identifies the weighted combination of embedding dimensions that varies consistently with the concept. Even when the underlying features and their interactions are complex and nonlinear, pretraining can therefore transform them into signals that are linearly accessible. Consistent with this interpretation, features often become increasingly linearly separable with depth in neural networks (Alain and Bengio 2016), and structural information can be recovered from protein language-model embeddings using linear projections (Rives et al. 2021).

CAVs have been applied to biological and biomedical models since their introduction, most often in the context of concept attribution. The original TCAV study used retinal lesions to interpret predictions from a diabetic-retinopathy classifier (Kim et al. 2018), and subsequent work has used concept vectors to relate histological features to model predictions (Graziani et al. 2020; Gamble et al. 2021) and to identify and remove unwanted demographic signals in medical-imaging models (Correa et al. 2024). More recently, TPCAV extended concept attribution to genomics, enabling regulatory motifs and chromatin features to be related to deep-learning predictions (Yang and Mahony 2026). These studies establish concept attribution as a useful way to connect biologically meaningful features to model behavior.

Concept directions have also begun to support uses beyond attribution. Text2Concept derives CAVs from natural-language descriptions and uses similarity to these directions to rank images and perform zero-shot classification (Moayeri et al. 2023). CAVs have likewise been used to modify generated 3D shapes by adding or combining concept-associated directions in latent space (Druc et al. 2022) and to steer large language models by adding or removing learned concept directions during inference (H. Zhang et al. 2025). Here, we take a different view of concept directions: biological knowledge can be converted into a growing set of reusable directions within pretrained representation spaces, and those directions can themselves be compared, localized, contrasted, combined, and used to search for related examples.

This framework could complement task-specific prediction systems by providing a lightweight way to define and reuse biological properties within a common representation. Once embeddings have been generated, new concepts are inexpensive to define and can be added independently without retraining the underlying foundation model or existing CAVs. At the same time, CAVs can recover only information that is present in the underlying representation, and their performance will inherit the limitations, biases, and model-specific properties of that embedding space. Concept signal should therefore be established independently for each application rather than assumed to be present. The property captured by a CAV also depends on the background or contrast used to define it, allowing related signals to be progressively refined but making the choice of reference set an important part of concept definition.

The applications explored here likely represent only a small subset of what can be captured as reusable directions in biological embedding spaces. Because the framework requires only vector representations, it can in principle be applied to many other biological data types and models. As foundation models increasingly integrate sequence, structure, expression, imaging, and other modalities, CAVs could provide a common way to define and relate concepts across otherwise distinct forms of biological information. The broader opportunity is therefore not any single downstream application, but an extensible vocabulary of biological concepts that can grow with new models, datasets, and biological questions.

SUPPLEMENTAL FIGURE 1

Supplemental Figure 1. Distinct fibroblast expression programs vary along the colorectal cancer CAV transition axis.

(A) Comparison of normal-versus-cancer differential expression with gene-expression correlation along the fibroblast colorectal cancer transition axis (L2). Selected genes marking the OGN-associated fibroblast program (blue), CXCR4 (green), and ADAMDEC1/CXCL14 (orange) are highlighted. (B) Pairwise co-expression among highlighted genes across fibroblasts, showing strong correlation within the OGN-associated module and limited co-expression between CXCR4 and ADAMDEC1/CXCL14. (C) Fraction of normal and cancer fibroblasts with high OGN-module scores (top quartile) or no detectable expression of any of the five module genes. (D) OGN-module score across cells ordered by L2 transition score, colored by normal or cancer status; the black curve shows the smoothed trend.

SUPPLEMENTAL FIGURE 2

Supplemental Figure 2. Successive removal of cell-type- and tissue-associated variation improves agreement between CAV-defined cancer transitions and differential expression. For neutrophils in breast cancer, epithelial cells in lung cancer, and fibroblasts in colorectal cancer, gene-level correlations with each CAV-defined transition axis are compared with normal-versus-cancer differential expression. (A) Raw normal-to-cancer transition axis, before removal of cell-type- or tissue-associated variation. (B) Transition axis after removal of the cell-type baseline component (L0). (C) Final transition axis after removal of both cell-type- and tissue-associated components (L0 + L1; L2). Each point represents a gene; x-axes show differential-expression log2 fold change and y-axes show the Pearson correlation between gene expression and cell position along the corresponding CAV axis. Spearman correlations summarize agreement between the two gene-level measures.

Alain, Guillaume, and Yoshua Bengio. 2016. “Understanding Intermediate Layers Using Linear Classifier Probes.” In arXiv [Stat.ML]. October 5. arXiv. https://doi.org/10.48550/arXiv.1610.01644.

Becker, Winston R., Stephanie A. Nevins, Derek C. Chen, et al. 2022. “Single-Cell Analyses Define a Continuum of Cell State and Composition Changes in the Malignant Transformation of Polyps to Colorectal Cancer.” Nature Genetics 54 (7): 985–95.

Blum, Matthias, Antonina Andreeva, Luis Carlos Florentino, et al. 2025. “InterPro: The Protein Sequence Classification Resource in 2025.” Nucleic Acids Research 53 (D1): D444–56.

Correa, Ramon, Khushbu Pahwa, Bhavik Patel, Celine M. Vachon, Judy W. Gichoya, and Imon Banerjee. 2024. “Efficient Adversarial Debiasing with Concept Activation Vector - Medical Image Case-Studies.” Journal of Biomedical Informatics 149 (104548): 104548.

Crécy-Lagard, Valérie de, Rocio Amorin de Hegedus, Cecilia Arighi, et al. 2022. “A Roadmap for the Functional Annotation of Protein Families: A Community Perspective.” Database 2022: baac062.

Druc, Stefan, Aditya Balu, Peter Wooldridge, Adarsh Krishnamurthy, and Soumik Sarkar. 2022. “Concept Activation Vectors for Generating User-Defined 3D Shapes.” In arXiv [Cs.CV]. April 29. arXiv. https://doi.org/10.48550/arXiv.2205.02102.

Finn, Robert D., Alex Bateman, Jody Clements, et al. 2014. “Pfam: The Protein Families Database.” Nucleic Acids Research 42 (Database issue): D222–30.

Fischer, Samantha N., Erin R. Claussen, Savvas Kourtis, et al. 2025. “Hu.MAP3.0: Atlas of Human Protein Complexes by Integration of >25,000 Proteomic Experiments.” Molecular Systems Biology 21 (7): 911–43.

Gamble, Paul, Ronnachai Jaroensri, Hongwu Wang, et al. 2021. “Determining Breast Cancer Biomarker Status and Associated Morphological Features Using Deep Learning.” Communications Medicine 1 (1): 14.

Graziani, M., V. Andrearczyk, S. Marchand-Maillet, and H. Müller. 2020. “Concept Attribution: Explaining CNN Decisions to Physicians.” Computers in Biology and Medicine 123 (103865): 103865.

Guimarães, Gabriela Rapozo, Giovanna Resk Maklouf, Cristiane Esteves Teixeira, et al. 2024. “Single-Cell Resolution Characterization of Myeloid-Derived Cell States with Implication in Cancer Outcome.” Nature Communications 15 (1): 5694.

Hou, Chao, Di Liu, and Yufeng Shen. 2025. “MotifAE Reveals Functional Motifs from Protein Language Model: Unsupervised Discovery and Interpretability Analysis.” bioRxiv: The Preprint Server for Biology, ahead of print. https://doi.org/10.1101/2025.11.04.686576.