Informatics, digital & computational pathology

• Preparing graph structured data from whole slide images
  • Tissue preprocessing: using various image processing techniques to isolate large contiguous regions of tissue
  • Nodes: patches, nuclei / cells, clustered microarchitectures / tissue regions
    • Graph formation based on breaking down tissue into various constituent components (i.e., nodes)
      • Localized cells: use of cell detection / segmentation algorithms (e.g., UNET, YOLOv8, Detectron, Mask-RCNN, hematoxylin deconvolution) to assign positional coordinates to cells throughout a slide
        • Underappreciates diagnostic / prognostic information contained outside of cell dense regions (e.g., large scale interactions with fibrous stroma)
        • Highly dependent on cell detection accuracy and information extracted from cells, very complex and computationally intractable based on number of cells
        • Farthest point sampling reduces complexity
        • Tissue microarrays can also make cell graphs possible but can introduce sampling bias
      • Tissue patches
        • Patches can also be captured based on partitioning of tissue through segmentation based / superpixel approaches (e.g., SLIC) or regularly spaced
  • Edges: typically based on spatial proximity (e.g., k nearest neighboring cells or patches within a 50 micron radius)
  • Nodal attributes / embeddings: usually a set of features derived from applying a convolutional neural network (CNN) or vision transformer to an image associated with the node; can also represent various shape, color and texture features (e.g., gray level co-occurrence matrix, custom image filters, etc.)
    • Embeddings depend on the learning objective for CNN, e.g., supervised / unsupervised / self supervised
    • Methods to extract cell / patch features to form nodal attributes
      • Based on raw imaging features
        • Morphological / shape characteristics: texture based analysis (e.g., gray level co-occurrence), nuclei features (e.g., presence of grooves, frayed chromatin), nuclei shape (e.g., eccentricity), etc.
        • Multiplexed immunofluorescence / immunohistochemistry (IHC): staining intensities for detected cells
      • Convolutional / transformer approaches
      • Any combination of the above features
  • Targets
    • Node level: tissue region classification, spatial gene expression patterns, cell type prediction (e.g., immune cell), etc.
    • Graph level: prediction of nodal metastasis, various bulk molecular alterations, survival outcomes, etc.
  • Input data preprocessing
    • Routine tissue staining: utilizes hematoxylin for highlighting nuclei and eosin for visualizing cytoplasm and extracellular matrix
    • Immunostaining: detect and visualize specific proteins of interest
      • Immunohistochemistry: typically uses 2 to 3 stains through enzyme labeled antibodies, producing chromogenic reactions that deposit colors that can be visualized through brightfield imaging
      • Multiplexed immunofluorescence: relies on spectral unmixing to differentiate antibodies conjugated to fluorophores that may spectrally overlap, which are employed for labeling biomolecules like proteins
• Structure of GNNs
  • Graph convolution: shares information between neighboring nodes as added context; nodes pass messages along edges and update their states based on received messages
    • Graph attention: weighs the importance of nodes differently during convolution
    • GraphSAGE: aggregates features from a node's local neighborhood using sampling techniques
    • Graph convolutional network (GCN): uses a spectral based approach to define convolutions on graphs
  • Pooling methods: techniques to downsample nodes and edges for hierarchical representation; represents objects at higher abstraction; akin to decreasing the spatial resolution of an image to represent more complex phenomena
    • DiffPool: differentiable pooling that assigns nodes to clusters in a hierarchical manner
    • TopK pooling: retains the top k nodes based on a scoring function
    • SAGPool: self attention graph pooling that scores nodes using a self attention mechanism
      • Self attention: technique that provides a score for each node based on their relevance
  • Data augmentation: techniques to enhance the robustness and generalization of models
    • Drop node: randomly removes nodes during training
    • Drop edge: randomly removes edges during training
    • Subgraph sampling: samples subgraphs to create varied training instances
    • Feature masking: randomly masks node attributes during training
• Examples of types of GNNs for various prediction tasks
  • Node level classification / regression: predicting labels or continuous values for individual node
    • Examples: automatic annotation, spatial gene expression, cancer cell detection
  • Node level clustering: grouping nodes into clusters based on their features and connectivity
    • Examples: identifying functional modules in biological networks, customer segmentation in social networks
  • Graph level classification / regression: predicting labels or continuous values for entire graphs
    • Examples: risk stratification, cancer detection, bulk gene expression
  • Graph level clustering: grouping entire graphs into clusters based on their overall structure
    • Examples: identification of novel histological variants
  • Self supervised learning: learning representations without explicit labels by leveraging the structure of the graph; can help cluster nodes and graphs and reduce the amount of data needed for supervised analysis (Pac Symp Biocomput 2024;29:464, BioData Min 2023;16:23)
    • Examples: contrastive learning, node embeddings, graph embeddings
      • PyGCL: self supervised contrastive learning (ArXiv: An Empirical Study of Graph Contrastive Learning [Accessed 20 August 2024])
    • Graph clustering (ArXiv: Spectral Clustering with Graph Neural Networks for Graph Pooling [Accessed 20 August 2024])
  • Link prediction: predicting the existence or likelihood of edges between nodes
    • Examples: recommending friends in social networks, predicting protein - protein interactions
      • Graph autoencoder (IJCAI (US) 2018;2609)
  • GNN interpretation: explaining and interpreting the predictions made by GNNs; can highlight important nodes, edges, subgraphs, features
    • Examples: GNNExplainer, Captum, integrated gradients, attention mechanisms
  • Transductive versus inductive: approaches for making predictions
    • Transductive: making predictions on specific nodes seen during training
    • Inductive: generalizing to unseen instances and graphs beyond the training data

Contributed by Joshua Levy, Ph.D., Minh-Khang Le, M.D., Ph.D. and Louis Vaickus, M.D., Ph.D.

In a pathology graph constructed from a whole slide image, what is typically represented by nodal attributes?

1. Diagnosis associated with each slide
2. Metadata of the slide including the date of preparation and the patient's details
3. Numerical characteristics describing features such as cell morphology or staining intensity
4. Type of scanner used to digitize the whole slide image

C. Numerical characteristics describing features such as cell morphology or staining intensity. Information must be stored at and shared between specific locations in the tissue slide that encompasses the underlying histomorphology / molecular information. Predictive modeling with graph neural networks requires the extraction and sharing of information organized spatially within a slide and could not be accomplished without this information. Answer A is incorrect because the diagnosis is an example of an outcome for a graph level classification task and can be derived from summarizing nodal attributes across the patches in a slide. Answer B is incorrect because metadata are patient level characteristics and do not represent local, patch / cellwise imaging features. Answer D is incorrect because the type of scanner is unrelated to the nodal attributes in the context of whole slide graphs.

Comment Here

Reference: Application of graph neural networks to whole slide images

What is the primary function of convolutional layers in a graph neural network (GNN) when applied to pathology images?

1. To adjust the color balance of histology images for better visualization
2. To classify the entire slide image into benign or malignant
3. To encrypt patient data for secure transmission
4. To share information between neighboring nodes, providing context to biological entities

D. To share information between neighboring nodes, providing context to biological entities. Graph convolution operators are used to capture spatial dependencies between adjacent tissue regions / cells to improve predictive modeling above and beyond modeling this information independently without graph convolutions. Answer A is incorrect because adjusting the color balance or normalizing the tissue stain can help improve the predictive accuracy of the algorithm; however, this task is typically performed during preprocessing and not by convolutional layers. Answer B is incorrect because classifying entire slide images is generally achieved through pooling layers that aggregate information at a higher level. Answer C is incorrect because data encryption is unrelated to the function of convolutional layers in GNNs.

Comment Here

Reference: Application of graph neural networks to whole slide images