Multimodal fusion of CT and MRI for liver tumorsementation and classification using attention-based CNN's
DOI:
https://doi.org/10.63682/jns.v14i31S.8335Keywords:
Liver, CT scan, Machine learning, Regression, CNNAbstract
Accurate segmentation and classification of liver tumors are critical for effective clinical diagnosis and treatment planning. While Computed Tomography (CT) and Magnetic Resonance Imaging (MRI) are commonly used modalities for liver imaging, each offers complementary anatomical and functional information. This study presents an attention-based convolutional neural network (CNN) framework for fusing CT and MRI modalities to enhance liver tumor segmentation and classification. The proposed architecture employs dual-branch CNN encoders to extract modality-specific features, which are fused using spatial and channel attention mechanisms for joint representation learning. A U-Net-inspired decoder reconstructs tumor masks for segmentation, while a fully connected classifier predicts tumor type (benign or malignant).
A synthetic multimodal dataset was generated to simulate real-world CT and MRI feature distributions, incorporating segmentation quality (Dice scores) and class labels. The model achieved Dice scores in the range of 0.75–0.92, indicating strong tumor boundary delineation. For classification, the model obtained a macro-averaged F1-score of 0.47 and an AUC of 0.64, demonstrating the potential of attention-guided fusion even under simulated conditions. Attention heatmaps further validated the model’s spatial focus on tumor-relevant regions. These results suggest that multimodal attention-based fusion significantly improves the diagnostic capabilities of CNNs in liver cancer imaging tasks, with promising implications for future clinical deployment.
Downloads
Metrics
References
Siegel, R. L., Miller, K. D., & Jemal, A., “Cancer statistics, 2023,” CA Cancer J. Clin., vol. 73, no. 1, pp. 17–48, 2023.
Gao, Y., Lim, J., & Teo, E. C., “MRI and CT imaging for liver tumor characterization: A review,” Eur. J. Radiol., vol. 141, pp. 110802, 2021.
Casanova, R., et al., “MR and CT imaging of liver tumors: Comparison and hybrid applications,” J. Magn. Reson. Imaging, vol. 53, no. 4, pp. 1020–1032, 2021.
Zhang, Y., Zhang, Y., & Wang, L., “Multimodal medical image fusion via convolutional neural networks: A survey,” Inf. Fusion, vol. 72, pp. 48–71, 2021.
Chen, M., et al., “Multi-level feature fusion for multimodal medical image segmentation,” IEEE Access, vol. 9, pp. 89230–89241, 2021.
Litjens, G., et al., “A survey on deep learning in medical image analysis,” Med. Image Anal., vol. 42, pp. 60–88, 2017.
Rajpurkar, P., et al., “CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning,” arXiv preprint arXiv:1711.05225, 2017.
Hu, J., Shen, L., & Sun, G., “Squeeze-and-Excitation Networks,” in Proc. CVPR, 2018, pp. 7132–7141.
Woo, S., Park, J., Lee, J. Y., & Kweon, I. S., “CBAM: Convolutional Block Attention Module,” in Proc. ECCV, 2018, pp. 3–19.
Guo, M. H., et al., “Attention mechanisms in computer vision: A survey,” Comput. Vis. Image Underst., vol. 211, pp. 103287, 2021.
Pham, D. L., Xu, C., & Prince, J. L., “Current methods in medical image segmentation,” Annu. Rev. Biomed. Eng., vol. 2, no. 1, pp. 315–337, 2000.
Ronneberger, O., Fischer, P., & Brox, T., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Proc. MICCAI, 2015, pp. 234–241.
Çiçek, O., et al., “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation,” in Proc. MICCAI, 2016, pp. 424–432.
Schlemper, J., et al., “Attention gated networks: Learning to leverage salient regions in medical images,” Med. Image Anal., vol. 53, pp. 197–207, 2019.
Christ, P. F., et al., “Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields,” in Proc. MICCAI, 2016, pp. 415–423.
Zhou, T., et al., “Multimodal medical image fusion via convolutional neural networks,” Comput. Math. Methods Med., vol. 2020, Article ID 8279342, 2020.
Wang, G., et al., “Automatic brain tumor segmentation using cascaded anisotropic convolutional neural networks,” Front. Comput. Neurosci., vol. 12, pp. 55, 2018.
Valindria, V. V., et al., “Multi-modal learning from unpaired images: Application to multi-organ segmentation in CT and MRI,” in Proc. CVPR, 2018, pp. 9252–9260.
Zhao, Y., et al., “Multimodal medical image classification with hybrid fusion and attention mechanism,” IEEE J. Biomed. Health Inform., vol. 25, no. 9, pp. 3626–3636, 2021.
Wang, X., et al., “Non-local Neural Networks,” in Proc. CVPR, 2018, pp. 7794–7803.
Chen, L., et al., “Dual attention network for scene segmentation,” in Proc. CVPR, 2019, pp. 3146–3154.
Oktay, O., et al., “Attention U-Net: Learning where to look for the pancreas,” arXiv preprint arXiv:1804.03999, 2018.
Li, H., et al., “Cross-modal attention network for joint segmentation of brain tumor in PET and MRI,” Neurocomputing, vol. 410, pp. 102–111, 2020.
Gao, F., et al., “Deep learning for the classification of liver tumors based on multi-phase CT images,” BMC Med. Imaging, vol. 21, pp. 1–13, 2021.
Zhang, L., et al., “Automatic liver tumor classification using convolutional neural network with texture and wavelet features,” IEEE Access, vol. 7, pp. 138438–138447, 2019.
Zhang, Y., et al., “Dual-path CNN model for classification of liver tumor using multimodal imaging,” Comput. Biol. Med., vol. 134, pp. 104529, 2021.
Zhou, Y., et al., “Hybrid radiomics and deep learning approach for hepatic lesion classification in multiphase CT,” Eur. Radiol., vol. 31, pp. 5301–5311, 2021..
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
You are free to:
- Share — copy and redistribute the material in any medium or format
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
Terms:
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
