Evaluation of Transformer Models for Summarizing Lengthy Clinical Notes
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V6I4P113Keywords:
Transformer Models, Clinical Notes, Text Summarization, BART, T5, Longformer, MIMIC-III, ROUGE, Natural Language Processing (NLP), Healthcare AIAbstract
Digitization of healthcare records are set to grow at a phenomenal rate, particularly unstructured clinical data (such as long clinical notes). These reports usually contain irrelevant and repetitive data and it is difficult to get important information needed by the healthcare providers in a short span of time. Automatic text summarization is an effective way out since it gives the shortened form of these notes but does not lose any important medical data. The paper will analyze the best transformer-based models such as BART, T5 and Longformer on the task of abstractive summarizations of clinical notes. We source our experiments on MIMIC-III data and we test the quality of the model on ROUGE scores and qualitative evaluation of humans. As we can see, more generic transformer models are adequate to work, but more architectures have proven to be better in longer sequences (like Longformer). Our research indicates the advantages and disadvantages of each model and gives future considerations to the future improvement of clinical natural language processing
Downloads
References
[1] Chaves, C. Kesiku, and B. Garcia-Zapirain, “Automatic text Summarization of Biomedical text data: A Systematic review,” Information, vol. 13, no. 8, p. 393, Aug. 2022, doi: 10.3390/info13080393.
[2] L. Agilandeeswari, A. Dagar, A. Deepthi, and R. Arangasakthivel, “Automatic Text Summarization for Medical Dataset-An Analysis,” in Lecture notes in networks and systems, 2024, pp. 336–352. doi: 10.1007/978-3-031-64813-7_35.
[3] S. Madan, M. Lentzen, J. Brandt, D. Rueckert, M. Hofmann-Apitius, and H. Fröhlich, “Transformer models in biomedicine,” BMC Medical Informatics and Decision Making, vol. 24, no. 1, Jul. 2024, doi: 10.1186/s12911-024-02600-5.
[4] K. Denecke, R. May, and O. Rivera-Romero, “Transformer Models in Healthcare: A survey and thematic analysis of potentials, shortcomings and risks,” Journal of Medical Systems, vol. 48, no. 1, Feb. 2024, doi: 10.1007/s10916-024-02043-5.
[5] R. Mehta, N. Mehta, V. Purohit, I. Saha, and P. Mishra, “Text Summarization for Research Papers using Transformers,” 2022 IEEE 7th International Conference for Convergence in Technology (I2CT), Apr. 2024, doi: 10.1109/i2ct61223.2024.10543503.
[6] J. Whitton and A. Hunter, “Automated tabulation of clinical trial results: A joint entity and relation extraction approach with transformer-based language representations,” Artificial Intelligence in Medicine, vol. 144, p. 102661, Sep. 2023, doi: 10.1016/j.artmed.2023.102661.
[7] Y. Dash, A. Kumar, S. S. Chauhan, A. V. Singh, A. Ray, and A. Abraham, “Advances in Medical text Summarization: Comparative performance analysis of PEGASUS and T5,” 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT), pp. 1–5, Jun. 2024, doi: 10.1109/icccnt61001.2024.10724845.
[8] A. Siebra, M. Kurpicz-Briki, and K. Wac, “Transformers in health: a systematic review on architectures for longitudinal data analysis,” Artificial Intelligence Review, vol. 57, no. 2, Feb. 2024, doi: 10.1007/s10462-023-10677-z.
[9] R. Haruna, A. Obiniyi, M. Abdulkarim, and A. A. Afolorunsho, “Automatic Summarization of Scientific documents using Transformer Architectures: A review,” 2022 5th Information Technology for Education and Development (ITED), pp. 1–6, Nov. 2022, doi: 10.1109/ited56637.2022.10051602.
[10] L. B. Elvas, A. Almeida, and J. C. Ferreira, “Natural language processing in medical text processing: A scoping literature review.,” PubMed, vol. 204, p. 106049, Jul. 2025, doi: 10.1016/j.ijmedinf.2025.106049.
[11] Y. Gao, D. Dligach, T. Miller, M. M. Churpek, O. Uzuner, and M. Afshar, “Progress Note Understanding — Assessment and Plan Reasoning: Overview of the 2022 N2C2 Track 3 shared task,” Journal of Biomedical Informatics, vol. 142, p. 104346, Apr. 2023, doi: 10.1016/j.jbi.2023.104346.
[12] T. Lai, “Interpretable Medical Imagery Diagnosis with Self-Attentive Transformers: A Review of Explainable AI for Health Care,” BioMedInformatics, vol. 4, no. 1, pp. 113–126, Jan. 2024, doi: 10.3390/biomedinformatics4010008.
[13] S. Sharma, S. Srivastava, P. Verma, A. Verma, and S. N. Chaurasia, “A comprehensive analysis of Indian legal documents summarization techniques,” SN Computer Science, vol. 4, no. 5, Aug. 2023, doi: 10.1007/s42979-023-01983-y.
[14] E. Kotei and R. Thirunavukarasu, “A Systematic Review of Transformer-Based Pre-Trained Language Models through Self-Supervised Learning,” Information, vol. 14, no. 3, p. 187, Mar. 2023, doi: 10.3390/info14030187.
[15] Z. A. Nazi and W. Peng, “Large language models in healthcare and medical domain: A review,” Informatics, vol. 11, no. 3, p. 57, Aug. 2024, doi: 10.3390/informatics11030057.
[16] Hands and R. Kavuluru, “A survey of NLP methods for oncology in the past decade with a focus on cancer registry applications,” Artificial Intelligence Review, vol. 58, no. 10, Jul. 2025, doi: 10.1007/s10462-025-11316-5.
