Automated identification of snake, scorpion bites and other skin lesions using a new multimodal vision transformer (ViT) / Hibatallah Djokhdem

Public ISBD Titre : Automated identification of snake, scorpion bites and other skin lesions using a new multimodal vision transformer (ViT) Type de document : document multimédia Auteurs : Hibatallah Djokhdem, Auteur ; Wafaa Asmahan Taifouri, Auteur ; Nardjes Hamini, Directeur de thèse Editeur : Laghouat : Université Amar Telidji - Département d'informatique Année de publication : 2026 Importance : 37 p. Accompagnement : 1 disque optique numérique (CD-ROM) Note générale : Option : Artificial intelligence and data science Langues : Anglais (eng) Mots-clés : Multimodal learning  Vision transformer (ViT)  BERT  Cross-attention fusion  Skin lesions and bite injuries Résumé : This work presents a multimodal framework for the automated classification of injuries resulting from bites and stings (such as snakes, scorpions, spiders, and insects) and distinguishing them from nonbite skin lesions. The proposed approach integrates medical images using a Vision Transformer (ViT) and descriptive clinical text using a BERT model, driven by a bidirectional crossattention fusion mechanism. A custom dataset comprising 1,526 images and textual descriptions was compiled via a web interface following a Humanintheloop approach, and class imbalance was addressed using a specialized data augmentation strategy. Experimental results demonstrate that the proposed multimodal network achieves a superior test accuracy of 90.52%, significantly outperforming the unimodal baselines, namely ViTonly (73.0%) and BERTonly (67.8%). Furthermore, model interpretability was rigorously analyzed using visual and textual attention heatmaps. note de thèses : Mémoire de master en informatique Automated identification of snake, scorpion bites and other skin lesions using a new multimodal vision transformer (ViT) [document multimédia] / Hibatallah Djokhdem, Auteur ; Wafaa Asmahan Taifouri, Auteur ; Nardjes Hamini, Directeur de thèse . - Laghouat : Université Amar Telidji - Département d'informatique, 2026 . - 37 p. + 1 disque optique numérique (CD-ROM). Option : Artificial intelligence and data science Langues : Anglais (eng) Mots-clés : Multimodal learning  Vision transformer (ViT)  BERT  Cross-attention fusion  Skin lesions and bite injuries Résumé : This work presents a multimodal framework for the automated classification of injuries resulting from bites and stings (such as snakes, scorpions, spiders, and insects) and distinguishing them from nonbite skin lesions. The proposed approach integrates medical images using a Vision Transformer (ViT) and descriptive clinical text using a BERT model, driven by a bidirectional crossattention fusion mechanism. A custom dataset comprising 1,526 images and textual descriptions was compiled via a web interface following a Humanintheloop approach, and class imbalance was addressed using a specialized data augmentation strategy. Experimental results demonstrate that the proposed multimodal network achieves a superior test accuracy of 90.52%, significantly outperforming the unimodal baselines, namely ViTonly (73.0%) and BERTonly (67.8%). Furthermore, model interpretability was rigorously analyzed using visual and textual attention heatmaps. note de thèses : Mémoire de master en informatique

---

1 (1 - 1 / 1) Par page : 25 50 100 200