publications

2024

1. 

   Harnessing Large Language Models for Training-free Video Anomaly Detection

   Luca Zanella, Willi Menapace, Massimiliano Mancini , and 2 more authors

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024

   Abs Bib HTML PDF Code

   Video anomaly detection (VAD) aims to temporally locate abnormal events in a video. Existing works mostly rely on training deep models to learn the distribution of normality with either video-level supervision, one-class supervision, or in an unsupervised setting. Training-based methods are prone to be domain-specific, thus being costly for practi- cal deployment as any domain change will involve data collection and model training. In this paper, we radically depart from previous efforts and propose LAnguage-based VAD (LAVAD), a method tackling VAD in a novel, training- free paradigm, exploiting the capabilities of pre-trained large language models (LLMs) and existing vision-language models (VLMs). We leverage VLM-based captioning models to generate textual descriptions for each frame of any test video. With the textual scene description, we then devise a prompting mechanism to unlock the capability of LLMs in terms of temporal aggregation and anomaly score estimation, turning LLMs into an effective video anomaly detector. We further leverage modality-aligned VLMs and propose effec- tive techniques based on cross-modal similarity for cleaning noisy captions and refining the LLM-based anomaly scores. We evaluate LAVAD on two large datasets featuring real- world surveillance scenarios (UCF-Crime and XD-Violence), showing that it outperforms both unsupervised and one-class methods without requiring any training or data collection.

   @inproceedings{zanella2024harnessing,
     title = {Harnessing Large Language Models for Training-free Video Anomaly Detection},
     author = {Zanella, Luca and Menapace, Willi and Mancini, Massimiliano and Wang, Yiming and Ricci, Elisa},
     booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
     pages = {18527--18536},
     year = {2024},
   }

2023

1. 

   Delving into clip latent space for video anomaly recognition

   Luca Zanella, Benedetta Liberatori, Willi Menapace , and 3 more authors

   arXiv preprint arXiv:2310.02835, 2023

   Abs Bib HTML PDF Code

   We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, the first to combine Large Language and Vision (LLV) models, such as CLIP, with multiple instance learning for joint video anomaly detection and classification. Our approach specifically involves ma- nipulating the latent CLIP feature space to identify the normal event subspace, which in turn allows us to effectively learn text-driven directions for abnormal events. When anomalous frames are pro- jected onto these directions, they exhibit a large feature magnitude if they belong to a particular class. We also introduce a computationally efficient Transformer architecture to model short- and long-term temporal dependencies between frames, ultimately producing the final anomaly score and class pre- diction probabilities. We compare AnomalyCLIP against state-of-the-art methods considering three major anomaly detection benchmarks, i.e. ShanghaiTech, UCF-Crime, and XD-Violence, and empirically show that it outperforms baselines in recognising video anomalies.

   @article{zanella2023delving,
     title = {Delving into clip latent space for video anomaly recognition},
     author = {Zanella, Luca and Liberatori, Benedetta and Menapace, Willi and Poiesi, Fabio and Wang, Yiming and Ricci, Elisa},
     journal = {arXiv preprint arXiv:2310.02835},
     year = {2023},
   }

2. 

   Confmix: Unsupervised domain adaptation for object detection via confidence-based mixing

   Giulio Mattolin, Luca Zanella, Elisa Ricci , and 1 more author

   In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2023

   Abs Bib PDF Code

   Unsupervised Domain Adaptation (UDA) for object detection aims to adapt a model trained on a source domain to detect instances from a new target domain for which annotations are not available. Different from traditional approaches, we propose ConfMix, the first method that introduces a sample mixing strategy based on region-level detection confidence for adaptive object detector learning. We mix the local region of the target sample that corresponds to the most confident pseudo detections with a source image, and apply an additional consistency loss term to gradually adapt towards the target data distribution. In order to robustly define a confidence score for a region, we exploit the confidence score per pseudo detection that accounts for both the detector-dependent confidence and the bounding box uncertainty. Moreover, we propose a novel pseudo labelling scheme that progressively filters the pseudo target detections using the confidence metric that varies from a loose to strict manner along the training. We perform extensive experiments with three datasets, achieving state-of-the-art performance in two of them and approaching the supervised target model performance in the other.

   @inproceedings{mattolin2023confmix,
     title = {Confmix: Unsupervised domain adaptation for object detection via confidence-based mixing},
     author = {Mattolin, Giulio and Zanella, Luca and Ricci, Elisa and Wang, Yiming},
     booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
     pages = {423--433},
     year = {2023},
   }

2022

1. 

   Graph-based generative face anonymisation with pose preservation

   Nicola Dall’Asen, Yiming Wang, Hao Tang , and 2 more authors

   In International Conference on Image Analysis and Processing , 2022

   Abs Bib PDF Code

   We propose AnonyGAN, a GAN-based solution for face anonymisation which replaces the visual information corresponding to a source identity with a condition identity provided as any single image. With the goal to maintain the geometric attributes of the source face, i.e., the facial pose and expression, and to promote more natural face generation, we propose to exploit a Bipartite Graph to explicitly model the relations between the facial landmarks of the source identity and the ones of the condition identity through a deep model. We further propose a landmark attention model to relax the manual selection of facial landmarks, allowing the network to weight the landmarks for the best visual naturalness and pose preservation. Finally, to facilitate the appearance learning, we propose a hybrid training strategy to address the challenge caused by the lack of direct pixel-level supervision. We evaluate our method and its variants on two public datasets, CelebA and LFW, in terms of visual naturalness, facial pose preservation and of its impacts on face detection and re-identification. We prove that AnonyGAN significantly outperforms the state-of-the-art methods in terms of visual naturalness, face detection and pose preservation.

   @inproceedings{dall2022graph,
     title = {Graph-based generative face anonymisation with pose preservation},
     author = {Dall’Asen, Nicola and Wang, Yiming and Tang, Hao and Zanella, Luca and Ricci, Elisa},
     booktitle = {International Conference on Image Analysis and Processing},
     pages = {503--515},
     year = {2022},
     organization = {Springer},
   }