Muhammad Uzair Khattak - Learning to Prompt with Text Only Supervision for Vision-Language

Speaker: Muhammad Uzair KhattakAbout Speaker: I am Muhammad Uzair, recently graduated from MBZUAI with a master's degree in Computer Vision. During my master's, I was fortunate to work under the supervision of Dr. Salman Khan and Dr. Fahad Khan at the IVAL lab. I am also grateful to have been supervised and mentored by Dr. Muzammal Naseer. My research focus is on adapting foundational multi-modal models for vision tasks, including image recognition, object detection, and video action recognition. The goal is to guide these foundational models for downstream tasks with limited data (few-/zero-shot) while maintaining their pre-trained generalization for novel tasks. About the talk:ProText: Prompt Learning with Text Only Supervision, allows one to finetune CLIP by leveraging contextual knowledge derived from Large Language Models (LLMs) without relying on visual samples. ProText exhibits strong transferability towards unseen datasets and classes and effectively overcomes the transferability limitations of LLM-based Prompt Ensembling methods.Through text-only training, ProText improves over previous prompt ensembling and image-supervised methods in challenging cross-dataset transfer settings.We have open-sourced our checkpoints and source code. ProText offers a drop-in replacement for CLIP’s text transformer. We look forward to the impact of ProText on applications beyond what we tested in the manuscript.abs: https://arxiv.org/abs/2401.02418 This session is brought to you by the Cohere For AI Open Science Community - a space where ML researchers, engineers, linguists, social scientists, and lifelong learners connect and collaborate with each other. Thank you to our Community Leads for organizing and hosting this event. If you’re interested in sharing your work, we welcome you to join us! Simply fill out the form at https://forms.gle/ALND9i6KouEEpCnz6 to express your interest in becoming a speaker. Join the Cohere For AI Open Science Community to see a full list