BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Models
Yibin Wang*, Haizhou Shi*, Ligong Han, Dimitris Metaxas, Hao Wang
Arxiv Pre-print, 2024.
TLDR: We propose a Bayesian Low-Rank Adaptation framework for LLMs that continuously and jointly adjusts both the mean and covariance of LLM parameters throughout the whole fine-tuning process, which effectively alleviates the problem of LLMs' overconfidence.
[paper] [code]
Continual Learning of Large Language Models: A Comprehensive Survey
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Hao Wang
Arxiv Pre-print, 2024.
TLDR: In this paper, we provide a comprehensive review of the past, present, and future of Continual Learning and its application to LLMs, enabling them to adapt to ever-changing envorinments in an efficient and reliable way.
[paper] [code]
MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning
Yun Zhu, Haizhou Shi, Zhenshuo Zhang, Siliang Tang
Arxiv preprint, 2023.
TLDR: We propose a shared recipe, MARIO, for improving out-of-distribution (OOD) generalization for graph unsupervised learning, by introducing (i) the principle of Information Bottleneck and (ii) the principle of Invariant Features.
[paper] [code]
A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm
Haizhou Shi, Hao Wang
Thirty-Seventh Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
TLDR: A novel theoretical framework, UDIL, is proposed for domain incremental learning, unifying multiple existing methods. The key insight of our study is that UDIL allows for adaptive coefficients during training, yielding a tighter generalization bound compared to its counterparts.
[paper] [code] [slides] [talk]
Structure-Aware Group Discrimination with Adaptive-View Graph Encoder: A Fast Graph Contrastive Learning Framework
Zhenshuo Zhang, Yun Zhu, Haizhou Shi, Siliang Tang
Twenty-Sixth Annual European Conference on Artificial Ingelligence (ECAI), 2023.
TLDR: A fast graph contrastive learning framework is proposed, which consists of two components: (i) adaptive view graph encoder (AVGE) and (ii) structure-aware group discrimination loss (SAGD). The proposed method manages to bring down the training and inference cost by a significant margin compared to its counterparts.
[paper] [code]
On the Efficacy of Small Self-Supervised Contrastive Models without Distillation Signals
Haizhou Shi, Youcai Zhang, Siliang Tang, Wenjie Zhu, Yaqian Li, Yandong Guo, Yueting Zhuang
Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI), 2022.
TLDR: This paper studies the issue of training self-supervised small models without distillation signals. Key observations about the learned representation distribution are made and several empirical measures addressing the problem are evaluated.
[paper] [code] [slides]
Towards Communication-efficient and Privacy-preserving Federated Representation Learning
Haizhou Shi, Youcai Zhang, Zijin Shen, Siliang Tang, Yaqian Li, Yandong Guo, Yueting Zhuang
International Workshop on Trustable, Verifiable and Auditable Federated Learning in Conjunction with AAAI (FL-AAAI), 2022.
TLDR: Based on the observed robustness against data i.i.d.-ness of contrastive methods, a similarity-based distillation method is used to design a novel federated representation learning framework, FLESD, which is verified to be more communication-efficient and privacy-preserving.
[paper] [code] [talk]
Revisiting Catastrophic Forgetting in Class Incremental Learning
Zixuan Ni*, Haizhou Shi*, Siliang Tang, Longhui Wei, Qi Tian, Yueting Zhuang
Arxiv preprint, 2021.
TLDR: Three causes of catastrophic forgetting in class incremental learning are analysed, based on which a novel framework, C4IL, is proposed to reduce the representational overlaps among different tasks.
[paper] [code]
CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction
Tao Chen, Haizhou Shi, Siliang Tang, Zhigang Chen, Fei Wu, Yueting Zhuang
Fifty-Ninth Annual Meeting of the Association of Computational Linguistics (ACL), 2021.
TLDR: This paper goes beyond typical multi-instance learning (MIL) framework and propose a novel contrastive instance learning (CIL) framework for distantly supervised relation extraction. Specifically, we regard the initial MIL as the relational triple encoder and constraint positive pairs against negative pairs for each instance.
[paper] [code]
Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
Haizhou Shi*, Dongliang Luo*, Siliang Tang, Jian Wang, Yueting Zhuang
Arxiv preprint, 2020.
TLDR: In this paper, we suggest understanding BYOL from the view of our proposed interpretable self-supervised learning framework, Run Away From your Teacher (RAFT). It optimizes two objectives at the same time: (i) aligning two views of the same data and (ii) running away from the model’s Mean Teacher (MT). Theoretical justification is also provided for equating the proposed RAFT and BYOL under certain conditions.
[paper] [code]
Informative Visual Storytelling with Cross-modal Rules
Jiacheng Li, Haizhou Shi, Siliang Tang, Fei Wu, Yueting Zhuang
Twenty-Seventh ACM International Conference on Multimedia (ACM MM), 2019.
TLDR: To solve problem of generating too-general descriptions in the field of visual storytelling, we propose a method to mine the cross-modal rules to help the model infer these informative concepts given certain visual input.
[paper] [code]