Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models

doi:10.48550/arXiv.2501.14406

Paper

Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models

Published Jan 24, 2025 · Fei Wu, Jia Hu, Geyong Min

ArXiv

0

Citations

0

Influential Citations

PDF

Abstract

Pre-trained Language Models (PLMs) have demonstrated their superiority and versatility in modern Natural Language Processing (NLP), effectively adapting to various downstream tasks through further fine-tuning. Federated Parameter-Efficient Fine-Tuning (FedPEFT) has emerged as a promising solution to address privacy and efficiency challenges in distributed training for PLMs on resource-constrained local devices. However, our measurements reveal two key limitations of FedPEFT: heterogeneous data across devices leads to significant performance degradation, and a fixed parameter configuration results in communication inefficiency. To overcome these limitations, we propose FedARA, a novel Adaptive Rank Allocation framework for federated parameter-efficient fine-tuning of language models. Specifically, FedARA employs truncated Singular Value Decomposition (SVD) adaptation to enhance similar feature representation across clients, significantly mitigating the adverse effects of data heterogeneity. Subsequently, it utilizes dynamic rank allocation to progressively identify critical ranks, effectively improving communication efficiency. Lastly, it leverages rank-based module pruning to automatically remove inactive modules, steadily reducing local computational cost and memory usage in each federated learning round. Extensive experiments show that FedARA consistently outperforms baselines by an average of 6.95% to 8.49% across various datasets and models under heterogeneous data while significantly improving communication efficiency by 2.40$ \times$. Moreover, experiments on various edge devices demonstrate substantial decreases in total training time and energy consumption by up to 48.90% and 46.95%, respectively.

Preprint

Study Snapshot

FedARA improves communication efficiency and performance in language model fine-tuning, reducing training time and energy consumption by up to 48.90% and 46.95%, respectively.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Adaptive Rank Allocation for Federated Parameter-Efficient Fine-Tuning of Language Models

References

Citations