Accepted to ICML 2025 Workshop on Efficient Systems for Foundation Models
Low-Rank Adapters (LoRAs) have transformed the fine-tuning of Large Language Models (LLMs) by enabling parameter-efficient updates. However, their widespread adoption remains limited by the reliance on GPU-based training. In this work, we propose a theoretically grounded approach to LoRA fine-tuning designed specifically for users with limited computational resources, particularly those restricted to standard laptop CPUs. Our method learns a meta-operator that maps any input dataset, represented as a probability distribution, to a set of LoRA weights by leveraging a large bank of pre-trained adapters for the Mistral-7B-Instruct-v0.2 model. Instead of performing new gradient-based updates, our pipeline constructs adapters via lightweight combinations of existing LoRAs directly on CPU.
@article{arabpour2025lora,title={LoRA Fine-Tuning Without GPUs: A CPU-Efficient Meta-Generation Framework for LLMs},author={Arabpour, Reza and Sáez de Ocáriz Borde, Haitz and Kratsios, Anastasis},journal={ICML 2025 Workshop on Efficient Systems for Foundation Models},year={2025},month=jul,eprint={2507.01806},archiveprefix={arXiv},primaryclass={cs.LG},url={https://arxiv.org/abs/2507.01806},}
2024
Under Review
Low-dimensional approximations of the conditional law of Volterra processes: a non-positive curvature approach
Reza Arabpour, John Armstrong, Luca Galimberti, and 2 more authors
Under review at Analysis and Applications, May 2024
Predicting the conditional evolution of Volterra processes with stochastic volatility is a crucial challenge in mathematical finance. While deep neural network models offer promise in approximating the conditional law of such processes, their effectiveness is hindered by the curse of dimensionality caused by the infinite dimensionality and non-smooth nature of these problems. To address this, we propose a two-step solution. Firstly, we develop a stable dimension reduction technique, projecting the law of a reasonably broad class of Volterra process onto a low-dimensional statistical manifold of non-positive sectional curvature. Next, we introduce a sequentially deep learning model tailored to the manifold’s geometry, which we show can approximate the projected conditional law of the Volterra process. Our model leverages an auxiliary hypernetwork to dynamically update its internal parameters, allowing it to encode non-stationary dynamics of the Volterra process, and it can be interpreted as a gating mechanism in a mixture of expert models where each expert is specialized at a specific point in time.
@article{arabpour2024volterra,title={Low-dimensional approximations of the conditional law of Volterra processes: a non-positive curvature approach},author={Arabpour, Reza and Armstrong, John and Galimberti, Luca and Kratsios, Anastasis and Livieri, Giulia},journal={Under review at Analysis and Applications},year={2024},month=may,eprint={2405.20094},archiveprefix={arXiv},primaryclass={math.NA},url={https://arxiv.org/abs/2405.20094},}