T-REX AI Model Documentation
Overview
T-REX is an advanced AI model designed to address significant challenges in multitask learning, particularly those associated with the Mixture-of-Experts (MoE) paradigm. By leveraging a novel approach that incorporates ultra-lightweight rank-1 experts and semantic-aware routing mechanisms, T-REX enhances computational efficiency, optimizes resource usage, and improves adaptability in multi-task learning scenarios.
Architecture
T-REX builds on established frameworks such as Low-rank Adapter (LoRA) and Mixture of Experts (MoE). Its architecture features:
- Rank-1 Experts: These experts maximize parameter efficiency and enhance flexibility within the MoE framework by decoupling row and column subspaces, allowing for a more expressive model with fewer parameters.
- Mix-and-Match Mechanism: This mechanism enables the combination of multiple rank-1 vectors into LoRA experts, effectively expanding the model's capacity without a proportional increase in parameter overhead.
- Semantic-Aware Router: This component assigns tasks to experts based on the semantic similarity between input embeddings and predefined clusters, improving task allocation and model performance.
Goals
T-REX aims to:
- Mitigate parameter and computational overheads in multitask finetuning.
- Address unfair task allocation in traditional MoE systems.
- Enhance scalability and adaptability in multi-task learning environments.
- Improve the correlation between expert routing and specific tasks, thereby optimizing resource usage.
Dataset Info
T-REX is designed to work with multitask datasets, particularly across 14 distinct datasets. It uses semantic clustering to group embeddings, which enhances its ability to learn from diverse data distributions.
Outputs
The outputs of T-REX include:
- Improved mean accuracy, achieving up to 1.78% gain over traditional methods.
- Reduction in trainable parameters by approximately 30%-40%.
- Enhanced performance across various benchmarks, including significant improvements in complex reasoning tasks.
Evaluation
T-REX has been evaluated using several base models, including LLaMA-2, Mistral 7B, and others. It has demonstrated:
- Consistent performance improvements across both in-distribution (ID) and out-of-distribution (OOD) scenarios.
- Notable accuracy gains on specific benchmarks, such as a 7.54% improvement on WSC performance with Mistral 7B.
Techniques and Modules
T-REX incorporates several key techniques and modules:
- Mix-and-Match (MaM): Enhances performance while reducing parameter overhead.
- Semantic-Aware Routing: Improves expert allocation based on semantic clusters, leading to better generalization.
- MoE Module: Enhances transformer blocks for multitask learning, targeting query, key, value, and feed-forward network components.
Practicalities
The model operates with specific hyperparameters:
- Batch size: 64
- Learning rate: 5e-5
- Epochs: 3
Compute and Systems
T-REX requires substantial computational resources, specifically NVIDIA A100 GPUs. It offers significant reductions in computational overhead compared to other methods, with only 39.9M additional parameters and 5.13G FLOPs.
Limitations and Open Questions
While T-REX shows strong performance, there are areas for improvement, particularly regarding accuracy drops in certain scenarios, such as with LLaMA-2 and Gemma on OpenCompass 2.
Conclusion
T-REX represents a significant advancement in the field of multitask learning, offering a robust solution to the limitations of traditional MoE approaches. Its innovative architecture and techniques enable enhanced performance and efficiency, making it a valuable tool for complex reasoning tasks and diverse data environments.