Editor: Ma Qinghe
Images: Qin Mingli
Layout: Su Yayun
Press release portal: https://news.zhenrobot.com
Google Research Releases Cappy: Enhancing Multi-Task Large Language Models with a 360-Million Parameter Model
Google Research recently introduced a new method called Cappy. It is a lightweight scoring model designed for multi-task large language models (LLMs), aimed at improving performance and adaptation efficiency in multi-task scenarios without directly fine-tuning the large model's parameters. Experiments show that Cappy, with only 360 million parameters, outperforms OPT-175B and OPT-IML-30B on multiple language understanding tasks and achieves accuracy comparable to existing mainstream multi-task models like T0-11B and OPT-IML-175B [source].
For AI developers, the key takeaway from this work is clear: beyond simply building "larger models," the collaborative mechanism of small models focused on "scoring, filtering, and reranking" is becoming an important path for enhancing large model capabilities and reducing deployment costs.
Multi-Task Large Language Models Face Dual Challenges in Performance and Cost
In recent years, LLMs have driven natural language processing into a new paradigm centered on "instruction following." Multi-task models like T0, FLAN, and OPT-IML typically convert different tasks into "instruction-answer" formatted data and then train the model to generate answers based on input instructions. It has been proven that these methods can not only handle tasks seen during training but also generalize to new tasks to some extent.
However, this paradigm also brings significant issues. Since the model needs to understand and complete a large number of different types of tasks, the difficulty of unified modeling solely through instructions is high. Therefore, multi-task LLMs often have billions to hundreds of billions of parameters, such as FLAN-11B, T0-11B, OPT-IML-175B, and even the larger FLAN-PaLM-540B. This size means that both training and inference require massive computing power and memory support, leading to high deployment costs and limited adaptation efficiency.
Furthermore, some of the most capable multi-task models are not open-source, making it difficult for developers to directly modify them. In practical applications, a single model struggles to consistently cover all complex tasks under zero-shot conditions, especially personalized tasks or those difficult to describe with concise instructions. Meanwhile, downstream supervised data is often insufficient, and training a standalone task model may not fully leverage the existing knowledge of large models.
Cappy's Core Concept: Teaching a Small Model to "Score Answers"
Cappy is not a new generative large model; rather, it is a scoring model. Its input is an "instruction-answer" pair, and its output is a score between 0 and 1, used to estimate the correctness of the answer relative to the instruction.
The key to this design is that Cappy does not directly generate answers but learns to determine "which answer is better." From an engineering perspective, this is equivalent to adding a lightweight quality evaluation layer outside the large model, helping the system select better results at a lower cost or adapt more efficiently in downstream tasks.
Training Data Construction: Learning Correctness Judgments from Weakly Supervised Regression Data
For data construction, Google Research used multi-task datasets from the same source as T0, comprising 39 diverse datasets from PromptSource, covering various task types like question answering, sentiment analysis, and summarization. Each dataset was converted into paired instructions and standard answers using templates.
Since Cappy needs to learn a continuous score for "answer correctness," the research team further constructed regression data with correctness annotations. Specifically, for each generative task sample, multiple answers were first sampled using an existing multi-task LLM based on the same instruction. These answers were then compared with the standard answer, and Rouge-L was used to calculate the similarity between them, serving as a weak supervision signal between 0 and 1.
Ultimately, the team obtained approximately 160 million regression training samples with score annotations and performed continued pre-training on the RoBERTa model to obtain the final Cappy. The entire training process ran on Google's TPU-v4 using the lightweight distributed training tool RedCoast.
Compared to Common Adaptation Methods, Cappy Requires Less Memory and is Better Suited for Closed-Source Models
In terms of adaptation, Cappy addresses a long-standing industry problem: how to improve task performance using downstream supervised data without extensively modifying the LLM.
Existing parameter-efficient fine-tuning methods, such as prompt tuning and adapters, reduce storage costs but typically still require backpropagation through the large language model's parameter path during training, leading to non-trivial memory overhead. Another class of in-context learning methods guides the model by concatenating a few examples in the input, but due to context length limitations, the number of training examples that can be utilized is limited.
Cappy's advantage lies in the fact that it does not require backpropagation on the LLM's parameters nor access to the underlying model weights. Therefore, it can be used with closed-source multi-task models that only provide services via Web APIs. Compared to in-context learning, Cappy is not constrained by the model's maximum input length and can theoretically integrate an unlimited number of downstream training samples.
The research team also noted that Cappy can be used independently or combined with fine-tuning and in-context learning to further enhance overall performance.
Evaluation Results: A 360-Million Parameter Model Matches Mainstream Multi-Task Large Models
On 11 language understanding classification tasks provided by PromptSource that were not included in training, Cappy demonstrated strong parameter efficiency. Experimental results show that Cappy, with 360 million parameters, outperformed OPT-175B and OPT-IML-30B and achieved accuracy comparable to the current best-performing multi-task LLMs, T0-11B and OPT-IML-175B.
This result indicates that the pre-training strategy based on scoring may possess unique value. Unlike traditional multi-task LLMs that primarily rely on teacher-forcing training and only learn standard answers, Cappy explicitly distinguishes between high-quality and low-quality answers during training, introducing a stronger contrastive signal. This ability to "judge good from bad" becomes a crucial reason for its competitive performance at a much smaller parameter scale.
Notably, the paper also compared Cappy with pre-trained RLHF reward models, and the results showed its overall performance is at the forefront of existing multi-task models.
On Complex BIG-Bench Tasks, Cappy Significantly Improves FLAN-T5's Performance
Beyond standard language understanding tasks, the research team also tested Cappy's adaptation capabilities on complex tasks. The experiment selected 45 generative tasks from BIG-Bench, which have no preset options and are generally considered highly difficult tasks beyond the basic capabilities of many LLMs. The evaluation metric used Rouge-L, and the average score across all 45 test sets was calculated.
In this experiment, different scales of FLAN-T5 were used as the backbone LLM, while the parameters of the base FLAN-T5 remained frozen. Results showed that Cappy consistently improved FLAN-T5's performance across all model sizes and significantly outperformed baseline methods relying on the model's own scoring. The "own scoring" mentioned here refers to using the LLM's cross-entropy to select candidate answers.
This means that Cappy is not only effective on standard benchmarks but also provides strong enhancement for complex generation tasks. For developers focusing on Agents, complex workflows, and task decomposition, this is particularly noteworthy, as many real-world problems are closer to open-ended complex tasks like BIG-Bench rather than standard classification questions.
Implications for Developers: Small Scoring Models Could Become New Infrastructure for Large Model Deployment
The significance of Cappy goes beyond "achieving more with less"; it provides a productization approach different from the traditional fine-tuning route. In the past, the industry focused more on training larger generative models or low-cost fine-tuning of the large models themselves. Cappy demonstrates an alternative path: providing selection, filtering, and task adaptation capabilities for large models through an independent scorer.
For startups and application teams, this method offers several practical benefits. First, it reduces dependence on memory and training resources, lowering the adaptation barrier. Second, it can serve closed-source API models, broadening the scope of application. Third, unconstrained by context window limits, it is better suited for absorbing large volumes of downstream samples. Fourth, as an external module, it is easier to integrate with existing generation, retrieval, and ranking pipelines, entering RAG, Agent, and tool invocation systems.
From an industry competition perspective, Cappy also shows that improving multi-task LLM capabilities does not necessarily rely solely on scaling up parameters. Systematic optimization around reranking, reward modeling, response selection, and weakly supervised data construction is becoming an essential part of enhancing model capabilities.
Conclusion
Cappy, introduced by Google Research, is a scoring-based solution for enhancing multi-task LLMs. By learning correctness scores for "instruction-answer" pairs, it improves the performance of multi-task models on downstream tasks without requiring backpropagation on the large model's parameters or access to closed-source model weights.
Experimental results show that Cappy, with 360 million parameters, can match or even surpass much larger models on multiple tasks; on 45 complex generative tasks from BIG-Bench, it also brings significant gains to the frozen-parameter FLAN-T5. For the broader AI community, the most noteworthy aspect of Cappy is that scoring models are gradually evolving from auxiliary components into a critical capability layer within large model application architectures.
As large model deployment enters a new phase emphasizing cost, efficiency, and controllability, small model collaboration solutions like Cappy may become a key direction for the next round of product innovation and engineering optimization.
References