Orca 2: Enhancing Reasoning in Smaller Language Models - Technical Details

This story was originally published on HackerNoon at: https://hackernoon.com/orca-2-enhancing-reasoning-in-smaller-language-models-technical-details . Orca 2 enhances small language models' reasoning by teaching diverse strategies for tasks, outperforming models up to 10x larger in complex benchmarks. Check more stories related to programming at: https://hackernoon.com/c/programming . You can also check exclusive content about #language-models , #orca-2 , #reasoning-techniques , #machine-learning , #small-models , #imitation-learning , #ai-benchmarks , #model-training , and more. This story was written by: @textmodels . Learn more about this writer by checking @textmodels's about page, and for more stories, please visit hackernoon.com . The Orca 2 dataset has four main sources:FLAN: Our main source of prompts for synthetic data generation is the FLAN-v2 Collection 33, which consists of five sub-collections. Following Orca 1 42, we consider tasks from only CoT, NiV2, T0, Flan 2021 and Dialogue. Some of the tasks are associated with an associated answer. For the Cautious Reasoning dataset we selected ~602 zero-shot user queries from the split of 1448 high quality tasks out of 1913.