In-Context Learning in Language Models Has Many Moving Parts

A short prompt with a few examples can change how a language model answers a question. This method, called in-context learning (ICL), lets a model infer a task from examples in its input without updating its parameters. A survey by Qingxiu Dong and colleagues shows that the apparent simplicity hides choices that affect the result.

Examples are part of the method

The survey reviews training approaches, prompt design, scoring methods, applications and research on how ICL works. Prompt design includes which examples to select, how to format them and what order to place them in. A prompt can act as a lightweight interface for task-specific knowledge, but that interface is sensitive to its contents (Dong et al., 2024, pp. 1107–1118).

The authors also report their own comparison of three unsupervised demonstration-selection methods. They tested four models—GPT-2, GPT-J, Qwen2 and Llama 3—on five datasets covering sentiment, commonsense questions, natural-language inference and news classification. The method that performed best depended on the model: “mdl” led on GPT-2, while “topk” led on the other three models (Table 2, p. 1111). That result does not identify one universally best recipe. It shows that example selection and model choice interact.

Faster scoring can cost more

A second experiment compared scoring functions used to choose among candidate answers. Direct scoring was fastest. On average, perplexity scoring took 2.61 times as long and channel scoring 2.90 times as long in the paper’s setup, which used eight examples in each prompt (Table 4, p. 1125). The survey also compares how stable results were when demonstration examples changed across five random seeds (Table 5, p. 1125).

Those comparisons matter because prompt workflows are often judged only by answer quality. A selection or scoring method may raise performance in one task while adding inference time or sensitivity to example choice. For a team using ICL at scale, those costs affect latency and repeatability. The figures come from a specific four-model, five-dataset experiment, not a service-level estimate.

A broad field with open gaps

Beyond the experiments, the review covers applications in data engineering, model augmentation and knowledge updating. It discusses possible explanations for ICL, including whether some behavior resembles learning algorithms. The authors note that studies on text do not automatically transfer to image or speech tasks, and that many analytic studies use smaller models and simpler tasks than those deployed now (pp. 1123–1127).

The survey has limits of its own. The literature is large, so relevant work may have been missed. Some experiments in the reviewed papers used older models. Their added benchmark is evidence about model dependence, but covers only part of the field and was run on one NVIDIA A100 (Appendix B, p. 1125).

Possible implications for practice

ICL turns prompt construction into part of a system’s operating behavior. Teams may need to consider example selection, order, format, model version and scoring time together. A template that works with one model or dataset may not carry over to another. This suggests prompt changes deserve the same kind of version tracking as other system settings.

The practical question is less “does few-shot prompting work?” than “which examples make this task reliable, for this model, under this cost constraint?” The survey’s evidence points to evaluation across prompts and tasks, especially when user-facing behavior depends on a handful of demonstrations. Results remain task-specific, and the prompt can shape both capability and variation.

ICL also puts pressure on the boundary between product configuration and model capability. Two teams using the same model can create meaningfully different behavior through example choice or ordering. A prompt library therefore carries operational knowledge that can be lost when a model is updated or a workflow is copied. The researchers do not test such workplace handoffs, but their model-dependent results suggest why configuration provenance may be useful when a system’s answers change.

Bibliography & sources

1. Qingxiu Dong; Lei Li; Damai Dai; Ce Zheng; Jingyuan Ma; Rui Li; Heming Xia; Jingjing Xu; Zhiyong Wu; Baobao Chang (常宝宝); Xu Sun; Lei Li; Zhifang Sui. 2024. “A Survey on In-context Learning.” In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 1107–1128. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.emnlp-main.64.