(Research or White) Papers in 2026: Goal: 52 Finished: 19 In Progress: 8

In-context Learning

https://arxiv.org/pdf/2506.13109

• ICL prompt with trajectory and snippet demo is best performance; trajectory traces are expensive though, but yield most improvements
• automatic trajectory annotation: use strong LLM executor to generate traces + solver function to generate training data; keep low temperature to generate deterministic outputs
• ReAct: state → thought → action → repeat
• PnE: plan → split into subtasks → execute each subtask
• SETBSR: greedily select the k set of BSR embeddings which most match the input embedding
• ICL is like RAG, but for trajectory demonstration traces

Context Rot

https://research.trychroma.com/context-rot

• LLM context windows are longer and still achieve good needle-in-the-haystack performance, but NITH only captures direct lexical matching; other benchmarks show degradation but its hard to isolate if its from task complexity or longer context; NITH contains irrelevant content meant to fill up context, distractors which are related to the topic, but don’t answer the question
• LLM context windows are longer and still achieve good needle-in-the-haystack performance, but NITH only captures direct lexical matching; other benchmarks show degradation but its hard to isolate if its from task complexity or longer context; NITH contains irrelevant content meant to fill up context, distractors which are related to the topic, but don’t answer the question
• evaluate model with GPT-4.1 judge; sets model temperature=0; does this hurt performance?
• needle-question similarity: aligns needle to be topically similar to the rest of the text to be more realistic; use cosine similarity between needle and question; less similarity results in worse performance as context grows; uses vector DB to verify that answer to question doesn’t already exist in the data, which feels flaky
• impact of distractors: how many distractors are in the haystack; performance tends to degrade with more needles; non-uniform performance with more variance as input length increases (claude says idk more, while GPT hallucinates more); did not test cases where distractors had higher cosine similarity, which could be interesting
• needle-haystack similarity: use cosine similarity between chunked haystack and needle because intuitively higher similarity means it should be more difficult to find; non-uniform, semantically different needles can have same results as semantically similar needles
• haystack structure: keep original logical ordering of ideas or randomly shuffle them so it’s incoherent ideas; contrary to intuition, random shuffling improves performance, showing logical flow doesn’t affect retrieval
• LongMemEval: give agent context, and have it reason over it to provide a response; focused on knowledge-update, temporal reasoning, multi-session conversations; tested with focused prompts and irrelevant longer prompts; results showed that model performed way better with focused prompts; thinking mode helped, but there was still a delta; claude did worse with ambiguity so often just returned idk if it got confused
• repeated words: get model to do trivial task of repeating a string with a unique word at an index ex. apple, apple, apples, apple; shows performance degrades as input length increases, makes sense because it’s hard for the model to keep track of indices; declines responding more often