back

Learning Next Action Predictors from Human-Computer Interaction (10/16/26) arXiv

Today's agents excel at executing well-specified tasks, but have limited contextual reasoning about a user's future behavior. While recommendation systems can predict user preferences within bounded settings (such as social media feeds), reasoning about open-ended human workflows remains an open problem.

Here, Shaikh et al. propose next-action prediction (NAP) as a learning objective for more proactive, personalized agents. Given a user's computer-use history and recent actions, the model predicts the user's most likely next step.

The authors introduce two systems:

Data: NAPsack

NAPsack is a passive data collection and annotation pipeline that converts multimodal computer use recordings into natural-language action descriptions. It records screenshots around user interactions (I/O mouse clicks/movements, scroll, and keyboard) where nearby interactions are grouped into bursts. The screenshots captured before and after bursts (after chunking to maintain VLM context lengths) are fed in temporal order to a VLM, which generates natural language action captions.

The authors compare NAPsack against:

  1. Naively captioning the 10-minute recording with a single prompt.
  2. + split: chunking recordings and captioning separately.
  3. + compress: retaining only interaction-relevant screenshots before chunking.
  4. + input/output: augmenting the VLM with keyboard and mouse events.

Each chunk of predicted labels ℰ^t+1:t+h={e^t+1,…,e^t+h} is compared with the corresponding ground-truth events ℰt+1:t+h*={et+1*,…,et+h*}. Gemini 3.0 Flash as a judge computes a similarity score sim(ℰ^t+1:t+h,ℰt+1:t+h*)∈[0,1].

Results:

Annotation methodLLM-judge score
Naive: caption entire recording0.48
+ Split into chunks0.57
+ Event-driven compression0.60
+ I/O event information*0.70

These improvements are validated with small-scale human comparisons as well as the LLM judge (claimed to be more useful than embedding or lexical metrics).

The training data for LongNAP comes from the Screenomics dataset, a collection of smartphone activity recorded in 2021. This dataset contains screenshots but no I/O events, so the actions used to train LongNAP do not seem to benefit from NAPsack's strongest annotation.

Model: LongNAP

LongNAP is a VLM trained with RL to predict a user’s future actions, using retrieved memories for in-context learning. It is split into two phases: first, reasoning over recent actions to retrieve past reasoning traces; then, using those traces alongside recent screenshots to refine its reasoning and predict future actions.

Phase 1: reason to retrieve

Given recent observations ℰt−k:t, the model reasons about possible next steps, generating a reasoning trace zretrieve∼pθ(·∣ℰt−k:t). Using this reasoning as a query, it retrieves entries D=ℛ(zretrieve,ℳt) from memory with BM25.

Phase 2: reason to predict

The model refines its initial reasoning with the retrieved memory, generating zpredict∼pθ(·∣ℰt−k:t,zretrieve,D). This reasoning now incorporates the user's past behavior and preferences, and the model predicts future actions ℰt+1:t+h∼πθ(·∣ℰt−k:t,D,zpredict).

During training, the prediction trace with the highest reward (scored by the LLM judge) is saved back into memory ℳt+1=ℳt∪{(ℰt−k:t,zpredict*,t)}. The same rewards are used by GRPO with a group size of 4 to train the two-phase policy with LoRA.

LongNAP: reason to retrieve, then reason to predict A hand-drawn two-phase diagram. Qwen-2.5-VL-7B, the VLM policy pi, reasons over recent observations. Its reasoning becomes a BM25 query into memory M at time t. Each memory entry pairs past observations with a previous reasoning trace z; only timestamps tau at or before t are accessible. Retrieved entries and recent observations inform revised reasoning and future action predictions. As events unfold and new traces are generated, memory is updated; during training the highest-reward prediction trace is saved. Phase 1: reason to retrieve Recent observations screenshots + action context VLM policy π Qwen-2.5-VL-7B reason about what might come next Reasoning → query Memory ℳt observations + trace z past entries only Lexical retriever ℛ BM25 only timestamps τ ≤ t Retrieved observations + traces Phase 2: reason to predict Refine reasoning retrieved context + recent observations Predict future actions update memory with new traces

Takeaways

LongNAP outperforms prompting and SFT baselines, but only 17.1% of its predictions align with observed behavior. Combining parametric learning with retrieval of past reasoning improves prediction of open-ended human behavior, but anticipating a user's actions is not equivalent to understanding intent or identifying what assistance would actually help them (for example, a chronic procrastinator may worsen their habits via a learned procrastinator assistant without the proper goal alignment ).