Research Reality Cards

Science fiction exhibits a stronger tendency to degender terms and place masculine terms in feminine contexts compared to general fiction.

The model predicts a masculine term about twice as often when masking 'woman' compared to predicting a feminine term when masking 'man'.

The study's findings may be limited by the specific corpora used and the potential biases in the RoBERTa model.

ClawLoop enables self-improvement of LLM-based agents without external teacher models, resulting in consistent performance gains across multiple benchmarks.

ClawLoop yields consistent improvements on SkillsBench, Claw-Eval-Live, and ClawGym-Bench.

The framework's reliance on the initial loop model may limit generalizability to other models.

LHTB 2.0 provides a broader and more process-aware testbed for evaluating agent execution over extended periods, correcting previous defects and extending tasks.

LHTB 2.0 includes 77 tasks across 13 domains, with a significant increase in evaluation horizon from 90 minutes to three hours.

The reliance on specific parameterized reference implementations may limit generalizability and reproducibility of results.

The proposed correction step significantly enhances sample quality in diffusion models, achieving performance competitive with classifier-free guidance (CFG).

Substantial improvement in FID scores for unguided and unconditional generation.

The paper does not specify authors or provide detailed reproducibility metrics.

MonoMIP achieves state-of-the-art performance on KITTI and Waymo benchmarks for monocular 3D object detection without requiring extra training data.

Achieved 25.84% Moderate AP3D|R40 on KITTI validation set.

The method may require specific configurations for the multi-height inverse perspective mapping that could affect reproducibility.

A small, self-hosted policy can be trained to follow regulated procedures it has never seen, achieving a scenario pass rate increase from 15/43 to 40/43.

Documented checks improved from 67-71% to 93-100% after training.

The audit of the shared specification found defects that were not exposed by its own metric.

The study demonstrates that a simple additive prediction method can outperform specialized neural predictors in terms of consistency and reliability when predicting unseen gene combinations.

Additive disagreement (AD) is shown to be a consistent and informative measure across all evaluated datasets.

The conventional confidence signals used in the models are inconsistent across datasets and models, which raises concerns about reproducibility.

WorldMemBench provides a systematic testbed for diagnosing memory failures and guiding the development of persistent, temporally coherent video worlds.

Evaluated nine representative models, revealing that strong static retention does not imply reliable dynamic state evolution.

The study indicates that memory degradation occurs beyond the first return, which may complicate reproducibility in dynamic scenarios.

The paper introduces the first test based on Levin's Kt using LLMs, enabling the generation of high-difficulty items that can be certified before being solvable by current AI or human testers.

A battery of certified items was administered to twelve LLMs, revealing agent characteristic curves that decrease with difficulty, indicating a new high-difficulty testing region.

The main limitation is the potential difficulty in reproducing the high-difficulty items and ensuring consistent certification across different LLMs.

The proposed Fisher Utility Weighting (FUW) method significantly improves performance in on-policy distillation across multiple benchmarks compared to matched uniform methods.

FUW improves over matched uniform OPD under reverse-KL by 1.35-4.0 macro points and forward-KL by 1.08-2.71 macro points.

The paper does not provide extensive details on the implementation of the proposed method, which may hinder reproducibility.

No existing class-incremental learning methods transfer reliably across different protein encoders, highlighting the need for encoder-aware approaches.

Nine CIL methods were evaluated across five protein encoders with no reliable transfer observed.

The instability in results is attributed to weak gradient signals and heterogeneity in embedding geometry, which complicates reproducibility.

The proposed VITAL framework improves cross-domain estimation performance on multiple rPPG datasets while maintaining the general visual understanding capability of VLMs.

VITAL enhances generative heart rate estimation reliability through structured numerical alignment.

The paper does not specify the authors or provide details on the reproducibility of the experiments.

The proposed inverse reasoning model significantly improves performance on reasoning tasks, doubling pass@1 on hard prompts and outperforming rejection-sampling self-distillation metrics.

A single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts.

The method relies on the quality of the inverse reasoning model and may not generalize across all tasks.

The Action Jet network allows planners to evaluate multiple action candidates over short horizons with over 300x speedup in inference time compared to traditional methods.

Scoring sixteen candidates from a single extraction is over 300x faster than with either source generator.

The method's performance may vary based on the quality of the data used for training the Action Jet network.

The paper demonstrates that existing evaluations of agentic data science overlook critical methodological failures, emphasizing the need for a comprehensive analysis of failure modes.

The analysis identified that the majority of agentic solutions contain at least one failure mode across 220 data science tasks.

The evaluations primarily focus on predictive performance, which may mask underlying methodological issues.

The evaluation reveals that strong English summarization performance does not necessarily transfer to Bengali, emphasizing the need for comprehensive evaluation frameworks for low-resource languages.

GPT-5.6 Luna leads several English lexical metrics but LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores in Bengali.

Different evaluation methods produce varying model rankings, indicating limitations in relying on a single metric for assessment.

Coding agents can achieve competitive results in compositional accuracy for image generation while maintaining a consistent internal representation of scene content.

Achieved competitive results in zero-shot drawing tasks on a 200-prompt benchmark.

The agents exhibit limitations in perceptual richness compared to traditional generative models.

LLMs can become effective event forecasters when provided with natural-language descriptions and a separation of event semantics from prediction space.

LLMs substantially outperform description-free variants and non-LLM baselines when using event descriptions, with a benchmark of ~150K events tested.

The study may be limited by the specific design of the DescribeEvent benchmark and the potential variability in LLM capabilities.

JSSA achieves up to 6.43 times the decoding throughput of full-KV attention at 128K context while maintaining high accuracy.

Achieves up to 6.43× the decoding throughput of full-KV attention.

The method may not provide measurable improvement in certain settings when refining the exact objective.

LOTUS achieves the best average performance on downstream tasks with minimal trainable parameters, outperforming the strongest matched baseline by 3.36%.

59.91% accuracy across six downstream evaluation datasets.

The paper does not specify potential limitations or reproducibility concerns.

AVA improves average accuracy by up to 2.40 points over the strongest same-setting baselines while significantly reducing the prior-only answer rate and hallucination rate.

Increases visual-dependency score from 0.28 to 0.49.

The framework's performance is evaluated without fine-tuning, which may limit its adaptability to specific use cases.

AutoIndexer enables post-generation context-editing capabilities that surpass those of most autoregressive and diffusion models without sacrificing performance on standard benchmarks.

Exhibits post-generation context-editing capabilities that most autoregressive and diffusion models lack.

The robustness to erroneous context may vary based on the quality of perturbation training data.

EvoSyn improves average task success by 13.3 points over the strongest synthesis baseline while producing higher-quality synthetic experiences.

Improved average task success by 13.3 points.

The paper does not specify potential limitations or reproducibility concerns.

AVA improves performance on vision-language benchmarks by an average of +1.7 points, with gains of up to +2.4 points on VizWiz, while maintaining robustness against noisy demonstrations.

+1.7 points average improvement on five vision-language benchmarks with interleaved 4-shot prompts.

No specific authors or detailed methodology provided, which may hinder reproducibility.

SWE-Duel successfully demonstrates a self-scaling benchmark where coding agents evaluate each other, leading to a significant performance difference based on task authoring capability.

GPT-5.5 with Codex CLI repaired 98.3% of the defects it reviewed.

The study's results may be limited by the specific repositories and languages used, which may not generalize to all coding environments.

DCOCO achieves up to 6.6x speedup over vanilla decoding while maintaining high throughput across various configurations.

DCOCO outperforms competing methods by 6-23% across 4-32 concurrent requests.

The paper does not specify the authors or provide detailed reproducibility guidelines.

The proposed method allows for training compact student encoders that are up to 10x more efficient than larger teacher foundation models while matching or outperforming their performance.

Achieved up to 10x efficiency in training compact student encoders compared to larger teacher models.

The paper does not specify potential limitations or reproducibility concerns.

The deployment regime significantly influences error variation in PV disaggregation models, more so than model choice itself.

The best model achieved a daylight mean absolute error (MAE) of 0.104 kW on seen households, increasing to 0.185 kW on unseen households.

The study's findings may vary based on the specific deployment regime and the adaptation to new regions, which could limit generalizability.

MIRAGE achieves accurate metric trajectory estimation, outperforming COLMAP in computational efficiency for vision-only benchmarks.

MIRAGE establishes accurate 6-DoF reference trajectories for a new vision-only benchmark.

Scarcity of long-horizon monocular benchmarks for large-scale indoor reconstruction may limit reproducibility.

The CLUE framework effectively automates event type induction and annotation, reducing reliance on manual processes.

The framework utilizes contrastive loss guided clustering for event type induction.

The effectiveness of the method may vary based on the dataset's domain and scope, which could affect reproducibility.

The implementation of Kairos optimizes convergence for a byte-level foundation model across multiple modalities, achieving a minimum loss near $10^{-3}$ with specific attention mechanisms.

The two attention variants tested reach a minimum loss near $10^{-3}$ after 3,000 steps.

The study does not address generalization, multimodal training, or comparison with existing models, which raises concerns about the reproducibility of results.

The input-dependent phase improves performance on longer sequences, reducing perplexity from 12.81 to 3.76 when tested beyond training length.

Achieved a perplexity reduction from 12.81 to 3.76 on 1024 tokens after training on 256 tokens.

The learned phase does not improve retrieval and is limited by the model's exposure to training history.

The paper identifies five major shortcomings of existing faithfulness metrics in XAI and suggests four research directions to enhance their reliability.

The paper outlines five major shortcomings of current faithfulness metrics.

Existing metrics for quantifying faithfulness are either infeasible or computationally expensive.

EliteTable demonstrates significant improvement in candidate scores across six Gymnasium control tasks, with some tasks achieving score ceilings in the first generation.

Candidate-score medians rose in all six tasks over ten generations.

Comparisons and ablations are needed to establish EliteTable's relative performance and the contributions of its components.

ELR enables looped language models to achieve non-recurrent model accuracy at one loop without additional cost, while allowing for improved accuracy with more loops.

At one loop, ELR models are 0.52 points below and 1.04 points above their matched baseline on a ten-task average.

The paper does not specify author names or provide detailed reproducibility metrics.

The proposed CircuitTree method enables the composition of multiple circuits into a single model that can classify syllables with 64.8% accuracy.

Achieved 64.8% accuracy in transcribing open syllables, within 1.2 points of a 66.0% ceiling.

The method's effectiveness may depend on the specific circuits used and their pruning from HuBERT.

SpireBench demonstrates that while frontier models can match average human players in Slay the Spire, they significantly lag behind expert players, particularly in long-term strategic decision-making.

20 large language models were tested, with frontier models matching average human performance but falling short against experts as run length increased.

Models do not improve from the memories they write across runs, often leading to incorrect and overly cautious decisions.

Algebraic integer quantization (AIQ) achieves competitive accuracy compared to the MX format for large language model inference while enabling error-free integer arithmetic.

AIQ demonstrates competitive accuracy across diverse models and benchmarks at equivalent bit-per-parameter budgets.

Generalizations to higher-order algebraic integers are still a promising direction and may not be fully explored in the current work.

MCTS is 2.88% worse than randomized-restart list scheduling under equal budget conditions, indicating that simpler methods can outperform MCTS in certain scenarios.

At a budget of 2048 evaluated schedules, MCTS is 2.88% worse than the randomized-restart method.

Results are confined to a controlled synthetic domain with limited task and processor configurations, raising concerns about generalizability.

Coral-KV improves PSNR by up to 4.95 dB while reducing cache memory usage by up to 48% compared to existing methods.

Improved PSNR by 4.95 dB while using 37% less cache memory on MAGI-1.

The paper does not provide specific details on the implementation or datasets used for reproducibility.

The dataset includes 1,211 samples of stereo video clips and stills, released in camera-native formats along with lens calibration and conversion tools.

Native-fisheye coding required 38% more bitrate than half-equirectangular coding at equal viewport quality across all tested clips.

The dataset is limited to recordings from specific cameras, which may affect generalizability to other systems.

SLVR significantly improves multimodal reasoning benchmarks with structured latent supervision, achieving absolute gains of +9.4 on MMVP and +14.2 on BLINK Relation.

Achieved absolute gains of +9.4 on MMVP and +14.2 on BLINK Relation.

The method's reliance on specific training signals may limit generalizability to other tasks or datasets.

The method achieved a 50% reduction in cumulative input tokens and significantly lowered API costs while maintaining functionality.

Used 50% fewer cumulative input tokens, resulting in an estimated API cost of USD 1.28-1.44 versus approximately USD 4.38.

The method did not fully satisfy follow-up evaluations and showed variability in performance across different tasks.

The framework significantly improves the efficiency of TMP preparation by fine-tuning multiple open-source LLMs.

Fine-tuning improves performance across standard text generation metrics.

LLMs tend to over-generate strategies and struggle with project-specific justifications and accurate cost estimates.

Machine learning models can serve as 'cognitive thermometers' that bridge the gap between symbolic logic and connectionist AI, offering a unified approach to measuring semantic complexity.

Emerging evidence shows that logic and machine learning often yield converging results on relative complexity.

The reliance on specific logical languages may limit the generalizability of the findings.

MESSENGER achieves state-of-the-art performance in long-horizon future extrapolation, reducing EPE3D by 71.6% on nuScenes and 67.7% on Argoverse 2.

Reduces EPE3D by 71.6% on nuScenes.

The computational overhead may increase with more input frames, potentially affecting reproducibility in resource-constrained environments.

The Synthesis Through Simulation (STS) framework guarantees structural validity of generated data while achieving 0.88 average marginal fidelity and 100% constraint satisfaction across ten environments without access to database schemas.

Achieved 0.88 average marginal fidelity and 100% constraint satisfaction across all ten environments.

The framework's effectiveness may be limited by the specific environments and the lack of access to database schemas.

The study proves that independent majority amplification preserves well-defined languages in the context of agentic AI self-improvement.

Exact verification is established as the zero-randomness case, with placement and completeness results.

The framework's reliance on explicit complexity assumptions may limit reproducibility and general applicability.

The proposed autoencoder achieves compressed representations comparable to lossless text compression algorithms at 2.24 bits per byte while maintaining good reconstruction and performance on downstream tasks.

Achieves 2.24 bits per byte compression on web text data.

The paper does not specify the authors, which may affect reproducibility and validation of results.

The proposed architecture effectively discovers disentangled object and attribute representations, demonstrating empirical evidence for the Linear Representation Hypothesis in slot space.

Improvements in DCI scores over state-of-the-art methods.

The paper does not specify the authors, which may limit reproducibility and verification of results.

Router telemetry can significantly enhance membership inference attacks on fine-tuned MoE models, increasing true positive rates by 2.7–9.4 percentage points.

Router telemetry improves membership inference over a strong output-signal ensemble by 2.7–9.4 percentage points at 1% false positive rate.

The study's findings may be limited by the specific architectures and data domains tested, which could affect generalizability.

The number of synthetic samples needed for activation probes is determined by coverage rather than the difficulty of the data, with a median half-gain size of 7-11 samples across different concepts.

Probes for high-stakes and harmful situations plateau with as few as 80 samples, while instruction probes require several times more.

The variance in results is significantly influenced by the concept and distribution, which may affect reproducibility across different contexts.

Diffu-LoRA achieves improved subject fidelity and prompt alignment in personalized diffusion models compared to traditional fine-tuning methods.

Experiments show improved overall subject fidelity and prompt alignment relative to evaluated fine-tuning baselines.

The paper does not specify the authors or provide detailed reproducibility guidelines.

The authors created a 6.75 billion token corpus from 294,369 documents, which is over three orders of magnitude larger than existing annotated datasets for pre-1918 Polish.

The character error rate on a hand-corrected sample is 0.68%, with 45% of sampled passages being uncorrectable.

The models trained on this corpus reproduce period-specific prejudices, including antisemitic statements.

SPERA achieves the highest average balanced accuracy across nine downstream tasks in clinical, cognitive, and BCI applications, demonstrating its effectiveness as a general-purpose EEG model.

Pretrained on approximately 80,000 hours of EEG data from 29,048 subjects across 106 datasets.

The model's performance may vary across different recording conditions, which could affect reproducibility.

OcuFlow achieves superior reflection removal quality and ocular fidelity, outperforming existing methods significantly in user studies.

OcuFlow received 67.32% of user selections in a blind study, 6.2 times the next-best share.

Existing datasets cover limited reflection conditions, which may affect generalization to complex real-world scenarios.

The framework effectively maps headers to 39 interpretable types and assesses data quality issues, providing a reusable workflow for semantic annotation and quality monitoring.

Evaluated across 120,000 header columns with broad practical coverage across noisy real-world metadata.

The official evaluation results were modest, indicating potential issues with benchmark granularity and ontology-selection effects.

Self-evolution of LLM agents can be effectively managed by confining changes to a runtime harness, allowing for reviewable adaptations without altering model weights.

The gated loop admitted 144 of 7,449 candidate changes, none of which worsened error on held-out history.

The gate's fixed tolerance blocked correct replacements in half the seeds at the highest structural severity.

The proposed framework improves anomaly detection in in-vehicle networks by effectively capturing subtle temporal and contextual anomalies through a lightweight Transformer encoder and federated learning.

Achieved robust anomaly detection performance while maintaining efficiency and privacy.

The reliance on open-source datasets may limit generalizability to real-world scenarios.

Freezing the decoder and healing only the encoder achieves parity with other methods while using 3x fewer trainable parameters and 3x less optimizer-state memory.

Achieved a real measured saving of 3x fewer trainable parameters and 3x less optimizer-state memory.

The results are based on a specific compression ratio and may not generalize across different configurations or models.