Publications
2026
- Using Human-LLM Disagreement to Improve Checklist-Based Quality AppraisalTimo van der Kuil, Bruno Messina Coimbra, Mirjam van Zuiden, Robert A. Bagheri, Rens van de Schoot, Klaas Dieleman, Berend Greijn, Stefan Houkes, Sebastiaan Rodenhuis, and Elizabeth M. Grandfield2026
Systematic reviews rely on quality appraisal of included studies, a process that is time-consuming and sensitive to ambiguity in checklist criteria. Although large language models (LLMs) offer opportunities to support these tasks, appraisal checklists are typically treated as fixed inputs, and it remains unclear how their design affects agreement with expert judgments. Therefore, we investigate (1) whether LLMs can approximate human judgments in checklist-based appraisal and (2) whether patterns of human-LLM disagreement can be used to identify and improve ambiguous checklist items. Using the Guidelines for Reporting on Latent Trajectory Studies (GRoLTS) checklist, we compare LLM-generated assessments with expert annotations across three research topics and two checklist versions. Agreement is assessed using item-level accuracy, chance-corrected agreement, and preservation of study-level rank ordering. We find that performance varies substantially across checklist items, with ambiguous and conditional criteria producing the greatest disagreement. Revising these items improves both raw and chance-corrected agreement. Although item-level misclassifications persist, LLM-generated scores often preserve the relative ranking of studies when high-agreement items are retained. These results indicate that reliable LLM-assisted appraisal depends not only on model choice but also on checklist design. The findings suggest that analyzing human-LLM disagreement can help identify problematic checklist items and support the iterative improvement of research synthesis workflows.
@article{vanderkuil2026using, title = {Using Human-LLM Disagreement to Improve Checklist-Based Quality Appraisal}, author = {{van der Kuil}, Timo and {Messina Coimbra}, Bruno and {van Zuiden}, Mirjam and {Bagheri}, Robert A. and {van de Schoot}, Rens and {Dieleman}, Klaas and {Greijn}, Berend and {Houkes}, Stefan and {Rodenhuis}, Sebastiaan and {Grandfield}, Elizabeth M.}, year = {2026}, eprint = {2608.20385}, archiveprefix = {arXiv}, primaryclass = {cs.CL}, url = {https://arxiv.org/abs/2608.20385}, doi = {10.48550/arXiv.2608.20385} } - PTSD Symptom Trajectories in the Wake of Potentially Traumatic Events: a Systematic Review and Meta-Analysis of 20 Years of ResearchBruno Messina Coimbra, Marit Sijbrandij, Iris Engelhard, Rutger Chris Neeleman, Elizabeth M. Grandfield, Elena Jalsovec, Timo van der Kuil, Lauke Stoel, Melissa Alderfer, and Cherie Armour2026
This systematic review, meta-analysis and meta-regression evaluates twenty years of literature on data-driven identification of common PTSD symptom trajectories following potentially traumatic events. Traditional, machine learning- and large language model–assisted systematic search approaches were combined to identify eligible articles applying statistical Growth Mixture Modeling to longitudinally assessed PTSD symptom severity. Ninety-nine studies representing 113 independent samples and 215,136 participants from 21 countries were included. We identified five prototypical PTSD symptom trajectories that commonly occurred across samples. We determined pooled relative trajectory prevalence estimates across samples observing these specific trajectories with the low symptoms trajectory being the most and the increasing symptoms trajectory the least prevalent. Using meta-regression, various socio-demographic, exposure-related and methodological moderators of sample-level relative trajectory prevalences were identified. Our findings provide empirical support and further refinements of leading theoretical PTSD conceptualizations. Furthermore, the obtained relative prevalence estimates and sample-level moderators may inform population-tailored trauma-related mental health care.
@article{coimbra2026ptsd, title = {PTSD Symptom Trajectories in the Wake of Potentially Traumatic Events: a Systematic Review and Meta-Analysis of 20 Years of Research}, author = {{Messina Coimbra}, Bruno and {Sijbrandij}, Marit and {Engelhard}, Iris and {Neeleman}, Rutger Chris and {Grandfield}, Elizabeth M. and {Jalsovec}, Elena and {van der Kuil}, Timo and {Stoel}, Lauke and {Alderfer}, Melissa and {Armour}, Cherie}, year = {2026}, url = {https://doi.org/10.31219/osf.io/fkjb2_v3}, doi = {10.31219/osf.io/fkjb2_v3} } - ASReview Dory: Bringing new and exciting models to ASReview LABSoftware Impacts, 2026
Systematic reviewing is a time-consuming process which can be accelerated through screening prioritisation via active learning. ASReview Dory enables researchers to test, validate, and apply a wide range of embedders and classifiers in systematic literature screening. It extends ASReview LAB, an open source, lightweight, and user-friendly environment with proven default models and extensibility through Python entry points. ASReview Dory adds ready-to-use transformer-based embedders, neural classifiers, and a framework for integrating custom models. Once installed, these models are directly available in ASReview LAB without additional configuration and can be systematically evaluated using the API or ASReview Makita.
@article{vanderkuil2026asreviewdory, title = {ASReview Dory: Bringing new and exciting models to ASReview LAB}, journal = {Software Impacts}, volume = {27}, pages = {100809}, year = {2026}, issn = {2665-9638}, doi = {https://doi.org/10.1016/j.simpa.2025.100809}, url = {https://www.sciencedirect.com/science/article/pii/S2665963825000697}, author = {{van der Kuil}, Timo and {Teijema}, Jelle Jasper and {de Bruin}, Jonathan and {van de Schoot}, Rens}, keywords = {Systematic reviews, Machine learning, Natural language processing, Reproducibility, ASReview} }
2025
- ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of expertsJonathan de Bruin, Peter Lombaers, Casper Kaandorp, Jelle Teijema, Timo van der Kuil, Berke Yazan, Angie Dong, and Rens van de SchootPatterns, 2025
ASReview LAB v.2 introduces an advancement in AI-assisted systematic reviewing by enabling collaborative screening with multiple experts (“a crowd of oracles”) using a shared AI model. The platform supports multiple AI agents within the same project, allowing users to switch between fast general-purpose models and domain-specific, semantic, or multilingual transformer models. Leveraging the SYNERGY benchmark dataset, performance has improved significantly, showing a 24.1% reduction in loss compared to version 1 through model improvements and hyperparameter tuning. ASReview LAB v.2 follows user-centric design principles and offers reproducible, transparent workflows. It logs key configuration and annotation data while balancing full model traceability with efficient storage. Future developments include automated model switching based on performance metrics, noise-robust learning, and ensemble-based decision-making.
@article{bruin2025asreview, title = {ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts}, journal = {Patterns}, pages = {101318}, year = {2025}, issn = {2666-3899}, doi = {https://doi.org/10.1016/j.patter.2025.101318}, url = {https://www.sciencedirect.com/science/article/pii/S2666389925001667}, author = {{de Bruin}, Jonathan and {Lombaers}, Peter and {Kaandorp}, Casper and {Teijema}, Jelle and {van der Kuil}, Timo and {Yazan}, Berke and {Dong}, Angie and {van de Schoot}, Rens} }
