Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen

Over the past decade, AI-assisted coding tools like GitHub Copilot and OpenAI Codex have revolutionized software development. Early studies demonstrated productivity gains at the individual level, but comprehensive understanding of team-level impacts, especially in open-source contexts, remains limited. As autonomous AI agents capable of submitting, reviewing, and merging code emerge, the dynamics of contribution, collaboration, and quality assurance are fundamentally shifting. Prior research has focused on short-term experiments or specific metrics, lacking a longitudinal perspective. Open-source repositories such as vLLM and SGLang, with their transparent PR histories, provide ideal datasets to explore these changes systematically. This study builds on prior work by integrating multiple engineering metrics over an extended timeline, capturing the evolution of human-AI collaboration in real-world, high-velocity environments, with implications for scientific software engineering and bioinformatics workflows.

The core challenge lies in quantifying how AI tools influence development velocity, contributor diversity, and collaboration quality over time. Existing metrics often focus on isolated aspects, failing to capture the complex, dynamic interplay between human effort and AI assistance. Additionally, the rise of autonomous agents raises questions about contribution attribution, team cohesion, and review processes. Understanding these shifts is critical for optimizing AI integration in scientific research, where maintaining data-contextual continuity and collaborative robustness is vital. The difficulty in distinguishing AI-generated code from human contributions complicates analysis, necessitating sophisticated attribution methods and multi-metric approaches to accurately reflect the evolving landscape.

This work introduces a multi-dimensional, longitudinal framework combining PR-level metrics with account-type-based bot attribution, enabling precise tracking of AI's impact on development dynamics. It innovates by segmenting development into eras aligned with AI tooling milestones, applying regression models to quantify contributor growth, and analyzing comment activity to gauge collaboration depth. Unlike prior studies limited to short-term or single-metric analyses, this approach offers a comprehensive, temporal view of how AI assistance reshapes open-source workflows, providing actionable insights for optimizing AI-human synergy in scientific software engineering.

• �� Select two open-source AI infrastructure projects (vLLM and SGLang) with publicly accessible PR histories. • Collect monthly PR data via GitHub API, including creation, merge times, comments, and code changes. • Segment development into four eras based on AI tool release milestones, aligning metrics with technological shifts. • Use GitHub account-type metadata to identify bot contributions, validating with comment content sampling. • Calculate seven engineering metrics per month: PR throughput, cycle time (median and P90), contributor count, comment density, merge rate, new contributor share, and PR size. • Apply Ordinary Least Squares (OLS) regression to analyze contributor growth trends over time. • Generate heatmaps of normalized metrics to visualize multi-dimensional evolution, and perform sensitivity analysis to confirm robustness against automated activity.

The analysis involved full PR datasets from both projects, segmented into eras reflecting AI tooling milestones. Metrics were computed monthly, with bot contributions identified via account metadata and comment sampling. Regression analysis quantified contributor growth, while heatmaps illustrated multi-metric patterns. Validation included comparing bot vs. human contribution ratios and assessing the stability of PR sizes. The experiments aimed to establish causal links between AI tooling adoption and changes in development dynamics, ensuring metrics accurately reflect real collaboration shifts. Sensitivity tests confirmed that the observed trends are not artifacts of automated activity, providing a reliable basis for interpretation.

The data demonstrate a dramatic increase in development velocity, with PR throughput soaring over 20 times, driven mainly by human effort. Contributor diversity expanded steadily, with monthly unique authors rising significantly, indicating broader engagement. Comment density increased fourfold, suggesting deeper collaboration, while PR sizes remained stable, ruling out scope expansion as a cause. Cycle times, although slightly longer in the latest era, still support rapid review processes. These findings collectively reveal that AI tools facilitate faster, more inclusive, and more interactive development workflows, challenging assumptions that automation leads to contribution concentration or reduced participation.

In the short term, these metrics can guide the optimization of open-source scientific software workflows, especially in bioinformatics, by identifying bottlenecks and fostering inclusive collaboration. Long-term, they support the design of AI-enhanced pipelines that maintain contextual integrity, improve reproducibility, and accelerate biomedical research. The insights can inform policy for AI integration in scientific teams, ensuring balanced human-AI synergy, and fostering sustainable, innovative research ecosystems.

The attribution of contributions to AI remains indirect, relying on account types and comment analysis, which may misclassify some activities. The study's focus on two projects limits generalizability; different domains or project sizes may exhibit distinct patterns. The analysis does not incorporate code quality or performance metrics, which are critical for scientific validity. Further research should include multi-project validation, direct code attribution, and integration of quality indicators to deepen understanding of AI's impact on scientific software development.