ScaleCall -- Agentic Tool Calling at Scale for Fintech: Challenges, Methods, and Deployment Insights
ScaleCall shows that ETR minimizes latency, while listwise TRR improves ambiguity resolution; instruction boosts Code N@10 from 14.26% to 32.38%.
Key Findings
Methodology
ScaleCall, developed at Mastercard, compares Embedding-based Tool Retriever (ETR), Tool Retrieval with Re-ranking (TRR), and hybrid designs. ETR encodes tool descriptions offline and queries online with all-MiniLM-L6-v2, ranking candidates by cosine similarity. TRR first retrieves the top 10 tools, then applies generative listwise ranking with LLaMA 3.1-8B or LLaMA 3.2-3B through vLLM. Experiments use query-only and query-plus-instruction inputs.
Key Results
- The observed ToolRet release contains 7,961 tasks and 44,453 tools. Adding instructions increased ETR N@10 from 14.26% to 32.38% on TOOLRET-Code and from 24.78% to 33.29% on TOOLRET-Custom, showing that task context is central to retrieval quality.
- TRR improved ranking under laboratory conditions but suffered frequent timeouts and high latency in on-premises deployment. With LLaMA 3.1-8B, Web N@10 rose from 15.92% to 34.90% with instructions, yet operational overhead often triggered fallback behavior.
- Performance was domain-dependent: Web was easiest, Code was hardest, and LLaMA 3.1-8B was particularly effective on nuanced Customized tools. No retrieval method dominated across repository size, semantic overlap, and query type.
Significance
The study shifts tool-calling research from open-web demonstrations toward regulated enterprise operation. It addresses persistent problems involving proprietary APIs, inconsistent documentation, overlapping names, authorization logic, and auditability. Academically, it shows that tool retrieval is not merely a conventional information-retrieval problem: domain semantics, access policies, and infrastructure jointly determine success. Industrially, it offers a deployment-oriented decision framework. The main lesson is that retrieval accuracy must be evaluated together with latency, timeout behavior, compliance, and execution safety rather than in isolation.
Technical Contribution
ScaleCall unifies offline indexing, candidate retrieval, LLM reranking, execution gating, permission validation, and error recovery in a flexible architecture. Unlike ToolRet’s pointwise cross-encoder reranker, its TRR performs comparative listwise reasoning over the complete candidate set. The authors also standardize CPU embedding Layer Normalization in float32 to improve numerical stability across hardware. Evaluation combines ChatGPT-based syntactic checks with subject-matter-expert review, creating a domain-aware assessment procedure for enterprise tool selection.
Novelty
The principal novelty is systems-oriented rather than a new embedding architecture. ScaleCall is among the few studies to compare embedding retrieval, generative listwise reranking, and hybrids inside a regulated, on-premises fintech environment. Relative to LangChain, ToolLLM, RankGPT, and ToolRet, it treats timeout, fallback, permissions, licensing, and auditability as first-class design variables. Its central finding is that algorithmic superiority is conditional on enterprise domain and infrastructure.
Limitations
- Model coverage was constrained by licensing, cost, hardware, and internal approval requirements; experiments used only all-MiniLM-L6-v2, LLaMA 3.1-8B, and LLaMA 3.2-3B, so conclusions do not generalize to every stronger retriever or reranker.
- The supplied paper text does not report complete Table III/IV values, throughput, or detailed latency distributions, limiting independent analysis of the accuracy–cost frontier.
- ToolRet is not a full proxy for financial workflows involving jurisdiction, role-based access, sensitive data, and multi-step recovery.
Future Work
Future research should learn routing policies that select ETR, TRR, or cascades according to repository size, overlap, risk, and latency budgets. Important extensions include permission-aware retrieval, confidence calibration, caching, quantized local models, audit-native evaluation, and real task-success metrics. Larger enterprise benchmarks should also test MCP integration, multi-turn recovery, and execution outcomes rather than retrieval scores alone.
AI Executive Summary
Large language models can call APIs, but choosing the correct API inside a bank is a harder problem than producing a syntactically valid function call. Enterprise repositories may contain hundreds of proprietary services with incomplete documentation and nearly identical names, while authorization rules, jurisdictional constraints, and on-premises policies differ. A wrong choice can cause silent failure or a compliance incident.
To study this problem, Mastercard researchers built ScaleCall, a prototype for internal API orchestration and data-engineering automation. Its Embedding-based Tool Retriever (ETR) uses all-MiniLM-L6-v2 to encode queries and tool descriptions, then retrieves candidates by cosine similarity. Its Tool Retrieval with Re-ranking (TRR) sends the top 10 candidates to LLaMA 3.1-8B or LLaMA 3.2-3B for generative listwise comparison. The evaluation uses the observed ToolRet release: 7,961 tasks and 44,453 tools, tested with query-only and query-plus-instruction inputs.
The results reject a universal winner. Adding instructions raised ETR Code N@10 from 14.26% to 32.38% and Customized N@10 from 24.78% to 33.29%. For LLaMA 3.1-8B, Web N@10 increased from 15.92% to 34.90% with instructions. Yet on-premises TRR frequently timed out or became too slow, triggering fallback and reducing its production value. ETR therefore remains attractive for large, latency-sensitive repositories, while listwise ranking is valuable when functional ambiguity is costly. ScaleCall’s broader contribution is a deployment perspective: retrieval, authorization, auditability, error recovery, and infrastructure must be designed as one system. Its evidence also points toward adaptive cascades and risk-aware routing rather than a single best algorithm.
Deep Analysis
Background
LLM tool calling has evolved from function prompting toward retrieval-augmented orchestration. Embedding retrieval appears in systems such as LangChain and OpenAI function calling; ToolLLM and RankGPT add stronger semantic reasoning and reranking. ToolRet, however, shows that conventional retrievers struggle when tools overlap functionally. Fintech adds local deployment, legacy integration, access control, sensitive data handling, and regulatory audit requirements.
Core Problem
Given a natural-language query, optional instruction, and a large tool library, the system must select the correct API and sometimes retrieve all relevant tools. The bottlenecks are semantic overlap, weak documentation, underspecified queries, hidden authorization context, and the latency of generative reranking. Selection errors can produce incorrect execution, policy violations, or failures that are difficult to detect.
Innovation
- ��ScaleCall provides an enterprise tool-calling lifecycle for Mastercard feature-store APIs and data workflows.
- ��The study compares ETR, listwise TRR, and hybrid retrieval under real deployment constraints rather than assuming one method is universally best.
- ��Evaluation combines ToolRet, synthetic query generation, ChatGPT validation, and expert review.
- ��Engineering adaptations include approved local models, CPU float32 Layer Normalization, permission checks, execution gating, and timeout fallback.
Methodology
- ��Input: ToolRet supplies Query, Instruction, tool name, description, schemas, and a ground-truth tool ID.
- ��ETR: encode tool documentation offline; encode the query at runtime; rank with cosine similarity and return top-k.
- ��TRR: pass ETR’s top-10 candidates to LLaMA 3.1-8B or 3.2-3B; jointly compare and reorder them in a generative listwise prompt.
- ��Metrics: Success/P@1, R@1, R@10, P@10, C@10, and nDCG@10.
- ��Deployment: serve models with vLLM and protect execution with permissions, gating, auditability, and fallback mechanisms.
Experiments
Experiments cover ToolRet Web, Code, Customized, and aggregate subsets under query-only and query-plus-instruction conditions. The observed release contains 7,961 tasks: 5,230 Web, 1,749 Code, and 982 Customized, over 44,453 tools. ETR uses all-MiniLM-L6-v2; TRR uses LLaMA 3.1-8B and 3.2-3B with k=10. CPU embedding computations use float32 Layer Normalization. The study also compares baseline retrieval with reranking and instruction ablations.
Results
Instruction context consistently improves retrieval. ETR Code N@10 rises from 14.26% to 32.38%, and Customized rises from 24.78% to 33.29%. For TRR, LLaMA 3.1-8B average N@10 increases from 13.82% to 15.80%; Web rises from 15.92% to 34.90%, while LLaMA 3.2-3B rises from 14.35% to 30.35%. Web is easiest and Code hardest. Despite better ambiguity reasoning, TRR’s local latency and timeout behavior limit production benefit.
Applications
ScaleCall targets account aggregation, Mastercard Open Banking API orchestration, internal feature-store access, and data-engineering automation. Deployment requires reliable tool metadata, role-based permissions, audit logs, isolated execution, and fallback behavior. ETR suits large repositories and strict latency budgets; TRR is most useful when similar tools have materially different authorization or business semantics.
Limitations & Outlook
The approved-model constraint limits model generality, while ToolRet may not represent real financial authorization, jurisdiction, and sensitive-data conditions. The available text lacks complete latency and throughput tables, preventing a full cost analysis. Future systems should use adaptive ETR–TRR routing, caching, quantized local models, permission-aware retrieval, confidence thresholds, MCP integration, and benchmarks based on end-to-end task success.
Plain Language Accessible to non-experts
Imagine a huge factory receiving the instruction, “Send this package to the right customer.” The factory has thousands of machines. Several machines have almost the same label, but one serves Europe, another serves Asia, and a third can only be used by a supervisor. Choosing the wrong machine may not look wrong immediately, yet the package can go to the wrong place.
ETR is the fast warehouse clerk. It reads the request and quickly finds the ten machines whose descriptions look most similar. This works well when the factory is huge and every second matters, but similar labels can fool it. TRR is the senior supervisor. The supervisor looks at all ten options together and compares their real purposes, so it is better at separating near-duplicates.
The study found that giving a fuller instruction matters greatly: ETR’s Code N@10 increased from 14.26% to 32.38%, and Customized N@10 from 24.78% to 33.29%. But the supervisor takes longer. In Mastercard’s local environment, reranking sometimes timed out, so the factory needed a backup route.
The practical message is simple: a dependable AI factory needs both a fast searcher and a careful checker, plus permission checks and records of every action. The best arrangement depends on the job’s risk, ambiguity, and time limit.
ELI14 Explained like you're 14
Picture an AI teammate in a video game. You say, “Open the reward chest,” but the game has 44,453 possible tools. Some chests look almost identical: one belongs to the forest level, one to a special event, and one needs admin permission. If the AI picks by name alone, yikes—it may choose the wrong chest!
ScaleCall gives the teammate two ways to search. ETR is like a super-fast search bar: it finds the ten options that sound most related. TRR is like a clever friend who compares those ten options side by side and asks, “Which one really fits this mission?” That makes TRR better at confusing look-alikes, but it uses more time and computer power.
The researchers tested the ToolRet benchmark with 7,961 tasks. When they added a helpful instruction instead of only a short question, ETR’s Code score at the top ten jumped from 14.26% to 32.38%, and its Customized score rose from 24.78% to 33.29%. Clearer instructions help the AI a lot!
But here is the plot twist: in a real finance company running models on its own computers, TRR sometimes took too long or timed out. So ScaleCall needs a backup plan. The lesson is not “the smartest model always wins.” A useful AI must choose correctly, respond quickly, respect permissions, and leave a record. It is a bit like gaming with both speed-running and safety rules!
Glossary
Tool Retrieval
The process of selecting APIs or functions relevant to a user request. Technically, it maps a query to one or more tools in a candidate library.
The paper treats retrieval as the central pre-execution stage of ScaleCall.
Embedding-based Retrieval
A method that maps text to vectors and ranks items using a similarity function such as cosine similarity. It is efficient but can confuse tools with similar meanings.
ETR implements this method with all-MiniLM-L6-v2.
Listwise Ranking
A ranking strategy in which a model evaluates a whole candidate list jointly rather than scoring each item independently. Joint comparison can expose subtle differences among tools.
TRR applies generative listwise reranking to the top 10 ETR candidates.
nDCG@10
A ranking metric that rewards relevant results, especially when they appear near the top ten positions. It measures ordering quality, not only whether an answer appears.
The paper uses it to compare retrieval and instruction effects.
ToolRet
A public benchmark for tool retrieval containing Web APIs, code functions, and customized tools. It provides queries, tool metadata, ground truth, and official metrics.
ScaleCall reproduces and adapts ToolRet for enterprise analysis.
Execution Gating
A control layer that decides whether a selected tool may actually run. It can check permissions, policy, confidence, and operational conditions.
It is required for safe fintech deployment alongside fallback and auditing.
Open Questions Unanswered questions from this research
- 1 How should a system automatically choose ETR, TRR, or a cascade from repository size, semantic overlap, risk, and latency budget? The paper identifies the trade-off but does not learn a unified routing policy.
- 2 It remains unclear whether ToolRet scores predict real financial task success when authorization, jurisdiction, sensitive data, and multi-step recovery are present. New enterprise benchmarks are needed.
- 3 MCP may standardize interfaces, but its integration with legacy APIs and role-based controls under on-premises compliance constraints lacks long-term production evidence.
Applications
Immediate Applications
Internal API routing
Data engineers can describe account aggregation or feature-store tasks in natural language. ETR rapidly narrows the search, while permission checks and selective TRR handle ambiguous, high-risk candidates; timeout fallback preserves service reliability.
Open Banking automation
Before invoking consent-management or Open Banking endpoints, ScaleCall can use business context, jurisdiction, and authorization state to distinguish similarly named services. This reduces silent failures and policy violations while producing an audit trail.
Long-term Vision
Compliance-native enterprise agents
A future platform could combine adaptive retrieval, permission-aware planning, sandboxed execution, recovery, and audit logging. Such systems could automate multi-step banking workflows locally without exposing regulated data to external services.
Abstract
While Large Language Models (LLMs) excel at tool calling, deploying these capabilities in regulated enterprise environments such as fintech presents unique challenges due to on-premises constraints, regulatory compliance requirements, and the need to disambiguate large, functionally overlapping toolsets. In this paper, we present a comprehensive study of tool retrieval methods for enterprise environments through the development and deployment of ScaleCall, a prototype tool-calling framework within Mastercard designed for orchestrating internal APIs and automating data engineering workflows. We systematically evaluate embedding-based retrieval, prompt-based listwise ranking, and hybrid approaches, revealing that method effectiveness depends heavily on domain-specific factors rather than inherent algorithmic superiority. Through empirical investigation on enterprise-derived benchmarks, we find that embedding-based methods offer superior latency for large tool repositories, while listwise ranking provides better disambiguation for overlapping functionalities, with hybrid approaches showing promise in specific contexts. We integrate our findings into ScaleCall's flexible architecture and validate the framework through real-world deployment in Mastercard's regulated environment. Our work provides practical insights into the trade-offs between retrieval accuracy, computational efficiency, and operational requirements, contributing to the understanding of tool-calling system design for enterprise applications in regulated industries.