PROGRAMMATIC TRAINING FOR TOOL-USE AGENTS

ProgAgentLearning Tool Orchestration
through Programmatic Training

A full PTC training stack for long-horizon tool orchestration.
Verifiable tasks, a lightweight PTC runtime, and a two-stage training recipe.

215 × 4training tasks × environment scales
30.7%LOCA-bench at 64k · ProgAgent-14B
45.6%MCP-Atlas · ProgAgent-14B

Verifiable Environment Synthesis

We construct verifiable tasks and scale their environment states from v₀ to v₃ while keeping instructions, task logic, and tool schemas fixed.

Overview of ProgAgent: task synthesis, a persistent PTC runtime, and two-stage SFT and RL training.
Overview of ProgAgent. (a) Task synthesis. We construct verifiable tasks, scale their environment states from v₀ to v₃ while keeping instructions, task logic, and tool schemas fixed, and apply independent auditing and difficulty filtering. The scaling plot shows DeepSeek V4 Pro’s declining DTC pass@8 as environment size increases. (b) PTC runtime. A persistent IPython kernel executes tool-calling programs, retains intermediate results in variables across turns, and returns printed outputs to the agent. (c) Two-stage training. Cold-start SFT on teacher trajectories is followed by RL with binary rewards from Python task verifiers.

Experimental Results

We evaluate our models across MCPMark, MCP-Atlas, and Toolathlon-Verified, and further use LOCA-bench at 8k–64k environment-state sizes. Each reported pass rate averages four independent runs.

Main Results

ProgAgent-14B · PTC training and evaluation
MCPMark29.7%

Four-suite average

LOCA-bench38.3%

Average over 8k–64k states

Toolathlon-Verified17.1%

Verified task subset

MCP-Atlas45.6%

Pre-v2.0 full task set

Full main-results table, including frontier proprietary models, open-source foundation models, open-source baselines, ProgAgent-8B, and ProgAgent-14B.
Main results. Mean pass rates (%) on four benchmarks. Best and second-best scores within each 8B/14B comparison are bolded and underlined.

ProgAgent uses PTC evaluation; the baselines in the main results table use DTC. MCPMark excludes the GitHub suite. Toolathlon scores marked † are from the original Toolathlon benchmark; all other scores in that column are from Toolathlon-Verified. “–” denotes an unreported score.

On MCPMark, ProgAgent achieves average pass rates of 18.3% at 8B and 29.7% at 14B, respectively. On Toolathlon-Verified, ProgAgent reaches 9.3% at 8B and 17.1% at 14B, compared with at most 2.8% and 1.9% among the 8B and 14B baselines.

Moreover, on MCP-Atlas, ProgAgent-14B achieves 45.6%, comparable to Claude Sonnet-4.5 (43.8%) under the DTC harness. The gap is most pronounced on LOCA-bench: the evaluated comparable-size baselines score zero at 32k and 64k, while ProgAgent mitigates context rot to maintain pass rates between 17.3% and 30.7% at 64k.

Training-data Scales

The resulting 215 tasks are each instantiated at 4 scales, yielding a shared training pool of 860 task instances.

Training-data scale comparison: TOUCAN 1.53M trajectories; Simulator 135k trajectories; EnvScaler 11,572; AWM 3,315; Agent-World 2,000+; EnvFactory 2,575; ProgAgent 215 tasks at four scales.
Training-data scales across baselines (traj.: trajectories). Simulator aggregates synthesis settings; EnvScaler counts SFT + RL samples (Xu et al., 2026).

Training Paradigms across Benchmarks

The table below compares each training recipe with its untrained backbone under the same harness. For untrained backbones, switching to PTC provides limited benefits across benchmarks and often yields the lowest scores among the three harnesses. Training reverses this pattern: after SFT and RL, the PTC recipe achieves the highest aggregate score on each benchmark at both model scales.

DTC, Hybrid, and PTC training at 8B and 14B, with untrained-backbone scores. The PTC recipe achieves the highest aggregate pass rate on each of the four benchmarks at both model sizes.
Training paradigms across benchmarks. Final pass rates (%) with the corresponding base model’s scores under the same harness shown in parentheses. Each trained model is evaluated under its training harness. The best score for each benchmark within each backbone is bolded.

ProgAgent-14B reaches 29.7% on MCPMark and 45.6% on MCP-Atlas, compared with 8.0% and 29.7% for DTC, respectively. ProgAgent-8B also outperforms the 8B DTC model on all four benchmarks.

Original stacked bar chart showing gains from SFT and RL. PTC averages 24.3% at 8B and 38.3% at 14B on LOCA-bench, and maintains 30.7% at 64k for 14B.
Training paradigms on LOCA-bench. Each bar decomposes the final pass rate into the untrained backbone’s score and the gains from SFT and RL. Labels above the bars show rounded final pass rates. Left: Results for the 8B and 14B backbones, averaged over four environment-state sizes. Right: Results for the 14B backbone at each environment-state size, showing PTC’s robustness as the environment grows.

The chart above compares the three training paradigms on LOCA-bench, showing gains from both SFT and RL. As the environment state scales, DTC suffers from context rot. In contrast, PTC effectively mitigates this degradation and sustains robust performance, achieving average pass rates of 24.3% (8B) and 38.3% (14B). Furthermore, despite providing a more flexible action space during both training and evaluation, the Hybrid paradigm performs worse than PTC.

Analysis

What Does PTC Change During Training?

At 14B, DTC achieves substantially lower rewards than PTC and Hybrid. During RL training, tool-output tokens under PTC decrease by roughly 20%, falling below 20k per episode; both programmatic paradigms use roughly half as many tool calls as DTC.

Original 14B training curves for reward, tool-output context tokens, and tool calls per episode. PTC and Hybrid achieve higher rewards with fewer tool-output tokens and tool calls than DTC.
RL training dynamics at 14B. Episode reward (left), tool-output tokens entering context (middle), and tool calls per episode (right) under DTC, Hybrid, and PTC.

Does Environment State Scaling Help?

Environment state scaling improves PTC performance on LOCA-bench across both backbones. Restricting training to v₀ reduces average pass rates from 24.3% to 16.0% at 8B and from 38.3% to 23.7% at 14B.

At v0, v1, v2, and v3, database sizes are 99, 173, 335, and 711 KB; mean workspace file counts are 12.7, 21.9, 31.1, and 40.3; DTC pass@8 is 0.95, 0.88, 0.65, and 0.42.
Environment state and task difficulty. Database size (KB), mean workspace file count, and DeepSeek V4 Pro’s DTC pass@8 across four scales.
Hybrid pass@8 across v0 through v3: 0.95, 0.92, 0.93, 0.88. PTC pass@8: 0.95, 0.94, 0.92, 0.90.
Environment scaling under Hybrid and PTC. DeepSeek V4 Pro’s pass@8 on synthesized tasks at scales v₀–v₃; the DTC counterpart appears in the method overview.

How Does PTC Facilitate Tool Orchestration?

PTC models use loops, conditional execution, and error handling to orchestrate tools during RL. PTC-trained models outperform DTC-trained models under both evaluation interfaces, with a 11.9-percentage-point gain on MCP-Atlas at 14B under DTC evaluation.

Loops, conditional execution, and error handling are used at rates of 82.6%, 65.6%, and 8.2% at 8B, and 82.1%, 64.0%, and 9.5% at 14B.
Programming structures used during RL. Percentage of code blocks using loops and conditional execution, and executions using error handling.
Full cross-harness evaluation for 8B and 14B. PTC-trained models outperform DTC-trained models on both benchmarks under both evaluation interfaces. At 14B with DTC evaluation, MCP-Atlas improves from 29.7% to 41.6%.
Cross-harness evaluation on LOCA-bench and MCP-Atlas. Pass rates (%) under DTC and PTC evaluation for untrained backbones (Base) and models trained with the DTC or PTC SFT+RL recipe. LOCA-bench scores are averaged over four environment-state sizes.

What Challenges Remain for PTC Agents?

DTC-like behavior: on Playwright, 40.7% of ProgAgent-14B’s PTC calls contain a single tool invocation without control flow, error handling, or selective printing. Silent server failures: disabling server-level error detection reduces LOCA-bench pass rates from 24.3% to 19.3% at 8B and from 38.3% to 34.5% at 14B.

Interaction costs of PTC on MCPMark
Interaction costs for 8B and 14B across Filesystem, PostgreSQL, Notion, and Playwright, including tool calls, tool-output tokens, context limit, and DTC-like calls.
Interaction costs of PTC on MCPMark. Tool calls and tool-output tokens are averaged per trajectory. Context limit reports the fraction of trajectories reaching the context limit. The “DTC-like calls” column reports the percentage of PTC calls containing a single explicit tool invocation without control flow, error handling, or selective printing; any printed tool return is reproduced in full without processing.

Toolathlon-Verified: From an Emailed Schedule to Published Canvas Courses

This case shows how PTC uses loops and returned identifiers to coordinate dependent course-creation and configuration calls.

Instruction

This is admin3, a registrar from XXX University. The university has sent me the new semester's course schedule via email with the subject "Course Schedule Notification." I need you to help me find the courses I'm responsible for from the attached master schedule and create corresponding courses on Canvas for the instructors. The specific requirements are as follows:

  • Assign the corresponding instructor as the teacher for each course on Canvas. The instructors' emails are saved in my local workspace.
  • Publish an announcement for each course that includes the instructor's name and the class time.
  • If an instructor is teaching the same course multiple times, these courses should be set up for resource sharing in Canvas.
  • The syllabi for each course are saved in my workspace. Please publish them for their respective courses.
  • All courses I am responsible for need to be published.
AVAILABLE MCP SERVERS

Filesystem, Canvas, Terminal, Emails.

Step 1 of 4
01

Select assigned courses and look up instructor IDs

PTC CALLPython
# data: parsed schedule rows (course, instructor, time, administrator).
# parse_tool_result: helper defined earlier to decode the tool response.
admin3_courses = [(c, i, t) for c, i, t, a in data if a == "admin3"]
instructor_map = {}
for name in set(i for _, i, _ in admin3_courses):
    result = tools["canvas_canvas_list_account_users"](
        account_id=account_id, search_term=name, page=1, per_page=20)
    users = parse_tool_result(result)
    if users:
        u = users[0] if isinstance(users, list) else users
        instructor_map[name] = u['id']
02

Create courses and use returned IDs to enroll instructors

PTC CALLPython
print(f"Processing {len(admin3_courses)} courses...")
created_courses = []
for idx, (course_name, instructor, class_time) in enumerate(admin3_courses):
    try:
        course = tools["canvas_canvas_create_course"](
            account_id=account_id, name=course_name,
            start_at=(date.today()).isoformat(),
            end_at=(date.today() + timedelta(days=365)).isoformat(),
            time_zone="America/Denver")
        course_id = course['id']
        tools["canvas_canvas_enroll_user"](
            course_id=course_id, user_id=instructor_map[instructor],
            role="TeacherEnrollment", enrollment_state="active")
        created_courses.append((course_name, course_id, instructor, class_time))
        print(f"[{idx+1}/{len(admin3_courses)}] Created {course_name} (ID={course_id}) for {instructor}")
    except Exception as e:
        print(f"ERROR {idx}: {e}")
print(f"\nCreated {len(created_courses)} courses.")
OUTPUTExecution result
Processing 27 courses...
[1/27] Created English Literature-7 (ID=31) for Melissa Sanchez
[2/27] Created Business Management-7 (ID=32) for Cynthia Gomez
[3/27] Created Philosophy-7 (ID=33) for Donald Reed
[4/27] Created Creative Writing-7 (ID=34) for Carolyn Nguyen
...
Created 27 courses.
03

Create announcements using course IDs

PTC CALLPython
# created_courses: 27 course tuples listed earlier in this call.
for course_name, course_id, instructor, class_time in created_courses:
    message = f"Welcome to {course_name}. This course is taught by {instructor} at {class_time}."
    try:
        tools["canvas_canvas_create_announcement"](
            course_id=course_id, title=f"{course_name} Class Info",
            message=message, published=True)
        print(f"Announcement for {course_name} (ID={course_id}): {message[:80]}...")
    except Exception as e:
        print(f"Announcement ERROR for {course_name}: {e}")
OUTPUTExecution result
Announcement for English Literature-7 (ID=31): Welcome to English Literature-7. This course is taught by Melissa Sanchez at Mon...
Announcement for Business Management-7 (ID=32): Welcome to Business Management-7. This course is taught by Cynthia Gomez at Wedn...
Announcement for Philosophy-7 (ID=33): Welcome to Philosophy-7. This course is taught by Donald Reed at Friday 1,2,3....
...
04

Check selected course settings and summarize results

The following program checks course publication, teacher count, the public-syllabus flag, and announcement presence. Its success message refers to these checks; it does not verify instructor identity, syllabus or announcement content, or resource sharing.

PTC CALLPython
# courses: 27 course-name/ID pairs (literal list omitted).
# instructor_map: Canvas instructor IDs to names (dictionary omitted).
print("=== FINAL COMPREHENSIVE VERIFICATION ===\n")

# Track instructor-to-course count
instructor_courses = {}
for cname, cid in courses:
    detail = tools["canvas_canvas_get_course"](course_id=cid)
    tid = detail['teachers'][0]['id'] if detail['teachers'] else None
    if tid:
        instructor_courses.setdefault(tid, []).append(cname)

# Verify
all_ok = True
failures = []

for course_name, course_id in courses:
    detail = tools["canvas_canvas_get_course"](course_id=course_id)
    state_ok = detail['workflow_state'] == 'available'
    teacher_ok = len(detail['teachers']) == 1
    # Check teacher is correct
    actual_teacher = detail['teachers'][0]['display_name'] if detail['teachers'] else 'NONE'

    syll_ok = detail['public_syllabus']
    anns = tools["canvas_canvas_list_announcements"](course_id=course_id)
    ann_ok = len(anns) >= 1

    if not state_ok:
        failures.append(f"{course_name}: state={detail['workflow_state']}")
    if not teacher_ok:
        failures.append(f"{course_name}: teachers count={len(detail['teachers'])}")
    if not syll_ok:
        failures.append(f"{course_name}: syllabus not public")
    if not ann_ok:
        failures.append(f"{course_name}: no announcements")

    print(f"{course_name}: state={detail['workflow_state']}, teachers={len(detail['teachers'])} ({actual_teacher}), syll={syll_ok}, anns={len(anns)}")

if failures:
    print(f"\n✗ {len(failures)} failures: {failures}")
else:
    print(f"\n✓ All {len(courses)} courses verified successfully!")

# Instructor summary
print("\nInstructor-Course Mapping:")
for tid, cnames in sorted(instructor_courses.items()):
    print(f"  {tid} ({instructor_map[tid]}): {len(cnames)} courses - {cnames}")

print(f"\nTotal unique instructors: {len(instructor_courses)}")
print(f"Total courses: {len(courses)}")
OUTPUTExecution result
=== FINAL COMPREHENSIVE VERIFICATION ===

English Literature-7: state=available, teachers=1 (Melissa), syll=True, anns=1
Business Management-7: state=available, teachers=1 (Cynthia), syll=True, anns=1
Philosophy-7: state=available, teachers=1 (Donald), syll=True, anns=1
Creative Writing-7: state=available, teachers=1 (Carolyn), syll=True, anns=1
...

✓ All 27 courses verified successfully!

Instructor-Course Mapping:
  62 (Dennis Robinson): 3 courses - ['Marketing Principles-7', 'Operations Management-7', 'Innovation Management-7']
  63 (Donald Reed): 3 courses - ['Philosophy-7', 'Optics-7', 'Business Strategy-7']
...

Total unique instructors: 11
Total courses: 27

BibTeX

@misc{li2026progagent,
  title = {ProgAgent: Learning Tool Orchestration through Programmatic Training},
  author = {Li, Hao and He, Keqing and Zheng, Longtao and Li, Junlong and Zeng, Weihao and Liu, Qian and Huang, Yuzhen},
  year = {2026},
  url = {https://llleoli.github.io/ProgAgent/}
}

Paper figure

Open original ↗
Enlarged paper figure

Use your browser’s zoom to inspect details.