Four-suite average
ProgAgentLearning Tool Orchestration
through Programmatic Training
A full PTC training stack for long-horizon tool orchestration.
Verifiable tasks, a lightweight PTC runtime, and a two-stage training recipe.
Verifiable Environment Synthesis
We construct verifiable tasks and scale their environment states from v₀ to v₃ while keeping instructions, task logic, and tool schemas fixed.
Experimental Results
We evaluate our models across MCPMark, MCP-Atlas, and Toolathlon-Verified, and further use LOCA-bench at 8k–64k environment-state sizes. Each reported pass rate averages four independent runs.
Main Results
Average over 8k–64k states
Verified task subset
Pre-v2.0 full task set
ProgAgent uses PTC evaluation; the baselines in the main results table use DTC. MCPMark excludes the GitHub suite. Toolathlon scores marked † are from the original Toolathlon benchmark; all other scores in that column are from Toolathlon-Verified. “–” denotes an unreported score.
On MCPMark, ProgAgent achieves average pass rates of 18.3% at 8B and 29.7% at 14B, respectively. On Toolathlon-Verified, ProgAgent reaches 9.3% at 8B and 17.1% at 14B, compared with at most 2.8% and 1.9% among the 8B and 14B baselines.
Moreover, on MCP-Atlas, ProgAgent-14B achieves 45.6%, comparable to Claude Sonnet-4.5 (43.8%) under the DTC harness. The gap is most pronounced on LOCA-bench: the evaluated comparable-size baselines score zero at 32k and 64k, while ProgAgent mitigates context rot to maintain pass rates between 17.3% and 30.7% at 64k.
Training-data Scales
The resulting 215 tasks are each instantiated at 4 scales, yielding a shared training pool of 860 task instances.
Training Paradigms across Benchmarks
The table below compares each training recipe with its untrained backbone under the same harness. For untrained backbones, switching to PTC provides limited benefits across benchmarks and often yields the lowest scores among the three harnesses. Training reverses this pattern: after SFT and RL, the PTC recipe achieves the highest aggregate score on each benchmark at both model scales.
ProgAgent-14B reaches 29.7% on MCPMark and 45.6% on MCP-Atlas, compared with 8.0% and 29.7% for DTC, respectively. ProgAgent-8B also outperforms the 8B DTC model on all four benchmarks.
The chart above compares the three training paradigms on LOCA-bench, showing gains from both SFT and RL. As the environment state scales, DTC suffers from context rot. In contrast, PTC effectively mitigates this degradation and sustains robust performance, achieving average pass rates of 24.3% (8B) and 38.3% (14B). Furthermore, despite providing a more flexible action space during both training and evaluation, the Hybrid paradigm performs worse than PTC.
Analysis
What Does PTC Change During Training?
At 14B, DTC achieves substantially lower rewards than PTC and Hybrid. During RL training, tool-output tokens under PTC decrease by roughly 20%, falling below 20k per episode; both programmatic paradigms use roughly half as many tool calls as DTC.
The 8B backbone reproduces the same trend: PTC and Hybrid achieve higher rewards with fewer tool-output tokens and tool calls than DTC.
Does Environment State Scaling Help?
Environment state scaling improves PTC performance on LOCA-bench across both backbones. Restricting training to v₀ reduces average pass rates from 24.3% to 16.0% at 8B and from 38.3% to 23.7% at 14B.
How Does PTC Facilitate Tool Orchestration?
PTC models use loops, conditional execution, and error handling to orchestrate tools during RL. PTC-trained models outperform DTC-trained models under both evaluation interfaces, with a 11.9-percentage-point gain on MCP-Atlas at 14B under DTC evaluation.
What Challenges Remain for PTC Agents?
DTC-like behavior: on Playwright, 40.7% of ProgAgent-14B’s PTC calls contain a single tool invocation without control flow, error handling, or selective printing. Silent server failures: disabling server-level error detection reduces LOCA-bench pass rates from 24.3% to 19.3% at 8B and from 38.3% to 34.5% at 14B.
Interaction costs of PTC on MCPMark +
Toolathlon-Verified: From an Emailed Schedule to Published Canvas Courses
This case shows how PTC uses loops and returned identifiers to coordinate dependent course-creation and configuration calls.
Instruction
This is admin3, a registrar from XXX University. The university has sent me the new semester's course schedule via email with the subject "Course Schedule Notification." I need you to help me find the courses I'm responsible for from the attached master schedule and create corresponding courses on Canvas for the instructors. The specific requirements are as follows:
- Assign the corresponding instructor as the teacher for each course on Canvas. The instructors' emails are saved in my local workspace.
- Publish an announcement for each course that includes the instructor's name and the class time.
- If an instructor is teaching the same course multiple times, these courses should be set up for resource sharing in Canvas.
- The syllabi for each course are saved in my workspace. Please publish them for their respective courses.
- All courses I am responsible for need to be published.
Filesystem, Canvas, Terminal, Emails.
Select assigned courses and look up instructor IDs
# data: parsed schedule rows (course, instructor, time, administrator).
# parse_tool_result: helper defined earlier to decode the tool response.
admin3_courses = [(c, i, t) for c, i, t, a in data if a == "admin3"]
instructor_map = {}
for name in set(i for _, i, _ in admin3_courses):
result = tools["canvas_canvas_list_account_users"](
account_id=account_id, search_term=name, page=1, per_page=20)
users = parse_tool_result(result)
if users:
u = users[0] if isinstance(users, list) else users
instructor_map[name] = u['id']Create courses and use returned IDs to enroll instructors
print(f"Processing {len(admin3_courses)} courses...")
created_courses = []
for idx, (course_name, instructor, class_time) in enumerate(admin3_courses):
try:
course = tools["canvas_canvas_create_course"](
account_id=account_id, name=course_name,
start_at=(date.today()).isoformat(),
end_at=(date.today() + timedelta(days=365)).isoformat(),
time_zone="America/Denver")
course_id = course['id']
tools["canvas_canvas_enroll_user"](
course_id=course_id, user_id=instructor_map[instructor],
role="TeacherEnrollment", enrollment_state="active")
created_courses.append((course_name, course_id, instructor, class_time))
print(f"[{idx+1}/{len(admin3_courses)}] Created {course_name} (ID={course_id}) for {instructor}")
except Exception as e:
print(f"ERROR {idx}: {e}")
print(f"\nCreated {len(created_courses)} courses.")Processing 27 courses...
[1/27] Created English Literature-7 (ID=31) for Melissa Sanchez
[2/27] Created Business Management-7 (ID=32) for Cynthia Gomez
[3/27] Created Philosophy-7 (ID=33) for Donald Reed
[4/27] Created Creative Writing-7 (ID=34) for Carolyn Nguyen
...
Created 27 courses.Create announcements using course IDs
# created_courses: 27 course tuples listed earlier in this call.
for course_name, course_id, instructor, class_time in created_courses:
message = f"Welcome to {course_name}. This course is taught by {instructor} at {class_time}."
try:
tools["canvas_canvas_create_announcement"](
course_id=course_id, title=f"{course_name} Class Info",
message=message, published=True)
print(f"Announcement for {course_name} (ID={course_id}): {message[:80]}...")
except Exception as e:
print(f"Announcement ERROR for {course_name}: {e}")Announcement for English Literature-7 (ID=31): Welcome to English Literature-7. This course is taught by Melissa Sanchez at Mon...
Announcement for Business Management-7 (ID=32): Welcome to Business Management-7. This course is taught by Cynthia Gomez at Wedn...
Announcement for Philosophy-7 (ID=33): Welcome to Philosophy-7. This course is taught by Donald Reed at Friday 1,2,3....
...Check selected course settings and summarize results
The following program checks course publication, teacher count, the public-syllabus flag, and announcement presence. Its success message refers to these checks; it does not verify instructor identity, syllabus or announcement content, or resource sharing.
# courses: 27 course-name/ID pairs (literal list omitted).
# instructor_map: Canvas instructor IDs to names (dictionary omitted).
print("=== FINAL COMPREHENSIVE VERIFICATION ===\n")
# Track instructor-to-course count
instructor_courses = {}
for cname, cid in courses:
detail = tools["canvas_canvas_get_course"](course_id=cid)
tid = detail['teachers'][0]['id'] if detail['teachers'] else None
if tid:
instructor_courses.setdefault(tid, []).append(cname)
# Verify
all_ok = True
failures = []
for course_name, course_id in courses:
detail = tools["canvas_canvas_get_course"](course_id=course_id)
state_ok = detail['workflow_state'] == 'available'
teacher_ok = len(detail['teachers']) == 1
# Check teacher is correct
actual_teacher = detail['teachers'][0]['display_name'] if detail['teachers'] else 'NONE'
syll_ok = detail['public_syllabus']
anns = tools["canvas_canvas_list_announcements"](course_id=course_id)
ann_ok = len(anns) >= 1
if not state_ok:
failures.append(f"{course_name}: state={detail['workflow_state']}")
if not teacher_ok:
failures.append(f"{course_name}: teachers count={len(detail['teachers'])}")
if not syll_ok:
failures.append(f"{course_name}: syllabus not public")
if not ann_ok:
failures.append(f"{course_name}: no announcements")
print(f"{course_name}: state={detail['workflow_state']}, teachers={len(detail['teachers'])} ({actual_teacher}), syll={syll_ok}, anns={len(anns)}")
if failures:
print(f"\n✗ {len(failures)} failures: {failures}")
else:
print(f"\n✓ All {len(courses)} courses verified successfully!")
# Instructor summary
print("\nInstructor-Course Mapping:")
for tid, cnames in sorted(instructor_courses.items()):
print(f" {tid} ({instructor_map[tid]}): {len(cnames)} courses - {cnames}")
print(f"\nTotal unique instructors: {len(instructor_courses)}")
print(f"Total courses: {len(courses)}")=== FINAL COMPREHENSIVE VERIFICATION ===
English Literature-7: state=available, teachers=1 (Melissa), syll=True, anns=1
Business Management-7: state=available, teachers=1 (Cynthia), syll=True, anns=1
Philosophy-7: state=available, teachers=1 (Donald), syll=True, anns=1
Creative Writing-7: state=available, teachers=1 (Carolyn), syll=True, anns=1
...
✓ All 27 courses verified successfully!
Instructor-Course Mapping:
62 (Dennis Robinson): 3 courses - ['Marketing Principles-7', 'Operations Management-7', 'Innovation Management-7']
63 (Donald Reed): 3 courses - ['Philosophy-7', 'Optics-7', 'Business Strategy-7']
...
Total unique instructors: 11
Total courses: 27BibTeX
@misc{li2026progagent,
title = {ProgAgent: Learning Tool Orchestration through Programmatic Training},
author = {Li, Hao and He, Keqing and Zheng, Longtao and Li, Junlong and Zeng, Weihao and Liu, Qian and Huang, Yuzhen},
year = {2026},
url = {https://llleoli.github.io/ProgAgent/}
}