{"schemaVersion":7,"rubricVersion":"0.3","canonicalUrl":"https://podium.do/compare/agents/rubric/","markdownUrl":"https://podium.do/compare/agents/rubric/index.md","dataUrl":"https://podium.do/compare/agents/rubric/data.json","generatedAt":"2026-11-22","publisher":{"name":"Podium","url":"https://podium.do","marketParticipant":true,"editorialControl":"Podium selects the products, rubric, sources and wording, including for comparisons that do not include Podium, and may add labeled Podium team research without linked sources.","disclosure":"Podium publishes this catalog and sells an agent development environment. Podium chooses the products, criteria, sources and wording, so treat these pages as vendor-authored research. Check important claims against the linked primary sources. Sourced findings follow the same evidence rules for Podium and its competitors. Facts labeled Podium team research come from Podium's own team and have no linked source. None of this removes the conflict of interest."},"attribution":{"dataSource":{"name":"adecomparison.com by Podium","url":"https://adecomparison.com","repository":"https://github.com/madeinorbit/ade-comparison"},"license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"dataLicenseUrl":"https://github.com/madeinorbit/ade-comparison/blob/4322f0c035f303791413639d1462a138443d8c1e/DATA-LICENSE.md","snapshot":{"commit":"4322f0c035f303791413639d1462a138443d8c1e","commitDate":"2026-11-22","url":"https://github.com/madeinorbit/ade-comparison/tree/4322f0c035f303791413639d1462a138443d8c1e"},"podiumTeamLabel":"Podium team research","podiumTeamText":"Podium team research is a fact the Podium team added without a linked source. It is labeled wherever it appears, and any differing adecomparison.com finding stays visible beside it.","statement":"Based on data from adecomparison.com by Podium (https://adecomparison.com; source repository https://github.com/madeinorbit/ade-comparison, snapshot 4322f0c of 2026-11-22), used under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Source links and check dates are kept. Podium arranges the findings into pairs, applies its publication checks, and added its own research where marked \"Podium team research\"."},"criteria":[{"id":"ACC-01","title":"Client access","group":{"id":1,"name":"Access and agent choice"},"judgment":"List desktop OS and minimum versions, iOS, Android, browser, mobile browser, CLI and editor extensions separately. For each client, record start, steer, approve, review and resume actions, notifications and plan restrictions. Distinguish an installable mobile app from a mobile website and from status-only monitoring.","subquestions":[{"id":"desktop_os","question":"Desktop os"},{"id":"mobile_apps","question":"Mobile apps"},{"id":"browser","question":"Browser"},{"id":"mobile_browser","question":"Mobile browser"},{"id":"cli","question":"Cli"},{"id":"editor_extensions","question":"Editor extensions"},{"id":"task_actions","question":"Task actions"},{"id":"notifications","question":"Notifications"},{"id":"cross_device_continuity","question":"Cross device continuity"}]},{"id":"ACC-02","title":"Agent and model choice","group":{"id":1,"name":"Access and agent choice"},"judgment":"List supported harnesses and model providers separately, including custom adapters. Record whether the product embeds a harness, controls its sessions through an API, or only launches its terminal command. Describe mixed-harness operation only when sources establish at least two different harnesses.","subquestions":[{"id":"harnesses","question":"Harnesses"},{"id":"model_providers","question":"Model providers"},{"id":"custom_adapters","question":"Custom adapters"},{"id":"session_integration","question":"Session integration"},{"id":"mixed_harness_operation","question":"Mixed harness operation"}]},{"id":"ACC-03","title":"Subscriptions, API keys and vendor credits","group":{"id":1,"name":"Access and agent choice"},"judgment":"Record bring-your-own agent subscription, bring-your-own API account, vendor credits and local models as separate routes. List supported plans, models/modes, multiple-account switching and what works without a vendor account or token purchase. Existing subscription support cannot be inferred from API-key support. Record required extra payments and documented provider restrictions.","subquestions":[{"id":"agent_subscriptions","question":"Agent subscriptions"},{"id":"api_accounts","question":"Api accounts"},{"id":"vendor_credits","question":"Vendor credits"},{"id":"local_models","question":"Local models"},{"id":"account_switching","question":"Account switching"},{"id":"vendor_account_requirement","question":"Vendor account requirement"},{"id":"provider_restrictions","question":"Provider restrictions"}]},{"id":"ACC-04","title":"Onboarding and project prerequisites","group":{"id":1,"name":"Access and agent choice"},"judgment":"Record the documented path from fresh install to a first task, with its manual steps. Record support for a local directory without Git, a local Git repo without a remote, and a GitHub-backed repo separately. Record the documented account, project, host, credential and permission prerequisites. Record managed-cloud setup and first-class BYO VPS installer/onboarding independently of hidden settings or a DIY shell recipe.","subquestions":[{"id":"first_task_setup","question":"First task setup"},{"id":"no_git","question":"No git"},{"id":"local_git_without_remote","question":"Local git without remote"},{"id":"hosted_git","question":"Hosted git"},{"id":"prerequisite_guidance","question":"Prerequisite guidance"},{"id":"managed_cloud_setup","question":"Managed cloud setup"},{"id":"byo_vps_onboarding","question":"Byo vps onboarding"}]},{"id":"ACC-05","title":"Programmable access","group":{"id":1,"name":"Access and agent choice"},"judgment":"Inventory actual CLI, API, SDK, MCP and webhook operations for create, inspect, steer, stop and review. Record authentication and API stability. Having an MCP client that uses external tools is different from exposing the product to other agents over MCP.","subquestions":[{"id":"cli","question":"Cli"},{"id":"api","question":"Api"},{"id":"sdk","question":"Sdk"},{"id":"mcp_server","question":"Mcp server"},{"id":"webhooks","question":"Webhooks"},{"id":"operations","question":"Operations"},{"id":"authentication","question":"Authentication"},{"id":"stability","question":"Stability"}]},{"id":"ACC-06","title":"External tools and extensions","group":{"id":1,"name":"Access and agent choice"},"judgment":"Record MCP client support, plugins, custom tools, skill/instruction import, integration authentication and permission scope. Name which component provides each connection and whether setup is product-managed or inherited from the underlying harness.","subquestions":[{"id":"mcp_client","question":"Mcp client"},{"id":"plugins","question":"Plugins"},{"id":"custom_tools","question":"Custom tools"},{"id":"instructions_import","question":"Instructions import"},{"id":"authentication","question":"Authentication"},{"id":"permission_scope","question":"Permission scope"},{"id":"integration_owner","question":"Integration owner"}]},{"id":"ACC-07","title":"Client implementation","group":{"id":1,"name":"Access and agent choice"},"judgment":"For each desktop/mobile client, record native platform UI, Electron, Tauri, React Native, web/PWA or unknown, plus hybrids. Require official technical docs or pinned source/build evidence. Tauri is not synonymous with fully native UI. Implementation choice says nothing about responsiveness or usability.","subquestions":[{"id":"desktop_implementation","question":"Desktop implementation"},{"id":"mobile_implementation","question":"Mobile implementation"},{"id":"hybrids","question":"Hybrids"},{"id":"implementation_evidence","question":"Implementation evidence"}]},{"id":"PLAN-01","title":"Intake and external tracker relationship","group":{"id":2,"name":"Planning and context"},"judgment":"Record local creation, prompt intake, issue import and chat/Git triggers. Identify the authoritative tracker, stable issue IDs, one-way or two-way sync, supported edits and operations requiring a trip back to Linear/GitHub/Jira. Importing or attaching a ticket is not native issue management. Answer PLAN-06 and PLAN-07 separately.","subquestions":[{"id":"local_creation","question":"Local creation"},{"id":"prompt_intake","question":"Prompt intake"},{"id":"external_import","question":"External import"},{"id":"chat_git_triggers","question":"Chat git triggers"},{"id":"authoritative_tracker","question":"Authoritative tracker"},{"id":"stable_ids","question":"Stable ids"},{"id":"sync","question":"Sync"},{"id":"supported_edits","question":"Supported edits"},{"id":"external_roundtrips","question":"External roundtrips"}]},{"id":"PLAN-02","title":"Durable task plans","group":{"id":2,"name":"Planning and context"},"judgment":"Record task plans and acceptance criteria, editors, revision history and links to implementation. Record whether a plan can be amended while work is running and revisited later. A task-specific starting brief, reusable memory and a persistent product specification are different objects; product specs are judged in PLAN-08.","subquestions":[{"id":"plans","question":"Plans"},{"id":"acceptance_criteria","question":"Acceptance criteria"},{"id":"editors","question":"Editors"},{"id":"revisions","question":"Revisions"},{"id":"implementation_links","question":"Implementation links"},{"id":"amendment","question":"Amendment"},{"id":"restart_persistence","question":"Restart persistence"}]},{"id":"PLAN-03","title":"Hierarchy, dependencies and evolving scope","group":{"id":2,"name":"Planning and context"},"judgment":"Record epics, arbitrary parent/child depth, blockers, readiness, dependency enforcement and state propagation. Record whether a simple task can grow into an epic while keeping its identity, discussion, history and outputs, and whether blockers prevent dispatch or are only displayed. Record whether a rigid preselected workflow is required, and whether changing scope requires recreating work.","subquestions":[{"id":"epics","question":"Epics"},{"id":"hierarchy","question":"Hierarchy"},{"id":"blockers","question":"Blockers"},{"id":"readiness","question":"Readiness"},{"id":"enforcement","question":"Enforcement"},{"id":"state_propagation","question":"State propagation"},{"id":"task_to_epic","question":"Task to epic"},{"id":"identity_retention","question":"Identity retention"},{"id":"workflow_constraints","question":"Workflow constraints"}]},{"id":"PLAN-04","title":"Shared context and memory","group":{"id":2,"name":"Planning and context"},"judgment":"List repository instructions, skills, project docs and cross-session/cross-harness memory. Record scope, provenance, citation, correction/deletion and stale-context controls. Record what survives a new session or colleague takeover. A notes page or shared file is not automatically a maintained spec or managed memory.","subquestions":[{"id":"instructions","question":"Instructions"},{"id":"skills","question":"Skills"},{"id":"project_docs","question":"Project docs"},{"id":"memory","question":"Memory"},{"id":"scope","question":"Scope"},{"id":"provenance","question":"Provenance"},{"id":"correction","question":"Correction"},{"id":"deletion","question":"Deletion"},{"id":"freshness","question":"Freshness"},{"id":"takeover_continuity","question":"Takeover continuity"}]},{"id":"PLAN-05","title":"Architecture and API awareness","group":{"id":2,"name":"Planning and context"},"judgment":"Record code search, repository indexing, external knowledge and multimodal inputs, with source freshness and access scope. Record retrieval, citation, conflict detection and review-time enforcement separately. A large context window or an instruction file alone does not prove architectural consistency.","subquestions":[{"id":"code_search","question":"Code search"},{"id":"indexing","question":"Indexing"},{"id":"external_knowledge","question":"External knowledge"},{"id":"multimodal_inputs","question":"Multimodal inputs"},{"id":"freshness","question":"Freshness"},{"id":"access_scope","question":"Access scope"},{"id":"api_retrieval","question":"Api retrieval"},{"id":"citations","question":"Citations"},{"id":"conflict_detection","question":"Conflict detection"},{"id":"review_enforcement","question":"Review enforcement"}]},{"id":"PLAN-06","title":"Native issue management and idea backlog","group":{"id":2,"name":"Planning and context"},"judgment":"Enumerate create, edit, discuss, prioritize, label, assign, search, filter, archive, reopen and project/epic views within the product. Record whether an idea can be saved without starting an agent, refined with another person, and marked ready or dispatched later. Record separate task/session entities, metadata depth, multiple sessions per task, durable state transitions and whether an external tracker remains necessary.","subquestions":[{"id":"create","question":"Create"},{"id":"edit","question":"Edit"},{"id":"discuss","question":"Discuss"},{"id":"prioritize","question":"Prioritize"},{"id":"label","question":"Label"},{"id":"assign","question":"Assign"},{"id":"search","question":"Search"},{"id":"filter","question":"Filter"},{"id":"archive","question":"Archive"},{"id":"reopen","question":"Reopen"},{"id":"project_epic_views","question":"Project epic views"},{"id":"idea_parking","question":"Idea parking"},{"id":"explicit_dispatch","question":"Explicit dispatch"},{"id":"task_session_separation","question":"Task session separation"},{"id":"metadata","question":"Metadata"},{"id":"durable_states","question":"Durable states"},{"id":"external_tracker_requirement","question":"External tracker requirement"}]},{"id":"PLAN-07","title":"Agent-native issue operations","group":{"id":2,"name":"Planning and context"},"judgment":"Record whether an agent can read/create/update the same durable issue IDs and fields humans use, claim/assign work, change state, create dependencies/children and report blockers/results through documented CLI/API/MCP. Record whether human views reflect agent writes with attribution, permissions and conflict handling. Distinguish shared authoritative records, synchronized external records, agent-private metadata and prompt-only conventions.","subquestions":[{"id":"shared_ids","question":"Shared ids"},{"id":"read","question":"Read"},{"id":"create","question":"Create"},{"id":"update","question":"Update"},{"id":"claim","question":"Claim"},{"id":"assign","question":"Assign"},{"id":"state_changes","question":"State changes"},{"id":"dependencies","question":"Dependencies"},{"id":"children","question":"Children"},{"id":"results","question":"Results"},{"id":"human_visibility","question":"Human visibility"},{"id":"attribution","question":"Attribution"},{"id":"permissions","question":"Permissions"},{"id":"conflicts","question":"Conflicts"}]},{"id":"PLAN-08","title":"Living product specifications","group":{"id":2,"name":"Planning and context"},"judgment":"Record persistent requirements across tasks/releases, editable by humans and agents with attributable revisions and approval rules. Record requirement-to-task/implementation links, retrieval during later tasks, drift detection and consideration during review/quality control. A one-time spec-to-plan workflow, per-task brief, memory store or unreferenced document does not establish a living product spec.","subquestions":[{"id":"persistent_requirements","question":"Persistent requirements"},{"id":"human_edits","question":"Human edits"},{"id":"agent_edits","question":"Agent edits"},{"id":"revisions","question":"Revisions"},{"id":"approvals","question":"Approvals"},{"id":"implementation_links","question":"Implementation links"},{"id":"later_retrieval","question":"Later retrieval"},{"id":"drift_detection","question":"Drift detection"},{"id":"review_integration","question":"Review integration"}]},{"id":"COORD-01","title":"Parallel work","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record independently running sessions and tasks, documented quotas and concurrency limits. Describe how a user sees running, waiting, failed and finished work. Several chat tabs are not proof that runs execute concurrently.","subquestions":[{"id":"concurrent_sessions","question":"Concurrent sessions"},{"id":"concurrent_tasks","question":"Concurrent tasks"},{"id":"quotas","question":"Quotas"},{"id":"observed_concurrency","question":"Observed concurrency"},{"id":"status_visibility","question":"Status visibility"}]},{"id":"COORD-02","title":"Ownership and dispatch","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record assignment to humans or agents, claims, queues, routing and duplicate-work prevention. Distinguish a display label, an advisory lock and enforced exclusive ownership.","subquestions":[{"id":"human_assignment","question":"Human assignment"},{"id":"agent_assignment","question":"Agent assignment"},{"id":"claims","question":"Claims"},{"id":"queues","question":"Queues"},{"id":"routing","question":"Routing"},{"id":"duplicate_prevention","question":"Duplicate prevention"},{"id":"ownership_enforcement","question":"Ownership enforcement"}]},{"id":"COORD-03","title":"Inspectable delegation and communication","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record parent/child roles, model/harness/effort choice, messages, result return and cross-harness teams. Distinguish product orchestration, provider-native subagents, installed orchestration skills and scripts. Record activation, auth and experimental flags. Record child identity, live progress, retained results and ability to reopen or follow up after completion.","subquestions":[{"id":"roles","question":"Roles"},{"id":"model_harness_effort","question":"Model harness effort"},{"id":"messages","question":"Messages"},{"id":"results_return","question":"Results return"},{"id":"cross_harness","question":"Cross harness"},{"id":"orchestration_owner","question":"Orchestration owner"},{"id":"activation","question":"Activation"},{"id":"authentication","question":"Authentication"},{"id":"live_progress","question":"Live progress"},{"id":"retained_results","question":"Retained results"},{"id":"followup","question":"Followup"},{"id":"invocation","question":"Invocation"}]},{"id":"COORD-04","title":"Unattended triggers","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record schedules, event triggers, recurring tasks and completion conditions. Name where the scheduler runs, overlap handling, retries and whether closing the client stops it. A desktop cron recipe is a manual integration unless the product owns it.","subquestions":[{"id":"schedules","question":"Schedules"},{"id":"events","question":"Events"},{"id":"recurring_tasks","question":"Recurring tasks"},{"id":"completion_conditions","question":"Completion conditions"},{"id":"scheduler_location","question":"Scheduler location"},{"id":"overlaps","question":"Overlaps"},{"id":"retries","question":"Retries"},{"id":"client_independence","question":"Client independence"}]},{"id":"COORD-05","title":"Intervention and attention","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record pause, stop, resume, steering, notifications, pending decisions and escalation. Record documented behavior for silent waits, CLI-only questions and whether stop terminates execution while preserving history.","subquestions":[{"id":"pause","question":"Pause"},{"id":"stop","question":"Stop"},{"id":"resume","question":"Resume"},{"id":"steer","question":"Steer"},{"id":"notifications","question":"Notifications"},{"id":"pending_decisions","question":"Pending decisions"},{"id":"escalation","question":"Escalation"},{"id":"visible_errors","question":"Visible errors"},{"id":"chat_terminal_parity","question":"Chat terminal parity"},{"id":"retained_history","question":"Retained history"}]},{"id":"COORD-06","title":"Fleet and epic management UI","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record how the interface shows work across several epics, including blocked, failed, awaiting-input, reviewing and completed states. Record whether parent/child relationships, dependencies, attention needs, history and workload are visible without reading every transcript. Boards, graphs, waterfalls, timelines or another interface can qualify; no required visual form.","subquestions":[{"id":"fleet_fixture","question":"Fleet fixture"},{"id":"parent_child","question":"Parent child"},{"id":"dependencies","question":"Dependencies"},{"id":"attention","question":"Attention"},{"id":"history","question":"History"},{"id":"workload","question":"Workload"},{"id":"navigation_errors","question":"Navigation errors"},{"id":"fixture_mode","question":"Fixture mode"}]},{"id":"COORD-07","title":"Workflow flexibility and orchestration visibility","group":{"id":3,"name":"Coordination and automation"},"judgment":"Record mandatory workflow/preset selection, natural-language configuration, per-agent model/effort control, changes during execution and whether the UI exposes what actually ran, including coordinator and child roles, relationships and results during work and after completion. Record discoverability and auth requirements.","subquestions":[{"id":"quick_task","question":"Quick task"},{"id":"growing_epic","question":"Growing epic"},{"id":"presets","question":"Presets"},{"id":"natural_language","question":"Natural language"},{"id":"agent_configuration","question":"Agent configuration"},{"id":"in_run_changes","question":"In run changes"},{"id":"live_roles","question":"Live roles"},{"id":"retained_relationships","question":"Retained relationships"},{"id":"discoverability","question":"Discoverability"},{"id":"auth_friction","question":"Auth friction"}]},{"id":"RUN-01","title":"Laptop-independent execution","group":{"id":4,"name":"Execution and reliability"},"judgment":"List local, user-owned VPS/remote host, self-hosted service, vendor cloud and hybrid modes. Record provisioning, payer, compute resources, region, idle/max lifetime and disconnection rules. Record whether work continues after the originating client disconnects and can be continued from another device. A hosted UI or keep-awake setting is not evidence of remote execution. Record managed-cloud convenience separately from BYO setup effort in ACC-04.","subquestions":[{"id":"local","question":"Local"},{"id":"byo_remote","question":"Byo remote"},{"id":"self_hosted","question":"Self hosted"},{"id":"managed_cloud","question":"Managed cloud"},{"id":"hybrid","question":"Hybrid"},{"id":"provisioning","question":"Provisioning"},{"id":"payer","question":"Payer"},{"id":"resources","question":"Resources"},{"id":"region","question":"Region"},{"id":"expiry","question":"Expiry"},{"id":"client_disconnect","question":"Client disconnect"},{"id":"other_device_continuation","question":"Other device continuation"}]},{"id":"RUN-02","title":"Workspace and process isolation","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record shared checkout, branch, worktree, container, VM and OS sandbox separately. Name filesystem, process, network and secret boundaries. Do not claim hostile-code containment without separate security evidence.","subquestions":[{"id":"checkout","question":"Checkout"},{"id":"branch","question":"Branch"},{"id":"worktree","question":"Worktree"},{"id":"container","question":"Container"},{"id":"vm","question":"Vm"},{"id":"os_sandbox","question":"Os sandbox"},{"id":"filesystem","question":"Filesystem"},{"id":"process","question":"Process"},{"id":"network","question":"Network"},{"id":"secrets","question":"Secrets"},{"id":"code_separation","question":"Code separation"},{"id":"security_boundary","question":"Security boundary"}]},{"id":"RUN-03","title":"Recovery, expiry and continuity","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record behavior on client disconnect, worker crash, service restart and documented sandbox expiry. Separate transcript reload, process reconnection, task resumption and state reconstruction. Record lost context, duplicated actions, manual repair and ability to continue a multi-day epic after compute replacement. Short sandbox lifetimes are reported with their continuity behavior, not automatically treated as failure.","subquestions":[{"id":"client_disconnect","question":"Client disconnect"},{"id":"worker_crash","question":"Worker crash"},{"id":"service_restart","question":"Service restart"},{"id":"expiry","question":"Expiry"},{"id":"transcript_reload","question":"Transcript reload"},{"id":"process_reconnect","question":"Process reconnect"},{"id":"task_resume","question":"Task resume"},{"id":"state_reconstruction","question":"State reconstruction"},{"id":"context_loss","question":"Context loss"},{"id":"duplicate_actions","question":"Duplicate actions"},{"id":"manual_repair","question":"Manual repair"},{"id":"multi_day_continuity","question":"Multi day continuity"}]},{"id":"RUN-04","title":"Development environment reliability","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record dependency setup, node_modules/cache handling, snapshots, services, port allocation and secret injection. Record how two workspaces with the same service requirements avoid installation duplication and port collisions. Record which problems the product handles automatically and which the user must diagnose.","subquestions":[{"id":"dependencies","question":"Dependencies"},{"id":"caches","question":"Caches"},{"id":"snapshots","question":"Snapshots"},{"id":"services","question":"Services"},{"id":"ports","question":"Ports"},{"id":"secrets","question":"Secrets"},{"id":"duplicate_installs","question":"Duplicate installs"},{"id":"collisions","question":"Collisions"},{"id":"remote_setup","question":"Remote setup"},{"id":"automatic_repair","question":"Automatic repair"}]},{"id":"RUN-05","title":"Limits and failure handling","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record time, turn, token and spend limits; timeout, retry and provider-failure behavior. Record whether limits are warnings, enforced stops, or external billing controls.","subquestions":[{"id":"time_limits","question":"Time limits"},{"id":"turn_limits","question":"Turn limits"},{"id":"token_limits","question":"Token limits"},{"id":"spend_limits","question":"Spend limits"},{"id":"timeouts","question":"Timeouts"},{"id":"retries","question":"Retries"},{"id":"provider_failures","question":"Provider failures"},{"id":"enforcement","question":"Enforcement"}]},{"id":"RUN-06","title":"Truthful status and operational statistics","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record logs, traces, failure reasons, usage exports and statistics by task, agent, project and time. Record how agent, task and board state are kept consistent, including after reconnect. Distinguish working, blocked, failed, idle, ready for review and delivered. Attribute speed and success-rate figures to their source; this catalog measures none.","subquestions":[{"id":"logs","question":"Logs"},{"id":"traces","question":"Traces"},{"id":"failure_reasons","question":"Failure reasons"},{"id":"usage_export","question":"Usage export"},{"id":"task_stats","question":"Task stats"},{"id":"agent_stats","question":"Agent stats"},{"id":"project_stats","question":"Project stats"},{"id":"time_stats","question":"Time stats"},{"id":"failure_status","question":"Failure status"},{"id":"approval_status","question":"Approval status"},{"id":"reconnect_consistency","question":"Reconnect consistency"},{"id":"performance_method","question":"Performance method"},{"id":"responsiveness","question":"Responsiveness"}]},{"id":"RUN-07","title":"Per-task API and compute cost","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record visible model/API spend per task, session, child and epic, including aggregation across harness changes. Distinguish billed charges, estimates, tokens, subscription allowance and unavailable amounts; never render unknown subscription marginal cost as zero. Record compute fees separately, refresh delay, budget alerts, export and drill-down from summary statistics.","subquestions":[{"id":"task_cost","question":"Task cost"},{"id":"session_cost","question":"Session cost"},{"id":"child_cost","question":"Child cost"},{"id":"epic_cost","question":"Epic cost"},{"id":"harness_aggregation","question":"Harness aggregation"},{"id":"billed_vs_estimated","question":"Billed vs estimated"},{"id":"tokens","question":"Tokens"},{"id":"subscription_allowance","question":"Subscription allowance"},{"id":"provider_reconciliation","question":"Provider reconciliation"},{"id":"compute_cost","question":"Compute cost"},{"id":"refresh_delay","question":"Refresh delay"},{"id":"alerts","question":"Alerts"},{"id":"export","question":"Export"},{"id":"drilldown","question":"Drilldown"}]},{"id":"RUN-08","title":"Compute-aware scheduling and verification","group":{"id":4,"name":"Execution and reliability"},"judgment":"Record concurrency limits, queues, CPU/RAM limits, backpressure, per-host resource awareness and reuse/deduplication of checks. Record whether excess work queues or throttles, plus cancellation, fairness, recovery and whether overlapping epics share limits.","subquestions":[{"id":"concurrency","question":"Concurrency"},{"id":"queues","question":"Queues"},{"id":"cpu_ram","question":"Cpu ram"},{"id":"backpressure","question":"Backpressure"},{"id":"host_awareness","question":"Host awareness"},{"id":"check_reuse","question":"Check reuse"},{"id":"check_deduplication","question":"Check deduplication"},{"id":"bounded_tests","question":"Bounded tests"},{"id":"cancellation","question":"Cancellation"},{"id":"fairness","question":"Fairness"},{"id":"recovery","question":"Recovery"},{"id":"shared_limits","question":"Shared limits"}]},{"id":"TEAM-01","title":"Live human collaboration","group":{"id":5,"name":"Human collaboration and control"},"judgment":"Record shared visibility, issue discussion, comments, presence and simultaneous co-prompting as independent capabilities. For co-prompting, record input ordering/conflict handling, attribution and permission controls. Read-only links and shared issues do not establish co-steering. Record team/free-tier restrictions and whether the original owner must stay online.","subquestions":[{"id":"visibility","question":"Visibility"},{"id":"discussion","question":"Discussion"},{"id":"comments","question":"Comments"},{"id":"presence","question":"Presence"},{"id":"coprompting","question":"Coprompting"},{"id":"ordering","question":"Ordering"},{"id":"conflicts","question":"Conflicts"},{"id":"attribution","question":"Attribution"},{"id":"permissions","question":"Permissions"},{"id":"plan_restrictions","question":"Plan restrictions"},{"id":"owner_online","question":"Owner online"}]},{"id":"TEAM-02","title":"Roles and access boundaries","group":{"id":5,"name":"Human collaboration and control"},"judgment":"Record organization/project/task roles, membership lifecycle, SSO and provisioning where offered. Record which plan provides each control and whether enforcement is documented or visible in source.","subquestions":[{"id":"organization_roles","question":"Organization roles"},{"id":"project_roles","question":"Project roles"},{"id":"task_roles","question":"Task roles"},{"id":"membership","question":"Membership"},{"id":"sso","question":"Sso"},{"id":"provisioning","question":"Provisioning"},{"id":"unauthorized_access","question":"Unauthorized access"},{"id":"revocation","question":"Revocation"},{"id":"plan_scope","question":"Plan scope"}]},{"id":"TEAM-03","title":"Approvals and temporary access","group":{"id":5,"name":"Human collaboration and control"},"judgment":"Record command, tool, network, secret and delivery approvals, approver identity and scope lifetime. Record allow, deny and revocation, including what happens to temporary tunnels or shared access after their thread/session ends. Separate default full access, available safer modes and product-enforced policy from instructions the model is expected to follow.","subquestions":[{"id":"command","question":"Command"},{"id":"tool","question":"Tool"},{"id":"network","question":"Network"},{"id":"secret","question":"Secret"},{"id":"delivery","question":"Delivery"},{"id":"approver_identity","question":"Approver identity"},{"id":"lifetime","question":"Lifetime"},{"id":"allow","question":"Allow"},{"id":"deny","question":"Deny"},{"id":"revoke","question":"Revoke"},{"id":"tunnels","question":"Tunnels"},{"id":"defaults","question":"Defaults"},{"id":"safer_modes","question":"Safer modes"},{"id":"enforcement","question":"Enforcement"}]},{"id":"TEAM-04","title":"Decision and activity history","group":{"id":5,"name":"Human collaboration and control"},"judgment":"Record attributed actions, approvals, edits, discussion and decision links. Record whether history persists across users and sessions, can be exported, and can be altered. Use the word audit only with a defined retention and integrity model.","subquestions":[{"id":"actions","question":"Actions"},{"id":"approvals","question":"Approvals"},{"id":"edits","question":"Edits"},{"id":"discussion","question":"Discussion"},{"id":"decisions","question":"Decisions"},{"id":"cross_user_persistence","question":"Cross user persistence"},{"id":"cross_session_persistence","question":"Cross session persistence"},{"id":"export","question":"Export"},{"id":"mutability","question":"Mutability"},{"id":"retention","question":"Retention"},{"id":"integrity","question":"Integrity"}]},{"id":"TEAM-05","title":"Review experience and guidance","group":{"id":5,"name":"Human collaboration and control"},"judgment":"Record how a reviewer locates pending reviews, understands a change, navigates evidence, comments, requests changes and finds the next shipping action. Record documented keyboard and narrow-screen support, clarity of agent roles and preservation of review context. Generic diff/PR access and a guided review workflow are different capabilities. Link high-volume review findings to SHIP-06.","subquestions":[{"id":"pending_reviews","question":"Pending reviews"},{"id":"change_understanding","question":"Change understanding"},{"id":"evidence_navigation","question":"Evidence navigation"},{"id":"comments","question":"Comments"},{"id":"request_changes","question":"Request changes"},{"id":"shipping_guidance","question":"Shipping guidance"},{"id":"effort","question":"Effort"},{"id":"missed_evidence","question":"Missed evidence"},{"id":"keyboard","question":"Keyboard"},{"id":"narrow_screen","question":"Narrow screen"},{"id":"context_switching","question":"Context switching"},{"id":"roles","question":"Roles"},{"id":"retained_context","question":"Retained context"}]},{"id":"TEAM-06","title":"Colleague takeover and continuity","group":{"id":5,"name":"Human collaboration and control"},"judgment":"Record whether a second authorized person can take over ongoing work with the same task, session history, decisions, artifacts, permissions and runtime context, including after the first person disconnects. Record ownership transfer, attribution and lost context. Separately record transferring to another model/harness; a forked transcript or shared link alone is not full human takeover.","subquestions":[{"id":"ongoing_takeover","question":"Ongoing takeover"},{"id":"same_task","question":"Same task"},{"id":"session_history","question":"Session history"},{"id":"decisions","question":"Decisions"},{"id":"artifacts","question":"Artifacts"},{"id":"permissions","question":"Permissions"},{"id":"runtime_context","question":"Runtime context"},{"id":"owner_disconnect","question":"Owner disconnect"},{"id":"ownership","question":"Ownership"},{"id":"attribution","question":"Attribution"},{"id":"context_loss","question":"Context loss"},{"id":"harness_transfer","question":"Harness transfer"}]},{"id":"SHIP-01","title":"Retained artifacts and outputs","group":{"id":6,"name":"Verification and delivery"},"judgment":"Record diffs, documents, screenshots, recordings, previews and test/reviewer outputs. Record a searchable/browsable artifact gallery or equivalent retrieval, linked to task and exact revision, usable after sessions end, including a completed child reviewer's full findings. In-chat attachments, persistent review evidence and a cross-task gallery get distinct answers.","subquestions":[{"id":"diffs","question":"Diffs"},{"id":"documents","question":"Documents"},{"id":"screenshots","question":"Screenshots"},{"id":"recordings","question":"Recordings"},{"id":"previews","question":"Previews"},{"id":"test_outputs","question":"Test outputs"},{"id":"reviewer_outputs","question":"Reviewer outputs"},{"id":"gallery","question":"Gallery"},{"id":"task_links","question":"Task links"},{"id":"revision_links","question":"Revision links"},{"id":"persistence","question":"Persistence"},{"id":"child_findings","question":"Child findings"},{"id":"prompt_leakage","question":"Prompt leakage"}]},{"id":"SHIP-02","title":"Verification and spec gates","group":{"id":6,"name":"Verification and delivery"},"judgment":"Record tests, lint, types, CI, independent review and persistent-spec conformance checks as required versus optional. Record whether a failed check or a conflict with an existing product requirement blocks delivery. An agent saying checks passed is not a passing gate; record configured and enforced policy separately.","subquestions":[{"id":"tests","question":"Tests"},{"id":"lint","question":"Lint"},{"id":"types","question":"Types"},{"id":"ci","question":"Ci"},{"id":"independent_review","question":"Independent review"},{"id":"spec_checks","question":"Spec checks"},{"id":"required_checks","question":"Required checks"},{"id":"failure_blocking","question":"Failure blocking"},{"id":"requirement_conflicts","question":"Requirement conflicts"},{"id":"configured_policy","question":"Configured policy"},{"id":"enforced_policy","question":"Enforced policy"}]},{"id":"SHIP-03","title":"Git and pull request workflow","group":{"id":6,"name":"Verification and delivery"},"judgment":"Record branches/worktrees, commits, PR creation, review comments, CI response, rebase and conflict handling. Distinguish an integrated workflow from the agent running Git commands without product state.","subquestions":[{"id":"branches","question":"Branches"},{"id":"worktrees","question":"Worktrees"},{"id":"commits","question":"Commits"},{"id":"pull_requests","question":"Pull requests"},{"id":"comments","question":"Comments"},{"id":"ci_response","question":"Ci response"},{"id":"rebase","question":"Rebase"},{"id":"conflicts","question":"Conflicts"},{"id":"product_state","question":"Product state"}]},{"id":"SHIP-04","title":"Merge and deployment authority","group":{"id":6,"name":"Verification and delivery"},"judgment":"Record who can approve, merge, release and deploy, plus checks against the exact revision. PR creation, merge and production deployment are separate capabilities.","subquestions":[{"id":"approve","question":"Approve"},{"id":"merge","question":"Merge"},{"id":"release","question":"Release"},{"id":"deploy","question":"Deploy"},{"id":"exact_revision","question":"Exact revision"},{"id":"authority_enforcement","question":"Authority enforcement"}]},{"id":"SHIP-05","title":"Completion evidence and rollback","group":{"id":6,"name":"Verification and delivery"},"judgment":"Record task-to-commit/PR/build/deployment links, delivery status, retained evidence and rollback or revert support. Record whether approval and evidence are invalidated when the candidate revision changes after review. Agent exit alone does not prove delivery.","subquestions":[{"id":"commit_links","question":"Commit links"},{"id":"pr_links","question":"Pr links"},{"id":"build_links","question":"Build links"},{"id":"deployment_links","question":"Deployment links"},{"id":"delivery_status","question":"Delivery status"},{"id":"retained_evidence","question":"Retained evidence"},{"id":"rollback","question":"Rollback"},{"id":"revert","question":"Revert"},{"id":"stale_approval_invalidation","question":"Stale approval invalidation"}]},{"id":"SHIP-06","title":"Batch review and high-volume shipping","group":{"id":6,"name":"Verification and delivery"},"judgment":"For a queue of completed tasks, record summaries, grouping, risk/size filters, retained reviewer findings, artifact navigation, request-changes flow, approval of exact revisions, dependency-aware merge/release queues and clear next actions. Record whether stale approvals invalidate after edits and whether duplicate checks respect RUN-08. Distinguish basic diff/PR actions from a coordinated review-and-delivery queue; do not require any product's terminology.","subquestions":[{"id":"review_queue","question":"Review queue"},{"id":"summaries","question":"Summaries"},{"id":"grouping","question":"Grouping"},{"id":"risk_size_filters","question":"Risk size filters"},{"id":"findings","question":"Findings"},{"id":"artifacts","question":"Artifacts"},{"id":"request_changes","question":"Request changes"},{"id":"exact_revision_approval","question":"Exact revision approval"},{"id":"dependency_queues","question":"Dependency queues"},{"id":"next_actions","question":"Next actions"},{"id":"stale_approval_invalidation","question":"Stale approval invalidation"},{"id":"check_deduplication","question":"Check deduplication"}]},{"id":"OWN-01","title":"Source and license","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record license and revision per component, including client, server and hosted extensions. Distinguish open source, source-available and closed components. Repository visibility is not a license classification or proof of hosted parity.","subquestions":[{"id":"client_license","question":"Client license"},{"id":"server_license","question":"Server license"},{"id":"hosted_extensions","question":"Hosted extensions"},{"id":"source_visibility","question":"Source visibility"},{"id":"source_revision","question":"Source revision"},{"id":"hosted_parity","question":"Hosted parity"}]},{"id":"OWN-02","title":"Hosting and exit","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record self-hosting scope, required vendor services, data export/import, standard formats and account cancellation behavior. Record what remains usable without the vendor account. Exported code alone does not mean tasks, context and history are portable.","subquestions":[{"id":"self_hosting","question":"Self hosting"},{"id":"vendor_dependencies","question":"Vendor dependencies"},{"id":"export","question":"Export"},{"id":"import","question":"Import"},{"id":"formats","question":"Formats"},{"id":"cancellation","question":"Cancellation"},{"id":"account_independence","question":"Account independence"},{"id":"history_portability","question":"History portability"}]},{"id":"OWN-03","title":"Data handling and telemetry","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record where code, prompts, credentials, logs and artifacts go, retention/deletion, training use, residency and subprocessors. Separately record telemetry disclosure, collection defaults, opt-in/opt-out controls and documented first-run prompts. The absence of a documented consent dialog is not proof of no telemetry, secret telemetry or unlawful processing. Separate orchestration-vendor and model-provider policies.","subquestions":[{"id":"code_data","question":"Code data"},{"id":"prompts","question":"Prompts"},{"id":"credentials","question":"Credentials"},{"id":"logs","question":"Logs"},{"id":"artifacts","question":"Artifacts"},{"id":"retention","question":"Retention"},{"id":"deletion","question":"Deletion"},{"id":"training","question":"Training"},{"id":"residency","question":"Residency"},{"id":"subprocessors","question":"Subprocessors"},{"id":"telemetry_disclosure","question":"Telemetry disclosure"},{"id":"telemetry_defaults","question":"Telemetry defaults"},{"id":"telemetry_controls","question":"Telemetry controls"},{"id":"consent_prompt","question":"Consent prompt"},{"id":"provider_separation","question":"Provider separation"}]},{"id":"OWN-04","title":"Price and included usage","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record currency, billing interval, seats/seat minimums, workspace charges, included usage, credits, trial conditions, overages, taxes treatment and separate model/compute costs. Map billing routes to ACC-03. Cite public prices or dated quotes. Per-seat versus pooled billing is a fact, not a universal quality judgment.","subquestions":[{"id":"currency","question":"Currency"},{"id":"interval","question":"Interval"},{"id":"seats","question":"Seats"},{"id":"seat_minimum","question":"Seat minimum"},{"id":"workspace_fees","question":"Workspace fees"},{"id":"included_usage","question":"Included usage"},{"id":"credits","question":"Credits"},{"id":"trial","question":"Trial"},{"id":"overages","question":"Overages"},{"id":"taxes","question":"Taxes"},{"id":"model_cost","question":"Model cost"},{"id":"compute_cost","question":"Compute cost"}]},{"id":"OWN-05","title":"Scenario cost and controls","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Price stated workloads for one, two and five developers, with fixed tasks, concurrency, tokens and compute-hours. The two-person scenario models two developers collaborating. Include subscription, platform, model and compute charges; separate fixed, variable, included and unknown costs. No invented average task cost. Link cost visibility and reconciliation to RUN-07.","subquestions":[{"id":"one_person","question":"One person"},{"id":"two_people","question":"Two people"},{"id":"five_people","question":"Five people"},{"id":"task_assumptions","question":"Task assumptions"},{"id":"concurrency","question":"Concurrency"},{"id":"tokens","question":"Tokens"},{"id":"compute_hours","question":"Compute hours"},{"id":"fixed_cost","question":"Fixed cost"},{"id":"variable_cost","question":"Variable cost"},{"id":"included_cost","question":"Included cost"},{"id":"unknown_cost","question":"Unknown cost"}]},{"id":"OWN-06","title":"Availability and support","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record GA/beta/access restrictions, support channels, documented response commitments, release activity, status history and deprecation/export options. Funding, stars and popularity can be context but are not reliability scores. An archived repository does not prove the hosted service is discontinued.","subquestions":[{"id":"release_stage","question":"Release stage"},{"id":"access_restrictions","question":"Access restrictions"},{"id":"support","question":"Support"},{"id":"response_commitments","question":"Response commitments"},{"id":"release_activity","question":"Release activity"},{"id":"status_history","question":"Status history"},{"id":"deprecation","question":"Deprecation"},{"id":"export_options","question":"Export options"}]},{"id":"OWN-07","title":"Enterprise assurance","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record each claimed assurance separately: ISO standard/certificate scope, issuer and validity; SOC 2 report type, covered service and audit period; report/certificate access and verification date. Distinguish public documentation, vendor assertion, independently accessible evidence and unavailable evidence. A logo, enterprise pricing tier, SSO or a provider certification does not prove this product is certified or audited. Report controls in TEAM-02 separately; no legal-compliance verdict.","subquestions":[{"id":"iso_scope","question":"Iso scope"},{"id":"iso_issuer","question":"Iso issuer"},{"id":"iso_validity","question":"Iso validity"},{"id":"soc2_type","question":"Soc2 type"},{"id":"soc2_scope","question":"Soc2 scope"},{"id":"soc2_period","question":"Soc2 period"},{"id":"report_access","question":"Report access"},{"id":"verification_date","question":"Verification date"},{"id":"evidence_status","question":"Evidence status"}]},{"id":"OWN-08","title":"Useful free tier and evaluation access","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Distinguish an ongoing free tier, time-limited trial, credits-only trial, open-source self-hosting costs, paid-only access and gated early access. Map which shared scenarios work for free, especially two-person collaboration, BYO subscriptions, remote compute, mobile, issue management and review. Record card requirement, usage caps and whether meaningful evaluation needs payment or sales approval.","subquestions":[{"id":"ongoing_free","question":"Ongoing free"},{"id":"trial","question":"Trial"},{"id":"credits_trial","question":"Credits trial"},{"id":"self_host_cost","question":"Self host cost"},{"id":"paid_only","question":"Paid only"},{"id":"early_access","question":"Early access"},{"id":"free_scenarios","question":"Free scenarios"},{"id":"card_required","question":"Card required"},{"id":"usage_caps","question":"Usage caps"},{"id":"paid_evaluation","question":"Paid evaluation"}]},{"id":"OWN-09","title":"Pricing transparency","group":{"id":7,"name":"Ownership, privacy and economics"},"judgment":"Record publicly stated units, included limits, overage rates, compute/model pass-through or markup, minimum commitment, cancellation and sales-only unknowns. Use transparent, partially specified, quote-only or unknown, with named missing fields. A quote-only enterprise price is a disclosed procurement constraint, not proof of dishonesty.","subquestions":[{"id":"units","question":"Units"},{"id":"included_limits","question":"Included limits"},{"id":"overages","question":"Overages"},{"id":"compute_markup","question":"Compute markup"},{"id":"model_markup","question":"Model markup"},{"id":"minimum_commitment","question":"Minimum commitment"},{"id":"cancellation","question":"Cancellation"},{"id":"sales_only_unknowns","question":"Sales only unknowns"},{"id":"transparency_classification","question":"Transparency classification"}]}]}
