Factory
vs Warp

Judgment rule. List desktop OS and minimum versions, iOS, Android, browser, mobile browser, CLI and editor extensions separately. For each client, record start, steer, approve, review and resume actions, notifications and plan restrictions. Test moving the same task from desktop to phone; distinguish an installable mobile app from a mobile website and from status-only monitoring.

Judgment rule. List supported harnesses and model providers separately, including custom adapters. Record whether the product embeds a harness, controls its sessions through an API, or only launches its terminal command. Verify at least two different harnesses before describing mixed-harness operation as observed.

Judgment rule. Record bring-your-own agent subscription, bring-your-own API account, vendor credits and local models as separate routes. List supported plans, models/modes, multiple-account switching and what works without a vendor account or token purchase. Existing subscription support cannot be inferred from API-key support. Record required extra payments and documented provider restrictions.

Judgment rule. Measure fresh install to first successful task, with manual steps and failures. Test a local directory without Git, a local Git repo without a remote, and a GitHub-backed repo separately. Record whether account, project, host, credential and permission prerequisites are explained before accepting an unusable prompt. Evaluate managed-cloud setup and first-class BYO VPS installer/onboarding independently of hidden settings or a DIY shell recipe.

Judgment rule. Inventory actual CLI, API, SDK, MCP and webhook operations for create, inspect, steer, stop and review. Record authentication and API stability. Having an MCP client that uses external tools is different from exposing the product to other agents over MCP.

Judgment rule. Record MCP client support, plugins, custom tools, skill/instruction import, integration authentication and permission scope. Name which component provides each connection and whether setup is product-managed or inherited from the underlying harness. Verify one external-tool call and its access boundary where available.

Judgment rule. For each desktop/mobile client, record native platform UI, Electron, Tauri, React Native, web/PWA or unknown, plus hybrids. Require official technical docs or pinned source/build evidence. Tauri is not synonymous with fully native UI. Keep implementation choice separate from measured responsiveness and usability.

Judgment rule. Record local creation, prompt intake, issue import and chat/Git triggers. Identify the authoritative tracker, stable issue IDs, one-way or two-way sync, supported edits and operations requiring a trip back to Linear/GitHub/Jira. Importing or attaching a ticket is not native issue management. Answer PLAN-06 and PLAN-07 separately.

Judgment rule. Record task plans and acceptance criteria, editors, revision history and links to implementation. Test amending a task while work is running, restarting and revisiting it later. A task-specific starting brief, reusable memory and a persistent product specification are different objects; product specs are judged in PLAN-08.

Judgment rule. Record epics, arbitrary parent/child depth, blockers, readiness, dependency enforcement and state propagation. Start with a simple interactive task, grow it into an epic and retain its identity, discussion, history and outputs. Verify blockers actually prevent dispatch. Record whether a rigid preselected workflow is required, and whether changing scope requires recreating work.

Judgment rule. List repository instructions, skills, project docs and cross-session/cross-harness memory. Record scope, provenance, citation, correction/deletion and stale-context controls. Test what survives a new session or colleague takeover. A notes page or shared file is not automatically a maintained spec or managed memory.

Judgment rule. Record code search, repository indexing, external knowledge and multimodal inputs, with source freshness and access scope. Ask an agent to use an existing API under a documented architecture rule, then propose a conflicting change. Check retrieval, citation, conflict detection and review-time enforcement separately. A large context window or an instruction file alone does not prove architectural consistency.

Judgment rule. Enumerate create, edit, discuss, prioritize, label, assign, search, filter, archive, reopen and project/epic views within the product. Save an idea without creating a running agent, refine it with another person, and explicitly mark it ready or dispatch it later. Record separate task/session entities, metadata depth, multiple sessions per task, durable state transitions and whether an external tracker remains necessary.

Judgment rule. Verify an agent can read/create/update the same durable issue IDs and fields humans use, claim/assign work, change state, create dependencies/children and report blockers/results through documented CLI/API/MCP. Check that human views reflect actual writes with attribution, permissions and conflict handling. Distinguish shared authoritative records, synchronized external records, agent-private metadata and prompt-only conventions.

Judgment rule. Record persistent requirements across tasks/releases, editable by humans and agents with attributable revisions and approval rules. Verify requirement-to-task/implementation links, retrieval during later tasks, drift detection and consideration during review/quality control. Amend a requirement after one task ships, then implement another. A one-time spec-to-plan workflow, per-task brief, memory store or unreferenced document does not establish a living product spec.

Judgment rule. Record independently running sessions and tasks, documented quotas and observed concurrency. Describe how a user sees running, waiting, failed and finished work. Several chat tabs are not proof that runs execute concurrently.

Judgment rule. Record assignment to humans or agents, claims, queues, routing and duplicate-work prevention. Test two workers attempting the same task. Distinguish a display label, an advisory lock and enforced exclusive ownership.

Judgment rule. Record parent/child roles, model/harness/effort choice, messages, result return and cross-harness teams. Distinguish product orchestration, provider-native subagents, installed orchestration skills and scripts. Record activation, auth and experimental flags. Verify actual child identity, live progress, retained results and ability to reopen or follow up after completion. A final sentence claiming two reviewers ran is not proof. Test the documented invocation as well as the shared natural-language request.

Judgment rule. Record schedules, event triggers, recurring tasks and completion conditions. Name where the scheduler runs, overlap handling, retries and whether closing the client stops it. A desktop cron recipe is a manual integration unless the product owns it.

Judgment rule. Record pause, stop, resume, steering, notifications, pending decisions and escalation. Test an in-run amendment and a harmless approval prompt in both chat and terminal modes. Record silent waits, hidden CLI-only questions and whether stop terminates execution while preserving history. Require discoverable errors rather than inferring success from silence.

Judgment rule. Use a disclosed fixture of 12 task/agent states across several epics, including blocked, failed, awaiting input, reviewing and completed work. Locate parent/child relationships, dependencies, attention needs, history and workload without reading every transcript. Record navigation steps and errors. Boards, graphs, waterfalls, timelines or another interface can qualify; no required visual form. State whether the fixture was simulated or used live agents.

Judgment rule. Compare a quick interactive change with a growing epic. Record mandatory workflow/preset selection, natural-language configuration, per-agent model/effort control, changes during execution and whether the UI exposes what actually ran. Test a coordinator with two reviewer children; capture roles, relationships and results during work and after completion. Document discoverability and auth friction without judging an unsupported magic prompt as proof of missing capability.

Judgment rule. List local, user-owned VPS/remote host, self-hosted service, vendor cloud and hybrid modes. Record provisioning, payer, compute resources, region, idle/max lifetime and disconnection rules. Close/disconnect the originating client and continue from another device. A hosted UI or keep-awake setting is not evidence of remote execution. Record managed-cloud convenience separately from BYO setup effort in ACC-04.

Judgment rule. Record shared checkout, branch, worktree, container, VM and OS sandbox separately. Name filesystem, process, network and secret boundaries. Validate code separation with two changes; do not claim hostile-code containment without separate security evidence.

Judgment rule. Test client disconnect, worker crash, service restart and documented sandbox expiry safely. Separate transcript reload, process reconnection, task resumption and state reconstruction. Record lost context, duplicated actions, manual repair and ability to continue a multi-day epic after compute replacement. Short sandbox lifetimes are reported with their continuity behavior, not automatically treated as failure.

Judgment rule. Record dependency setup, node_modules/cache handling, snapshots, services, port allocation and secret injection. Start two workspaces with the same service requirements; note installation duplication, port collisions and repair steps. Repeat on a clean remote host. Record which problems the product handles automatically and which the user must diagnose.

Judgment rule. Record time, turn, token and spend limits; timeout, retry and provider-failure behavior. Test one controlled failure and a small safe budget. Record whether limits are warnings, enforced stops, or external billing controls.

Judgment rule. Record logs, traces, failure reasons, usage exports and statistics by task, agent, project and time. Induce a worker failure and an approval wait: check agent, task and board state agree and remain correct after reconnect. Distinguish working, blocked, failed, idle, ready for review and delivered. For speed/success claims disclose fixture, model, hardware, attempts and failures. UI responsiveness needs observed interaction timings; never infer a framework from lag.

Judgment rule. Record visible model/API spend per task, session, child and epic, including aggregation across harness changes. Distinguish billed charges, estimates, tokens, subscription allowance and unavailable amounts; never render unknown subscription marginal cost as zero. Reconcile totals with provider evidence where available. Record compute fees separately, refresh delay, budget alerts, export and drill-down from summary statistics.

Judgment rule. Record concurrency limits, queues, CPU/RAM limits, backpressure, per-host resource awareness and reuse/deduplication of checks. Queue multiple browser end-to-end tests under a declared safe capacity; verify excess work queues or throttles rather than overloading the host. Inspect cancellation, fairness, recovery and whether overlapping epics share limits. Do not run an uncontrolled load test.

Judgment rule. Record shared visibility, issue discussion, comments, presence and simultaneous co-prompting as independent capabilities. Use two named accounts to steer the same agent and record input ordering/conflict handling, attribution and permission controls. Read-only links and shared issues do not establish co-steering. Record team/free-tier restrictions and whether the original owner must stay online.

Judgment rule. Record organization/project/task roles, membership lifecycle, SSO and provisioning where offered. Test unauthorized access and revocation with consenting test accounts. Describe documented versus observed enforcement, including which plan provides it.

Judgment rule. Record command, tool, network, secret and delivery approvals, approver identity and scope lifetime. Demonstrate allow, deny and revocation, including temporary tunnels or shared access after its thread/session ends. Separate default full access, available safer modes and product-enforced policy from instructions the model is expected to follow.

Judgment rule. Record attributed actions, approvals, edits, discussion and decision links. Check whether history persists across users and sessions, can be exported, and can be altered. Use the word audit only with a defined retention and integrity model.

Judgment rule. Evaluate locating pending reviews, understanding a change, navigating evidence, commenting, requesting changes and finding the next shipping action. Measure steps/time and missed evidence on a shared fixture. Record keyboard/narrow-screen usability, context switching, clarity of agent roles and preservation of review context. Generic diff/PR access and a guided review workflow are different capabilities. Link high-volume review findings to SHIP-06.

Judgment rule. Have a second authorized person take over ongoing work with the same task, session history, decisions, artifacts, permissions and runtime context. Test after the first person disconnects. Record ownership transfer, attribution and lost context. Separately record transferring to another model/harness; a forked transcript or shared link alone is not full human takeover.

Judgment rule. Record diffs, documents, screenshots, recordings, previews and test/reviewer outputs. Check for a searchable/browsable artifact gallery or equivalent retrieval, linked to task and exact revision, usable after sessions end. Reopen a completed child reviewer and retrieve its full findings. Check whether internal prompts or tool metadata unintentionally appear in the produced application or other user-facing output. In-chat attachments, persistent review evidence and a cross-task gallery get distinct answers.

Judgment rule. Record tests, lint, types, CI, independent review and persistent-spec conformance checks as required versus optional. Cause a failed test and a conflict with an existing product requirement. Verify review catches the conflict and delivery blocks where claimed. An agent saying checks passed is not a passing gate; record configured and enforced policy separately.

Judgment rule. Record branches/worktrees, commits, PR creation, review comments, CI response, rebase and conflict handling. Complete one fixture change through a draft PR. Distinguish an integrated workflow from the agent running Git commands without product state.

Judgment rule. Record who can approve, merge, release and deploy, plus checks against the exact revision. Verify the documented authority boundary without publishing real changes. PR creation, merge and production deployment are separate capabilities.

Judgment rule. Record task-to-commit/PR/build/deployment links, delivery status, retained evidence and rollback or revert support. Change the candidate revision after review and examine whether approval/evidence is invalidated. Agent exit alone does not prove delivery.

Judgment rule. Return to a queue of at least six completed fixture tasks and review a subset. Record summaries, grouping, risk/size filters, retained reviewer findings, artifact navigation, request-changes flow, approval of exact revisions, dependency-aware merge/release queues and clear next actions. Check stale approvals invalidate after edits and duplicate checks respect RUN-08. Distinguish basic diff/PR actions from a coordinated review-and-delivery queue; do not require Podium terminology.

Judgment rule. Record license and revision per component, including client, server and hosted extensions. Distinguish open source, source-available and closed components. Repository visibility is not a license classification or proof of hosted parity.

Judgment rule. Record self-hosting scope, required vendor services, data export/import, standard formats and account cancellation behavior. Verify what remains usable without the vendor account. Exported code alone does not mean tasks, context and history are portable.

Judgment rule. Record where code, prompts, credentials, logs and artifacts go, retention/deletion, training use, residency and subprocessors. Separately record telemetry disclosure, collection defaults, opt-in/opt-out controls and observed first-run prompts. No consent dialog is an observation, not proof of no telemetry, secret telemetry or unlawful processing. Separate orchestration-vendor and model-provider policies.

Judgment rule. Record currency, billing interval, seats/seat minimums, workspace charges, included usage, credits, trial conditions, overages, taxes treatment and separate model/compute costs. Map billing routes to ACC-03. Cite public prices or dated quotes. Per-seat versus pooled billing is a fact, not a universal quality judgment.

Judgment rule. Price stated workloads for one, two and five developers, with fixed tasks, concurrency, tokens and compute-hours. The two-person scenario models Till and Michael collaborating. Include subscription, platform, model and compute charges; separate fixed, variable, included and unknown costs. No invented average task cost. Link cost visibility and reconciliation to RUN-07.

Judgment rule. Record GA/beta/access restrictions, support channels, documented response commitments, release activity, status history and deprecation/export options. Funding, stars and popularity can be context but are not reliability scores. An archived repository does not prove the hosted service is discontinued.

Judgment rule. Record each claimed assurance separately: ISO standard/certificate scope, issuer and validity; SOC 2 report type, covered service and audit period; report/certificate access and verification date. Distinguish public documentation, vendor assertion, independently accessible evidence and unavailable evidence. A logo, enterprise pricing tier, SSO or a provider certification does not prove this product is certified or audited. Report controls in TEAM-02 separately; no legal-compliance verdict.

Judgment rule. Distinguish an ongoing free tier, time-limited trial, credits-only trial, open-source self-hosting costs, paid-only access and gated early access. Map which shared scenarios work for free, especially two-person collaboration, BYO subscriptions, remote compute, mobile, issue management and review. Record card requirement, usage caps and whether meaningful evaluation needs payment or sales approval.

Judgment rule. Record publicly stated units, included limits, overage rates, compute/model pass-through or markup, minimum commitment, cancellation and sales-only unknowns. Use transparent, partially specified, quote-only or unknown, with named missing fields. A quote-only enterprise price is a disclosed procurement constraint, not proof of dishonesty.