Proof Before Purchase
How a Florida law firm can turn an AI sales promise into firm evidence before it signs, expands, or renews.
A former Big Law corporate equity partner at Steptoe (NYC Managing Partner) and Seyfarth Shaw • Florida-focused • Independent consultant • No software to sell
Last reviewed: October 1, 2026. This page is updated when Florida guidance, evaluation standards, or common vendor practices materially change.
A vendor demonstration is a stage play. The data is selected, the prompt is rehearsed, and no client is waiting for the answer at 4:47 p.m. on filing day.
The firm is not buying the demonstration. It is buying a change to a real workflow performed by real people using imperfect files under ordinary pressure. One clean answer proves possibility. It does not prove reliability, control, adoption, or value.
That is why the purchase process should begin with proof. Define one recurring problem. Measure the current process. Test the exact product and account against representative, approved material. Count review and correction time. Write the decision. Then buy, restrict, revise, defer, or walk away.
The short version
Seven rules prevent most expensive experiments.
- Test one defined workflow, not the product’s entire feature list.
- Record the current baseline before the trial changes anyone’s memory of it.
- Use the same approved test set, starting point, and measures for every candidate.
- Count collection, production, review, correction, handoff, exceptions, and administration.
- Set pass thresholds and hard stops before seeing the results.
- Require written evidence for material vendor claims, data practices, controls, and exit rights.
- End with a dated decision, named owner, approved use, limits, and next review date.
Make the next vendor demo earn a second meeting.
Get the AI tool-approval scorecard, five workflows worth testing, a Florida ethics guide, daily verification checklist, sample client language, incident-response protocol, and 90-day implementation outline.
Get the Free Florida AI Starter Playbook →On This Page
A Demo Is Not Proof
Sales demonstrations answer the vendor’s question: can the product look useful under favorable conditions? The firm has a different question: can this exact product improve our defined work under conditions we can control and repeat?
Those are not hostile questions. They are purchasing questions. A good vendor should welcome a disciplined pilot because it produces a customer who knows what the product is for. A vendor who insists that every workflow is a fit has confused a market with a mirror.
| Question | Vendor demonstration | Firm proof test |
|---|---|---|
| Input | Curated and clean | Representative, messy, incomplete, and approved |
| Instructions | Prepared by product experts | Written for ordinary users and saved for reuse |
| Output | Best feature shown | Exact format required by the next workflow step |
| Review | Fast visual reaction | Named reviewer checks against identified sources |
| Measurement | Speed of the visible task | Total process time, corrections, quality, service, and cost |
| Failure | Explained away or skipped | Recorded, classified, and applied to a stop rule |
| Decision | Schedule another call | Approve, limit, retest, defer, reject, or buy nothing |
A free trial without a baseline is free confusion.
If the firm does not know how the current process performs, any result can be called an improvement. Memory is an enthusiastic accountant.
Use the Five-Level Proof Ladder
Not all evidence deserves the same weight. Move the candidate up the ladder only when the lower level has answered its question.
- Claim. The vendor, article, colleague, or conference speaker says the product can do something. This creates a question, not an answer.
- Demonstration. The product performs the task under selected conditions. This shows that the capability may exist.
- Sandbox test. The firm uses fictional, public, anonymized, or otherwise approved material to test functions, limits, and review needs.
- Controlled workflow pilot. Representative users perform one defined task with a fixed input boundary, review process, measures, and stop rules.
- Production evidence. The approved workflow produces stable results over time, within its controls, after ordinary staff turnover, product changes, and difficult files.
A purchase may be justified after a controlled pilot. A broad rollout is justified by production evidence. Buying at the claim level is not innovation. It is retail therapy with a security questionnaire.
Write a One-Page Pilot Charter
The charter prevents the trial from expanding whenever the product does something interesting. It should be short enough to use and specific enough to stop an argument.
| Charter field | Decision to write before testing |
|---|---|
| Problem | What recurring delay, cost, inconsistency, or service failure are we trying to improve? |
| Current workflow | Who performs the work, from which sources, through which systems, and for which next step? |
| Candidate | What exact vendor, product, plan, feature, account type, settings, and version will be tested? |
| Test users | Which two to five representative users will perform the work, and who administers access? |
| Approved input | What fictional, public, anonymized, or otherwise approved material may enter? What must remain outside? |
| Expected output | What format, fields, source references, and quality requirements must the next person receive? |
| Review | Who checks what, against which source, before which action? |
| Measures | Which time, quality, correction, service, adoption, control, and cost measures decide the result? |
| Hard stops | Which event suspends the test immediately? |
| Decision | Who decides, on what date, among approve, limit, retest, defer, reject, or buy nothing? |
The charter names the proposed use. That matters. “The firm approved Vendor X” is not a useful decision. Approval should identify the exact feature, account, users, data boundary, workflow, review, limits, and review date.
Measure the Current Process Before the Tool
A baseline is the current workflow measured from the same starting point to the same finished result the candidate will be asked to produce. If the current process begins when documents arrive and ends when a reviewed update reaches the client, the test cannot begin with a clean summary and end with an unreviewed draft.
Measure several ordinary matters or work items. One unusually easy file flatters the current process. One unusually hard file flatters the tool. The point is not statistical theater. It is an honest picture of recurring work.
Count the whole workflow
| Time component | What belongs in the measurement |
|---|---|
| Collection | Find, request, open, sort, and prepare the source material. |
| Production | Read, extract, organize, calculate, summarize, or draft. |
| Review | Check the result against sources, instructions, law, and firm standards. |
| Correction | Repair omissions, false statements, formatting, tone, and unsupported conclusions. |
| Handoff | Move the result into the next system, person, approval, or client communication. |
| Exceptions | Resolve missing, conflicting, stale, privileged, or unusual material. |
| Administration | Manage access, prompts, templates, logs, training, support, and records. |
The same boundary applies to the candidate process. A three-minute draft followed by twenty minutes of source checking is a twenty-three-minute process. The stopwatch does not care which part had an AI label.
Record quality and service, not merely time
- Material corrections: facts, dates, amounts, names, duties, deadlines, quotations, citations, and legal conclusions.
- Completeness: required fields, issues, documents, next steps, and exceptions included or flagged.
- Source traceability: ability to locate the source supporting each important statement.
- Consistency: compliance with the firm’s format, terminology, tone, and approved process.
- Cycle time: elapsed time from trigger to usable result, including waiting and handoffs.
- Client service: speed, clarity, predictability, fewer repeat requests, or fewer avoidable follow-ups.
- User effort: mental load, extra copying, workarounds, training, and support required.
A faster draft that needs a slower lawyer is not efficiency.
It is a costume. Count the reviewer’s time and the correction burden or the pilot will manufacture its own success.
Set Pass Criteria Before Seeing the Results
A pilot should not end with everyone looking at the same spreadsheet and choosing the story they prefer. Write the decision thresholds before the test begins. There is no universal percentage that makes a tool worthwhile. The threshold should match the task, risk, volume, client impact, and cost.
| Decision measure | Threshold to set in advance |
|---|---|
| Total process time | What reduction would matter after review, correction, handoff, and administration? |
| Material corrections | What error types or correction rate would make the use impractical or unsafe? |
| Completeness | Which required fields or issues may never be omitted without a clear flag? |
| Traceability | Which statements must point to a source, page, message, record, or authority? |
| Reviewer effort | How much review can the task support before the expected gain disappears? |
| User performance | What percentage of test users can follow the workflow after short, written instruction? |
| Control performance | Which data, permission, connector, sharing, or action limits must hold every time? |
| Net cost | What license, implementation, training, support, and administration cost can the gain justify? |
Write the hard stops
A hard stop suspends the pilot regardless of the time saved. Examples include:
- The feature receives, retrieves, exposes, or shares information outside the approved boundary.
- A required human review is bypassed or the system sends, files, updates, or commits without approval.
- The tool mixes matters, users, tenants, or source sets in a way the firm cannot reliably prevent.
- A material vendor answer about data use, retention, access, deletion, or security cannot be confirmed in writing.
- The product fabricates or alters a source, quotation, citation, deadline, amount, or client commitment and the workflow cannot reliably catch it.
- Users must evade the written process, copy sensitive material into unapproved locations, or rely on personal accounts to make the tool work.
- The review and correction burden exceeds the decision threshold.
A hard stop is not a declaration that the product is permanently bad. It is a statement that this test, use, configuration, or data boundary is not ready. Precision beats drama.
Build a Representative Test Set
The test set should look like the work the firm actually receives, including the parts that make people sigh. Start with fictional, public, anonymized, or otherwise approved material until the exact tool, account, terms, controls, and proposed use have been reviewed.
| Test case | What it should reveal |
|---|---|
| Ordinary | A clean, common example that represents the normal workload. |
| Long | A file large enough to expose context, retrieval, and output-limit problems. |
| Incomplete | Missing documents, fields, dates, or instructions that should be flagged, not invented. |
| Conflicting | Sources that disagree about a name, date, amount, obligation, or status. |
| Messy | Scans, poor formatting, email chains, duplicates, tables, handwriting, or inconsistent labels. |
| Edge case | An uncommon but foreseeable variation with higher review or escalation needs. |
| Outdated | Material containing superseded terms, law, status, or instructions that should not control. |
| Hostile instruction | A document or webpage containing text that tries to redirect the AI or expose information. |
The last row tests prompt injection: an attempt to make a connected AI follow instructions hidden inside content instead of the firm’s instructions. A small firm does not need a laboratory. It does need to know whether an ordinary document can persuade the feature to do something the workflow never approved.
Use the same test set for every serious candidate. Changing the inputs whenever a product struggles turns comparison into choreography.
Require Four Separate Proof Gates
Do not collapse every issue into one total score. A large time saving cannot cancel a confidentiality failure. A strong security packet cannot make a useless workflow useful. Each gate answers a different question.
| Gate | Question | Evidence |
|---|---|---|
| 1. Workflow fit | Does the feature solve a frequent, valuable problem and produce the format the next step needs? | Problem, trigger, inputs, outputs, owner, handoff, exceptions |
| 2. Work quality | Can the result be checked, corrected, and trusted for the defined use? | Accuracy, completeness, source traceability, consistency, reviewer effort |
| 3. Control fit | Can the firm limit data, access, sharing, connectors, retention, and actions? | Account, terms, permissions, logs, data path, deletion, incident handling |
| 4. Operating value | Will representative users follow the process and does the finished result justify total cost? | Adoption, training, support, cycle time, client service, administration, net cost |
Every required gate passes, or the use changes.
The firm may narrow the input, remove a connector, add review, restrict users, change the task, or choose another product. It may not average away a failed control.
Build a Vendor Evidence File
Florida Bar Ethics Opinion 24-1 directs lawyers using third-party generative AI with confidential information to understand matters such as retention, sharing, self-learning, security, provider obligations, liability limits, breach or process notice, ownership, and what happens after service ends. A logo and a sales representative’s assurance do not answer those questions. The AI Vendor Evaluation Worksheet and the Vendor AI and Data Governance Audit give you the questions to put in that file.
Keep the evidence small enough to maintain. Save the exact documents and written answers that support the decision, with the date reviewed. Screenshots are useful for settings. Contracts and current terms carry more weight than a slide with a shield icon.
What to collect
- Product identity: exact vendor, product, plan, feature, account type, version, administrator, and proposed use.
- Claim support: written evidence for material accuracy, performance, source, security, and efficiency claims; test the claims that drive the purchase.
- Data terms: collection, use, model training, retention, location, human access, ownership, sharing, subprocessors, deletion, and notice of changes.
- Security evidence: current security documentation, independent reports or certifications where relevant, incident history, breach notice, encryption, logs, and access controls.
- Workflow controls: roles, permissions, connectors, source boundaries, sharing, automated actions, audit history, and administrator visibility.
- Reliability evidence: limitations, known failure modes, service commitments, support path, product-change process, and status history.
- Commercial terms: pricing unit, seat minimums, usage charges, implementation, support, renewal, price changes, cancellation, and liability limits.
- Exit evidence: export formats, prompt or template portability, account closure, access removal, deletion process, backup treatment, and confirmation.
Use the AI Vendor Evaluation Worksheet for the initial gate and the Vendor AI & Data Governance Audit when the candidate may process client or sensitive firm information.
Unknown is not the same as yes.
When a material vendor question remains unanswered, record it as unknown, assign the risk, and decide whether the use can proceed with a narrower boundary. Do not let an empty cell quietly become approval.
Run a 10-Business-Day Proof Sprint
Ten business days is enough to test one narrow workflow. It is not enough to approve the entire platform, every practice area, or every data type. Good. Broad approval is usually where useful discipline goes to disappear.
Days 1–2: Charter and baseline
Write the problem, exact candidate, users, input boundary, output, reviewer, measures, hard stops, and decision date. Measure several current work items from the same trigger to the same reviewed result. Confirm the firm owns no current feature that should be tested as the comparison candidate.
Days 3–4: Evidence and configuration
Review the exact account, terms, data practices, security materials, permissions, connectors, logs, sharing, and actions. Limit the test users, information, sources, access, connectors, and automation to the minimum required. Assemble the representative test set and written instructions. Train users on the manual path and hard stops.
Days 5–7: Perform the controlled work
Run the same test cases through the current process and the candidate process where practical. Record collection, production, review, correction, handoff, exception, and administration time. Log every material error, omission, unsupported statement, control exception, user workaround, and support request.
Day 8: Test failure and recovery
Use incomplete, conflicting, messy, outdated, and hostile-instruction examples. Confirm users can stop, reject, correct, escalate, and return to the manual process. Test export, access removal, and deletion steps where the product and trial terms permit.
Days 9–10: Calculate and decide
Compare both processes against the prewritten thresholds and all four proof gates. Calculate full-process time, usable capacity, quality, service, adoption, control performance, and total cost. Record approve, approve with limits, revise and retest, defer, reject, or buy nothing. Name the owner and next review date.
The sprint ends with a decision. “The team liked it” is feedback. “We will revisit next quarter” without an owner or trigger is storage.
Worked Example: A Client-Update Draft
This is an illustrative pilot, not a claim about a product. Replace each field with the firm’s actual workflow, account, test material, and thresholds.
| Pilot field | Illustrative entry |
|---|---|
| Problem | Routine matter updates take too long to assemble, vary by lawyer, and often wait until the client asks. |
| Current trigger and finish | Trigger: approved weekly matter summary is complete. Finish: lawyer-approved update is ready to send. |
| Candidate use | Draft a four-part update: what happened, what comes next, what the client must do, and when the firm will communicate again. |
| Input boundary | Approved matter summary only. No unrestricted mailbox, drive, calendar, or matter-system search. |
| Output requirements | Plain English; named next step; accurate dates; no invented promise; unresolved questions clearly flagged. |
| Review | Matter lawyer checks every fact, deadline, commitment, judgment call, and client instruction against the summary and file. |
| Hard stops | Invented deadline or commitment; cross-matter information; missing required next step; automated send; unapproved source access. |
| Measures | Total time, material corrections, completeness, client clarity, reviewer effort, user compliance, control exceptions, and cost. |
| Decision | Approve only for the named workflow and input boundary if every required gate and threshold passes. |
Need candidate workflows? Use A Better Tuesday to choose one recurring bottleneck and the Small-Firm Workflow Playbook for practical examples across intake, documents, drafting, client communication, calendar, and billing.
Calculate Full-Process Value
Time saved is not revenue. It becomes value only if the firm uses the capacity to handle more work, reduce delay, improve client service, avoid write-offs, reduce overtime, or remove expense. The pilot should say which outcome matters.
Use the same boundary on both sides.
- Current total minutes = collection + production + review + correction + handoff + exceptions + administration.
- Candidate total minutes = the same seven components after the tool is added.
- Net usable minutes per item = current total minutes minus candidate total minutes.
- Monthly capacity hours = net usable minutes multiplied by realistic monthly volume, divided by 60.
- Monthly realizable benefit ($) = additional collected revenue or avoidable expense from capacity actually used. Do not treat every saved hour as revenue.
- Monthly net value ($) = monthly realizable benefit minus monthly license, support, and incremental administration costs, minus the monthly allocation of one-time implementation and training costs.
- Report faster service and quality outcomes separately when they have no defensible dollar value. Do not count administration time on both sides of the calculation.
Do not hide these costs
- Partner, associate, paralegal, and administrator time spent learning, configuring, testing, reviewing, correcting, documenting, and supporting the process.
- License minimums, add-ons, usage charges, integration fees, consultants, training, security review, data migration, and contract review.
- Work displaced from the test users and reviewers during the pilot.
- Renewal increases, unused seats, duplicate capabilities, cancellation limits, and switching or export work.
- Client-service harm, rework, delay, write-offs, or incident response when the process fails.
Do not value every saved hour at a lawyer’s full billing rate unless the firm can realistically turn that hour into collected revenue. A capacity hour may instead reduce backlog, shorten response time, or allow staff to perform higher-value work. Name the benefit. Fantasy invoices do not pay software renewals.
Practical resources. One sensible starting point.
Get the Playbook and occasional Small Law Firm AI Workflow Brief emails about useful workflow changes, Florida guardrails, and tool updates for firms under 25 lawyers. Unsubscribe anytime.
- Firm AI-readiness checklist
- AI tool-approval scorecard
- Five workflows worth testing first
- Daily lawyer verification checklist
- Plain-English guide to Florida Bar Opinion 24-1
- Sample client language
- Incident-response protocol
- 90-day implementation outline
Make the Buy-or-Do-Not-Buy Decision
The decision should fit on one page. It should identify the evidence, the approved use, the limits, the owner, and the next trigger. The purpose is not paperwork. It is to prevent “we tested it” from becoming folklore six months later.
| Decision | Evidence required |
|---|---|
| Approve or buy | Every required gate passes; thresholds are met; controls, owner, users, training, support, and review are ready. |
| Approve with limits | The evidence supports a narrower use, data boundary, user group, feature set, or review process. |
| Revise and retest | A specific workflow, instruction, configuration, input, or review change may cure the failed measure. |
| Defer | The need is real, but a material answer, integration, control, budget, owner, or product capability is not ready. |
| Reject | A required gate fails, the risk cannot be acceptably limited, the workflow fit is poor, or the total value is negative. |
| Buy nothing | The current process, current software, a template, a checklist, better delegation, or no change produces the better result. |
The decision record should include
- Date, decision maker, pilot owner, reviewer, administrator, and users.
- Exact product, plan, feature, account, settings, and version tested.
- Approved workflow, input boundary, output, review, restrictions, and prohibited uses.
- Baseline, test set, results, failures, exceptions, thresholds, and proof-gate findings.
- Contract, pricing, renewal, evidence-file location, support path, and exit steps.
- Training, rollout group, monitoring measures, incident escalation, next review date, and change triggers.
Run the Exit Test Before Signing
A product is easier to admire before the data arrives and the renewal notice follows. Test the exit while the vendor still wants the sale.
- Can the firm export its data, prompts, templates, logs, results, and configuration in usable formats? What cannot be exported?
- Can an administrator remove a user immediately and preserve the records the firm needs?
- What happens to active data, retained data, backups, derived data, and logs after termination?
- Will the vendor confirm deletion, and what exceptions remain?
- Can the firm reduce seats, features, storage, or usage before renewal? What notice period applies?
- What happens if the model, feature, subprocessor, data practice, price, or material term changes?
- What manual process keeps the work moving during an outage, suspension, dispute, or migration?
Cancel it on paper before you buy it. The easiest subscription to leave is the one whose exit was designed before the first invoice.
After Approval, Keep Testing
Approval is not permanent. Products, models, terms, data practices, permissions, connectors, users, and workflows change. The firm should monitor the measures that justified the decision and reopen the review when the evidence changes.
Review triggers
- A new model, feature, connector, automated action, data source, or administrator setting.
- A material change to terms, training, retention, access, subprocessors, security, pricing, or liability.
- A new practice area, data type, client requirement, user group, or external communication.
- A material error, missed review, unauthorized disclosure, complaint, incident, outage, or unexplained result.
- A renewal, seat expansion, major price increase, ownership change, or proposed replacement.
- Measured value, adoption, quality, service, or control performance falling below the decision threshold.
A product that passed last year may fail the current use. A product that failed a broad use may later pass a narrow one. The decision belongs to the evidence, not the logo.
Frequently Asked Questions
How long should a small-firm AI pilot last?
For one narrow, repeatable workflow, ten business days often produces enough evidence for a limited decision. A complex integration, higher-risk data use, or low-volume task may require longer. Time is not the main requirement. The pilot needs enough representative work to test ordinary cases, difficult cases, controls, review, and recovery.
How many users should participate?
Use a small group that represents the people who will perform and review the work, often two to five users in a small firm. Include at least one ordinary user, not merely the person most excited about the product. Enthusiasm is useful. Representativeness is better.
Can we use a free trial with client information?
Do not assume that free means approved. Review the exact account, terms, retention, training, access, security, deletion, and proposed use. Analyze Florida confidentiality duties and any need for disclosure or informed consent. Start with fictional, public, anonymized, or otherwise approved material until the firm has made a documented decision.
Should the vendor design the pilot?
The vendor can help configure the product and explain capabilities. The firm should own the workflow, test set, baseline, measures, hard stops, review, and decision. Otherwise the pilot becomes an extended demonstration.
How should we compare two products?
Use the same workflow, input boundary, representative test set, output requirements, users, reviewer, timing boundary, and decision measures. Record differences in configuration and control. A different test for each product produces anecdotes, not a comparison.
What if the vendor will not answer a material question?
Record the answer as unknown. Decide whether the use can proceed with public or fictional information, a narrower connection, or another control. If the proposed use depends on the missing answer, pause or reject it. Silence does not mature into due diligence with age.
What counts as ROI for a law firm?
Count the outcome the firm can actually use: collected revenue capacity, faster cycle time, fewer write-offs, lower overtime, reduced outside cost, better client communication, fewer errors, or less backlog. Subtract review, correction, training, administration, and subscription cost. Saved minutes alone are inventory.
What if the tool is accurate but lawyers will not use it?
The operating-value gate fails. Find the friction: wrong workflow, too many transfers, unclear instructions, weak integration, poor training, bad output format, or no owner. Fix one identified cause and retest. Do not buy more seats to improve morale.
When is a new product justified instead of software we already own?
Start with the current-stack audit. A new product is justified when the current tools cannot meet a valuable workflow need or acceptable control, quality, integration, or review requirement, and the candidate proves a net gain against the same baseline and test set.
How often should an approved tool be reviewed?
Review before renewal and after a material change to the product, model, terms, data practices, security, connectors, permissions, users, workflow, client requirement, or incident. Also monitor the measures that justified approval. A dated decision needs a dated recheck.
Bring one candidate tool, one recurring workflow, and whatever evidence the vendor has supplied.
We will identify the test boundary, baseline, measures, reviewer, hard stops, and whether the firm should use what it owns, run a pilot, narrow the proposed use, or stop shopping.
No sales presentation. No product quota. No need to pretend a free trial is a strategy.
About Greg Yates
Greg Yates is the founder of Actionable AI and a former Big Law corporate equity partner at Steptoe (NYC Managing Partner) and Seyfarth Shaw. He helps Florida law firms with fewer than 25 lawyers improve recurring work using AI capabilities in software they may already own. He is independent, does not sell software, and will say when a product or AI itself is not the answer.
Learn more about Greg, browse the Florida AI Guides, or review Actionable AI services.
Primary Authorities and Evaluation Resources
- Florida Bar Ethics Opinion 24-1: Generative AI, confidentiality, provider review, oversight, fees, and advertising.
- Rules Regulating The Florida Bar: Current professional-conduct rules and updates.
- The Florida Bar Guide to Getting Started with AI: Florida Bar educational guidance on cautious experimentation, review, and firm controls.
- ABA Formal Opinion 512: Additional guidance on competence, confidentiality, communication, supervision, candor, and fees.
- NIST AI Risk Management Framework: Voluntary framework for governing, mapping, measuring, and managing AI risk.
- NIST Generative AI Profile: Generative-AI risks and suggested risk-management actions.
- NIST AI RMF Playbook: Suggested actions aligned to the Govern, Map, Measure, and Manage functions.
- CISA Secure by Demand Guide: Questions software customers can use to demand stronger security evidence and defaults.
- FTC Workado AI Claims Order: A buyer reminder that impressive AI accuracy percentages require substantiation.