RFP scorecard: how to evaluate a digital agency before you sign
A practical scorecard non-technical buyers can use to evaluate a digital agency: proof, references, ownership, accessibility, scope, and exit terms.
The short version: five things that actually predict a good outcome
If you only have ten minutes before you sign with a digital agency, score them on five things. Can they show relevant proof of work like yours, with references you can actually call? Do you know who will do the hands-on work, not just who sold you? Is it written down that you own your code, content, data, and accounts? Is the scope clear, with a defined way to handle changes? And do you know what happens if you part ways? Everything else is a tiebreaker. An agency that answers those five well rarely surprises you later. One that gets vague on any of them usually does.
The rest of this is a scorecard you can put next to any proposal. For each criterion there is what a good answer sounds like and what a bad answer sounds like, so you can score it yourself even if you are not technical. Use it to evaluate a digital agency the same way you would any vendor you are trusting with something that matters: by what they will commit to in writing, not by how good the pitch deck looks.
Why a scorecard beats a gut feeling
Most buyers choose an agency on chemistry and the lowest reasonable price. Chemistry matters, but it is the easiest thing to fake for the length of a sales call. A scorecard forces the same questions across every bidder and surfaces the answers that get quiet when the work gets hard. It protects you from the most common failure mode for small organizations: signing with people who are great at selling and absent during delivery.
You do not need technical knowledge to run it. Listen for whether the answers are specific and put in writing, or general and verbal. Specific and written is good. General and verbal is the warning sign.
The scorecard
1. Relevant experience
What you are checking: have they done work like yours, for an organization like yours, recently?
Good answer: two or three projects close to your size, sector, and problem. A small-town government site, a nonprofit donation flow, a local booking system, whatever maps to you. They can explain the goal and what changed because of the work.
Bad answer: a long client logo wall with no detail, or impressive work that is nothing like yours. A glossy enterprise rebrand tells you little about whether they can ship a reliable site for a 12-person nonprofit.
2. References you can call
What you are checking: will real past clients vouch for them when the agency is not in the room?
Good answer: two or three references you may contact directly, ideally including one project that hit a bump. Ask them: did they hit deadlines, who actually did the work, would you hire them again? An agency proud of its delivery hands these over without friction.
Bad answer: stalling, only one reference, or references who turn out to be partners and friends rather than clients. Hesitation here is the single most useful signal in the scorecard.
3. Who does the work versus who sells it
What you are checking: are the people impressing you in the pitch the people who will build your thing?
Good answer: they name the actual team and their roles and tell you how much is in-house versus subcontracted. Subcontracting is fine if they own the relationship and quality. You should know who to call when something breaks.
Bad answer: a polished salesperson who cannot say who does the work, or a promise of senior attention that quietly becomes the most junior person on staff after signing. Ask plainly: who will I be emailing in week six?
4. Ownership of your code, content, data, and accounts
What you are checking: when the project ends, do you own everything needed to keep running without them?
Good answer: the contract states you own the source code, content, and data, and that all accounts are registered in your name with you as administrator. Your domain, hosting, analytics, content system, and any AI tools should be yours, with the agency given access rather than the reverse. They hand over credentials and documentation at the end as a matter of course.
Bad answer: accounts created under the agency’s name, a refusal to commit to code or content ownership in writing, or a platform you can only operate by continuing to pay them. This is how organizations get held hostage. If leaving them means losing your website or your data, you do not own your website or your data. For government and nonprofit buyers especially, public records and donor data must stay under your control.
5. Accessibility competence
What you are checking: can they build something usable by people with disabilities, and do they treat it as a requirement rather than an upsell?
This matters everywhere and is a legal requirement for many of you. Federal agencies and their vendors work to Section 508, which references the WCAG accessibility guidelines. State and local governments are now covered by a 2024 Department of Justice rule under ADA Title II that makes WCAG 2.1 Level AA the binding standard for public websites and apps, with compliance deadlines arriving over the next couple of years depending on population size. Nonprofits that serve the public, or take government funding, are often held to the same bar.
Good answer: they mention WCAG 2.1 AA without being prompted, describe how they test (including with a keyboard and a screen reader, not only an automated scanner), and treat accessibility as part of the build rather than an upsell.
Bad answer: a blank look, “our templates are accessible” with nothing behind it, or accessibility added on at the end. An automated scan alone is not accessibility testing, and any agency serving government or nonprofits should know that.
6. Clear scope and a change process
What you are checking: is it written down what you are getting, and is there a sane way to handle the changes that always come up?
Good answer: a specific deliverables list, what is explicitly out of scope, and a defined change process. When you ask for something new mid-project, you get a quick written note on cost and timeline impact before work starts. No surprises on the invoice.
Bad answer: a vague one-line scope, or a process so rigid every small request becomes a billable negotiation. Out-of-scope should be as clear as in-scope.
7. Communication cadence
What you are checking: will you know what is happening without having to chase them?
Good answer: a named point of contact, a regular check-in rhythm, and a clear answer on response times. You should know how status gets reported and where decisions are tracked.
Bad answer: “we will be in touch,” no named owner, or a single overloaded person who goes dark for two weeks. Without a technical staffer of your own, going dark is expensive, because you cannot tell whether silence means progress or a stalled project.
8. What happens if you part ways
What you are checking: can you leave cleanly, on terms you agreed to up front?
Good answer: a written exit and offboarding plan. Notice period, what gets handed over (code, content, data, credentials, documentation), in what format, and any transition support. The end of the relationship is described before it begins.
Bad answer: no exit terms, auto-renewing contracts with long lock-ins, or handover treated as a favor. The cleaner an agency makes it to leave, the more confident they are you will want to stay.
How to score it
Rate each of the eight criteria green, yellow, or red. Any red on ownership, references, or exit terms should stop you until it is fixed in writing, because those are the ones that trap you. A couple of yellows elsewhere are normal and worth a follow-up. The goal is not a perfect score. It is no nasty surprises six months in.
Where AI fits, honestly
If an agency uses AI in the work, the right question is what it gives back to your team. Used well, AI handles the repetitive load so your people spend their limited hours on the work only they can do. It should reclaim your team’s time, not replace the judgment and relationships that make your organization yours. Be wary of anyone selling AI as a way to cut staff, and interested in anyone who can show how it removes busywork while keeping your people in charge.
Hold us to the same scorecard
Rudder is itself a studio and agency. We build AI agents and provide senior technology leadership for small businesses, nonprofits, and local government, and we currently run 12 agents across 3 products. So we mean this plainly: run this scorecard on us too. Ask who does your work, ask for references you can call, ask us to put your ownership of your code, content, data, and accounts in writing, and ask how you would leave us cleanly.
If you want a second set of eyes, send us your situation and we will name the smallest useful next step, even when that step is not hiring an agency at all. Sometimes the honest answer is a small fix you can do yourselves, or one account you should move into your own name this week. We would rather tell you that than sell you a project you do not need.
Reading is free. so is the first call.
Bring us the problem behind the search that got you here. We'll tell you honestly whether we can help, and what the smallest useful engagement looks like.