Hiring a Conversion Rate Optimization (CRO) agency or consultant can feel overwhelming. Everyone claims they can boost your conversions, but the reality is: not all partners have the depth of expertise, research discipline, or technical chops to deliver consistent, meaningful results.

The stakes are high. Choose the wrong partner, and you could end up wasting precious budget on vanity tests, shallow insights, or empty promises. Choose the right one, and you’ll embed a system of rigorous testing and research that drives sustainable revenue growth.

To help you separate true CRO experts from the pretenders, here are crucial questions to ask before signing any contract—plus what to look for (and what to avoid) in the answers.

How do you estimate the potential ROI for your CRO projects?

Keep in mind that any agency or consultant that is projecting annualized numbers for results or revenue because this is not an accurate way to represent CRO success. Too many variables can influence annualized projections.

Why it matters: CRO doesn’t guarantee a neat, linear ROI. Real results depend on test velocity, traffic volume, baseline conversion rates, and dozens of external variables (seasonality, ad spend, competitor activity, etc.).

What to look for:

• Agencies that avoid annualized projections (“We’ll get you $2.4M in extra revenue per year!”). Those are misleading because they ignore variability and compounding effects.
• Honest answers that focus on ranges, scenarios, and confidence intervals, not guarantees.
• An explanation of how they calculate potential ROI using sample size, test duration, and lift estimates.

Red flags:

• They show exact dollar projections (“we’ll make you $2.3M in 12 months”).
• They present annualized revenue lifts as proof of CRO results.
• They avoid discussing assumptions or variability behind their numbers.
• They guarantee results within a specific time frame (e.g., “20% lift in 3 months”).

How does your team stay updated with the latest CRO tools and strategies?

This question can help you make sure the team actually specializes in CRO and isn’t trying to be a jack-of-all-trades.

Why it matters: CRO evolves quickly. From new testing methodologies to privacy regulations to analytics tools, what worked five years ago may be irrelevant (or even harmful) today.

What to look for:

• Specific mentions of conferences, social platforms, blogs, communities, etc. they participate in (e.g., CXL, experimentation forums, analytics Slack groups).
• Evidence that CRO is a dedicated practice area, not a bolt-on to their SEO or PPC service.
• Familiarity with advanced tools (e.g., experimentation platforms, behavioral analytics, CDPs).

Red flags:

• They only mention tool certifications (e.g., Google Optimize badge) but not active learning communities.
• They position themselves as generalist marketers (SEO, PPC, branding, CRO) with no focus.
• They can’t name recent conferences, publications, or communities they follow.
• Agencies that describe themselves as “full-service” across SEO, PPC, email, branding, and CRO without clear specialization.

Can you walk me through your process for planning and conducting a test?

Make sure this process includes pre-test calculations and tons of research. If it doesn’t, that’s a red flag!

Why it matters: Running tests without rigor leads to bad decisions. CRO isn’t about slapping together “button color” tests—it’s about combining research, hypothesis design, statistical planning, and disciplined execution.

What to look for:

• Clear explanation of pre-test calculations (approached appropriately for the stats being used).
• Heavy emphasis on research inputs—heatmaps, analytics, surveys, user testing, heuristic reviews.
• A structured process for hypothesis development, test design, launch, monitoring, and post-test analysis.

Red flags:

• They skip pre-test calculations (sample size, power, MDE).
• They can’t explain where hypotheses come from (research, data, user input).
• Their “process” is just launching lots of tests quickly.
• They emphasize cosmetic changes (button colors, font tweaks) over real hypotheses.

Can you provide an example of a time where you were unable to meet a client’s expectations? How did you handle the situation?

No matter how hard an agency tries, sometimes you might fall short. If an agency or consultant is telling you that they always deliver, they might not be providing an accurate picture of their experience, or they may be lacking experience. 

Why it matters: CRO is inherently uncertain. Not every test wins. What matters is how the agency handles setbacks and what systems they have in place to learn from them.

What to look for:

• Transparency: They should be comfortable admitting not every test wins.
• A focus on learning value—failed tests still provide insights that guide future strategy.
• Evidence they communicated clearly with the client and adjusted expectations proactively.

Red flags:

• They say they’ve never failed or that “every test is a win.”
• They avoid answering or pivot to another topic.
• They blame clients or circumstances without taking responsibility.
• They can’t describe what they learned from a failed test.
• They lack a process for turning failed tests into insights.

What is your perspective relative to how many variables there can be per test?

Make sure your potential agency or consultant is not proposing only testing one thing at a time or a lot of small tests, if so they are using outdated approaches. 

Why it matters: CRO has moved past testing one button at a time. Mature programs run tests with multiple variables (copy, layout, interaction patterns) and prioritize impactful hypotheses over “micro-optimizations.”

‍What to look for:

• A nuanced answer: sometimes single-variable tests are necessary, but they should also be comfortable with multivariate or multi-change tests when appropriate.
• Emphasis on test design that maximizes insight without compromising interpretability.

Red flags:

• They insist on testing one change at a time with no exceptions.
• They suggest running as many variables as possible without explaining risk to interpretability.
• They can’t discuss interaction effects or how multiple variables can be structured.
• Their examples focus on low-impact tweaks (headline changes, color tests) as their core strategy.
• They don’t connect variable strategy to traffic levels or sample size needs.

Talk to me about the difference/preference on bayesian vs. frequentist, fixed-horizon vs. sequential, and one-tailed vs. two-tailed testing.

If they can’t speak to the differences or have no idea about what you’re asking about, then they don’t have the depth of knowledge needed to be a true CRO partner for your business. You also want to make sure your agency or consultant has opinions for the gray areas of CRO. If they are just willing to say “yes” and never push back against a client, you will be unlikely to see significant results from your CRO efforts. 

Why it matters: If an agency can’t explain the basics of statistical frameworks, they shouldn’t be touching your data. Methodology choices directly affect how decisions are made.

What to look for:

• A clear preference and the ability to explain why they use it (Bayesian for continuous monitoring, Frequentist for fixed horizon, etc.).
• Awareness of trade-offs between speed, rigor, and error control.
• A willingness to push back on clients if the requested approach isn’t sound.

Red flags:

• They don’t know what Bayesian vs. Frequentist means.
• They say “we just use the testing tool’s defaults” without knowing what that implies.
• They can’t articulate when to use one-tailed vs. two-tailed tests.
• They avoid taking a stance and give vague, generic answers.
• They dismiss stats as unnecessary “technical stuff.”

What specific benefits have your clients seen from using AI in your projects?

If they are overselling you on AI, run. While AI can be useful for some aspects of digital marketing, it should not be the leading focus of your CRO strategy.

Why it matters: AI is everywhere—but in CRO, it’s more support tool than silver bullet. Overselling AI is a sign the agency is chasing trends, not building substance.

What to look for:

• Examples of AI applied in practical, limited contexts (e.g., clustering qualitative feedback, predicting test durations, generating copy variations).
• A clear stance that AI is supplementary, not a replacement for human judgment and research.

Red flags:

• They position AI as their core differentiator.
• They can’t provide specific, measured examples of AI usage.
• They promise AI can predict winners automatically.
• They avoid discussing the limitations of AI.
• They recommend replacing human research or hypothesis-building with AI entirely.

How do you prioritize which tests to run first?

You’ll likely have dozens of potential ideas to test, but not all are equal in value. The way an agency prioritizes experiments determines whether your CRO program drives impact quickly or wastes time on low-value tweaks.

Why it matters: Test prioritization determines how quickly you see meaningful results and whether your program scales strategically.

What to look for:

• Use of prioritization frameworks like PIE (Potential, Importance, Ease), ICE (Impact, Confidence, Ease), or PXL (a more granular model).
• Emphasis on testing high-impact opportunities first (critical funnel steps, high-traffic pages, major barriers).
• Ability to balance quick wins with strategic, deeper tests.
• Willingness to adapt prioritization to your business goals (e.g., revenue focus vs. lead quality).
  

Red flags:

• “We’ll just test whatever you want.” → No strategic filter.
• Overemphasis on volume over impact (“20 tests a month!”).
• Always going for “easy” tests instead of high-value ones.

How do you ensure results are documented and shared?

The value of CRO isn’t just in one winning test—it’s in building a knowledge base of what works for your users. Without systematic documentation, insights are lost, and mistakes get repeated.

Why it matters: Documentation ensures learnings compound over time instead of vanishing after a single campaign.

What to look for:

• A structured test library or knowledge base (e.g., Airtable, Notion, or a proprietary repository).
• Regular reporting cadences that summarize not just results but insights and implications.
• Sharing learnings with both marketing and product teams to amplify impact.
• Ability to tie test results back to broader hypotheses and research themes.
  

Red flags:

• Only sending screenshots from testing tools.
• No central place where learnings are stored.
• “We’ll just email you the results.”

Can you integrate with our analytics and data stack?

CRO depends on accurate, trustworthy data. If an agency can’t integrate with your existing stack, you risk siloed reporting or misleading insights.

Why it matters: CRO doesn’t live in isolation. It must align with your company’s analytics, BI, and data warehouse to provide context and accuracy.

What to look for:

• Ability to connect with GA4, Adobe Analytics, Mixpanel, Amplitude.
• Comfort integrating with CDPs (e.g., Segment).
• Understanding of data warehouses and BI tools (BigQuery, Snowflake, Redshift, Looker, Tableau).
• Proven processes for data QA and consistency checks across tools.
  

Red flags:

• “We only use the testing tool’s data.”
• Lack of familiarity with your analytics setup.
• Resistance to technical collaboration with your data team.

How do you handle traffic segmentation and targeting?

Different users behave differently. Without segmentation, you risk flattening insights into misleading averages.

Why it matters: Segmentation can reveal which audiences respond positively or negatively, guiding smarter personalization and strategy.

What to look for:

• Ability to segment by device, traffic source, geography, customer type.
• Awareness of sample size trade-offs when segmenting.
• Experience with personalization tools (Optimizely, Dynamic Yield, VWO, etc.).
• Clear process for testing across vs. within segments.
  

Red flags:

• No mention of segmentation in their methodology.
• Over-segmentation without traffic to support it.

What's your process for QA and technical validation of tests?

Poorly built tests can break site functionality, distort analytics, or erode trust. QA is non-negotiable.

Why it matters: CRO tests touch live users. A solid QA process prevents embarrassing or costly mistakes.

What to look for:

• A documented QA checklist covering browsers, devices, and key flows.
• Testing in staging environments before going live.
• Procedures for data validation—ensuring test data matches analytics data.
• Post-launch monitoring for anomalies.
  

Red flags:

• No clear QA process.
• “We only rely on the client’s dev team for QA.”
• History of broken tests or inaccurate tracking.

How do you handle conflicts when stakeholders disagree with your CRO recommendations?

Stakeholders often come with their own strong opinions. A real CRO partner can navigate conflict with evidence and confidence.

Why it matters: You need a partner who stands up for rigor, not one who folds under pressure.

What to look for:

• A diplomatic but firm stance: willing to educate, but not afraid to push back.
• Frameworks for balancing business priorities with testing discipline.
  

Red flags:

• Agencies that always agree with the client.
• No examples of past conflicts (“we’ve never had that happen”).
• Fear of being opinionated.

How do you measure success beyond conversion rates?

Conversion rate is just one metric. True CRO impacts revenue, retention, and long-term value.

Why it matters: If you only look at CR, you might celebrate shallow wins while ignoring deeper revenue drivers.

What to look for:

• Metrics like Revenue per Visitor (RPV), Average Order Value (AOV), funnel progression, and retention.
• Awareness of how CRO impacts customer LTV and acquisition efficiency (CAC payback).
• Integration with marketing and finance KPIs.
  

Red flags:

• Focus on “conversion rate lift” only.
• No connection to revenue or business outcomes.
• Reporting that stops at percentages, not dollars.

Can you describe your approach to qualitative research?

Quantitative data tells you what happened; qualitative tells you why. CRO without user research is half-blind.

Why it matters: Research-driven CRO produces deeper hypotheses and smarter tests.

What to look for:

• Use of surveys, polls, usability studies, and interviews.
• Incorporation of session recordings, heatmaps, and heuristic reviews.
• Ability to triangulate qual insights with quantitative data.
• Regular cadence of continuous discovery (not one-time audits).
  

Red flags:

• “We only look at analytics.”
• No experience running surveys or usability tests.
• Over-reliance on tools without human interpretation.

What does your typical client engagement look like over 6-12 months?

CRO is not a one-off project—it’s a program. You need clarity on how the agency evolves their approach over time.

Why it matters: Long-term CRO should scale from quick wins to deeper structural optimization.

What to look for:

• Month-by-month roadmap that evolves (research → quick wins → deeper testing → program maturity).
• Plan for scaling test velocity as research and processes mature.
• Emphasis on building a sustainable experimentation culture, not dependency.
  

Red flags:

• “We’ll run X tests per month indefinitely.”
• No plan beyond 90 days.
• No roadmap for growing sophistication.

How do you align CRO with broader business goals?

CRO should accelerate—not operate separately from—your company’s growth strategy.

Why it matters: Without alignment, CRO risks becoming a silo that optimizes micro-metrics instead of revenue.

What to look for:

• Understanding of your acquisition and retention strategy.
• Ability to tie CRO efforts into campaign performance, product development, or pricing strategies.
• Collaboration with marketing, product, and finance teams.
  

Red flags:

• “We just optimize the site.”
• No knowledge of your business model or goals.
• No effort to integrate insights across departments.

‍

Asking these questions will help you uncover any major issues before committing to a partnership with your next CRO agency or a consultant.

While on the search for CRO expertise, make sure you don't miss out on our post on 10 Red Flags When Choosing Your CRO Agency.