Survey Software for Randomized Controlled Experiments

Most survey software wasn't designed for experiments. It was designed for customer feedback, then marketed to researchers. The features that matter for experimental control are often missing or poorly implemented.

Running a randomized controlled experiment through a survey platform requires capabilities that go beyond what many tools offer. You need true random assignment to conditions, not just shuffled answer options. You need randomization that's logged in your data, not just applied invisibly. You need branching logic that maintains experimental integrity when paths diverge.

This guide covers what experimental researchers should look for in survey software, where platforms commonly fall short, and how to evaluate whether a tool can actually support your research design.

TL;DR:

• Block-level randomization (assigning participants to conditions) is different from option-order randomization. Many platforms only offer the latter.
• Randomization logging in exported data is essential for analysis. If you can't see which condition a participant was assigned to, you can't analyze the experiment.
• Stratified randomization ensures balanced groups across key variables. Without it, you risk confounds in small samples.
• Counterbalancing for within-subjects designs requires Latin square or similar controls. Few survey tools support this natively.
• Design integrity means your randomization and branching logic don't conflict. Test thoroughly before launch.

→ Try Lensym for Experimental Research

What Experimental Research Requires

Experimental designs impose specific technical requirements that distinguish them from descriptive surveys or customer feedback collection.

Between-Subjects Designs

In a between-subjects experiment, different participants experience different conditions. The survey platform must:

1. Randomly assign participants to conditions at the start of the survey (not partway through)
2. Route participants through condition-specific content using branching logic
3. Log condition assignment in the exported data so you know who saw what
4. Maintain assignment integrity even if participants navigate backward, refresh, or partially complete and return later

A 2×2 factorial design requires four conditions. The platform needs to assign participants to one of four paths with equal probability (or specified probabilities for unequal allocation designs).

Common failure mode: Many platforms can randomize which questions appear, but cannot randomly assign a participant to a complete condition path at survey start. You end up with participants who see a random mix of elements rather than a coherent experimental condition.

Within-Subjects Designs

In a within-subjects experiment, the same participant experiences multiple conditions. Requirements include:

1. Order randomization of condition blocks (or counterbalancing)
2. Carry-forward of responses across conditions for analysis
3. Timing controls if conditions require minimum exposure duration
4. Fatigue management through proper survey length estimation

The key challenge is order effects. If every participant experiences Condition A before Condition B, you cannot separate treatment effects from order effects.

Common failure mode: Platforms that randomize question order within a section often cannot randomize entire sections (condition blocks) against each other.

Factorial Designs

Factorial experiments cross multiple independent variables. A 2×3 design has six cells; a 3×3 has nine. Requirements:

1. Cell assignment with equal or specified allocation ratios
2. Factor-level tracking in exported data (not just cell ID, but which level of each factor—e.g., Framing = Gain vs Loss, Incentive = Present vs Absent)
3. Stratification to ensure balance across cells when sample sizes are small

Common failure mode: Platforms may support simple A/B assignment but struggle with multi-factor designs that require coordinated assignment across multiple dimensions.

Critical Features for Experimental Survey Software

1. Block-Level Randomization

This is the most important capability and the most commonly missing one.

What it means: Participants are randomly assigned to one of several complete survey paths (conditions) at the survey's start. Each path may contain different questions, different stimuli, different response options, or different framings.

What to evaluate:

• Can you define multiple "blocks" or "paths" that constitute different conditions?
• Can you control assignment probability (50/50, 60/40, unequal allocation)?
• Is assignment determined at survey start, not at the first branching point?
• Does assignment persist if the participant refreshes or returns later?

Why it matters: Without block-level randomization, you cannot implement a true between-subjects design. You're limited to within-subjects comparisons or quasi-experimental approaches.

2. Randomization Logging

Randomization that isn't logged is randomization that can't be verified or analyzed.

What it means: Your exported data includes a variable indicating which condition each participant was assigned to, which order they saw randomized elements in, and (for factorial designs) which level of each factor they experienced.

What to evaluate:

• Does the platform create a condition assignment variable automatically?
• For order randomization, is the actual order logged?
• Can you access randomization data at the individual response level?
• Is the randomization seed logged for reproducibility?

Why it matters: During analysis, you need to know exactly what each participant experienced. "Randomization was applied" is not sufficient for reporting methods sections or for debugging unexpected results.

3. Stratified Randomization

Simple randomization can produce imbalanced groups, especially with smaller samples.

What it means: Randomization is constrained to ensure that key demographic or stratification variables are balanced across conditions. If you're stratifying by gender, you ensure roughly equal gender distribution in each condition.

What to evaluate:

• Can you specify stratification variables?
• Does stratification work with your other randomization requirements?
• Is the stratification algorithm documented (block randomization within strata, minimization, etc.)?

Why it matters: An experiment where 80% of one condition is female and 80% of another is male has a confound. Stratified randomization prevents this.

Note: For large samples (n > 200 per condition), simple randomization usually produces adequate balance. Stratification matters most for smaller studies.

4. Counterbalancing Support

For within-subjects designs, counterbalancing controls order effects.

What it means: Rather than randomizing order for each participant individually (which may not achieve balance), the platform implements a systematic counterbalancing scheme like Latin square design.

What to evaluate:

• Can you implement Latin square counterbalancing?
• Can you specify complete counterbalancing (all possible orders)?
• Is the counterbalancing condition logged in exported data?
• Can you combine counterbalancing with other randomization?

Why it matters: If you have three conditions and simply randomize order, you might end up with 50% of participants seeing ABC order and 50% seeing other orders combined. Latin square ensures each order occurs equally often.

5. Seed-Based Reproducibility

For academic research and pre-registered studies, randomization should be documentable and reproducible.

What it means: You can specify a random seed that makes your randomization sequence deterministic. Given the same seed and the same participant order, the same assignments occur.

What to evaluate:

• Can you set a seed for the randomization algorithm?
• Is the seed (or effective seed) logged in exported data?
• Can you reproduce assignments for auditing or replication?

Why it matters: Pre-registration and peer review increasingly require evidence that randomization was implemented as described. Seed-based logging provides that verification. For product experiments where auditability matters less, this is a nice-to-have rather than essential.

6. Branching Logic That Maintains Integrity

Experimental surveys often combine randomization with conditional logic. These must work together.

What it means: If a participant is assigned to Condition A and then answers a screening question that triggers a branch, they should remain in Condition A along that branch. Randomization and branching should be orthogonal.

What to evaluate:

• Does branching logic correctly handle randomized paths?
• What happens if a randomized element is inside a conditional branch?
• Can you visualize the interaction between randomization and branching?
• Does the platform warn about potential conflicts?

Common failure mode: Branching logic resets randomization state, causing participants to "switch" conditions mid-survey.

7. Stimulus Presentation Controls

Some experiments require controlled stimulus exposure.

What it means: You can control how long participants see a stimulus (minimum display time, maximum display time), prevent skipping, and log actual viewing time.

What to evaluate:

• Can you set minimum page display times?
• Can you prevent "back" navigation after stimulus exposure?
• Is actual page time logged (not just total survey time)?
• Can you embed multimedia stimuli with timing controls?

Why it matters: If participants can skip through stimulus pages in 2 seconds, your manipulation may not have been experienced as intended.

Common Failure Patterns

Many survey platforms were designed for market research or customer feedback. They've added "randomization" as a feature, but the implementation often doesn't meet experimental research requirements. Here's what to watch for:

"Randomization" that isn't experimental randomization. Many platforms advertise randomization but only offer answer option shuffling, question order shuffling within sections, or random question display (showing a subset). None of these constitute experimental condition assignment. You need block-level assignment to complete condition paths.

Randomization without logging. A platform may randomize correctly but fail to record it. If your exported data doesn't include which condition each participant was assigned to, which order they experienced, or which factor levels they saw, you cannot analyze the experiment properly. We've seen researchers discover mid-analysis that their platform logged "randomization applied" without specifics. The data was unusable for the intended analysis.

Branching that breaks randomization. When experimental surveys combine randomization with conditional logic, things can fail silently. Randomization state may reset at branch points. Participants may "switch" conditions mid-survey without any error message. The survey appears to work, but the data contains impossible patterns: the same participant ID appearing in multiple conditions, impossible transitions between randomized branches, or cell distributions too uneven to be explained by chance. These issues often aren't discovered until analysis, when it's too late.

Stratification that doesn't scale. Basic platforms may offer no stratification, single-variable stratification only, or stratification that conflicts with other randomization features. Complex factorial designs require stratification across multiple variables simultaneously.

Evaluation Protocol for Experimental Use

Before committing to a platform for experimental research, test systematically.

Phase 1: Basic Capability Check

1. Create a simple 2-condition between-subjects design

   • Define two condition paths with different content
   • Implement random assignment at survey start
   • Complete the survey 20+ times
   • Export and verify: Is condition assignment logged? Is distribution approximately 50/50?

2. Create a within-subjects design with three conditions

   • Define three blocks that every participant sees
   • Implement order randomization
   • Complete 20+ times
   • Export and verify: Is actual order logged? Are all orders represented?

Phase 2: Complex Design Test

3. Create a 2×2 factorial design

   • Four conditions crossing two factors
   • Implement random assignment to cells
   • Add a branching question within one condition
   • Complete 30+ times
   • Verify: Are all four cells represented? Does branching work correctly within conditions? Are both factor levels logged separately?

4. Test stratified randomization

   • Add a stratification variable (e.g., simulated gender)
   • Verify balance across conditions
   • Check that stratification is logged

Phase 3: Edge Case Testing

5. Test navigation edge cases

   • Complete survey, use back button, continue forward
   • Refresh browser mid-survey
   • Complete on mobile with interruption
   • Verify: Does condition assignment persist? Is data integrity maintained?

6. Test data export thoroughly

   • Export to your analysis software
   • Verify all randomization variables are present and correctly coded
   • Check that timing data is available
   • Confirm codebook/metadata is adequate for analysis

Documentation Requirements

For any platform you're considering, verify you can document:

• The randomization algorithm used
• How stratification is implemented (if applicable)
• What variables are logged and their coding
• How the platform handles edge cases (back navigation, timeout, etc.)

This documentation is necessary for methods sections and may be requested by reviewers.

Implementation Recommendations

For Simple Between-Subjects Designs

If you're running a straightforward A/B or A/B/C experiment:

1. Create separate survey branches for each condition
2. Use block-level randomization at survey start
3. Verify logging in a pilot test (10-20 responses)
4. Include a condition identifier question as backup
5. Document the randomization approach in your pre-registration

For Factorial Designs

Multi-factor experiments require more careful setup:

1. Map out all cells before building (2×2 = 4, 2×3 = 6, 3×3 = 9)
2. Create the full branching structure first, then add randomization
3. Ensure each factor level is logged separately (not just cell ID)
4. Test all cells with multiple completions each
5. Calculate required sample per cell for power analysis

For Within-Subjects Designs

Order effects are the primary concern:

1. Decide on counterbalancing approach (full, partial, Latin square)
2. Implement using whatever mechanism the platform provides
3. Log actual presentation order per participant
4. Include manipulation checks at the end of each condition
5. Plan analysis to test for order effects

For Complex Designs

If your design combines multiple features (e.g., factorial + within-subjects + stratification):

1. Diagram the full design before touching the survey platform
2. Build incrementally, testing each component
3. Run a full pilot with 30+ responses before launch
4. Have a colleague attempt to complete the survey in unexpected ways
5. Build in data quality checks (attention checks, timing thresholds)

What This Means for Platform Selection

Must-Have Capabilities

For any experimental research, the platform must provide:

• Block-level condition assignment (not just option shuffling)
• Randomization logging in exported data
• Branching logic that works with randomization
• Question-level timing data
• Reliable data export to statistical software

Nice-to-Have Capabilities

Depending on your design:

• Stratified randomization
• Latin square counterbalancing
• Seed-based reproducibility
• Stimulus timing controls
• Visual logic editor showing randomization paths

Deal-Breakers

If a platform has any of these issues, it's unsuitable for experimental research:

• Randomization not logged in data
• Condition assignment resets on back navigation
• No way to implement block-level randomization
• Branching and randomization conflict silently
• Data export loses critical metadata

The Testing Imperative

No amount of feature documentation substitutes for testing. Platforms may claim capabilities they implement poorly. The only way to know is to build a representative experiment and stress-test it.

Budget time for this evaluation:

• Quick evaluation (2-4 hours): Simple between-subjects test, basic export check
• Thorough evaluation (1-2 days): Full design implementation, edge case testing, export validation
• Pilot study (1 week): Real participants, full analysis pipeline, methods section draft

The time invested prevents discovering problems after data collection, when it's too late to fix them.

---

Why Lensym Works for Experiments

Lensym was designed with experimental research in mind, not retrofitted for it.

Analyzable data from day one. Block-level condition assignment with full logging means your exported data includes everything you need: which condition, which order, which factor levels. No post-hoc guessing about what participants actually saw.

Catch design errors before launch. The visual graph editor shows your entire experiment flow, including how randomization and branching interact. You can simulate participant paths through all conditions and spot logic conflicts before they become data quality problems.

Documentable methodology. Seed-based randomization, version-controlled designs, and exportable survey documentation support pre-registration and peer review workflows.

Research-first workflow. Pilot mode keeps test data separate. Collaboration features support research teams. Ethics documentation exports cleanly.

→ Evaluate Lensym for Your Experiment

---

Related Reading:

• Survey Randomization: When It Helps, When It Hurts
• Survey Branching Logic: A Complete Guide for Researchers
• How to Evaluate Survey Software for Academic Research
• Survey Tools for Academic Research: What Features Actually Matter
• Types of Survey Bias: A Taxonomy for Researchers

---

Experimental design requirements vary by discipline and specific research questions. This guide covers common needs; consult your methodology resources for design-specific guidance. Always pilot test before launching experiments.