Survey Software for Complex Between-Subjects Experimental Design

A questionnaire and an experiment have different infrastructure requirements, even when both are delivered as a survey. A questionnaire collects responses. An experiment assigns participants to conditions, delivers treatments, measures outcomes, and requires the data structure to preserve the entire design. Most survey platforms were built for questionnaires. That's the problem.

Between-subjects experimental designs are the backbone of causal inference in social science. They require random assignment, condition isolation, manipulation delivery, outcome measurement, and analysis-ready data structures. Survey platforms are the most common delivery mechanism for online experiments, but the platform's suitability depends on whether it supports experimental infrastructure or just questionnaire features.

The difference is structural. A questionnaire platform needs question types, skip logic, and data export. An experimental platform needs all of that—plus a randomization engine, parallel condition paths, manipulation check integration, blinding controls, and exports that preserve the experimental design.

This guide covers the specific platform capabilities that between-subjects experiments require, organized by the workflow stages of experimental survey research.

TL;DR:

• Randomization must be truly random, balanced, and logged. Simple "random page" features are often insufficient for proper experimental design.
• Condition paths must be fully isolated. Participants should never see content from other conditions or indicators of which condition they are in.
• Manipulation delivery can include text vignettes, images, videos, or interactive elements. The platform must support the stimulus format your design requires.
• Outcome measures must be identical across all conditions. Any variation in the dependent variable section confounds the manipulation with measurement differences.
• Data exports must include condition tags, randomization metadata, and timestamps. Without these, the experimental structure is lost in analysis.

Try Lensym for Experimental Research

Stage 1: Random Assignment

What Proper Randomization Requires

Random assignment is what makes an experiment an experiment. The quality of your randomization determines whether your causal claims hold up.

Simple randomization: Each participant has equal probability of assignment to each condition. This is the minimum requirement. Many "randomization" features in survey platforms are actually systematic rotation—alternating conditions in sequence—which is not random and can introduce order-dependent confounds if participant characteristics change over the recruitment period.

Block randomization: Participants are assigned in blocks (e.g., blocks of 8 for a 4-condition study). Within each block, each condition appears exactly twice. This guarantees balanced cell sizes at every block boundary, which is essential for studies with smaller samples or rolling recruitment.

Stratified randomization: Assignment is balanced within strata defined by key covariates (e.g., equal men and women in each condition). This requires the platform to evaluate covariate responses before making the assignment—real-time logic that most skip-logic systems cannot provide.

Randomization Verification

After data collection, you must be able to verify that randomization worked:

• Cell size balance: Are condition groups approximately equal in size?
• Covariate balance: Do conditions differ on measured covariates? (They should not, if randomization is working.)
• Randomization log: Can you audit the assignment sequence to confirm it was not systematic?

The platform should store the randomization seed or assignment log so that the randomization sequence is reproducible and auditable.

Common Randomization Failures

Rotation instead of randomization: Alternating conditions (participant 1 gets A, participant 2 gets B, participant 3 gets A...) is not random. If recruitment patterns are non-random (e.g., morning participants differ from evening participants), rotation creates systematic assignment bias.

Client-side randomization: If randomization is performed in the respondent's browser (via JavaScript), it can be observed, manipulated, or fail silently. Server-side randomization is more secure and reliable.

No block structure: Simple randomization without blocking can produce substantial cell size imbalances, especially in studies with fewer than 100 participants per cell.

If your experiment depends on balanced randomization across conditions, a platform with built-in block randomization and allocation monitoring prevents you from discovering balance problems after data collection. See how Lensym handles experimental randomization →

Stage 2: Condition Delivery

Stimulus Presentation

Between-subjects experiments deliver different stimuli to different participants. The stimulus format varies by research domain:

Stimulus type	Platform requirement	Example
Text vignette	Rich text support per condition	News article manipulation
Image	Condition-specific image display	Product packaging study
Video	Embedded video per condition	Training effectiveness study
Interactive	Condition-specific page logic	Gamified task manipulation
Scale manipulation	Different response options per condition	Anchoring study

The platform must support the stimulus format your design requires. No workarounds. If your manipulation involves video and the platform doesn't support embedded video, you can't run your study.

Condition Isolation

Participants must experience only their assigned condition. No leakage of information about other conditions should occur.

Technical isolation: Content from non-assigned conditions should never be loaded in the participant's browser. Some platforms load all content and hide non-relevant sections with CSS—meaning the content is technically accessible.

Visual isolation: No visible indicators of condition assignment. URL parameters ("?condition=2"), page titles ("Version B"), or different styling between conditions reveal the assignment.

Temporal isolation: All conditions should have similar structure and timing. If Condition A has 3 pages and Condition B has 5 pages, the different experience length may itself become a confound.

Manipulation Checks

Manipulation checks verify that the experimental manipulation worked as intended. They belong after the manipulation but before (or integrated with) outcome measures.

Example for a credibility manipulation:

• Stimulus: News article attributed to a credible vs. non-credible source
• Manipulation check: "How credible did you find the source of this article?" (1-7 scale)
• Expected result: Higher credibility ratings in the high-credibility condition

If manipulation checks fail—no significant difference between conditions—the experiment cannot be interpreted. The platform must support conditional analysis: examining outcomes only for participants who passed the manipulation check.

Stage 3: Outcome Measurement

Identical Measures Across Conditions

This is a strict requirement: the dependent variable measures must be absolutely identical across all conditions. Same question text, same response options, same order, same page layout. No exceptions.

Any difference in the outcome section between conditions confounds the manipulation with measurement. If Condition A has the dependent variable on page 6 and Condition B has it on page 8—because Condition B had a longer manipulation—the additional survey length could affect responses.

The implementation pattern is:

1. Condition-specific content (stimulus + manipulation check)
2. Convergence point (all paths merge)
3. Shared outcome measures (identical for all participants)
4. Shared demographics and debriefing

The convergence must be explicit in the survey design. In graph-based editors, this is a visible merge node. In skip-logic systems, it requires careful management of question numbering to ensure all paths arrive at the same point.

Attention Checks

Attention checks identify participants who are not engaging with the survey content. Common approaches:

• Instructed response: "Please select 'Strongly disagree' for this item." Participants who select other options were not reading carefully.
• Content-based: Questions about the stimulus that require having read/watched it. "What was the main topic of the article you just read?"
• Timing-based: Flagging participants who complete the survey faster than is plausible for careful reading.

Attention check failure rates in online experiments typically range from 5-20%. Pre-registering your exclusion criteria (e.g., "participants who fail 2 or more of 3 attention checks will be excluded") is important for transparent reporting.

If you're running between-subjects experiments with multiple condition paths, visual editors that show each arm and its convergence point make isolation errors obvious before launch. See how Lensym's graph editor handles experimental design →

Stage 4: Data Export and Analysis

Required Export Variables

Variable	Purpose	Required for
participant_id	Unique identifier	All analyses
condition_[factor]	Assignment per factor	ANOVA, regression
cell	Combined condition	Interaction analysis
randomization_block	Block membership	Randomization verification
manipulation_check_[n]	Check responses	Manipulation verification
attention_check_[n]	Check responses	Data quality filtering
timestamp_start	Session start	Duration analysis
timestamp_condition_start	Condition section start	Condition-specific timing
timestamp_complete	Survey end	Completion analysis
completed	Full completion flag	Filtering incomplete data

Analysis-Ready Structure

The ideal export is a single flat file (one row per participant) with all variables ready for analysis. The alternative—separate files per condition, or missing condition variables that must be manually coded—introduces error and delays analysis.

For factorial designs, the export should include separate variables for each factor's level assignment, not just a single "condition" variable. This allows main effect and interaction analysis without recoding.

Platform Evaluation for Experimental Research

Capability	Essential	Nice to have
True random assignment	Yes
Block randomization	Yes
Stratified randomization		Yes
Parallel condition paths	Yes
Condition-tagged exports	Yes
Video/image stimulus support	Depends on design
Server-side randomization	Yes
Randomization logging		Yes
Real-time cell count monitoring		Yes
Attention check item types		Yes
Manipulation check integration	Yes
Identical outcome measure enforcement	Yes

Lensym for Experimental Research

Lensym's graph-based architecture is designed for experimental survey research:

Randomization engine: True server-side randomization with block, stratified, and adaptive options. Randomization logs are stored for reproducibility and auditing.

Visual condition management: Each condition arm is a visible path in the survey graph. Convergence nodes explicitly show where conditions merge for shared outcome measures.

Condition isolation: Content from non-assigned conditions is never rendered. No URL parameters, client-side condition indicators, or style differences between conditions.

Stimulus support: Rich text, images, embedded video, and interactive elements within condition paths.

Export with experimental metadata: Condition assignments, randomization blocks, manipulation check data, attention check results, and per-section timestamps in a single analysis-ready file.

EU-hosted infrastructure ensures compliance for research involving European participant data.

Run Your Next Experiment on Lensym

---

Related Reading:

• Survey Software for Randomized Controlled Experiments
• Managing Conditional Branching in Multi-Condition Factorial Studies
• Graph-Based Survey Logic: Visual Conditional Design for Complex Research
• Survey Randomization: When It Helps, When It Hurts
• How to Evaluate Survey Software for Academic Research
• Survey Tools for Academic Research: What Features Actually Matter