Baseline Survey

Developing a High Quality Baseline

Overview

What is a baseline?

Why you should care

The phases of conducting a baseline

The errors to avoid in each phase

How to manage the common errors

Elements of a baseline survey (TOR)

What is a baseline?

Fixing the time at the base – a benchmark from which you measure progress

Snapshot of indicators at a time

Instrument used to: 

Test hypotheses of project (assess results)

Planning (refine targeting, indicators to monitor)

Why you should care …

To identify whether there were any benefits for the investments made

Were objectives met?

What factors explain the result?

How can the program be improved?

Compare alternative models to get the biggest bang for your buck

To inform next generation projects

Evidence-based policy making – demonstration effect for government

The phases of conducting a baseline

Design

Implementation – actual survey

Data entry and analysis

Report writing

Phase 1: Design Phase   

P1: Check-list

Clear objectives (what is the problem?) 

Clear idea of how you will achieve the objectives (causal chain or hypotheses)

Clear and measurable indicators

Clear (precise and unambiguous)

Relevant (to objectives)

Monitorable

Example of a causal chain

P1: Check-list (cont’d)

Design survey instrument – keep it simple and related to the objectives and hypotheses you want to test

Link surveys to GIS – use consistent units

Select controls/counterfactuals to attribute change (causality)

Timing of baseline

Before project (what if project never materializes)?

2 years into the project (intervention has begun)?

Other factors (seasonality)

P1: Check-list (cont’d)

Sampling strategy 

Random allows inferences about a population 

Random

Stratified random (include groups which could be excluded)

Non-random – use a group smaller than the population 

Introduces selection bias. Note: every stratification introduces a level of bias. 

P1: Check-list (cont’d)

Over sample for attrition

Design the database system for data entry

Translate the questionnaire

Back translate to verify

Provide adequate training on 

The objectives and importance of the study

How the sections are linked and what the questions mean

Attend the training if possible (stay involved)

P1: Check-list (cont’d)

Test the questionnaire

Ask yourself, can you answer these questions?

Are they relevant to the outcomes? 

Will they be understood?

Field test: To test both how the surveyor administers the instrument AND how the respondent understands the question.

P1: Why it is important to field test (i)

When asking a question about the level of awareness, the surveyor used a word that could mean awareness or knowledge – the respondent understood it to mean education. : 

The question was: “Ram/ Gita knows about everything that happens (vikas) in the village. For instance, they know [the name of the sarpanch, when and where the Gram Panchayat meets, nature and type of development work in village, etc.]” 

P1: Why it is important to field test (ii)

CASE STUDY: Domestic Violence Study in India

“In studying domestic violence, a question in the survey instrument asked if female respondents had ever been beaten by their husbands in the course of their marriage. Only 22 per cent of the women responded positively to this question – a domestic violence rate much lower than studies in Britain and the US had shown. In probing the issue with in-depth interviews we discovered that the women had interpreted the word ‘beating’ to mean extremely severe beating – when they had lost consciousness or were bleeding profusely and needed to be taken to the hospital. Hair pulling, ear twisting, etc, which were thought to be more everyday occurrences, did not qualify as beating. Reponses to a broader version of the abuse question, comparable to the questions asked in the US and UK surveys, elicited a 70 per cent positive response.” 

Source: Vijayendra Rao (1998) – “Wife-Abuse, Its Causes and Its Impact on Intra-Household Resource Allocation in Rural Karnataka” 

Phase 2: Implementation

What would you do if you …

Were a city person who didn’t speak the local language very well

Had to travel to several villages and spend hours asking people questions that have no relevance to you.

Were paid a small sum per questionnaire

Not monitored by supervisors

This is not your full time job

The answer is simple

Sit at home or in a bar and fill out the questionnaires!!

P2: Providing incentives and motivation

Sub-contracting surveyors from the state who speak the language

Include women surveyors

Include a supervisor who conducts data scrutiny

If possible pay reasonable wages 

Randomly verify questionnaires to reduce the likelihood of false responses (inform them beforehand - during the training)

Phase 3: Data Entry and Analysis

P3: Check-list

Data Entry

Make the data entry system as fool proof as possible - has unique identifiers to link both household, village and GIS data

Ensure database allows for merging of data

Do not change/erase data on questionnaires

Raw data should always be input as is, changes can then be made in the database software (programatically) with documentation

P3: Check-list (cont’d)

Often data entry is contracted out. 

Name variables corresponding to the question and section in the questionnaire – include a dictionary

Code descriptive answers (to facilitate analysis)

All fields should be filled (NA or NR)

Units should be uniform by district

Totals calculated by formula not from summary column

Consistency checks – check for missing entries, wrong entries, sample statistics, patterns (queries should be inbuilt)

Validity checks – similar questions in different places on the questionnaire (RCH example)

P3: Check-list (cont’d)

Data analysis

Common mistakes in interpreting data

No analysis! 

No correlations, crosstabs, statistical significance levels or regressions

Over generalizing the results

Mis-reporting statistics

Using % when the numbers are small

Attributing causality when it is not demonstrated

Phase 4: Report Writing

P4: What the report should be …

Simple, Clear and Relevant

State limitations (attribution, causality)

Major findings should be upfront

Focus on quality rather than quantity

Technical details in an appendix

Should always 

include the questionnaire in the appendix

ask for electronic copy of data 

Request copies of filled out surveys

Essential if you change consultants at midterm or want to conduct internal analysis to compare modes of delivery (data lost example). 

How to manage the common errors

Phase 1: Design

Clear objectives and hypotheses – know what you want to test

Identify a person in your unit who will manage this process

Write a good TOR, remember the baseline determines the quality of your panel

You can add questions as project evolves but cannot change questionnaire – threat to internal validity

Identify consultants 

Procurement – focus on quality not the cheapest bid “if you throw peanuts you’ll attract monkeys”

Ideally you should have a black-list of organizations

How to manage the common errors

Phase 2: Implementation

Organize an impact evaluation workshop if necessary

Randomly verify questionnaires to reduce the likelihood of false responses (no filling it in a bar)

Pay reasonable wages to surveyors (if possible)

Show the client and firm that you care

How to manage the common errors

Phase 3: Data entry and analysis

Double-data entry (2 separate organizations and verify. Payment based on quality of data entry)

Select 15 questionnaires at random and check data entry – person in your unit managing

Check data quality (consistency and validity checks)

Hold an IE workshop to build data analysis capacity (if necessary)

How to manage the common errors

Phase 4: Report writing

Agree on an outline beforehand

Dedicate a chapter on indicators you are tracking

Focus on quality not quantity

Think “Big Picture”

Elements of a Baseline Survey
 Terms of References 

Background:  Project objectives and components

Survey design:  Consult a sampling expert!!!

Survey instruments

Guidance on survey implementation

Data processing and analysis

Staffing

Duration and time schedule

Submission of reports and datasets

Support to the firm

Budget & Payment Schedule

Annexes:  Draft questionnaires, Results Framework

Baseline Survey Design:
Typical Tasks for Consultants

Recommend the methodology for sampling

Calculate the optimal sample size

Develop the sample frame and select the sample 

The final sample and details of the statistical methodology used to select the sample need to be cleared by the project

Construct the sample weights and provide documentation on the methodology used to construct the weights  

Survey Instruments:  Questionnaires

Design or refinement and adaptation of the data collection instruments

Specify levels of data collection

Length of questionnaires

Prepare all support documentation including  coding guides, interviewer and supervisor manuals and  the data entry manual

Translation and back-translation

Skip patterns, coding open ended questions

  

Guidance on Survey Implementation

Implementation plan

Selection and training of field workers:  specify minimum duration of training

Pilot testing should be explicitly specified in ToR

Responsibility for all field operations, including logistical arrangements for data collection and obtaining household consent lies with Consultants.

Ask for field-work progress reports (bi-weekly/monthly)

Staffing

Sampling expert/statistician

Technical specialists as relevant

Economist

Sociologist. 

Core survey staff: the survey manager, the field manager, the data manager

Enumerators, supervisors and data entry staff 

Baseline Report & Data

Explicitly request final electronic datasets—with complete documentation.

Agree on outline of baseline report up-front.

Managing a Baseline Survey

Consult the experts—survey specialist and sampling specialist and develop the ToR in consultation.

Selection committee should include a survey expert and social scientists in addition to technical experts.

You can never over-supervise!!!  Hire third-party supervision consultant if needed.

Question the data and the findings.  

Lets recap what you have learned

The devil lies in the detail 

Be watchful

No pain, No gain