Tender: Feasibility Study Regarding Methodology, Design and Mode of the European Company Survey
Dublin, Ireland, 14 June 2016
With this tender the Organisation is seeking to award a framework contract (FWC) to be prepared for the fourth improved wave of the European Company Survey. To achieve this, lessons learnt from the last ECS wave (2013) shall be drawn, recommendations in terms of sampling and modes be elaborated and the potential of setting up a a panel be discussed.
Scope of the work
This tender is not divided into lots. The tenderer must be in a position to be able to provide all services requested. However, specific tasks under the envisaged FWC will be ordered through specific contracts. Estimated timing of these specific contract is stipulated under point 3.5 (Provisional timetable) but might be subject to change due to budget availability.
Below, there are four tasks required by the Organisation. The contractor is requested to outline the approach taken to fulfil these tasks and to develop an overall framework explaining how the different work packages are interlinked.
The order of the tasks is just provisional and may be subject to change depending on the budget availability.
Task 1: Methodological lessons learnt from last ECS wave
The first task to be achieved is the submission of a methodological lessons learnt report summarising and reviewing insights from the third wave of the ECS in 2013. The objective is to identify pitfalls to be avoided for the next round in terms of preparation of the field phase, sampling, contacting respondents, identification of employee representatives, reducing non-response, mode of interview and quality control and to draw recommendations. The feasibility of which shall be elaborated in the subsequent tasks.
Task 2: Sampling modes and frames
The previous survey waves adopted random probability sampling. However, due to registers of very variable quality, the question arose if this approach results in a sample that allows for the production of robust and reliable estimates. The first question therefore to be addressed is if it would be possible to improve quality frames in terms of access to better registers in some of the countries.
Alternatively, other approaches can be envisaged that depart from a strict random probability framework. Notably, quota sampling approaches might be an alternative potentially providing more control over the structure of the sample, while at the same time allowing for the use of different information sources as selection frames (e.g. membership lists of employer organisations). Non-probability sample would on the other hand limit possibilities for the production of unbiased point estimates at EU or country level.
A further scenario could be to continue with probability sampling but to invest in a thorough screening process which would allow for knowing more about the profile of the non-respondents and understanding whether they differ significantly from respondents on key characteristics.
It will be part of this Task to review what was done in comparable other business surveys and to assess various scenarios of sampling also considering the size of the sample per country and its stratification. This might entail accepting smaller samples for very small countries and the most adequate size for bigger countries. Although reducing the size of the sample per country would reduce the cost of the survey, it would also limit the possibilities for country level estimates and analysis, which is why this question should be addressed.
Regardless of the option followed, an elaborate screening exercise will be required. the Organisation has concerns that depending on the complexity of the design this could be quite expensive and feasibility needs to be explored in order to prevent long duration screening interviews before the actual interview can start.
The exercise concludes with an exploration of the critical methodological issues identified and proposals to solve them. The work should result in an overview of various options for designing the sample, including an overview of the implications for cost (Task 4), quality, and possibilities for analysis.
Task 3: Survey modes
The third ECS was a survey of establishments in Europe based on telephone interviews, involving 24,251 interviews with a management representative and 6,860 interviews with an employee representative responsible for the establishment. The target population was all establishments with 10 or more employees in all economic sectors except agriculture, activities of households as employers and activities of extraterritorial organisations. The countries covered were all 28 EU Member States, Iceland, the Former Yugoslav Republic of Macedonia, Montenegro and Turkey.
With survey response rates dropping throughout Europe, there is a challenge to both find ways to involve selected respondents and to learn more about the non-respondents (to better control for non-response error). Because of the special circumstances and characteristics of business surveys, response rates are relatively low due to the difficulties involved in reaching a respondent (gate-keeper). The overall response rate of the third ECS 2013 for the management interviews was 38%, ranging between 18% in Austria and 62% in Slovenia. It is assumed that the very low response rates in some countries are in part due to the survey mode. Response rates were higher for the employee representative interviews with an average of 60% ranging from 39% in Ireland to 83% in Croatia.
Presenting multiple survey modes in sequence has been demonstrated as a successful way to improve response rates. A major advantage offered by mixed-mode survey designs is an increase in the coverage of the population of interest. Mixed-mode designs may offer a more inclusive frame with respect to the target population. Other advantages include a reduction of cost/increased efficiency, the establishment of credibility and trust with the respondent. Problematically, however, since each mode of data collection is unique in terms of the transmission of information and the environment of the interview, survey questions are received and processed by respondents in different ways. Mode effects are an important downside to the advantage(s) offered by a mixed-mode design.
In this task, the contractor will therefore be requested to develop several scenarios for potential mixed-mode data collection for the next ECS which is believed to improve data collection and overall response rates. Is an option to respond on-line a viable way to collect some characteristics from at least a part of non-respondents and to design better survey modes to have them take part in the future?
In this task, the contractor will be also requested to provide cost evaluation associated with introduction of new survey modes.
The overall objective of this task is to scrutinise and compare several survey mode scenarios as well as mode effects and to develop a proposal for an optimal scenario in consultation with the Organisation.
Task 4: Panel design
A potentially fruitful possibility to improve the cost-effectiveness of the process of respondent recruitment is to use the respondents more than once, in other words, to set up a panel. This might require a relatively large initial investment. However, it would reduce the cost of recruitment in future iterations, when the panel will only need to be supplemented to deal with attrition (loss of respondents over the years). A further advantage of a panel approach compared to a repeated cross-sectional data collection is that it allows for making causal inferences. By following a fixed set of companies over time we would be able to gain insight into causes and effects in terms of company policies and practices and company outcomes.
In this Task the potential of setting up a panel or integrating at least a panel component to future waves of the ECS should therefore be scrutinised.
Submission Deadline: 12 July 2016
Also view the ECN Procurement Notice (accessible by ECN Executive Members - Login to View)