Section 3: TEC Cymru’s Four-Step Phased Approach

TEC Cymru have developed a tried and tested four-phase approach as part of their Research & Evaluation framework. This approach is highly recommended as a robust method for data collection and analysis across a wide range of interventions or programmes.

In simple terms, Phase 0 sets the stage; Phase 1 and Phase 2 capture data from adoption through to full implementation; and Phase 3 tests it in its full form and determines long-term sustainability.

The Phase Zero is the discovery phase of any intervention or programme evaluated within TEC Cymru. This phase sets out to understand the project's rationale and objectives in order to determine its value and worth for TEC Cymru as a programme, and the need for time and resources spent on research and evaluation.

At this phase, evidence is sought, literature reviews are conducted, appropriate ethical approvals are applied for and baseline data is captured to understand the public opinion on the proposed intervention or programme. This can be gathered in a number of methods including baseline survey capture, public consultations or via patient and public involvement (PPI) – (discussed in Section 5). Access to the Welsh e-library can be found here.

It is also important in this phase to consider “The extent to which an activity or project can be evaluated in a reliable and credible fashion” (OECD-DAC 2010; p.21) and to undertake an evaluability assessment. More information about how evaluability works and assessment templates can be found here. This can include structured engagement with stakeholders to clarify the goals of the intervention and how they might be achieved. It can be helpful to develop a driver diagram, logic model or theory of change to articulate a shared understanding of the work, which evaluation models will be used and to seek advice on whether or not an evaluation can be carried out at reasonable cost.

At the beginning of an intervention/programme, despite previous evidence and early baseline data capture, often very little is known about the targeted participant group required for the proposed intervention/programme, particularly in terms of the likely uptake of the intervention/programme, or its likely response or outcome. Therefore, at this point, very little is also likely to be known about the best method or approach to take to capture the best evidence from this targeted participant group.

From the perspective of TEC Cymru, it would be wasteful to spend several months on designing a flawless data collection method, instrument or measurement, and spending months applying for and awaiting the response of IRAS ethical approval to later realise that the participants were not willing to participate, or that the intervention/programme was to not demonstrate value/worth, and thus goes against the QIs dimensions (e.g., dimension 5 'efficiency’ and dimension 1 ‘safety’ by prolonging a service).

TEC Cymru therefore recommend that if the intervention/programme has passed all necessary safety and quality checks then holding its go-live/start date up due to research and evaluation delays may perhaps do more harm than good to its potential participants; but also, to go live without an evaluation component attached could do harm (or at least produce errors) to the evidence base.

We have therefore developed a four-phase approach to their Research & Evaluation strategy, which allows them as a team to determine the ‘need’ or requirements for further phases as they learn more and progress.

NOTE: It is important to note that some of the phases or ethical approval applications will not be necessary for all types of interventions/programmes. This approach is merely an ‘ideal guide’ used by TEC Cymru.

By the time your intervention or programme reaches Phase 1, Phase Zero has led your team to believe that the proposed intervention/programme is of value and worth to the overarching intervention/programme and requires evaluation and research support.

At Phase 1 you merely want to know ‘Is it working?’

Phase 1 often attempts to answer this question by simply capturing data from service users (patients/families/professionals) via basic live feedback surveys (often attached to the intervention), which aim to capture measures such as the use and value of the intervention/programme.

Often within the NHS, all that is required to capture Phase 1-type data  and begin evaluation is Service Evaluation approval from a local Research & Development (R&D) department. As Phase 1 progresses, and more is learned about the participant groups, additional ethical approvals (e.g., IRAS) and more in-depth planning and resources may be required for the progression of further phases.

From TEC Cymru experience this phased approach allows for less waste, better planning, and provides a better understanding of the participant group, allowing the tailoring of the next phases more appropriately. This ultimately improves the intervention or programme and its likely outcomes.

By the time your intervention or programme reaches Phase 2, Phase 1 has led your team to believe it is working, but you are yet to understand how it is working, for whom, under which circumstances and to which extent?

Phase 2 often attempts to answer this question by continuing to capture data from service users (patients/families/professionals) but by digging deeper. This is often via more in-depth feedback surveys which aim to capture measures around benefits and challenges of the intervention/programme and to begin to explore the longer-term sustainability of it.

TEC Cymru split their Phase 2 work into 3-6 month increments and refer to them as Phase 2a, b, c and so on. Ideally, TEC Cymru would recommend that Phase 2 would be ongoing until the end of the intervention or programme to ensure there are no gaps in data capture moments and timeframes.

In addition, Phase 2 will seek to capture qualitative data to provide a richer understanding of its participant group and the context for which the intervention/programme is based, e.g., via interviews and focus groups.

By the time your intervention or programme reaches Phase 3, you should have a good understanding of your participant group and the context for which the intervention/programme is based. Phase 2 has led you to understand how it is working, for whom, under which circumstances and to which extent. But it’s important to understand that this ‘how’ is still merely a judgment and still will not tell you if it works.

Phase 3 however, asks ‘did it work?’

Knowing if something officially works needs to be proven, and proof can only derive from in-depth or experimental research testing measures such as efficacy and effectiveness e.g., cost or clinical effectiveness studies.

Phase 3 often attempts to answer this question by working closely with service users and teams (relationships developed in Phases 1 & 2) to understand more specific areas of need and requirement for in-depth research. This is followed by reaching out and collaborating with others (e.g., academia, international experts) to apply for more advanced ethical approvals and conduct more in-depth and experimental research  - such as Randomised Controlled Trials (RCTs), cost effectiveness studies and more in-depth, research led qualitative approaches extending on specialised areas. 

It is very important to note that by the time you reach Phase 3 additional support and resources within your intervention or programme are needed.

For example:

• In Phases 1 and 2, key questions and requirements are generally based on intervention or programme remits and must-haves (e.g., what the stakeholder has requested), Phase 3 however, operates more independently and has potential ‘should haves’ – in that it is now generating new knowledge which is distinctive and unique from original remits (e.g., the unknown).

• Unlike the structure that Phases 1 & 2 allows, Phase 3 research requires the freedom and creativity of a research team to explore new themes that emerge from Phases 1 and 2, and therefore, at this point, need to be able to step outside of its original intervention or programme remit. There are likely to be newly emerging areas of interest which are unlikely to be in the original intervention or programme remit. If you attempt to restrict natural data emergence and progression by preventing movement of intervention or programme remit change, you are potentially restricting true data findings which is the essence of research - and it is this essence that puts research over the top of evaluation in terms of error of judgement, non-bias, validity and reliability.

• If your intervention or programme is unable to support the requirements of Phase 3, it should not be labelled as Research & Evaluation, but rather as a Service Evaluation component of a intervention or programme only, as the evidence in Phases 1 & 2 is merely provide a judgement on ‘how to improve’ and not as ‘proven to work’.

• Remember: to improve conduct service evaluation and to prove conduct research; and to do it the TEC Cymru way – do both!