All you need to know about the evaluation process
What do the Academy’s ratings mean?
Is the Academy’s rating final?
How often are the quality ratings reviewed?
If I don’t like the rating, can my programme be removed from the Toolkit?
Is inclusion in the Toolkit an endorsement of a programme?
How does the parenting academy evaluate programmes?
Our research team has developed a measure for evaluating the quality of parenting programmes – the Parenting Programme Evaluation Tool (PPET) [PDF, 107 KB]. The PPET uses objective criteria developed after an intensive review of the international literature. This makes it possible to compare programmes against the same set of criteria and therefore provide commissioners with consistent evaluations of all programmes.
Developers need to be aware that evaluations are based on the evidence submitted and are rated strictly against the 4 criteria specified in the PPET (see also the following questions).
Evaluations are conducted independently by at least two researchers who are blind to the developer’s self rating and each other’s initial ratings. Programme developers may be contacted to clarify information and details of the programme and/or its description on the Toolkit. At this stage researchers may recommend changes to the submission form to ensure that it is accurately represented on the Toolkit.
Back to top
What is the difference between a self-rating and an Academy-rating?
When a programme is first published on the Toolkit it appears with the developer’s own self-ratings which they have based on the PPET criteria. Academy-ratings are arrived at through evaluations that are conducted independently by at least two evaluators who are blind to the developer’s rating. The two evaluators’ ratings are then submitted to an Evaluation Panel that reaches consensus on the final ratings and feedback to the developer. The Panel ensures consistency of approach between the evaluators, challenges evaluators when ratings differ, ensures that evidence exists to support claims and ensures a consistent standard across all programmes.
Once the programme has been through the Academy's evaluation process the self-rating is replaced by an Academy-rating. The Toolkit shows clearly whether a programme is self-rated or Academy-rated.
The order in which programmes are Academy-rated is determined by their relevance to government-set priority groups and parents, and the date the programmes were submitted.
Back to top
What are the four elements of the PPET that programmes are evaluated against?
Element 1: Targeted populations and explicit recruitment processes: this element looks at the quality of the specification of the target population and matching to the programme.
Element 2: Explicit content based on sound theory: this element considers the quality of the programme content and the process in place to make sure it is delivered well.
Element 3: Best practice in training, supervision and fidelity of programme: this looks at the quality of implementation and what processes are in place to make sure that the programme is delivered consistency and correctly and with enough support for practitioners.
Element 4: Robust evidence base: this element looks at what evidence has been presented to demonstrate that the programme is effective for its targeted outcomes.
Back to top
What do the Academy’s ratings mean?
The evaluation tool provides a rating for each of the four elements above (see previous question) using a scale of quality ratings ranging from 4 to 0.
4 = all criteria are met (Demonstrates best practice with reference to international standards)
3 = most criteria are met
2 = some criteria are met
1 = a few criteria are met
0 = no criteria are met
For example, a rating of 4 for Element 1 means that the programme clearly specifies all the criteria of best practice based on international standards: the target population is clearly specified and appropriate for the content of the programme, the needs assessment is appropriate to the target population, the target outcomes are specific, measurable, achievable, realistic and timely given the target population and content of the programme, and the classification level corresponds s to the target population and the content of the programme. However, a rating of 0 for Element 1 means that the programme does not meet any of the criteria for best practice for this element.
For guidance on the rating scale and criteria of best practice for each element please read the Parenting Programme Evaluation Tool (PPET) [PDF, 107 KB].
Back to top
Is the Academy’s rating final?
In submitting their programme details for inclusion in the Toolkit, developers agree to the Academy independently evaluating the programme and accept the independent Academy- ratings. The Academy’s rating is final. Developers should note that Academy-ratings may differ from self ratings and may, in fact, be lower. Developers will be informed of the Academy’s rating prior to its publication on the Toolkit.
Back to top
How often are the quality ratings reviewed?
Once programmes have been Academy-rated the Academy will aim to review each programme 12 months after the ratings are published on the Toolkit. Early annual reviews will be considered at our discretion in exceptional circumstances but no less than 3 months after publication of the Academy’s ratings. Once a programme has been through an early annual review it cannot be reviewed again for 12 months.
The expected review will be shown in the individual programme summary.
Developers of self-rated programmes awaiting Academy evaluation can request to change their self-evaluation to take account of relevant new information, for example, if you have just completed an evaluation demonstrating positive changes in child outcomes.
For more information on the evaluation process please read the Toolkit User Guide [PDF, 94 KB].
Back to top
If I don’t like the rating, can my programme be removed from the Toolkit?
Programmes may not be removed from the database but may be archived at the request of the developer, if the programme or approach is no longer available and in use in England or at the suggestion of the Academy if the programme does not meet the definition of a ‘parenting programme’. Full details of archived programmes will not be available for public view but the reason for archiving, a brief description of the programme and date of withdrawal will be viewable.
Back to top
Is inclusion in the Toolkit an endorsement of a programme?
No. Inclusion of a programme or approach on the Toolkit database is not an endorsement, recommendation, validation or Academy accreditation/seal of approval. The Academy-ratings only serve as a guide to commissioners and developers about the quality of each programme. Programmes initially appear with developers’ self-ratings and subsequent Academy-ratings may be lower or higher.
Back to top
For more information on the evaluation process please read the Toolkit User Guide [PDF, 94 KB].