Mental Health is a significant challenge for health services in the Netherlands. Depression has the highest burden of disease of all health conditions (8.2%), including cardiovascular diseases and cancer (1). To address this growing demand, professionals have identified that digital health can be useful in treatment, enhancing and extending services. But they haven’t used it extensively, as there has been uncertainty around which products are good. Professionals wanted digital health to be assessed before recommending it to patients.
Yet at the same time, people experiencing mental health conditions have been increasingly using apps. This has typically been approached informally by trial and error or asking peers. But this approach takes time, requires persistence and isn’t informed by evidence.
So in 2019, the Ministry of Mental Health Services put in place the funding to establish a programme to evaluate health apps, putting in place the tools to enable digital health to be safely part of mental health recovery services. With this funding at the end of 2019 ‘MIND’ – in co-creation with ‘de Nederlandse ggz’ – started the development of www.ggzappwijzer.nl, which was eventually launched in the beginning of 2021.
To be guided by evidence MIND reviewed scientific papers to understand if apps really do help. Once it was confirmed that there are good apps that do provide a positive impact, they researched how people find and choose apps.
As there is very little research into how people choose health apps, the team conducted experiments themselves. They discovered that searches result in hundreds of app suggestions, with no trusted information to indicate which are good.
Focus groups revealed the information people want when choosing an app. They don’t just want a score, they want the facts behind this. Basically they want to know three things:
Also people are interested in the opinion of others about an app. Do peers and/or professionals give some kind of appreciation or depreciation of an app?
Whilst researching the sector, the team learned about the Organisation for the Review of Care and Health Apps (ORCHA), and the hundreds of assessments it had already conducted on mental health apps. The team felt that the ORCHA assessment was objective and thorough; they liked how the assessment publishes not only a score but also the additional information sought. Further research revealed that every question the MIND team had developed to assess an app, ORCHA had already asked it. In fact ORCHA asked more questions the team couldn’t possibly answer within a reasonable balance between costs and benefits.
ORCHA’s comprehensive base of reviews, together with its work with the NHS, led MIND to work with ORCHA. This collaboration was more appropriate for the programme than trying to reinvent the wheel.
The programme decided to utilise ORCHA’s objective reviews and layer on top its own qualitative assessment information. For this additional layer, MIND arranged for 4 to 8 people to test an app for two weeks, after which each participant answers a structured questionnaire containing 20 (of quantitative and qualitative) questions. Questions ranged from ‘How easy was this app for you to keep using?’, ‘To what extent would you recommend this app to other people?’ to ‘What improvements do you suggest to improve the app?’. Participants included both professionals and people with mental health conditions. The feedback has been consistent from both groups, each bringing a different but each other reinforcing perspective. The individual testing reports were transformed by the MIND team into one consolidated summary, which functions as a preview of what someone can expect from this app.
As part of this programme, almost 100 digital health products have been through the ORCHA evaluation and 20 of these have been through the additional MIND structured questionnaire layer. It is hoped this second figure will reach 50 by the end of this year. The evaluated products are hosted on a website that can be easily searched by everybody who is interested in mental health apps. The website is also useful for health care professionals to give advice about using a mental health app.
Since its launch in January 2021, the site has been visited 35,000 times and feedback has been extremely positive. People welcome the trusted information on the apps that are in the library. This has been especially important during lock-down, when an increasing number of apps from un-reputable sources have been launched.
To reach more people, the library of reviews is now also integrated into QULI, one of the major personal health records in use in the Netherlands, as part of the Medmij programme.
The combined ORCHA assessment and MIND reviews have also enabled the team to have informed discussions with developers of digital health. They have provided developers with clear and actionable advice on how their product can be improved.
Commenting on the programme, Rimmert Brandsma, Project Leader, eHealth, MIND, said:
“The relationship between MIND, de Nederlandse ggz and ORCHA has also been very successful. There is huge synergy and I feel the end product is better having all partners working actively together.
“We look forward to the next steps, of making multilingual versions of the website and increasing the number of apps assessed, possibly including AI, VR and wearables into the assessment structure. This will help us to support even more people to find the best digital health.”
(1) https://www.oecd.org/els/health-systems/MMHC-Country-Press-Note-Netherlands.pdf
Serving a population of 1.3 million, Surrey and Borders Partnership NHS Foundation Trust is an ambitious provider of mental health, drug and alcohol and learning disability services for people of all ages. It achieves ‘Good’ across all five domains assessed by the CQC.
Having listened to clinicians, people using services, and digital steering groups, the Trust set out its ambition to create the best personalised experience and outcome for people, leveraging the potential that technology presents to do things differently and in an individualised way.
This includes three main goals:
As part of this journey, the Trust looked to establish key digital foundations, including a library of health apps. The focus on apps reflected the growing use of health apps as an additional source of support, both within the Trust and among the general population, and sits well within the Trust’s early intervention strategies within CYPS (Children & Young People’s Services) where the platform is under pilot.
The launch of such a service requires a robust system to minimise risk, with particular reference to the clinical environment where apps may be used as part of professional practice. Areas of concern included clinical effectiveness and data protection along with a requirement for the chosen solution to be kept up to date.
The Trust chose to work with ORCHA. With a track record in assessing health apps for mental health Trusts, ORCHA offers a comprehensive app review service and uses a closed-loop process for quality. The Trust would benefit from the many thousands of app reviews carried out by ORCHA to date and would seek to be an active partner in driving future development of the project.
The Trust’s implementation of ORCHA has been led by the Digital Healthtech team and has been designed with young people in mind, reflecting the CYPS focus. Although the site is primarily intended for the local population to access independently, the Trust are also piloting the use of Pro licences, which allow clinicians to recommend apps from the platform directly to the young people they are working with, either by phone or email.
A small team of clinicians were included in the pilot, as digital champions, including Child Wellbeing Practitioners based in schools across the county. These practitioners are a key part of the Trust’s CYPS early intervention strategy, working 1:1 directly with young people. Working with the project team, the ORCHA champions have identified key points in the young person’s treatment journey where apps may offer a good fit. Following training provided by the SABP Healthtech team, the Child Wellbeing Practitioners are trialling the pro accounts in their daily practice.
The team have created a list of favourite apps, in consultation with ORCHA, and this is being rolled out to all team members, giving them a common baseline to work from and streamlining the process of recommending apps to young people. The Digital Healthtech team form a vital link between the clinicians and supplier, offering regular check-ins, further training and articulating requests for change.
In addition to the work with the clinical team, the Healthtech team have been actively engaged with promoting ORCHA with partner organisations and the wider population of Surrey. Most of the promotion has been digital, although focus groups with young people suggest they prefer traditional promotional material such as posters and business cards in A&E waiting rooms and GP surgeries – things that are simply not possible under current restrictions. Instead, the Healthtech team have had to find other approaches – listing the site on local council and charity directories, as well as making direct contact with schools, other NHS organisations, youth organisations and charities. A QR code linking to the ORCHA site has been added to the footer of all referral letters and a paid social media campaign is in the pipeline.
The promotional campaign was underpinned by the fantastic selection of digital comms materials created by ORCHA from graphics supplied by the Trust. The pack includes materials for Facebook, Twitter, Instagram, etc., and features the cast of characters from our ORCHA site, providing a striking and accessible digital comms package.
The Trust now has a library of rated and approved apps, removing much of the risks previously associated with including unregulated apps in clinical practice. Any update or change to an app automatically triggers a re-review, ensuring the platform is always up to date and governance standards are maintained.
Before the ORCHA implementation, informal lists of top apps were circulated by well-meaning individuals and were unverified – some were found still circulating two or three years later, leading to increased risk of inappropriate apps being recommended or of apps no longer being available. Implementing the ORCHA platform has enabled a culture shift, directing people using services, clinicians, and the wider population, to a selection of apps that is always current and underpinned by the assurance offered by the ORCHA review process.
Early feedback has been positive both from focus group participants and clinicians alike. Clinicians appreciate the pro functionality as it includes a favourites list, enabling lists of favourite apps to be saved and quickly accessed in order to make recommendations directly from the platform in just a few clicks. The platform automatically logs activity, such as recording recommendations made.
The most engaged clinicians are acting as champions, recommending to others.
Elzbieta Vitkauskaite, Child Wellbeing Practitioner, and health app champion said:
“Young people are always on their phone. So, apps are a natural additional tool to be able to offer young people to help them to encourage to engage with their support. Apps offer more control and flexibility over when and where clients can engage, and many say they prefer an app to writing in a worksheet.
“We have found apps work really well as an additional support option at certain stages of our client’s treatment. When on a waiting list, after their first assessment and at the start of their review period – giving them tools to use when increasing their independence. There are some fantastic apps that have worked very well.”
Through listening to the needs of young people, their carers, and the clinicians that support them, the ORCHA apps library is becoming an established tool, empowering individuals to access high quality apps as part of our overall ambition to create the best personalised experience and outcome for the people we support.
Good Thinking is London’s Digital Mental Wellbeing Service, backed by the London Mayor’s Office, Public Health England, London Councils, NHS and Thrive LDN. It was developed in response to repeated studies that demonstrated that mental health difficulties were common but accessing support was complicated.
Intensive user testing with over 450 Londoners with mental health charity Mind revealed that people do not always fit into categories for, or want, traditional treatment options, and committing to a course of therapy is not always an option. However the research also revealed that people are willing to use online wellbeing resources about the four most common mental health concerns: anxiety, sleep, stress and low mood. There is also a desire for trusted interventions that are accessible 24/7, private, secure, online and where possible free. And people want to avoid stigma by being able to access services in private and at any time.
These insights led to a model of service for Good Thinking that is agile, responsive and hugely flexible. The elements of the service include using digital marketing to help reach people, a comprehensive self-assessment, clear urgent support information, support in finding offline support, a quiz to find personalised support and relatable helpful content.
Alongside these elements, it was also identified the programme should feature NHS-approved, class-leading Apps that offer choice and a range of ways of improving mental health and well-being, that are based on best-evidence, are safe and secure, and are free for users to use. But the team realised that trust in digital health may be an obstacle to progress and that trust and confidence must be earned.
To address the trust concerns, the products would need to be completely free, without any in-app purchases and they would need to be fully assessed for compliance with the latest regulations for safety.
To help identify the best in class apps, the team developed a Standard Operating Procedure, to ensure the programme would meet the health priorities, reduce inequalities, and ensure people are offered the best options safely, with an expectation that use will improve health and well-being.
To ensure the programme was best informed, Good Thinking chose to work with the Organisation for the Review of Care and Health Apps (ORCHA). Experts in health app review and accreditation, ORCHA worked closely with Good Thinking to advise on each step of the process.
It was decided that rather than offer a long list of options, the site would instead provide one best app for each health priority, accompanied by a clear usage explanation. This would enable people to quickly spot the most relevant apps for them.
Good Thinking identified a short list of the best apps for each health priority. ORCHA then identified which standards each app had to meet and conducted the relevant assessments of every app. This enabled Good Thinking to select the final class-leading apps.
Examples of apps selected that provide support to a distinct user group include:
As standards evolve and apps regularly update, unlike traditional approaches, ORCHA’s unique platform conducts ongoing monitoring, automatically identifying when an app requires a new review and conducting this. This enabled the programme to put in place an important governance step of identifying when an app needs to be decommissioned and removed from the platform.
Alongside the other support services, the Good Thinking site includes a best in class app for a range of situations for adults and children, all of which when tested, exceed quality thresholds.
Since its launch in November 2017, Good Thinking has attracted 500,000 users, or one in twelve of the region’s adult population. They’ve visited the service over 650,000 times, 24 hours a day, every day of the week, including national holidays.
Alongside all digital health services, it saw an increase in use during COVID, and especially during the first lockdown period. Since in-person services started to open-up, visitor numbers have dipped, but the team expects to see an increase in use of digital services over the next five years, much as we might expect in the retail sector.
Over the past year, there has been a great deal of work to ensure Good Thinking can complement other care pathways and services, and assist with demand management. For example, the apps that Good Thinking hosts can be prescribed directly by GPs.
A partnership of health and social care organisations working together, Our Dorset holds the ambition for its 750,000 residents to lead healthier, fulfilling lives supported by sustainable health and care services. But the organisation faces real challenges. Dorset’s population is ageing, bringing more long-term conditions, which places a growing demand on services. Its funding cannot keep pace with this growth in demand and without changing its services, in five years it would face an annual shortage of £229 million a year(1).
To address its ambitions and challenges, the Integrated Care System has identified a clear plan, in which digital plays an enabling role. It established a ‘digitally-enabled Dorset programme’ to increase the use of technology in the health and care system, to support new approaches to service delivery.
The programme first researched and established the building blocks needed to establish public facing digital health. Alongside videos and the NHS choices website, apps were identified as a key building block to enable patients to better manage their own health.
To understand more about if and how health professionals could recommend the use health apps and what would be needed to support them, a pilot was run with 20 nurses. This revealed the team didn’t know where to find good health apps, how to know if they could trust them, if they met policy or who to ask for advice.
Without the expertise to establish a closed-loop quality assured programme, or capacity to be able to test health apps, let alone test them again when they are updated, the team recommended partnering with ORCHA. They selected ORCHA as they had seen how it had delivered testing at a national level and for other regional providers. With ORCHA, Our Dorset could build a programme to mitigate risk and assure clinical teams they are recommending safe apps.
ORCHA tested apps against 350 standards and measures and worked with Our Dorset to identify the best health apps across each priority health area. A dedicated app library was built to house the apps and enable patients and staff to search. It was also agreed that if an app doesn’t appear in the library it will not be recommended by any member of staff. If an app is ever nominated that isn’t in the library it will first be reviewed by ORCHA before being used by Our Dorset.
Our Dorset also opted to include a feature to enable staff to recommend apps via email or text message from the app library directly to residents and patients. This eliminated human error and enabled recommendations to be monitored.
Once ready, the team ran a series of 30 onboarding sessions across all ICS providers and governance leads, including all non-clinical teams, including link workers, health coaching and social prescribers. Over one or two sessions, teams were briefed on the value proposition of health apps, walked through the system and key apps were demonstrated. The health app programme also saw the emergence of a local Community of Practice which became a regular fixture on team meetings, prompting people to share learnings, tips and good practice including hearing about cohorts that are responding well to health apps.
Alongside training staff, the app library was launched to the public, forming part of its #HereForYou campaign which reassures Dorset residents that their health services are still available should they need them and that they should seek help and advice despite the Covid-19 pandemic. This included PR, social and paid digital activities.
Since introduction, the teams have actively embedded the app library and the practice of recommending health apps into their service offer. The teams drive home the concept of self-management and pick out effective tools that are available.
There have been almost 25,000 pages viewed on the site (up to the end of November 2020), and in one month alone, the app library achieved nearly 5,500 page views. Social prescribers are actively recommending apps to service users and vitally, this advice is being acted upon, with almost 1,500 apps recommended to date and 56% of recommended apps downloaded, which is a very good engagement rate, especially compared with other services.
The most popular search terms include Mental Health MSK, Dementia, Diabetes and Cancer. Some of the most popular apps to be viewed are Wysa: Stress, Depression & Anxiety Therapy, Sleepio, NHS Weight Loss Plan and Musculoskeletal (MSK) Self Care, providing a range of health interventions and show that apps are being selected to improve the physical and mental health of the population.
One of the most active recommenders of health apps is George Mitchell, Social Prescriber, Live Well Dorset. One app from the library he finds useful is Quit Now, as he shares that “it gives people support 24×7.” He regularly shares app tips with other prescribers, as he says “Don’t assume that just because you know about an app, everyone else does.”
Commenting on the programme Crystal Dennis, Interim Lead for Public Facing Digital Health Services, Our Dorset Digital said:
“ORCHA power our health app library. They help us to break down the barriers and mitigate issues around digital health. Previously our clinical leads didn’t recommend digital health technologies as they had no idea where to look, were concerned about implied liability and couldn’t tell if a technology was of a good standard. Thanks to ORCHA we are building the trust with clinical teams and have put in place the tools and governance they need.”
Source: (1) https://ourdorset.nhs.uk/wp-content/uploads/2018/07/Our_Dorset_STP.pdf