Evaluating complex interventions such as integrated care programs can be problematic or fail to identify impacts for several reasons – identifying appropriate controls, isolating impact, and interrelationships between specific interventions. For these and many other reasons, existing evidence about the broad impact of integrated care is inconclusive.
We propose to discuss how we are evaluating the complex intervention of the Integrated Care Alliance (ICA) on the Gold Coast, Queensland. The ICA is a joint initiative of Gold Coast Hospital and Health Service and Gold Coast Primary Health Network to create an integrated, coordinated, seamless and streamlined world-class health system that strengthens the link between primary, secondary and tertiary care services to ensure a truly person-centred approach to service delivery. We intend to answer the research question: what are the impacts of the interventions of the ICA, and how have these impacts been achieved? We will report how the research strategy and interventions are both continually adapting in response to formative feedback.
We have developed a deliberately broad and iterative research strategy, covering a range of methods and timelines, to triangulate and provide a comprehensive evaluation of the overall impact and the processes associated with those impacts.
We consider impact on immediate, medium-term and long-term measurement of utilisation and cost of services, patient outcomes and experiences, and staff experiences and work processes.
A fundamental part of the intervention program and its evaluation is to measure and ensure long-term sustainability and the most efficient use of specific program resources. That is, whether the interventions have the capacity and capability to maintain service delivery, sustaining (or improving) health outcomes and experiences as well as containing delivery costs. This will support the ICA to reflect on progress against objectives, situate itself against current evidence and best practice, and identify next steps in its development.

Background
In 2017, the NSW Central Coast Local Hospital District (CCLHD) designed and implemented the Outcomes Based Care program to vulnerable older people healthy and at home. Two not for profit providers were commissioned to deliver their own form of care coordination for one year. Provider payments were based on their capacity to reduce unplanned public hospital bed days for an allocated patient cohort.

Objectives
Our study objectives were to explore how provider and service characteristics, along with patient behaviours, and mediators and moderators of service delivery, may have impacted outcomes. These included hospital utilisation, health outcomes and patient experience with care coordination.

Methods
The process evaluation explored mechanisms of impact and contextual factors, including the risk stratification approach, frailty of the patient cohort and impact of the funding incentives for care coordination between the CCLHD, community providers and hospitals. Evaluation criteria were developed after a systematic literature review. Information was collected through patient surveys, interviews with patients, providers, the CCLHD and additional program documentation.

Results
The primary outcomes showed a significant reduction in hospital activity for both the intervention and the control group. While there was an overall decrease in hospital use, modelling results suggest Outcomes Based Care increased ED visits for the intervention group. The risk stratification approach resulted in the selection of a cohort of older patients with more complex patients than anticipated. The funding incentives shifted financial risk to providers, but all allocated unplanned hospital bed days were used within six months with the contract needing to be renegotiated. Survey data suggests providers supported patients in the self-management of their medical conditions but is unclear whether the level of patient engagement was appropriate to achieve intended outcomes. Providers note the lack of General Practitioner and patient engagement hampered efforts to improve health outcomes.

Discussion
International experience presents similar results for care coordination approaches. A review of 15 randomised trials developed within the US found only three coordinated care programs experienced a significant difference in hospitalisations. One program increased hospitalisations by 19 per cent. Similar findings were identified in the UK with significantly increased ED visits and unplanned hospitalisations compared to control groups, though these weren’t part of a commissioning pilot.

Limitations
Survey data was intended to be collected at enrolment and again when Outcomes Based Care finished. They were not administered as planned and the timings varied for each survey and between the enrolled and control groups. Response rates also varied by survey type and mode of administration. One provider, in particular did not capture adequate evaluation data.

Conclusions
The study has important evidence based implications for the development of approaches designed to keep older people healthy and at home and to incentivise the delivery of care.

Lessons learned
Fourteen evaluation recommendations explore health outcomes, cost effectiveness, patient experience and unintended consequences that specifically relate to outcomes based commissioning designed to keep older people healthy and at home.

Suggestions for future research
Further models of outcomes based funding should be piloted within the NSW healthcare system.

Introduction
Case management is a special case of care coordination making an important contribution to person-centred integrated and coordinated cross sector health and social care. Case management is highly variable and therefore complex, which impedes practice, quality analysis, policy and planning. Case management for people with a disability is a high cost in social insurance schemes such as icareNSW and NDIS. Yet the characteristics of best practice person-centred case management models is not clear, whether change occurs over time and the person’s outcomes. The research aim is to better understand the patterns of characteristics including the person, context, mechanisms and outcomes for people with brain injury and spinal cord injury living in the community.

Methods
We developed an a priori theoretical model of temporal change to case management using realistic evaluation methods and an expert group. Characteristics related to the mechanisms, context and people related outcomes. Retrospective real world, real time data extracted from files of people severely injured in motor vehicle crashes in 2008 participants of Lifetime Care and Support (NSW). Frameworks to map characteristics include: the case management taxonomy; WHO International Classification of Functioning, Disability and Health. A multi-disciplinary data science approach used for analysis involved statistical analysis, pattern analysis and text mining combined with expert practice knowledge, interpretation for pre-processing and extracting meaningful information.

Results
There were 107 participant’s files reviewed and over 500 characteristics from each file extracted. The two file reviewers adopted a data extraction protocol, discussed and resolved issues as they arose to ensure consistency. Results include analysis of demographic characteristics, case management characteristics of intensity, interventions and duration and participant outcomes at 2 years, 5 years and 9 years post injury. Correlations, patterns and typologies emerged related to context, case management and outcomes.

Discussions
The range of data extracted provided extensive and rich descriptors of the population, participant progress, goals and outcomes. Patterns partially confirmed the theory of case management for intensity and duration, less the recovery phase and person-centred planning evident in the person’s goals. The patterns across time confirm inconsistencies and suggest differences in care coordination roles.

Conclusions
The new knowledge derived from analysis has emerging uses and planned uses in case management policy, quality analysis and practice in icareNSW. The results highlight factors and patterns that inform future research, quality analysis, policy and practice.

Lessons learned
The realist evaluation method provided an essential foundation for analysis aligned to context-mechanism-outcome theory. The frameworks were critical to manage the substantial number of characteristics and categories. Using a data science approach consisting of intensive data-driven methods combined with practice knowledge perspectives enabled new knowledge relevant to practice and service policy and planning.

Limitations
The sample size limited more definitive analysis of correlations. Pattern recognition limited the selection of variables for analysis and determined by the mechanism-context-outcome theory.

Suggestions for future research
Further research will involve a similar approach and comparisons for people with other health conditions, potentially other countries and to build real world knowledge on case management practice.

The monitoring, evaluation and economic assessment of the benefits of integrated care has been a topic of much recent discussion and ongoing academic discourse. Many approaches, frameworks and models exist, however there are not many cases in which they were truly applied to a specific integrated care (IC) project.
In 2018, Poland’s National Health Fund (NFZ) has decided to pilot-test an integrated primary health care model (POZ Plus). The NFZ has sub contracted out some of its M&E functions in the pilot to the World Bank. The ultimate aim of the M&E is to establish IC mechanisms that can be rolled out nationwide. To establish the M&E system for the project, the World Bank team looked closely at the methodologies used in evaluations and created their own M&E Framework.

Methods
The M&E Framework for POZ Plus pilot was created based on literature review of integrated care M&E concepts, framework and systems; and consultations with experts. Considering the mechanisms and characteristics of the pilot, as well as IT infrastructure and data management capacity of the stakeholders, the applicability of M&E models and tools was reviewed and further included in the M&E plan.

Results
M&F Framework was based on the nine notions that are true for all IC interventions, including the fact that IC evaluation requires measurements on multiple levels, measuring the quality of chronic care is essential; and that there is a need to focus on patient outcomes and experiences.
The Framework contains a set of descriptions about how to measure the achievements of the POZ Plus pilot. It describes the indicators, definitions, tools (eg. PROM,), processes, principles, and procedures (eg. COMIC model) through which the functionality of the M&E system to measure the pilot’s results is strengthened. Areas of evaluation of the pilot include impact evaluation, scalability assessment and process evaluations and operations research.

Discussions and conclusions
Due to the multidimensionality of IC projects, there is no golden standard for M&E of IC. Many approaches to evaluation of IC appeared in recent years, however one must always remember to fit the evaluation model to the context of the project that is being evaluated.

Lessons learned
Framework for M&E of IC should be created depending on the IC project. In the final evaluation, it is important to look at the IC project as a whole, rather than to evaluate individual elements such as payment mechanisms or new care pathways. At the same time, it is important to look at the project as a joint of different elements and point the interventions that work best or need improvement.

Limitations
Due to the scope of the contract between the NFZ and the World Bank, the in-depth evaluation of IT systems and the medical substantiation of the clinical care pathways was not included in the consideration for M&E Framework.

Suggestions for future research
There is a need for more research in evaluation of integrated care.

An introduction
Sydney Local Health District (SLHD) is a densely populated and socioeconomically and culturally diverse region in inner-Sydney experiencing significant population growth. 45% of the District’s residents were born overseas and 58% spoke a non-English language at home. With 15% of the population children aged 0-12 years. Annually 8,500 babies are born to mothers residing in the District. In the last year our child and family health services delivered care to 22,779 children on 77,783 occasions and managed 27,746 phone calls through our contact centre.

The aim of this project was to challenge existing child and family health models of care and distribution of resources by redesigning services to be more patient centred, integrated, accessible, flexible and responsive.

Description of practice change implemented
A multi-layered, multi-level redesign strategy was implemented including:
• Partnerships and integration: with the wider social care sector, research partners and internally within a local health district
• Consumers: engagement and co-design
• Workforce: maximising scope and roles, supporting innovation and celebrating success
• Models of care: developing new and revising existing models and delivery modes
• Policy: influencing and reshaping policy
• Funding: positioning services for funding opportunities
• Governance and leadership: across the health system and social care sector

Aim
To maximise workforce potential and create a service environment that met the needs of all families and enabled integration, flexibility, innovation and change. All initiatives were assessed for efficient resource allocation and utilisation. An iterative approach to implementation and redesign was taken with ongoing review and evaluation.

Population and stakeholders
Families with children antenatal to five years old in SLHD.

Timeline
The redesign strategy commenced in mid-2015 and is ongoing.

Highlights
• Partnerships and integration: united leadership across key government agencies and primary health
• Consumers: informed a communications plan, strategic priorities, programs and resource allocation
• Workforce: trust and permission to take risks and trial. Maximised capacity to deliver new models of care within existing resources, improved job satisfaction and autonomy to apply clinical judgement
• Models of care: right care, rather than prescribed care, determined by family need
• Policy: state policy influenced and other health districts following our lead
• Additional funding
• Governance and leadership: created an enabling environment for ongoing collaboration

Sustainability and transferability
The service redesign has secured significant funding and service models are being scaled.

Conclusions
This redesign has enabled changes in service delivery that have been evaluated. We have attracted increased investment and reallocated existing resources to sustain engagement with families requiring additional intervention, more accessible universal services for all families and stimulated statewide discussion regarding care models.

This work is ongoing in order to remain responsive, flexible and innovative. We will continue to measure, evaluate and refine. By creating an enabling environment, continued opportunities will emerge.

Lessons learned
Be courageous and challenge state directives when there is misalignment with local need. Embrace the complexity of delivering universal and targeted health improving services across a diverse population. Value and maximise the potential of your workforce.