Monitoring use over time

Monitoring the use of a technology over time in an aged care setting is crucial for understanding its impact on the people using it or affected by it. [1] Ongoing, planned monitoring processes can reveal any unintended consequences associated with using the technology, as well as any deviations in the way it is being used that do not reflect its original intended purpose. [2] Monitoring is also important for checking that a product continues to align with the goals of the organisation, as these goals may have changed over time with the introduction of new regulations, policies, and procedures. [3]

Monitoring may also reveal new and unexpected ways in which end-users engage with or employ a technology. This is not necessarily a negative thing. It may indicate that end-users have evolved workflows to improve the fit between the product and local needs. [4] Customisation of the product itself may also signify efforts to improve the product’s usefulness in the local context. [5]

To help aged care organisations identify what they might monitor throughout the life cycle of a new technology, we provide a list of considerations.

User knowledge and skills

Some technologies require a certain level of skills and knowledge to use them effectively. Training may have been provided at an early stage of implementation but over time, people may not retain knowledge, especially if they use the product irregularly, or if they have declining health or cognitive function. New staff may not have received training or received a reduced version of it based on the assumption that the technology has become routine and familiar. An organisation might track the ongoing training needs of staff or care recipients required to use a technology by monitoring their knowledge, skills, and confidence. Relevant questions to ask include:

• Are end-users comfortable and confident in using the technology?
• Do they think the training they received in its use was adequate?
• Do they believe they have the necessary skills and knowledge to use the technology to its fullest potential?
• Are they using its features and functions as intended? Or have they created workarounds to make it work better? [5]

User experience

User experience is a person’s overall experience while interacting with a product, service, or system, such as a website, mobile app, or physical device. It encompasses the individual’s perceptions, emotions, and behaviours in relation to the technology. [1, 6] To understand user experience, consider asking:

• Do people like using the technology? Are they satisfied with it?
• Do they find it easy to access and use?
• Are certain people excluded from using the product due to its features or design (e.g., small font size, complex website navigation, or reliance on fine motor skills)?
• Can users suggest improvements?
• Has the product reduced workload or increased it?

Effectiveness

For a product to be considered ‘effective’ it must demonstrate that it achieves the outcomes expected of it. Measuring effectiveness can be a complex undertaking, requiring an understanding of appropriate research methods. Effectiveness might also be monitored less formally by asking the questions:

• Do end-users or the people indirectly or directly affected by use of a technology believe it is producing the specific, tangible outcomes expected of it as outlined in the business case? For example, has the technology increased or helped maintain the health and wellbeing of people receiving care?   
• Do staff think that the technology has helped them get more work done, or to get it done faster? 
• Has the technology caused any problems that reduce efficiency or impact negatively on people? 

Cost-effectiveness

Cost-effectiveness is about getting value for money. This form of effectiveness can be measured by comparing the benefits gained from the product against all the costs associated with it. Costs may be financial or involve the investment of other types of resources such staff time spent in training or loss of productivity due to unreliable technology performance.  Consider asking the following questions to monitor cost-effectiveness:

• What benefits has it delivered, both tangible and intangible?
• Do the benefits of the product, service, or program outweigh the costs?
• Has the technology reduced costs or saved time?

Safety

It is important to monitor the safety of people and their personal information within the aged care setting. This might involve tracking the number of incidents that have taken place over a defined time range or recording the severity and frequency of hacking or phishing attempts or other data security issues. Knowledge of cybersecurity events over time can help plan strategies to avoid risks or reduce the effects of any breaches. Questions to consider asking here include:

• Has the technology caused any physical harms through use, both appropriate and inappropriate (e.g., falls, shocks, or reliability failures leading to harm)?
• Has it caused any emotional or mental harms to the people using it (e.g., stress, anxiety, or increased loneliness)?
• Have there been issues with cybersecurity (e.g., data or confidentiality breaches, viruses, or hacking attempts)?
• Are the current compliance regulations being met?

Technical performance

The technology should continue to perform at a certain level over time. Indicators for measuring this performance can be captured by asking questions such as: [7,8]

• Has the technology integrated well with other systems in the organisation?
• Does the technology continue to meet the defined user specifications?
• Has it worked reliably over time, performing its intended tasks consistently and dependably? Or is it prone to glitches such as network dropouts, buffering, slow processing times, or constant need for recharging? If so, how frequently and for what duration?
• Is the technology being maintained or serviced appropriately on a regular basis?
• Is the infrastructure able to continue supporting the use of the product?

If the answer to many of these questions is ‘no’, it may be time to reassess the technology or the available technology infrastructure. In time, this process might lead to a search for a different solution.

Regulatory compliance

Where there are laws and regulations surrounding the use of certain technologies (e.g., surveillance technologies), it is important that the aged care organisation continues to monitor its compliance with them. [3]

There are many ways to capture useful data on technology use for product or process refinement or for deciding on a product’s continued sustainability. The most straightforward approach may be to ask end-users or other stakeholders about their experiences and perceptions of a product. This could be as simple as providing a physical or virtual ‘suggestion box’ for capturing ongoing feedback. More structured and targeted approaches include conducting surveys, in-depth interviews, or focus groups with stakeholders. [10]

Numerical data to complement this subjective data could be captured by establishing an agreed-upon set of indicators that are routinely measured and reported on. Indicators are metrics that provide information on a product’s performance and achievement in meeting a certain defined outcome. This might be the number of times a specific event occurred or the number of people who engaged with a particular technology over a set period of time. [10] Indicators focus on narrow, well-defined aspects of technology use that can be easily measured, and which can serve to create a picture of how a technology is being used and its effectiveness over time. [10]

Examples of measurable indicators are:

• The number of times a remote sensor technology raises an alert (e.g., for when a person falls out of bed). This might include a count of the false alarms as well as the true emergencies.  
• Frequency of network ‘drop-outs’  
• Reductions in medication errors (e.g., with the introduction of an e-prescribing system) 
• The amount of time it takes a staff member to input data in a care management system.  

A more formal and systematic way of determining if a technology is benefiting an organisation is to conduct a research study. [9] This might be achieved through a collaborative partnership with a researcher or research team. However, it is likely to require a considerable investment of time in data collection and analysis over a lengthy period. There are also associated ethical considerations. Types of research studies that could provide evidence of a technology’s benefits or explain people’s lack of acceptance or use include:

• Studies aimed at eliciting people’s attitudes, beliefs, or perspectives
• Comparative effectiveness studies, such as randomised controlled trials, that provide information of the relative effectiveness of different products or care practices in producing a measurable benefit (or preventing a harm of some kind).  
• Cost-benefit studies that identify whether the benefits of the technology outweigh its costs 
• Implementation studies that focus on specific approaches to implementing a technology and their success.  

Rather than a one-off activity, monitoring should be a planned continuous quality improvement process that runs across the life cycle of a technology. Feedback and monitoring mechanisms embedded into the prototype testing and/or local piloting phase will provide the project team with useful information to guide product improvements prior to implementation. Similarly, monitoring activities that seek to understand people’s experiences, perspectives, and interactions with a technology during the implementation phase can highlight areas for further refinement. Even when a product has become fully integrated into people’s daily lives or the routines of a workplace, monitoring can offer insights into its ongoing value. [9]

Monitoring should be an ongoing process rather than a one-off event. It involves regularly gathering information and using it to make improvements. It is therefore an important part of continuous improvement practice. By regularly monitoring technology use and making changes where indicated, organisations can improve the fit between technology assets and the context in which they are applied. This may support decisions on whether a product provides ongoing value for money or if it can no longer be sustained.

1. Agboola S, Hale TM, Masters C, Kvedar J, Jethwani K. "Real-world" practical evaluation strategies: A review of telehealth evaluation. JMIR Res Protoc. 2014;3(4):e75. 
2. Di Pelino S, Lamarche L, Carr T, Datta J, Gaber J, Oliver D, et al. Lessons learned through two phases of developing and implementing a technology supporting integrated care: Case study. JMIR Form Res. 2022;6(4):e34899. 
3. Australian Government Department of Health and Aged Care. Regulation and compliance [Internet]. Canberra, ACT: Commonwealth of Australia; 2022 [cited 2023 Feb 28]. Available from: https://www.health.gov.au/about-us/what-we-do/regulation-and-compliance. 
4. Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: An interpretative review. Int J Med Inform. 2013;82(5):e73-86. 
5. Cresswell K, Williams R, Sheikh A. Developing and applying a formative evaluation framework for health information technology implementations: Qualitative investigation. J Med Internet Res. 2020;22(6):e15068. 
6. Lee TT. Evaluation of health information technology: Key elements in the framework. J Nurs Res. 2016;24(4):283-285. 
7. World Health Organization. Monitoring and evaluating digital health interventions: A practical guide to conducting research and assessment [Internet]. Geneva: WHO; 2016 [cited 2023 Feb 21]. Available from: https://apps.who.int/iris/handle/10665/252183. 
8. Rai HK, Cavalcanti Barroso A, Yates L, Schneider J, Orrell M. Involvement of people with dementia in the development of technology-based interventions: Narrative synthesis review and best practice guidelines. J Med Internet Res. 2020;22(12):e17531. 
9. van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011;13(4):e111. 
10. Adler KG. How to successfully navigate your EHR implementation. Fam Pract Manag. 2007;14(2):33-39.