In the US, Alzheimer’s disease (AD) is the single most expensive disease, the only disease in the top six for which the number of deaths is increasing. The greatest cost contributors are frequent hospitalizations, where falls are the largest culprit, and frequent need for assistance with the activities of daily living. A fall safety system shows the potential to reduce costs and increase quality of care by reducing the likelihood of emergency events (e.g., detecting falls before a fracture occurs and reducing the number of repeat falls). Unfortunately, current safety devices require wearable or sensor technology not suitable for individuals with dementia and incapable of showing caregivers how falls occur.
The goal of this project over the years was to build an online fall detection system with off-the-shelf wall-mounted cameras to automatically detect falls for residents with AD and related dementias (ADRD), enabled by a human-in-the-loop (HIL). It first started with experiments on sensor network technology (Estimotes, Samsung Smartthings etc.) and wearables (smart watches), leading to preliminary deployments in private homes in California and Nebraska, in collaboration with UCSF and University of Nebraska, which led to the conclusion that the use of this type of technology is not appropriate for AD. We subsequently designed a system, now commercialized by an off-spin company, SafelyYou. [URL https://www.safely-you.com/].
SafelyYou Guardian is designed to primarily operate in memory care facilities (which for the purpose of our work includes skilled nursing and assisted living). The HIL, who can monitor several facilities at a time, confirms the fall detection alerts provided by our artificial intelligence algorithms, and places a call to the facilities, so an intervention can happen within minutes of the fall detection (as opposed to hours after, when the next scheduled visit to the room of the resident happens). Subsequently, an Occupational Therapist (OT) reviews the fall videos to make recommendations on how to re-organize the resident space (intervention), to prevent future falls. SafelyYou Guardian does not require action of individuals / caregivers such as wearing a fall pendant and is therefore well-suited for individuals with ADRD. We leverage our HIL paradigm, in which our deep learning (a subfield of artificial intelligence) approaches identify and pre-filter falls well enough to leave the last check to a human, who will call the facilities in case of detected safety critical events (falls).
Fig 1: architecture of the fall detection system.
This project focuses three main objectives that advance science and technology. (1) we want to fully demonstrate the ability for deep learning algorithms to automatically perform safety critical tasks by learning over a sufficiently rich set of data. For this, the project will integrate and expand upon the state-of-the-art deep learning approaches including RCNN, domain adaption, few shot learning, human pose estimation, and advanced methods for forecasting occurrence of events from visual data. (2) The project will enable demonstration of a system which can run with a fully operational HIL paradigm at scale in a sustainable manner over time. (3) It will collect data on safety-critical events for ADRD care at unprecedented scales which could enable significant breakthroughs for tracing bio-markers of disease progression, pushing towards pro-active preventative care, and enabling the development of novel AI and robotics applications for supporting the needs of this vulnerable population.
Fig. 2: example of video data collected by the system
This project leverages the already recruited 400+ residents in our partner memory care facilities, recruited through our previous IRB-approved pilots, which we leverage:
- Pilot 1: We demonstrated the feasibility of the system by collecting proof-of-concept data containing 200 acted falls of healthy subjects and showed accurate fall detection.
- Pilot 2: We demonstrated acceptance of privacy/safety tradeoffs by residents, family and staff, through the collection of 3 months of video data at one of our partner memory care facilities, in which we identified 4 total hours of fall data. This led to clinical benefits including an 80% fall reduction through the intervention of an Occupational Therapist (OT) to re-organize resident space.
- Pilot 3: We demonstrated scalability and acceptance by deploying the system in 11 facilities, totaling 100 residents monitored by our system (offline, no HIL intervention).
- Pilot 4 (ongoing): We demonstrated the ability to perform online (real-time) fall detection, with real-time intervention of the HIL through our partner company Magellan-Solutions, which provides the 24/7 monitoring service for the facilities, and confirmed the decrease of the number of falls, with the 100 residents.
Fig. 3: left: reduction of monthly falls after our first trial, in one of our partner memory care facilities. Right: scheme of how the system works.
The trials were initially conducted together by the University and SafelYou [URL], through the SkyDeck [URL] campus incubator, and then subsequently The House [URL of THE HOUSE]. At this stage, SafelYou [URL] is running all the clinical trials, funded by a variety of grants, and leveraging past work with the university through NSF and NIH STTR and SBIR grants, as well as other sources of funding such as the Signature Fellows program [https://vcresearch.berkeley.edu/signatures/2017-18-fellows] and partnerships with other institutions such as Baycrest in Canada through the CABHI program [http://www.cabhi.com/].