QuickFind
Overview
The long lines and too many items to look from at the grocery stores can get overwhelming, especially for individuals with mild cognitive impairment(MCI) . In order to improve and speed up the grocery shopping experience for them, we built a new device based on Raspberry pi to t locate the items via indoor navigation.
In the development process, the pandemic happened unexpectedly, which hindered further field testing of the hardware equipment. Simultaneously, we found that public people also want to minimize their shopping time in the supermarket, so we pivoted the development from hardware devices to mobile applications, so the QuickFind app version was born.
Prototypes Figures:
Left side is Version One - a Desktop app based on Raspberry PI device;
Right side is Version Two - a Android Application with phone holder
Approaches
Finding items in a grocery store can be challenging and time-consuming. It becomes even more tricky with cognitive impairments like Mild Cognitive Impairment(MCI) or unusual pandemic situations like COVID-19. About 30 million shoppers in the US are facing this challenge each day. Hence, we decided to attack this problem.
IDEA
How might we improve and speed up the grocery shopping experience for individuals with mild cognitive impairment(MCI) and general public specific to COVID-19 pandemic ?
UX
- Interviews - Helped us identify problem and exam our solution feasibility.
- Survey - Helped us make appropriate and intuitive design choices for the potential users.
DEVELOP PHASE ONE
PyQt and Raspberry Pi based indoor navigation device.
DEVELOP PHASE TWO
Grocery navigation mobile app implemented with React Native.
Our Solution - Pivot Version One to Version Two
Considering that the items are placed on the shelves of the relevant aisles according to categories, we could locate products' areas according to the user's shopping list and the store database information. Then we can carry out route planning through the preset map anchor points. At the same time, we use the aisle information, including aisle number, label, etc., to locate.
What I do
Device Development
Hardware. I configured raspberry pi and a compatible touch screen, and corporation with the team to design hardware components installation constructions.
Software. I designed a desktop app using PyQt for managing shopping lists and showing shopping paths. To detect the aisle number for indoor positioning, I used the OpenCV East text detection model and screened the retrieved information to confirm the user's current location.
MobileApp Development
To do grocery store shopping, except getting shopping list function ,our main function also include doing pathfinding and locating to navigation. For pathfinding, I used DFS to do pathfinding as before, but optimized map and anchor data. For locating, using number detection, which is another challenge during the development. Since we use react native platforms, we can leverage a firebase ml kit to save our time for a while, but the challenge is that it will cause a storage problem since it will autosave image file for a while, I need to delete them in a certain time.
Software Architecture
By creating a user's shopping list on the mobile application, our navigation app will search for the aisle information of the desired items in the database and plan the route. On Clicking the button to start navigation, the app displays a navigation route, scans the surrounding environment through image processing, detects the aisle number and label, and corrects the position through the beacon signal and locating the user's real-time position as well. When the user is close to the desired product, it prompts the information about that product. If the user fails to follow the planned route for shopping, the system re-routes the path based on the user's current location.