Lumino

Innovation:
To convenience our users, we strongly believe in precognition: actively providing suggestions or finishing tasks for users based on their app usage habits instead of passively responding to their input. We have adopted a hybrid model: data unique to each user, such as their daily routines and idiosyncrasies, is stored locally on their device, while data beneficial to all users such as those regarding typical structures and objects (walls, cars, keys) are stored in the cloud. Our self-developed algorithm processes the user’s surrounding environment obtained from the live feed of their smartphone camera, cross-references objects of interests from similar data stored in local and cloud storages, and produces suggestions for the user’s reference, or completes tasks for them in the background. Visually impaired people no longer have to scour their house for misplaced keys, nor worry about impending hazards just out of reach of their white canes: Lumino is able to anticipate their needs and provide context-contingent assistance, backed up by an extensive local and cloud database.

Functionality:
Lumino’s software has three main functions: object identification, hazard detection and navigation. All three main functions output feedback to the user via a virtual assistant in the same vein as well-established programs of the sort, such as Siri and Google Assistant.

As mentioned above, the user’s surrounding environment is captured by their smartphone camera. The user can tap different parts of the screen to learn about the relevant objects present in that quadrant (for example, tapping on the left half of the screen would reveal objects of interest to the left of the user). Our algorithm crawls local and cloud storages, taking into account the user’s daily habits, and reveals the object most relevant to the user’s needs. The software, by logging user behaviour, can also provide suggestions: a path to the dairy section will be suggested to a user who buys a carton of milk from the supermarket every Tuesday.

Object identification also extends to user protection by recognizing common obstructions such as walls, potholes and steps. Additionally, the user’s movements – such as their pace – are logged to accurately gauge the distance of any potential hazards from the user and issue timely warnings. This function is further supported by compact hardware: a forearm-length device with the potential to replace white canes. With an in-built three-axis motor, the turning force produces angular momentum, which acts as pseudo-propulsion to mimic a pulling sensation, which steers the user clear of any potential dangers in their path.

Lumino is also seamlessly integrated with contemporary web mapping services such as Google Maps and Apple Maps to provide its users with navigational assistance. An overlay is added on top of such mapping services to insert waypoints at turns and crossings of the suggested routes, and the virtual assistant notifies the user at such junctions. Smartphone haptic feedback also ensures the user remains on the route by vibrating until the user faces the correct direction.