Coughlan Lab

Principal Investigator:

James Coughlan

Contact Information:

Email: coughlan@ski.org

Mobile Phone: (415) 345-2146

2318 Fillmore St.
San Francisco, CA 94115

The goal of our laboratory is to develop and test assistive technology for blind and visually impaired persons that is enabled by computer vision and other sensor technologies.

L to R: Huiying Shen, Ali Cheraghi, Brandon Biggs, James Coughlan, Charity Pitcher-Cooper, Giovanni Fusco
  • Active
  • Completed
  • Other
Active

Audiom Map of Smith-Kettlewell

An Audiom map of the main Smith-Kettlewell building at 2318 Fillmore St., San Francisco has been created by Brandon Biggs. This is an audio-visual map that allows users to explore a detailed map of the building with or without vision. The map runs in any browser and is available to anyone who visits the building.
Read more
Active

MapIO: a Gestural and Conversational Interface for Tactile Maps

For individuals who are blind or have low vision, tactile maps provide essential spatial information but are limited in the amount of data they can convey. Digitally augmented tactile maps enhance these capabilities with audio feedback, thereby combining the tactile feedback provided by the map with an audio description of the touched elements. In this context, we explore an embodied interaction paradigm to augment tactile maps with conversational interaction based on Large Language Models, thus enabling users to obtain answers to arbitrary questions regarding the map. We analyze the type of…
Read more
Person with visual impairment using a walk light detector app at a traffic intersection. Left photo shows user holding smartphone, right photo shows image acquired by app of traffic intersection including walk light.
Active

Using VR to Help Train Visually Impaired Users to Aim a Camera

People with visual impairments increasingly rely on camera-enabled smartphone apps for tasks like photography, navigation, and text recognition. Despite the growing use of these applications, precise camera aiming remains a significant challenge. This project explores the impact of virtual reality (VR) exploration in the context of learning to use a camera-based app. So far we have studied this approach in the context of training a visually impaired person to use walk-light detector app at traffic intersections.
Read more
Photo of Magic Map, a bronze scale map of the Magical Bridge Playground in Palo Alto. The bronze map is bordered by panels on both sides with descriptions of the map.
Active

Magic Map

The Magic Map is an interactive 3D map installed at the Magical Bridge Playground in Palo Alto, California. It consists of a 1/100 scale 3D bronze representation of the playground, which includes over seventy play structures organized into multiple play zones and paths. When the user's index fingertip touches a specific feature on the map, the name and description of the feature are read aloud in audio. This interactivity allows visitors with visual impairments to navigate the map without requiring them to read braille.
Read more
basf-05187
Active

Outreach at Smith-Kettlewell

Smith-Kettelwell is deeply committed to supporting our community. Please see the news tab for our upcoming and ongoing events. In the News tab, you will find information about our involvement in past outreach activities.
Read more
Image of an audio map
Active

Audiom

Audiom is a tool that allows blind and visually impaired individuals to view maps completely in audio. It is a web component and can be embedded into any webpage, similar to Google Maps. It allows non-visual use of route, landmark, and survey knowledge, which is the critical information needed for navigation.
Read more
map pose and exit detection image
Active

A Computer Vision-Based Indoor Wayfinding Tool

The ability to navigate safely and confidently is a fundamental requirement for independent travel and access to many settings such as work, school, shopping, transit and healthcare. Navigation is particularly challenging for people with visual impairments, who have limited ability to see signs, landmarks or maps posted in the environment.
Read more
Active

Computer Vision Journal Club

The Computer Vision Journal Club meets periodically to discuss papers on topics in computer vision, machine learning and other topics […]
Read more
Completed

Crowd-Sourced Description for Web-Based Video (CSD)

The Descriptive Video Exchange Project (funded by the National Eye Institute of the National Institutes of Health, (grant # R01 EY020925-01) focuses on crowd-sourced techniques for describing DVD media. CSD will expand DVX to include Internet-based media such as YouTube, iTunes U, and other streamed video found on a wide variety of web sites. Many streamed Internet-based video sources provide well-defined, public APIs for accessing all the information DVX requires. Using these APIs will allow the VDRDC to expand DVX to include streamed content so that seamless, simple, crowd-sourced descriptions can be added to Internet-based video by volunteers or professionals anywhere.
Read more
Talking Signs
Completed

Talking Signs

Created by William Loughborough in 1979, Talking Lights was a system of infrared transmitters and receivers allowing blind and visually impaired travelers to quickly and easily "read signs" at a distance.
Read more
ZoomBoard overview
Completed

ZoomBoard: an Affordable, Portable System to Improve Access to Presentations and Lecture Notes for Low Vision Viewers

The goal of the project is to develop a “ZoomBoard” system that students with low vision can use to better access visual material on a whiteboard or blackboard. The prototype version of the system that we plan to develop in this grant will consist of a dedicated camera system placed by the teacher to capture a view of the board, which wirelessly transmits a video stream that will be displayed on a student’s iPad. The student will use the ZoomBoard app to view this video stream, zoom in on any region of interest using a pinch gesture on the iPad, and apply image enhancements such as contrast…
Read more
Graphic of lines of braille with finger trace trajectory superimposed over braille
Completed

Regressions in Braille Reading

This project explores regressions (movements to re-read text) in braille reading. The image on the right plots the braille reading finger movements in blue and regressions in black.
Read more
Sample indoor sign showing automatic detection in yellow
Completed

Sign Finder

This project seeks to develop a computer vision-based system that allows a visually impaired traveler to find and read informational signs, such as signs labeling office doors, streets, restrooms and Exit signs. Link open source code
Read more
Completed

Tutorials and Reference

These are tutorials and reference materials I have written on various topics in probability and geometry over the years.
Read more
Icon showing a person's finger pointing to a tactile graphic that includes braille text
Completed

Tactile Graphics Helper (TGH)

Tactile graphics use raised lines, textures, and elevations to provide individuals with visual impairments access to graphical materials through touch. Tactile graphics are particularly important for students in science, technology, engineering, and mathematics (STEM) fields, where educational content is often conveyed using diagrams and charts. However, providing a student who has a visual impairment with a tactile graphic does not automatically provide the student access to the graphic's educational content. Instead, the student may struggle to decipher subtle differences between textures or…
Read more
ESTVI '13 logo
Completed

Workshop Series on Computer Vision and Sensor-Enabled Assistive Technology for Visual Impairment

Recent workshop: Workshop on Environmental Sensing Technologies for Visual Impairment (ESTVI '13 in San Francisco) ESTVI '13 focused on emerging technologies capable of sensing environmental features for applications in access technologies for persons with visual impairment, including low vision and blindness. The development of environmental sensing technologies (ESTs) and the study of their potential to support the activities of daily living for visually impaired persons is progressing at a rapid pace, and engages many disparate research fields, including computer vision, wearable sensors…
Read more
Zoomed-in view of appliance display partially obscured by glare
Completed

Display Reader

The goal of the Display Reader project is to develop a computer vision system that runs on smartphones and tablets to enable blind and visually impaired persons to read appliance displays. Such displays are found on an increasing array of appliances such as microwave ovens, thermostats and home medical devices. Click here for Display Reader software download.
Read more
BLaDE (Barcode Localization and Decoding Engine) smartphone app in action
Completed

BLaDE

BLaDE (Barcode Localization and Decoding Engine) is an Android smartphone app designed to enable a blind or visually impaired user find and read product barcodes. The primary innovation of BLaDE, relative to most commercially available smartphone apps for reading barcodes, is that it provides real-time audio feedback to help visually impaired users find a barcode, which is a prerequisite to being able to read it. Link to BLaDE software download: http://legacy.ski.org/Rehab/Coughlan_lab/BLaDE/ Click here  for YouTube video demo of BLaDE in action.
Read more
Virtual aerial view of intersection area near a pedestrian's feet, reconstructed by Crosswatch algorithms
Completed

Crosswatch

Crosswatch is a smartphone-based system developed for providing real-time guidance to blind and visually impaired travelers at traffic intersections. Using the smartphone's built-in camera and other sensors, Crosswatch is designed to tell blind and visually impaired travelers what kind of traffic intersection they are near, how to align themselves properly to the crosswalk, and when the walk light or other traffic light indicates it is time to cross. Click here  for link to zip file containing Crosswatch code (open source) and documentation.
Read more
Graphic showing STEM icons including a chemistry beaker, atom with electron orbits, and a mathematical square root symbol

Smith-Kettlewell Summer Institute

The Smith-Kettlewell Summer Institute program is a multi-year, learning opportunity designed to provide training for blind and visually impaired students, recent graduates, and early career researchers interested in data skills that will enhance Science, Technology, Engineering and Math (STEM) jobs within industry, government, non-profit organizations or academia. In today’s research climate, individuals are expected to have many skills at their fingertips. Often researchers are required to build software and hardware environments to run studies, to gather and analyze data and to prepare…
Read more
Close-up photo showing person pointing right index finger to Ireland on a map of the British Isles with other fingers closed.

CamIO Hands

This project builds on the CamIO project to provide point-and-tap interactions allowing a user to acquire detailed information about tactile graphics and 3D models. The interface uses an iPhone’s depth and color cameras to track the user’s hands while they interact with a model. When the user points to a feature of interest on the model with their index finger, the system reads aloud basic information about that feature. For additional information, the user lifts their index finger and taps the feature again. This process can be repeated multiple times to access additional levels of…
Read more

Shen, H. ., Chan, K.-Y. ., Coughlan, J. ., & Brabyn, J. . (2008). A mobile phone system to find crosswalks for visually impaired pedestrians. Technology and Disability, 20, 217–224.

Crabb, R. ., Cheraghi, S. A., & Coughlan, J. . (2023). A Lightweight Approach to Localization for Blind and Visually Impaired Travelers. Sensors, 23(5). https://doi.org/10.3390/s23052701 (Original work published 2023)

Hong, J. ., & Coughlan, J. . (2023). VR Training to Facilitate Blind Photography for Navigation. Journal On Technology And Persons With Disabilities. (Original work published 2023)

Biggs, B. ., Pitcher-Cooper, C. ., & Coughlan, J. . (2022). Getting in Touch with Tactile Map Automated Production: Evaluating Impact and Areas for Improvement.Journal on Technology and Persons With Disabilities, 10. (Original work published 2022)

Coughlan, J. ., & Shen, H. . (2013). Crosswatch: a system for providing guidance to visually impaired travelers at traffic intersections. Journal of Assistive Technologies, 7.

Tekin, E. ., Vásquez, D. ., & Coughlan, J. . (2013). S-K Smartphone Barcode Reader for the Blind. Journal on Technology and Persons With Disabilities, 1.

Manduchi, R. ., & Coughlan, J. . (2012). (Computer) vision without sight. Communications of the ACM, 55, 96–104.

Coughlan, J. ., & Manduchi, R. . (2009). Functional assessment of a camera phone-based wayfinding system operated by blind and visually impaired users. International Journal on Artificial Intelligence Tools, 18, 379–397.

Shen, H. ., Coughlan, J. ., & Ivanchenko, V. . (2009). Figure-ground segmentation using factor graphs. Image and Vision Computing, 27, 854–863.

Yuille, A. L., Coughlan, J. ., & Konishi, S. . (2003). The KGBR viewpoint-lighting ambiguity. Journal of the Optical Society of America (JOSA) A, 20(1). (Original work published 2003)

Yuille, A. L., Coughlan, J. ., & Konishi, S. . (2001). The KGBR viewpoint-lighting ambiguity and its resolution by generic constraints. Computer Vision, 2001. ICCV 2001. Proceedings. Eighth IEEE International Conference on, 2, 376–382.

Tekin, E. ., Coughlan, J. ., & Shen, H. . (2011). Real-time detection and reading of LED/LCD displays for visually impaired persons. Applications of Computer Vision (WACV), 2011 IEEE Workshop on, 491–496. IEEE.

Biggs, B. ., Toth, C. ., Stockman, T. ., Coughlan, J. ., & Walker, B. N. (2022). Evaluation of a Non-Visual Auditory Choropleth and Travel Map Viewer. International Conference on Auditory Display (ICAD) 2022. Presented at the International Conference on Auditory Display (ICAD) 2022. Virtual. Retrieved from PMCID: PMC10010675 (Original work published 2022)

Coughlan, J. ., Biggs, B. ., & Shen, H. . (2021). Point and Listen: Bringing a 3D Map to Life with Audio-Based AR. 6th Annual Frameless XR Symposium. Presented at the 6th Annual Frameless XR Symposium. (Original work published 2021)

Biggs, B. ., Coughlan, J. ., & Coppin, P. . (2021). Design and evaluation of an interactive 3D map. RESNA 2021 Conference. Presented at the RESNA 2021 Conference. (Original work published 2021)

Fusco, G. ., Cheraghi, S. A., Neat, L. ., & Coughlan, J. . (2020). An Indoor Navigation App using Computer Vision and Sign Recognition. 17th International Conference on Computers Helping People With Special Needs. Presented at the 17th International Conference on Computers Helping People with Special Needs. (Original work published 2020)

Coughlan, J. ., Biggs, B. ., Rivière, M.-A. ., & Shen, H. . (2020). An Audio-Based 3D Spatial Guidance AR System for Blind Users. 17th International Conference on Computers Helping People With Special Needs. Presented at the 17th International Conference on Computers Helping People with Special Needs. (Original work published 2020)

Biggs, B. ., Coughlan, J. ., & Coppin, P. . (2019). Design and Evaluation of an Audio Game-Inspired Auditory Map Interface. The 25th International Conference on Auditory Display (ICAD 2019). Presented at the The 25th International Conference on Auditory Display (ICAD 2019). Northumbria University, Newcastle-upon-Tyne, UK. (Original work published 2019)

Fusco, G. ., & Coughlan, J. . (2018). Indoor Localization using Computer Vision and Visual-Inertial Odometry. International Conference on Computers Helping People With Special Needs (ICCHP ’18). Presented at the International Conference on Computers Helping People with Special Needs (ICCHP ’18). Linz, Austria. (Original work published 2018)

Fusco, G. ., & Morash, V. S. (2015). The Tactile Graphics Helper: Providing Audio Clarification for Tactile Graphics Using Machine Vision. ASSETS. Presented at the ASSETS. (Original work published 2015)

Manduchi, R. ., & Coughlan, J. . (2014). The last meter: blind visual guidance to a target. Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, 3113–3122. ACM.

Murali, V. N., & Coughlan, J. . (2013). Smartphone-based crosswalk detection and localization for visually impaired pedestrians. Multimedia and Expo Workshops (ICMEW), 2013 IEEE International Conference on, 1–7. IEEE. (Original work published 2013)

Coughlan, J. ., & Shen, H. . (2012). The crosswatch traffic intersection analyzer: a roadmap for the future. Computers Helping People With Special Needs, 25–28. Springer Berlin Heidelberg.