FORD GLOBAL TECH LLC - 2021-11-1127 Nov 2021 07:13
VISUAL BEHAVIOR GUIDED OBJECT DETECTION
[0021] According to at least one additional illustrative example, a system is described that comprises: an eye-tracking system including a sensor; a computer communicatively coupled to the sensor and the eye-tracking system, the computer comprising one or more processors and memory storing instructions executable by the one or more processors, wherein the instructions, when executed by the processor, cause the processor to: receive, from the eye-tracking system, an image frame captured while an operator is controlling a vehicle; receive, from the eye-tracking system, eyeball gaze data corresponding to the image frame; and iteratively train a deep neural network to determine an object of interest depicted within the image frame based on the eyeball gaze data. The deep neural network generates at least one feature map and determine a proposed region corresponding to the object of interest within the at least one feature map based on the eyeball gaze data.
[0040] As shown in FIGS. 1 and 2, eye-tracking system 14 may comprise any device or apparatus that monitors eyeball gaze data of the operator and repeatedly determines a gaze point (e.g., where the operator is looking) that corresponds with a captured image. In some examples, the eye-tracking system 14 comprises a forward-looking sensor 22 (e.g., such as a red-green-blue (RGB) camera) that concurrently captures video representative of the operator's field of view (FOV), multiple eye-monitoring cameras 24, multiple illuminators 26 directed toward the operator's eyes, and a computer 28 configured to determine repeatedly eyeball gaze data of the operator for a plurality of image frames of video data from the sensor 22. The eye-tracking system 14 may be a so-called wearable eye-tracking system or it may be a system mounted in the vehicle 12. A non-limiting commercial implementation of the eye-tracking system 14 is the “Pro Glasses 2” by Tobii AB which comprise an eyeglasses frame, lenses, and the hardware discussed above. In other examples, the sensor 22 and/or computer 28 may not be part of a common assembly with the camera(s) 24 and illuminator(s) 26—e.g., the sensor 22 could be mounted elsewhere in the vehicle 12 or the like.
https://worldwide.espacenet.com/publicationDetails/biblio?II=1&ND=3&adjacent=true&locale=en_EP&FT=D&date=20211111&CC=US&NR=2021350184A1&KC=A1
https://www.freepatentsonline.com/20210350184.pdf