Member Info for johnchucka

Thanks TLS , that's very helpful.

Hi TLS,
I was looking at Microsoft EYE TRACKING SYSTEMS AND METHODS FOR NEAR-EYE-DISPLAY (NED) DEVICES and hoped you could comment on the Eye tracking using 'ambient light' part of the description?
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2020124844A1&KC=A1&FT=D&ND=3&date=20200423&DB=&locale=en_EP#
[0043] Turning now to FIG. 3, illustrated in an exemplary ellipse 204 that is projected from a circular feature of an eye 104 (e.g., an Iris 108) onto a sensor plane 302 of a sensor 102. The sensor plane 302 may correspond to a substantially planar surface within the sensor 102 that is angularly skewed with respect to a corresponding Iris-Pupil plane 206 (not shown in FIG. 3) so that circular features on the Iris-Pupil plane appear elliptical on the sensor plane 302. In some embodiments, the sensors 102 may be image sensors such as, for example, complementary metal oxide semiconductor (CMOS) sensors and/or charge-coupled device (CCD) sensors. In such embodiments, the sensors 102 may generate eye tracking data in the form of pixel data that defines images of the eyes. These images may be formed based on ambient light surrounding the user. Thus, in contrast to conventional eye tracking systems that rely on illuminating the eye(s) with near infrared light to cause first Purkinje reflections (e.g., “glints”) that are distributed around the iris, the techniques disclosed herein do not require active emission of near infrared light toward the user's eyes. The numerous benefits of the techniques disclosed herein include providing a system that can track the user's eyes using ambient light rather than having to expend battery resources to generate near infrared light. Moreover, the disclosed techniques provide a system that is highly sensitive and accurate in the detection of eye movements (e.g., the systems are sensitive enough to even accurately track saccadic eye movements).

VIDEO-BASED PHYSIOLOGICAL MEASUREMENT USING NEURAL NETWORKS
Frames of a video frame sequence capturing one or more skin regions of a body are provided to a first neural network. The first neural network generates respective appearance representations based on the frames. An appearance representation generated based on a particular frame is indicative of a spatial distribution of a physiological signal across the particular frame. Simultaneously with providing the frames to the first neural network, the frames are also provided to a second neural network. The second neural network determines the physiological signal based on the frames. Determining the physiological signal by the second neural network includes applying the appearance representations, generated by the first neural network, to outputs of one or more layers of the second neural network to emphasize regions, in the frames, that exhibit relatively stronger presence of the physiological signal and deemphasize regions, in the frames, that exhibit relatively weaker presence of physiological signal.

[0027] With continued reference to FIG. 2, the physiological signal measurement module 202 may operate on a motion representation 214, generated based on frames of the video frame sequence 208, to recover a physiological signal p(t) from the motion representation 214. The motion representation 214 corresponds to the motion representation 114 generated by the motion representation generator 112 of FIG. 1, in an embodiment. In other embodiments, the motion representation 214 is generated in other suitable manners. The appearance module 204 may operate on each respective frame of the video frame sequence 208 to generate an attention mask 220. The attention mask 220 may be a representation, in each current frame 216, of distribution of the physiological signal p(t) being recovered by the physiological signal measurement module 202. The attention mask 220 may generally emphasize regions of the frame that exhibit relatively stronger measure of the physiological signal 206 and deemphasize regions of the current frame that exhibit relatively weaker measures of the physiological signal 206. The attention mask 220 may be provided to the physiological signal measurement module 202, and may be utilized in the process of recovering the physiological signal 206 by the physiological signal measurement module 202. For example, the attention mask 220 may be applied to adjust weights in the physiological signal measurement module 202, such as weights in one or more certain layers of a neural network of the physiological signal measurement module 202, so as to emphasize regions in the frames of the video frame sequence 208 that exhibit stronger measures of the physiological signal 206 and deemphasize regions that exhibit relatively weaker measures of the physiological signal 206.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2020121256A1&KC=A1&FT=D&ND=3&date=20200423&DB=&locale=en

CAMERA BASED DISPLAY METHOD AND SYSTEM FOR SIMULATORS
A display system for a simulator comprising: a main display for displaying a simulated scene, the first display being positionable away from a user; a see-through display for displaying a portion of the simulated scene, the see-through display being wearable by the user; a filter for filtering a portion of a field of view of the user; and a control unit configured for: receiving environment images, the environment images comprising a first set of images of the simulated scene, a second set of images of the simulated scene and a third set of images of at least a portion of the simulated scene; displaying the first and second set of images on the first display; and displaying the third set of images on the see-through display.

There is described a method and system for displaying images in a simulator. The simulator may be used for concurrently training two users such as a pilot and a copilot of an aircraft. The simulator comprises a main display on which images of a scene are displayed to the two users who each have a respective position relative to the main display. Each user is provided with a second display on which respective images are displayed. For example, the images displayed on a wearable display may correspond to a section of the scene displayed on the main display according to the point of view of the respective user. Each user is further provided with a respective filter adapted to partially filter the field of view of the user outside of the wearable display.
https://worldwide.espacenet.com/publicationDetails/biblio?II=43&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200402&CC=WO&NR=2020065497A1&KC=A1#

METHODS AND SYSTEMS FOR GENERATING ADAPTIVE INSTRUCTIONS
Systems and methods for generating adaptive instructions for a driver. The system includes a memory that stores instructions for generating adaptive instructions for a driver. The system also includes a processor configured to execute the instructions. The instructions cause the processor to: monitor one or more factors including a gaze of the driver, a physiological signal of the driver, information related to the operation of a vehicle, and information related to an external environment of the vehicle; analyze the one or more factors to determine when and how to provide the adaptive instructions to the driver; generate the adaptive instructions based on the analysis; and provide the adaptive instructions to the driver via a vehicle user interface.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=2019375426A1&KC=A1&FT=D&ND=3&date=20191212&DB=&locale=en_EP#

SYSTEM AND METHOD FOR LEARNING AND PREDICTING NATURALISTIC DRIVING BEHAVIOR
A system and method for learning naturalistic driving behavior based on vehicle dynamic data that include receiving vehicle dynamic data and image data and analyzing the vehicle dynamic data and the image data to detect a plurality of behavioral events. The system and method also include classifying at least one behavioral event as a stimulus-driven action and predicting at least one behavioral event as a goal-oriented action based on the stimulus-driven action. The system and method additionally include building a naturalistic driving behavior data set that includes annotations that are based on the at least one behavioral event that is classified as the stimulus-driven action. The system and method further include controlling a vehicle to be autonomously driven based on the naturalistic driving behavior data set.
https://worldwide.espacenet.com/publicationDetails/biblio?II=1&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200206&CC=US&NR=2020039521A1&KC=A1#

SYSTEM AND METHOD FOR UTILIZING A TEMPORAL RECURRENT NETWORK FOR ONLINE ACTION DETECTION
A system and method for utilizing a temporal recurrent network for online action detection that include receiving image data that is based on at least one image captured by a vehicle camera system. The system and method also include analyzing the image data to determine a plurality of image frames and outputting at least one goal-oriented action as determined during a current image frame. The system and method further include controlling a vehicle to be autonomously driven based on a naturalistic driving behavior data set that includes the at least one goal-oriented action.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2020114924A1&KC=A1&FT=D&ND=3&date=20200416&DB=&locale=en_EP#

[0053] The camera control scenario described with respect to FIGS. 3 and 4 provides an example of eye gaze/facial/pupil tracking and object recognition that may be performed by an in-vehicle computing system. In other examples, such eye tracking may be performed to provide user identification and monitoring, and may be used to learn behaviors and preferences of a user. For example, a pupil diameter of a user may be measured and compared to diameters associated with different user states and environmental conditions. A pupil diameter may change based on ambient light conditions, an alertness of a user, a mood of a user, and/or other conditions. Accordingly, such conditions may be associated with different pupil diameters in a look-up table or other database. The association between diameters and user/environmental states may be customized for different users based on a calibration routine and/or on-going monitoring of the user by the in-vehicle computing system and associated sensors and/or by other computing devices and sensors. Furthermore, data from other sensors may be used to interpret the pupil tracking data. For example, a light sensor or other camera may determine an ambient light condition for the user in order to separate light reflex from cognitive load or other user state-based pupil diameter changes (e.g., to distinguish a cause of the pupil diameter changes).

[0056] The monitoring of biometric data of the user may also be used to determine the user's reaction to an action automatically triggered based on user/environment conditions. For example, rapid eye movement may indicate a heightened level of emotion of a user (e.g., excitement, frustration, fear, etc.). Based on learned behaviors of the user derived from the user's interaction with the in-vehicle computing system and/or other computing devices, the in-vehicle computing system may automatically trigger an action to calm the user responsive to detecting rapid eye movement. For example, the triggered action may include tuning a radio station to a user's favorite station and/or playing a song that has previously been observed to have a calming effect on the user. The user's eye movements may be continually monitored responsive to the action in order to determine whether the action has altered the user's state (e.g., whether the eye movements have slowed). In this way, multiple tiers of actions may be triggered based on whether or not a user responds to a triggered action. For example, if the user's eye movements are not slowed responsive to changing the music in the vehicle, the in-vehicle computing system may further trigger another action, such as changing interior cabin lights to emit a calming hue of light. The number of tiers of actions and the combination of actions in the tiers that result in an intended effect (e.g., a calming of a driver, a reduction of distraction or drowsiness of a driver, etc.) may be stored in association with a user profile or other identifier fo

PORTABLE PERSONALIZATION
[0016] The technologies may utilize the integration of multi-modal human-machine interface (HMI), e.g., eye gaze as an interaction method with a conversational system that simplifies the user interface. The technologies may combine object recognition with eye gaze tracking as a user interface to specify which object the user is attempting to recognize out of multiple ones with real-time segmentation. Facial/eye recognition measures pupil diameter via an eye tracker and separate light reflex from cognitive load. Deep learning engines are used to annotate and analyze parties via a dashboard that will allow for feature acceptance and personalization.

[0045] FIG. 3 schematically shows an example camera control system utilizing components of an in-vehicle computing system 300 which may include and/or be coupled to a one or more vehicle cameras 302 of a vehicle 304. In-vehicle computing system 300 may include and/or be included in in-vehicle computing system 109 of FIG. 1A and/or in-vehicle computing system 200 of FIG. 2. As an example, camera 302 may include a wide angle camera, a field of view of which is schematically shown at 306. An example camera view (e.g., based on the field of view 306) is shown at 308. A driver 310 may interact with and control the camera via gestures. For example, the driver 310 may point at an object of interest 312 in order to request that the camera focuses on or otherwise processes the object. Responsive to detecting the driver's pointing gesture (e.g., via one or more cabin-facing cameras configured to image the driver, passengers, and/or other objects in the cabin of the vehicle 304), the in-vehicle computing system 300 may send control instructions to the camera 302 and/or otherwise process images from the camera 302 in order to select a camera view based on the driver's pointing. An example of the camera view that is computationally selected (e.g., from a gesture such as a pointing gesture from the driver) is indicated at 314. An example final view is shown at 316, and may be sent to an object recognition module of the in-vehicle computing system 300 and/or an off-board (e.g., server) computing device. The final view 316 may include the object of interest 312 that intersects with the driver's pointing direction 318. For example, a sub-section 320 of the wide angle field of view of the camera may be selected based on the driver's pointing direction 318 to generate the final view 316 and/or to otherwise command the camera to focus on an object that intersects the driver's pointing direction. In this way, the driver may perform actions in the vehicle that are interpreted by the in-vehicle computing system 300 to control the camera 302 and/or the processing of images captured by the camera 302.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=2020114931A1&KC=A1&FT=D&ND=3&date=20200416&DB=&locale=en_EP#

TRAINING SALIENCY
Saliency training may be provided to build a saliency database, which may be utilized to facilitate operation of an autonomous vehicle. The saliency database may be built by minimizing a loss function between a saliency prediction result and a saliency mapper result. The saliency mapper result may be obtained from a ground truth database, which includes image frames of an operation environment where objects or regions within respective image frames are associated with a positive saliency, a neutral saliency, or a negative saliency. Neutral saliency may be indicative of a detected gaze location of a driver corresponding to the object or region at a time prior to the time associated with a given image frame. The saliency prediction result may be generated based on features extracted from respective image frames, depth-wise concatenations associated with respective image frames, and a long short-term memory layer or a recurrent neural network.
https://worldwide.espacenet.com/publicationDetails/biblio?II=0&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200326&CC=US&NR=2020097754A1&KC=A1#

INTERPRETING EYE GAZE DIRECTION AS USER INPUT TO NEAR-EYE-DISPLAY (NED) DEVICES FOR ENABLING HANDS FREE POSITIONING OF VIRTUAL ITEMS
A Near-Eye-Display (NED) device utilizes eye tracking to interpret gaze direction as user input for hands free positioning of virtual items. The NED device includes an eye tracking system to monitor gaze direction and a display component that renders virtual items within a user's view of a physical real-world environment. The NED device receives an indication that the user desires to adjust a position of the virtual item. In response, the NED device uses tracks the user's eye movements to dynamically change the position at which the display component is rendering the virtual item. In this way, a NED device may identify when a user desires to adjust a position at which a virtual item is being rendered and then may enable the user to make the desired adjustments simply by “dragging” the virtual item with deliberate and controlled eye movements.
https://worldwide.espacenet.com/publicationDetails/biblio?II=6&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200423&CC=US&NR=2020128231A1&KC=A1#
https://patents.justia.com/patent/20200128231

Integrative Cognition of Driver Behavior
By way of example, the technology disclosed by this document is capable of receiving signal data from one or more sensors; inputting the signal data into an input layer of a deep neural network (DNN), the DNN including one or more layers; generating, using the one or more layers of the DNN, one or more spatial representations of the signal data; generating, using one or more hierarchical temporal memories (HTMs) respectively associated with the one or more layers of the DNNs, one or more temporal predictions by the DNN based on the one or more spatial representations; and generating an anticipation of a future outcome by recognizing a temporal pattern based on the one or more temporal predictions.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2018053093A1&KC=A1&FT=D&ND=3&date=20180222&DB=&locale=en_EP#

Interactive driving system and method
An interactive driving system and method that improves a driving experience by allowing a passenger to interactively monitor and evaluate certain driving maneuvers in real-time with an augmented reality interface. The input provided by the passenger via the augmented reality interface is recorded for later processing and analysis. For instance, the passenger input may be used to generate driving reports for later review by the driver and passenger (e.g., when the passenger is teaching a new driver how to drive) or for the manufacturer of autonomous or semi-autonomous vehicles (e.g., when the manufacturer is seeking actionable feedback for improving various self-driving algorithms). In addition, the interactive driving system and method entertain the passenger with the augmented reality interface, which can be delivered in a game-like fashion via a computer program product installed on their own personal electronic device (e.g., a tablet, phone, laptop, etc.).
https://worldwide.espacenet.com/publicationDetails/biblio?II=2&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200421&CC=US&NR=10625676B1&KC=B1#
https://patents.justia.com/patent/10625676

USER-SPECIFIC EYE TRACKING CALIBRATION FOR NEAR-EYE-DISPLAY (NED) DEVICES
Technologies for performing user-specific calibration of eye tracking systems for Near-Eye-Display (NED) devices. The NED device may sequentially present different virtual stimuli to a user while concurrently capturing instances of eye tracking data. The eye tracking data reveals calibration ellipse centers that uniquely correspond to individual virtual stimuli. The calibration ellipse centers may be used define a polygon grid in association with a sensor plane. The resulting polygon grid is used during operation to interpolate the real-time gaze direction of the user. For example, a real-time instance of eye tracking data may be analyzed to determine which particular polygon of the polygon grid a real-time ellipse center falls within. Then, distances between the real-time ellipse center and the vertices of the particular polygon may be determined. A proportionality factor is then determined based on these distances and is used to interpolate the real-time eye gaze of the user.
https://worldwide.espacenet.com/publicationDetails/biblio?II=3&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200423&CC=US&NR=2020125166A1&KC=A1#
https://patents.justia.com/patent/20200125166
Drawings
http://www.freepatentsonline.com/20200125166.pdf

SYSTEM AND METHOD FOR DETECTING AND/OR PREVENTING AUTOMATION EXPECTATION MISMATCH IN VEHICLE
The present invention relates to a method of evaluating a current risk of mismatch between actual driving automation capabilities of a vehicle and driving automation capabilities of the vehicle expected by a driver. The method comprises monitoring at least one physical property of the driver indicative of a gaze direction; determining a first visual attention metric value indicative of a level of visual attention to the road ahead; comparing the first visual attention metric value to a first threshold value; and providing, when the comparison indicates that the current level of visual attention to the road is lower than the first threshold level, a signal indicative of an elevated risk of expectation mismatch.

[0011] The present inventors have performed behavioral tests indicating that as many as 28% of drivers crashed despite having their eyes on the conflict object (garbage bag, or parked car) while using highly reliable (but not perfect) driving automation. When analyzing these tests, the inventors surprisingly found that, although crashing drivers looked ahead when the crash occurred, there is a strong correlation between drivers exhibiting a low level of visual attention to the forward roadway prior to encountering the conflict object, and drivers who did not intervene, but allowed the vehicle to crash into the conflict object. In other words, the present inventors have surprisingly found that there is a strong correlation between low levels of visual attention to the forward roadway during autonomous vehicle operation and an elevated risk of mismatch between actual driving automation capabilities of a vehicle and driving automation capabilities of the vehicle expected by the driver operating the vehicle.

[0012] This mismatch, which may be referred to as "Automation Expectation Mismatch" (AEM) demonstrates that a key component of driver engagement while using automation is cognitive (understanding the need for action), rather than purely visual (looking at the threat), or having hands-on-wheel. Cognitive understanding of the need to act is a crucial component of driver engagement while using driving automation systems that are not perfect. AEM is thus a newly discovered cognitive state of mind.
https://worldwide.espacenet.com/publicationDetails/biblio?II=0&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200423&CC=US&NR=2020122745A1&KC=A1#
Drawings.
http://www.freepatentsonline.com/20200122745.pdf

LOW-POWER IRIS SCAN INITIALIZATION
Apparatuses, methods, and systems are presented for sensing scene-based occurrences. Such an apparatus may comprise a vision sensor system comprising a first processing unit and dedicated computer vision (CV) computation hardware configured to receive sensor data from at least one sensor array comprising a plurality of sensor pixels and capable of computing one or more CV features using readings from neighboring sensor pixels. The vision sensor system may be configured to send an event to be received by a second processing unit in response to processing of the one or more computed CV features by the first processing unit. The event may indicate possible presence of one or more irises within a scene.
https://worldwide.espacenet.com/publicationDetails/biblio?II=5&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200423&CC=US&NR=2020125842A1&KC=A1#
Drawings
http://www.freepatentsonline.com/20200125842.pdf

Assessing driver ability to operate an autonomous vehicle
Human operators of autonomous vehicles may exhibit varying levels of operating proficiencies and may respond differently to changing road conditions or requests from the vehicle to assume more or less control over navigation. A system for assessing driver capability of an autonomous vehicle assesses human operator capabilities in a variety of circumstances to assemble a driving capability profile of the human operator. The autonomous vehicle may rely on the driving capability profile as well as an environmental profile and human operator parameters describing the environment surrounding the vehicle and an alertness level of the human operator, respectively. The vehicle may condition requests to increase or decrease driving responsibilities of the human operator depending on the driving capability profile in combination with the environmental profile and/or the human operator parameters. An insurer may condition an insurance premium cost for a time period based on the driving capability profile in combination with the environmental profile and/or the human operator parameters.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=10618523B1&KC=B1&FT=D&ND=3&date=20200414&DB=&locale=en_EP#
https://patents.justia.com/patent/10618523

Systems and methods for improving situational awareness of a user
System, methods, and other embodiments described herein relate to improving situational awareness of a user. In one embodiment, a method includes, in response to acquiring, in a personal device, sensor data from a set of sensors that perceive a surrounding environment of the user, identifying from the sensor data one or more objects in the surrounding environment relative to the user. The method includes determining a threat to the user from the one or more objects according to at least trajectories of the one or more objects embodied in the sensor data. The method includes generating, via the personal device, an indicator to inform the user of the one or more objects according to the threat.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=10621858B1&KC=B1&FT=D&ND=3&date=20200414&DB=&locale=en_EP#
https://patents.justia.com/patent/10621858

MACHINE LEARNING ON BIG DATA IN AVIONICS
A method for handling aircraft data, includes the steps of receiving an input aircraft model and/or operational data; comparing the model and/or data with reference data previously obtained by machine learning performed on large collections of data of recorded flights, wherein the reference data defines safe boundaries for one or more aircraft models and/or operations data. Developments describe “big data” aspects, comprising aircraft or flight data of a plurality of airlines about a plurality of aircraft, engines, flight contexts, aircraft configurations and meteorological data; the handling of calculations about performances, weight and balance, fuel, crew duty times; etc., the use of machine deep learning (unsupervised pre-training); comparisons against a superset of data generated out from the collected large collections of data, with or without human intervention; automated validation tests of aircraft models and associated data. Hardware and software aspects are described.

[0022] In some embodiments, a "Deep Learning (Artificial Intelligence) Processing Engine" (acronym DLPE) can analyze the CDM or big data and establish relationships between variables, that is a "multi-dimensional model space". Deep learning is just one of the techniques which can be used (more generally: machine learning can be used). Such a model for example enables to determine if values entered by a pilot are within the expected boundaries (the "safe" boundaries).

[0023] Aviation or avionics "safety" designates the state of an avionics system (or organization) in which risks associated with aviation activities, related to, or in direct support of the operation of aircraft, are reduced and controlled to an acceptable level. "Safety" encompasses the theory, practice, investigation, and categorization of flight failures, and the prevention of such failures through regulation, education, and training.

[0024] In some embodiments, a "Sampling and Validation Engine" (acronym SVE) can generate the data scope (e.g. test fixture) needed to validate the results produced by the Operational Software against the Deep Learning Processing Engine. The SVE will also create a report for quality assurance purposes.
https://worldwide.espacenet.com/publicationDetails/biblio?II=59&ND=3&adjacent=true&locale=en_EP&FT=D&date=20200416&CC=US&NR=2020115066A1&KC=A1#

Drawings.
http://www.freepatentsonline.com/20200115066.pdf

Hi Redindi,
I have not had a chance to look into this much but the association that first came to mind was the CAD geometry references (below) in recent SEE patent DRIVER ATTENTION STATE ESTIMATION.
https://worldwide.espacenet.com/publicationDetails/description?CC=WO&NR=2020061650A1&KC=A1&FT=D&ND=3&date=20200402&DB=&locale=en_EP#

[0040] To perform method 400, an initial setup step 401 is performed. At step 401 , driving scene 500 is digitally represented such that the three-dimensional geometry of objects and regions within the scene are known. The scene geometry may be determined, at least in part, from a three-dimensional model of the vehicle such as a CAD model provided by a vehicle manufacturer. The scene geometry may also be determined from one or more two or three-dimensional images of scene 500 captured by camera 106 or other cameras in or around scene 500. In either embodiment, the digital representation of scene 500 may include positions and orientations of known features within scene 500, which may be defined in a reference coordinate frame. By way of example, the known features may include individual vehicle dashboard instruments, definable cabin contours, edges, or objects or the entire vehicle cabin itself. The features may be fixed in time and space relative to a frame of reference such as a vehicle frame of reference defined relative to a region of the vehicle frame.