London South East | The UK's favourite investor portal.

SEEING MACHINES 2019-12-26
HIGH FRAME RATE IMAGE PRE-PROCESSING SYSTEM AND METHOD
Described herein is an image pre-processing system and method. One embodiment provides a method (500) including: at step (501), receiving a plurality of images captured at a first frame rate, the plurality of images captured under at least two different image conditions; pre-processing the plurality of images by: at step (502) identifying one or more regions of interest within the images; at step (503), performing a visibility measure on the one or more regions of interest; and, at step (504), selecting a subset of the plurality of images based on the visibility measure; and, at step (505), feeding the subset of images to an image processing pipeline for subsequent processing at a second frame rate that is lower than the first frame rate.
https://worldwide.espacenet.com/publicationDetails/biblio?II=0&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191226&CC=WO&NR=2019241834A1&KC=A1#

[0038] The avatar intelligent behavior model 224 defines logic and data to support behavior and responses in order to simulate a cognitive state in the simulated co-pilot and generate realistic actions (e.g., conversation, emotions, physical task, etc.) to simulate the co-pilot in the scenarios. For instance, the avatar intelligent behavior model 224 can define behavior and responses to simulate expert pilot or co-pilot cognition and comprehension of flight deck 202 procedures and communications. Likewise, for instance, the avatar intelligent behavior model 224 can define behavior and responses that simulate actions in accordance with simulation scenarios and performance measure checkpoints. Also, for instance, the avatar intelligent behavior model 224 can define branching logic to invoke situations in the flight deck 202 (e.g., flight deck faults, equipment failures, crew communication issues, etc.) to allow measurement of the operator or student competencies. Further, for instance, the avatar intelligent behavior model 224 can define logic to recognize the cognitive and emotional state of the operator or student and respond to the cognitive and emotional state of the operator or student.

[0039] The avatar intelligent behavior model 224 (and/or the avatar physical model 222) also defines data output to support the operation of the immersive aircraft simulator system 200. For example, the avatar intelligent behavior model 224 (and/or the avatar physical model 222) outputs data to the co-pilot simulator 212 in order to respond to and to generate verbal communication, for instance, syntax and spoken words and voice inflection to indicate realistic behavior and emotion. Also, for example, the avatar intelligent behavior model 224 (and/or the avatar physical model 222) outputs data to the co-pilot simulator 212 in order to respond to and to generate non-verbal communication (e.g., facial expressions, gestures, etc.). Additionally for example, the avatar intelligent behavior model 224 (and/or the avatar physical model 222) outputs data to the co-pilot simulator 212 in order to generate olfactory cues (e.g., variance in hygiene, bodily function, etc.) and generate physical movement. The avatar intelligent behavior model 224 (and/or the avatar physical model 222) also receives input in order to simulate real-world interactions. For example, the avatar intelligent behavior model 224 (and/or the avatar physical model 222) can any one of receive data from the facial and gesture capture system 206, receive data from the voice recognition system 208, receive data from the physiological measure system 210, receive data from systems controlling the simulation scenario, receive data from an instructor or other user, and any combination thereof, and the like.

[0031] The flight deck 202 includes one or more devices and systems to capture biofeedback data. The biofeedback sensors can be any type of system that includes hardware, software, or combinations thereof and is capable of sensing or detecting actions, reactions, inputs, emotions, and the like of the operator 104. For example, the biofeedback sensors can be configured to capture the biofeedback data including speech and sound data, scent data, brain wave data (e.g., electroencephalogram (EEG) data), eye movement tracking data, micro-facial expressions data, body language, vital sign data (heart rate, blood pressure, respiration rate, etc.), muscle movement data, capillary dilation data, skin conductivity data, and the like.
[0032] For example, the biofeedback devices and system can include a facial and gesture capture system 206, a voice recognition system 208, and a physiological measurement system 210. The facial and gesture capture system 206 can be any type of system that includes hardware, software, or combinations thereof to detect, capture, and analyze facial gestures and body movements of the operator so that emotional and cognitive states of the operator can be determined. The voice recognition system 208 can be any type of system that includes hardware, software, or combinations thereof to detect, capture, and analyze the speech and sounds emitted by an operator or student. For instance, the voice recognition system 208 can analyze speech and sounds of the operator or student to detect the syntax and spoken words of the operator or student in order to identify speech directed to the simulation scenario and differentiate between actionable and non-actionable commands. Likewise, for instance, the voice recognition system 208 can analyze speech and sounds of the operator or student to detect the emotional aspects (e.g., inflection, tone, etc.) and interpolate the operator or student emotional state. The physiological measurement system 210 can be any type of system that includes hardware, software, or combinations thereof to detect, capture, and analyze physiological states of the operator or student (e.g., brainwaves, vital signs, capillary dilation, skin conductivity, etc.)

BOEING CO 2019-11-14
SYSTEMS AND METHODS FOR AN IMMERSIVE SIMULATOR
Systems, computer readable media, and method concern includes generating visual, auditory and other sensory depictions of a life-sized virtual crew member in an aircraft simulator. The virtual crew member simulates operational actions and behavioral and physiological responses of a crew member of an aircraft. The method includes collecting one or more responses of an operator using the aircraft simulator. The one or more responses comprise biofeedback data associated with the operator. The method includes generating, in response to the one or more responses, one or more simulated operational actions or simulated behavioral and physiological responses for the virtual crew member. The method includes collecting one or more additional responses of the operator. The one or more additional responses comprise additional biofeedback data of the operator that corresponds to an interaction with the one or more simulated operational actions or simulated behavioral and physiological responses for the virtual crew member.

[0017] The simulator 102 includes biofeedback sensors 110. The biofeedback sensors 110 can capture signals representing conscious and subconscious reactions, emotions, response, voice patterns, and the like of the operator 104. The reactions, emotions, response, and the like can be in response to the operation of the simulator 102 and can be in response to the interactions with the crew member simulated by the crew member simulator 108. The biofeedback sensors 110 can be any type of device that is capable of sensing or detecting actions, reactions, inputs, emotions, and the like of the operator 104. For example, the biofeedback sensors 110 can be configured to capture the biofeedback data including speech and sound data, scent data, brain wave data (e.g., electroencephalogram (EEG) data), eye movement tracking data, micro-facial expressions data, body language, vital sign data (heart rate, blood pressure, respiration rate, etc.), muscle movement data, capillary dilation data, skin conductivity data, and the like.

[0029] The immersive aircraft simulator system 200 includes a flight deck 202. The flight deck 202 can be configured to simulate the environment of an aircraft flight deck. The flight deck 202 can include the systems, hardware, control, displays, and the like found on an aircraft flight deck. In examples, the flight deck 202 can be a flight training device and/or full flight simulator. Flight deck 202 can be implemented using any commercially available flight deck simulator along with modifications according to aspects described herein. In some examples, the flight deck 202 can be implemented using a flight deck simulator available from CAE, Inc. and/or Thales.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=2019347952A1&KC=A1&FT=D&ND=3&date=20191114&DB=&locale=en_EP#

Could Toyota BSMs be a 'tailored 'Guardian BdMS ?

DETECTION OF A DROWSY DRIVER BASED ON VEHICLE-TO-EVERYTHING COMMUNICATIONS
[0004] The first solution provided by the drowsy detection system according to some embodiments is now described. Vehicles are increasingly equipped with Dedicated Short-Range Communication (DSRC) capabilities. DSRC-equipped vehicles transmit Basic Safety Messages ("BSM" if singular or "BSMs" if plural) at a regular interval (e.g., once every 0.10 seconds). BSMs have a mandatory payload that includes, among other things, information about the path history of the vehicle that transmits the BSM. A remote vehicle is driven by a drowsy driver. The remote vehicle is DSRC-equipped, and it regularly transmits BSMs that include path history data of the remote vehicle. The drowsy detection system includes software installed on an ego vehicle that receives the BSMs from the remote vehicle and analyzes the path history data to identify the presence of the drowsy driver. The ego vehicle is driven by an ego driver. The drowsy detection system is operable to notify the ego driver of the presence of the drowsy driver so that they can take steps to mitigate the danger posed to them by the drowsy driver.

[0005] The second solution provided by the drowsy detection system according to some embodiments is now described. An ego vehicle is driven by a drowsy driver and a remote vehicle is driven by a remote driver. Both the ego vehicle and the remote vehicle are DSRC-equipped, and both include an instance of the drowsy detection system installed in their onboard vehicle computer. The ego vehicle includes a driver monitoring system that detects that the driver is drowsy. In some embodiments, the drowsy detection system of the ego vehicle is installed in an Engine Control Unit (ECU) that operates the driver monitoring system. The drowsy detection system of the ego vehicle (1) identifies that the ECU is processing signals that indicate that the driver is drowsy and (2) determines whether the ego vehicle is in automated driving mode. The drowsy detection system of the ego vehicle inserts digital data into a BSM transmitted by the ego vehicle that describes (1) whether driver of the ego vehicle is drowsy; and (2) whether the ego vehicle is in automated driving mode [i.e., because the driver's drowsiness does not matter if automated driving is engaged]. The ego vehicle transmits the BSM. The remote vehicle receives the BSM. The drowsy detection system of the remote vehicle notifies the driver of the remote vehicle about the presence of the drowsy driver so that they can take steps to mitigate the danger posed to them by the drowsy driver. If the remote vehicle is an autonomous vehicle, then the remote vehicle automatically responds to the danger posed by the drowsy driver (e.g., by not assuming that the ego vehicle will be driven based on existing models of driver behavior).
https://tinyurl.com/r2u9xmk

SEE BdmS
https://tinyurl.com/

VOLKSWAGEN AG 2019-06-06
METHOD FOR CALCULATING THE MOVEMENT DATA OF THE HEAD OF A DRIVER OF A TRANSPORTATION VEHICLE, DATA GLASSES AND TRANSPORTATION VEHICLE FOR USE IN THE METHOD,

[0030] DE 102014206626 A1 discloses the principle of such a pair of data glasses for use in the transportation vehicle as a driver assistance system. The glasses are equipped with an electronic sensor and processing unit. This houses a unit for measuring rotation rates and acceleration values. Also, the data glasses have a communication module for wireless communication according to the BLUETOOTH® standard. Via the communication module, the glasses send the measurement data to an external control unit. The external control unit continuously receives measurement data for setting the steering angle as well as for accelerating the transportation vehicle. From this data, the control unit calculates the acceleration and rotation rate of the transportation vehicle. The external control unit subtracts the accelerations and rotation rates of the transportation vehicle calculated at the same time from the acceleration and rotation rate values that were measured in the glasses. The glasses are also equipped with at least one camera. The change in the mapping of objects in the interior of the transportation vehicle can be used to derive the motion of the data glasses, in other words the motion of the head. The data glasses are used for detecting driver fatigue.

[0031] The HoloLens data glasses from the Microsoft company are a known brand. HOLOLENS is a registered trade mark of the Microsoft company. These glasses also contain a special processing unit (holographic processing unit, HPU), which performs the calculation of the head pose based on the measurements of an IMU (Inertial Measurement Unit) which contains the rotation rate and acceleration sensors of the data glasses, and also based on the video images that are supplied by the video cameras with which the glasses are equipped. The video cameras are used to record the surroundings of the glasses. What the cameras record corresponds to a much larger area than that which is visible to the wearer of the glasses. The portion used for display is in fact specified using the calculated head pose. The images to be displayed are calculated using a graphical processing unit GPU. In addition, a central processing unit CPU, on which specific application programs can be executed, is also part of the glasses. The superimposed images calculated and output by the GPU are also adjusted to suit the display excerpt before they are actually displayed. This increases the accuracy of the image output, because the calculation of the images to be output is subject to a certain latency, and due to the continued movement of the head this would otherwise cause inaccuracy in the image output. In the case of the HoloLens glasses the display is performed by presentation on specific image output displays.
https://tinyurl.com/svolulj

[0076] The hypothetical future behavior models can be likened to introducing hypothetical “what ifs” to all road users and potentially hidden road users in the traffic environment of the host vehicle 1. Thus, assisting a driver of the host vehicle 1 in keeping a level of attention which is required by the situation with respect to different risks that may arise based on an assessment that is made by the at least one processing device 2 with respect to when the driver's attention is required, derived by introducing —“what if this road user behaves in this way?”—to all road users and potentially hidden road users in the traffic environment.

[0087] In this embodiment, eye movement plus other complimentary information, such as touchscreen use, other driver activity, seat recline status, etc., are used to detect an out-of-the-loop driver. Here, eye movement is not simply assessed relative to time and focus, but is also assessed relative to the threat assessment that has been made by the road vehicle. Out-of-the-loop drivers tend to have long eye glances (e.g., 1-40 seconds) and ignore warnings. Thus, these eye glances may be patterned and summarized with a buffer. Time looking at the road and time looking away from the road may be integrated over time, for example, with time looking at the road credited and time looking away from the road debited. In this manner, patterns may be established and rules may be established to detect an out-of-the-loop driver. Current methodologies analyze only “distracted” or “not distracted” at an instant in time. The present methodology analyzes a pattern of behavior and establishes a probability that a given driver is out-of-the-loop or not. This methodology is illustrated conceptually in FIG. 3. This out-of-the-loop detection pattern over time is exemplified as a buffer, but additional alternatives for pattern detection may also be implemented. Alternative pattern quantifications are exceedance of safe attention behavior rules, such as inappropriate off-path (or off-threat) long glance durations, inappropriate on-path (or on-threat) short glance durations, and the detection of a series of on-and-off glances over a window of time such as 6 s, 30 s, 60 s, or minutes. These patterns can be compared to reference patterns of attentive and skilled driver eye movement to determine inappropriateness. These reference patterns can be determined for different driving contexts, such as day/night, road curvature, friction levels, type of traffic environment (e.g. traffic density), different speeds, road types (motorway, rural roads, city driving, proximity to intersections, etc.). Reference patterns can also be adapted to individual differences, such as eye and facial features, and medical problems (e.g. strabismus), all of which impact eye movement patterns. Essentially, inappropriateness is determined for an individual in a driving context for a pattern over time or in response to a prompt, such as a warning.

[0046] Depending on the level of driver distraction, the warning shall hence be given to a driver in time to react to the warning and reach a high level of attention before the critical point in time. For a driver that is already attentive, no warning is needed, whereas for a distracted driver a warning shall be given e.g. 1 to 3 seconds before the critical point in time, depending on the level of distraction. To sum up, the driver need to be alerted 2 to 5 seconds before the critical point, or the speed of the host vehicle need to be reduced and/or the lateral position need to be adjusted to move the critical point forward in time. The critical point in turn may depend on the ability of the driver and/or the combined ability of the driver and ADAS systems to deal with the hypothetical threat. Which hypothetical threats that a driver should be alerted for and how the alert or speed reduction should be designed is a matter of choice for a system designer. Typically, a system designer might want to alert more frequently and earlier for hypothetical threats that are highly probable and later for threats that are less probable in the current traffic situation. Hypothetical threats can be applied both to observed objects and to obscured areas where objects and road users might be hidden. The probability of a threat event may in turn be estimated online/offline using environmental sensors and/or statistics from naturalistic driving data and accident data.

[0052] Optionally the method further comprises deriving, by the one or more driver-monitoring sensors 4 of the host vehicle 1, at least one of a gaze-direction and head-pose of a driver of the host vehicle 1. Such a driver-monitoring sensor 4 may e.g. be an interior camera that is arranged for deriving gaze and head-pose of a driver of a road vehicle 1, such that a viewing direction of the driver can be determined. From the derived gaze-direction and/or head-pose is determined if a host vehicle driver is attentive in a direction of the at least one hypothetical threat. If determined that the host vehicle driver is attentive in the direction of the at least one hypothetical threat, the threshold-time tthres can be reduced compared to if derived that the attention is not in a direction of the at least one hypothetical threat. Furthermore, if the estimated time remaining until a high level of attention is less than the threshold-time tthres, a driver may be encouraged, e.g. by the visual, acoustic and haptic stimuli provided to keep a high level of attention in the direction from which the threat may arise.

[0037] In the following will be described some example embodiments of a method and system for using information collected from road vehicle exterior sensing-systems with at least one road vehicle driver-monitoring sensor, e.g. camera-sensor, to assist road vehicle drivers in driving with precaution in traffic situations where a high level of attention is required.

[0038] Embodiments herein thus aim to assist a driver of a road vehicle in keeping a level of attention which is required by the situation with respect to different risks that may arise. In some situations, the required level of attention is low, whereas other situations require a higher level of attention.

[0042] A current host vehicle driver level of attention is derived 13 from one or more driver-monitoring sensors 4 of the host vehicle 1. Such a current host vehicle driver level of attention can e.g. be derived from gaze-directions and head-poses of a driver of the host vehicle determined by the one or more driver-monitoring sensors 4 of the host vehicle 1. The one or more driver-monitoring sensors 4 may e.g. comprise one or more driver-monitoring cameras and/or other types of sensors for monitoring driver behavior, such as e.g. sensors for monitoring driver interaction with a control panel, such as a touch-screen display or similar.

[0044] If determined that the estimated required host vehicle driver level of attention exceeds the current host vehicle driver level of attention and that the time until the estimated host vehicle driver level of attention will be required is less than the threshold-time tthres, according to the proposed method, there should be produced 15 at least one of visual, acoustic and haptic information to a host vehicle driver environment. Hence, if the time remaining until a high level of attention is required is less than the threshold-time tthres, a driver shall be encouraged to keep a high level of attention. Such information should be provided to promote increased host vehicle driver-attention to the at least one hypothetical threat. As an alternative or complement thereto, according to the proposed method, at least one of automated braking and steering of the host vehicle should be triggered. Such automated braking and/or steering of the host vehicle should thus be triggered to promote an increase in safety margin to the at least one hypothetical threat.

[0012] In a further embodiment the method further comprises: deriving, by the one or more driver-monitoring sensors of the host vehicle, at least one of a gaze-direction and head-pose of a driver of the host vehicle; and determining, from the derived gaze-direction and/or head-pose, if a host vehicle driver is attentive in a direction of the at least one hypothetical threat and if so determined reducing the threshold-time compared to if derived that the attention is not in a direction of the at least one hypothetical threat.
[0026] In still yet an additional embodiment, the method and system are operable for detecting and adequately warning a driver that is determined to be of a unique out-of-the-loop class of drivers that mistakenly believe that a road vehicle operating in a driver-assist or self-driving (i.e., piloted-driving) mode will automatically identify and react to a perceived or sudden threat. This situation requires a systematic analysis and classification of the behavior of a given driver and a higher level of coaching and warnings for this unique class of drivers. Of note, this out-of-the-loop detection may also be referred to as the “estimation of a predicted driver response to a crash situation” from patterns of behavior preceding a crash or “automation expectation mismatch” detection.

[0028] The present disclosure provides a method for assisting a driver of a host vehicle to drive with precaution, the method including: determining from one or more host vehicle sensors or cameras arranged to monitor an environment of the host vehicle other road users and at least one particular feature associated with a traffic situation of the host vehicle; applying at least one hypothesis related to at least one hypothetical threat that may arise based on the determined other road users and the at least one particular feature; analyzing an out-of-the-loop condition of the host vehicle driver based on one or more of host vehicle driver eye movement, host vehicle driver activity, host vehicle equipment sensor data, and host vehicle driver periods of attention and inattention over a period of time; setting or estimating, based on the applied at least one hypothesis and the out-of-the-loop condition, a host vehicle driver level of attention threshold required to handle the at least one hypothetical threat and estimating a time until that host vehicle driver level of attention will be required; deriving, from one or more host vehicle driver-monitoring sensors of the host vehicle, a current host vehicle driver level of attention; determining if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention; and, if the set or estimated required host vehicle driver level of attention threshold exceeds the current host vehicle driver level of attention, producing at least one of a visual, acoustic, haptic information, and brake impulse prompt to a host vehicle driver environ

METHOD AND SYSTEM FOR ASSISTING DRIVERS TO DRIVE WITH PRECAUTION
A method and system operable for detecting and adequately warning a driver that is determined to be of a unique out-of-the-loop class of drivers that mistakenly believe that a road vehicle operating in a driver-assist or self-driving (i.e., piloted-driving) mode will automatically identify and react to a perceived or sudden threat. A method and system operable for detecting and adequately warning an inattentive, drowsy, and/or out-of-the-loop driver and assessing and confirming the driver's adequate response to the associated warning prompt(s), which may include audio, visual, haptic, and, more preferably, brake pulse or the like warning prompt(s).

BACKGROUND
[0003] Modern road vehicles are equipped with an increasing amount of exterior sensors and communication devices in order to be able to collect information about their surroundings. Examples of such exterior sensors are radar and camera sensors that monitor traffic in front of a road vehicle, such that a driver of the vehicle can be assisted in avoiding intersection and rear-end collisions, for example.

[0004] Previous systems for assisting road vehicle drivers in keeping a high level of attention have primarily focused on detection and warnings for driver eyes off-road/driver-distraction situations. Such systems have usually been designed to either warn a driver of a road vehicle if the eyes off-road time exceeds a limit or to trigger warnings for threats at a low threat-level if the driver of the road vehicle is distracted.

[0005] Such existing solutions sometimes suffer some drawbacks while driving a road vehicle in a manual or supervised automation-mode. This is as a low level of driver-attention does not always imply a high level of risk. Warning a driver of a road vehicle of low attention levels may lead to unnecessary warnings if a traffic situation is considered as safe even with a low level of driver-attention. However, threats may also suddenly appear at any time while a driver is distracted, which can make it difficult to warn the driver in due time once a threat is detected.

[0006] To overcome this problem, solutions have been proposed to combine driver-monitoring cameras with exterior sensing-systems to warn the driver of a road vehicle if it is assessed that the driver has not perceived a threat which has been detected by an exterior sensing-system. These types of solutions can be used to reduce the number of unnecessary warnings, but they may however also lead to situations with late or missed warnings, e.g. due to occluded threats, such as pedestrians that walk out from an obscured area from which the exterior sensing-systems are unable to obtain sufficient information.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=2019367050A1&KC=A1&FT=D&ND=3&date=20191205&DB=&locale=en_EP#

Latest Share Chat

Member Info for johnchucka

Followers

Following

Posts (Last 30 Days)

Posts (All Time)

Last Post

Member Since

Share Chat Post Distribution (Last 30 Days)

Activity Log (Last 30 Days)

Latest Share Chat

Member Info for johnchucka

Followers

Following

Posts (Last 30 Days)

Posts (All Time)

Last Post

Member Since

Share Chat Post Distribution (Last 30 Days)

Activity Log (Last 30 Days)

Please enter at least 2 characters

No results found

Search results

Recently viewed

Trending