Member Info for johnchucka

Intel Gets IEEE to Ask ‘How Safe Is Safe Enough’ for AVs
https://www.eetimes.com/intel-gets-ieee-to-ask-how-safe-is-safe-enough-for-avs/#

Don't think I have posted this previously as the patent is from TOYOTA MOTOR CO and I have tended to focus on TRI. I thought the last sentence here sounded like could be SEE designed DMS but could equally be a standard design methodology.
TOYOTA MOTOR CO
DRIVER CONDITION DETECTION SYSTEM 2019-05-17

Hereinafter, a first embodiment of the present invention will be described with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing a configuration of a driver state detection device according to a first embodiment of the present invention. The driver state detection device 1 is mounted on a vehicle and detects the state of the driver of the vehicle. The driver state detection device 1 includes a driver monitoring camera 10 and an electronic control unit (ECU) 20.

FIG. 2 is a diagram schematically showing the inside of a vehicle equipped with a driver state detection device. The driver monitoring camera 10 captures the face of the driver of the vehicle 80 (own vehicle) to generate an image of the driver's face. The driver monitoring camera 10 is provided inside the vehicle 80. Specifically, as shown in FIG. 2, the driver monitoring camera 10 is provided above a steering column 81 of the vehicle 80. In FIG. 2, the projection range of the driver monitoring camera 10 is shown by a dotted line. In addition, the driver monitoring camera 10 may be provided on a steering wheel 82 of the vehicle 80, a rearview mirror, a dashboard, a dashboard, and the like. The driver state detection device 1 may include a plurality of driver monitoring cameras 10.

The driver monitoring camera 10 includes a camera and a light projector. For example, the camera is a CMOS (Complementary Metal Oxide Film Semiconductor) camera or a CCD (Charge Coupled Element) camera, and the projector is an LED (Light Emitting Diode). In addition, it is preferable that the light projector is a near-infrared LED, so that the driver's face can be photographed without discomfort to the driver even at low illumination such as at night.

For example, the projector is two near-infrared LEDs arranged on both sides of the camera. In addition, a filter such as a visible light cut filter may be provided in the camera. The driver's face image generated by the driver monitoring camera 10 is transmitted from the driver monitoring camera 10 to the ECU 20.

https://worldwide.espacenet.com/publicationDetails/biblio?CC=CN&NR=109758167A&KC=A&FT=D&ND=3&date=20190517&DB=&locale=en_EP#

FORD GLOBAL TECH
Vehicle Localization Using Cameras 2019-12-19
[0013] Localization is an important part of allowing for and improving operation of autonomous vehicles or driver assistance features. For example, a vehicle must know precisely where it is in the world (or with respect to a map) in order to navigate safely. A large variety of current approaches exist to localize with respect to a map of the world, but require a substantial quantity of computation to do so, or necessitate the use of expensive, power-hungry sensors, such as light detection and ranging (LIDAR) sensors.

[0015] In at least one approach present in the present disclosure, a camera-based system on a vehicle may be used to localize the vehicle with respect to a prior generated or obtained map. The prior map could be a vector-based semantic map that contains information like the locations of lane lines and other road markings. An example of a vector base semantic map is the Road Network Definition File (RNDF) provided by the Defense Advanced Research Projects Agency (DARPA) for the Urban Challenge. The prior map may include a LIDAR intensity map of the ground plane. The prior map may be compared a projected top-down image generated based on a captured camera image.

[0030] The localization component 104 extracts 302 a top-down image from the camera image. For example, the localization component 104 may extract a ground plane or driving surface from within the camera image. Extracting the ground plane may involve identifying pixels or regions within the camera image that correspond to the ground plane or driving surface. For example, the ground plane or driving surface may be identified using a RANSAC algorithm based on the camera image and another camera image (such as an image captured by another camera or an earlier or later image captured by the same camera). The localization component may then create the top-down image by projecting the ground plane or driving surface as a top down image. For example, the ground plane or driving surface may be skewed or rotated to create a top-down view of the ground plane or driving surface. The resulting top-down image or top-down view may be a top-down view of the pavement or driving surface only.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=2019385336A1&KC=A1&FT=D&ND=3&date=20191219&DB=&locale=en_EP#

Convolutional Neural Network

JAGUAR LAND ROVER 2019-12-12
DISPLAY CONTROL APPARATUS AND METHOD
The present disclosure relates to a display control apparatus (1) for dynamically controlling the display of information in a vehicle (3). The display control apparatus (1) comprises a controller (12) having at least one electronic processor (13). The at least one electronic processor (13) is configured to control first and second displays (4, 5). In dependence on a determined gaze direction of the vehicle driver, the at least one electronic processor (13) controls a switching module (16) to cause a first information data set (INF1) displayed on said first display (4) to be displayed on said second display (5). The present disclosure also relates to a method of controlling the display of information in a vehicle (3).

[0045] The display control apparatus 1 could optionally be configured to determine a driver workload or to receive a driver workload signal. The driver workload can be determined with reference to a vehicle control inputs, such as steering angle, and/or dynamic parameters, such as vehicle speed/acceleration. Data received from the vehicle sensors, such as traffic monitoring sensors, can also be used to determine driver workload. The display control apparatus 1 could be configured to inhibit switching of the information data set INF1 between the first and second displays 4, 5 when the driver workload is determined to be high. Alternatively, or in addition, the display control apparatus 1 can modify the time period for which the information data set INF1 is displayed on the second display 5 when it is switched from the first display 4.

[0046] The display control apparatus 1 can optionally be configured to monitor how the driver performs a particular task (or set of tasks), for example to learn how much time the user requires to perform a task and how often the driver looks at the first display 4 whilst carrying out that tasks. The display control apparatus 1 may monitor driver behaviour in different situations, for example based on the current workload of the driver or in particular driving conditions. The display control apparatus 1 can be configured to modify how the switching module 16 is controlled in dependence on the historic driver behaviour.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019377534A1&KC=A1&FT=D&ND=3&date=20191212&DB=&locale=en_EP#

https://www3.nhk.or.jp/nhkworld/en/news/20191218_16/

Toyota to use advanced self-driving tech in commercial vehicles first

https://www.reuters.com/article/us-toyota-autonomous/toyota-to-use-advanced-self-driving-tech-in-commercial-vehicles-first-idUSKBN1YL0G9

As Lester Freaman once said,
'You follow drugs, you get drug addicts and drug dealers. But you start to follow the patents , and you don’t know where the f*** it’s gonna take you'.

I get the impression they are all in the same room sharing a thesaurus.

Smokey,
I'm making a load of assumptions and guesses based on some things we know and somethings we don't. I'm starting with the known linkage that GM patents GM gaze tracking systems IP builds upon SEE IP and by associating similarly worded and closely dated patents from the likes of Toyota and Honda I'm suggesting their maybe also be a similar IP structure.

I'm also assuming again from the patents that the IP algorithms for in/out facing camera calibration described in the SEE patent of 2019-08-29 below is the same IP that Bosch , GM and Toyota etal build their systems around.

In the grand scheme of things I then extrapolate that these systems in the patents I post are all describing what will be part of the next global safety system architecture which will incorporate cloud based and local street scene data in part collected by SEE calibrated inward / out cameras.

Seeing Machines 2019-08-29
SYSTEM AND METHOD FOR IDENTIFYING A CAMERA POSE OF A FORWARD FACING CAMERA IN A VEHICLE
[0074] Thus, by estimating the position of the center of the forward road lane (or other object/region statistically definable by gaze definable), the gaze direction calculated form images of the driver's face captured from one of cameras 102-104 can be accurately mapped to the forward facing road scene captured by camera 101. The projection or mapping may be performed by deriving a transformation function or matrix which maps the gaze direction observed in the driver facing camera onto the forward facing road scene images captured from the forward facing camera. In some

On callibration
GM
THREE-DIMENSIONAL ALIGNMENT OF RADAR AND CAMERA SENSORS 2019-04-26
Bosch
Method for operating a driver assistance system for motor vehicles 2019-12-05

Toyota / Simon Stent / 2019-02-07
METHODS AND SYSTEMS PROVIDING AN INTELLIGENT CAMERA SYSTEM
[0027] Turning to FIG. 2, a diagram 200 depicts one embodiment showing an example of a vehicle passenger compartment 202 featuring a gaze detector 206, located on a rearview mirror 208, which tracks the gaze of a driver 204 looking out the windshield 210. In various embodiments, the gaze detector 206 may be located, by way of non-limiting example, on the top, bottom, front, back, on a side, and/or any other suitable portion located inside of or on the exterior of the vehicle 100. Any suitable type of saccading camera system may be utilized, which in some embodiments may be a camera system that provides for rapid camera movements. In other embodiments, two or more cameras may be coordinated in saccading movement. In various embodiments, saccading movement of one or more cameras may correspond to detected eye movement, such as with gaze detection. For example, a gaze detector 206 may utilize any suitable type of image-capture device (still, video, digital, analog, etc.) and utilize any suitable type of imaging (visible light, night-vision, infra-red, microwave, etc.). Any number of gaze detectors 206 may be utilized and may be located anywhere within or on a vehicle 100 to observe a gaze of any occupant, such as a driver and/or a passenger. Other embodiments (e.g., autonomous vehicles) may not utilize a gaze tracker. Output from the gaze detector 206 may be utilized to direct movement of a tracking camera 104. Some embodiments may track the gaze of a driver 204 based upon a gaze directed towards a windshield 210 (for example), such as based on the movement of one or both eyes of the driver 204 or another vehicle occupant. In other embodiments, gaze detection may be performed with respect to the gaze of the driver 204 or another vehicle occupant directed towards one or more displays 212, 214. In still other embodiments, other types of gaze detection that may be utilized with respect to the attention of the driver 204 or other vehicle occupants include by way of non-limiting examples, head movement, torso movement, limb movement, and/or verbal indications, any and/or all of which may also be used in combination with the tracking of eye movements. While a wide angle camera is stationary in some embodiments, other embodiments may feature a moving wide angle camera. For example, output from the gaze detector 206 may be utilized to direct movement of a wide angle camera. In various embodiments, gazes from different vehicle occupants may each independently direct movement of a camera.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019043327A1&KC=A1&FT=D&ND=3&date=20190207&DB=&locale=en_EP#

Patents which describe Driver gaze in an/other vehicle being detected.

Toyota 2019-02-07
METHODS AND SYSTEMS PROVIDING AN INTELLIGENT CAMERA SYSTEM
27
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019043327A1&KC=A1&FT=D&ND=3&date=20190207&DB=&locale=en_EP#

Honda Patent 2019-07-04
VEHICLE CONTROL DEVICE, VEHICLE CONTROL METHOD, AND PROGRAM
31/37/52/53/54/55/56
https://worldwide.espacenet.com/publicationDetails/description?CC=WO&NR=2019130483A1&KC=A1&FT=D&ND=3&date=20190704&DB=&locale=en_EP#

GM GLOBAL TECH OPERATIONS LLC 2019-10-17
METHODS AND SYSTEMS FOR PROCESSING DRIVER ATTENTION DATA
029
https://worldwide.espacenet.com/publicationDetails/biblio?II=0&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191017&CC=US&NR=2019318180A1&KC=A1#

TOYOTA RES INSTITUTE INC 2019-07-25
VEHICLE SYSTEMS AND METHODS FOR DETERMINING A TARGET BASED ON A VIRTUAL EYE POSITION
0028] The user detection system
https://worldwide.espacenet.com/publicationDetails/biblio?CC=US&NR=201922824

Chris,
Not too early imo especially as Honda and GM/ Cruise have some AV partnership arrangements if recall correctly .

RIVIAN IP HOLDINGS LLC 2019-12-12
SYSTEMS AND METHODS FOR OPERATING AN AUTONOMOUS VEHICLE IN A GUARDIAN MODE
Systems and methods are provided herein for operating an autonomous vehicle. A destination is received from a user. A mode selection input is also received from a user. A selected route is determined based on the destination input. In response to determining that the mode selection input identifies a regular driving mode, the autonomous vehicle is operated to travel along the selected route in the regular driving mode. In the regular driving mode at least one capability of a first plurality of capabilities of the autonomous vehicle is enabled. Further, in response to determining that the selection of the mode identifies a guardian mode, the autonomous vehicle is operated to travel along the selected route in the guardian mode. In the guardian mode at least one capability of a first plurality of capabilities of the autonomous vehicle is disabled.
https://patents.justia.com/patent/20190375409
https://worldwide.espacenet.com/publicationDetails/biblio?II=3&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191212&CC=US&NR=2019375409A1&KC=A1#

JAGUAR LAND ROVER LTD 2019-12-12
DISPLAY CONTROL APPARATUS AND METHOD
The present disclosure relates to a display control apparatus (1) for dynamically controlling the display of information in a vehicle (3). The display control apparatus (1) comprises a controller (12) having at least one electronic processor (13). The at least one electronic processor (13) is configured to control first and second displays (4, 5). In dependence on a determined gaze direction of the vehicle driver, the at least one electronic processor (13) controls a switching module (16) to cause a first information data set (INF1) displayed on said first display (4) to be displayed on said second display (5). The present disclosure also relates to a method of controlling the display of information in a vehicle (3).

The first image sensor 17-1 is located in a dashboard 21 below a centre-point of the front windshield WE, and the second image sensor 17-2 is located near a bottom corner of the front windshield WF on the driver side of the vehicle 3. The first and second image sensors 17-1, 17-2 are oriented inwardly into the occupant compartment 2. The first and second image sensors 17-1, 17-2 have respective first and second field of view matrices. In operation, the first and second image sensors 17-1, 17-2 each generate image data which is output to the image processing unit 19. The first and second image sensors 17-1, 17-2 are configured to detect the driver and determine a point of origin in three-dimensional coordinate space based on the position of the driver's head. The orientation of the driver's head can also be determined more accurately to model the gaze direction and/or to determine an approximate eye position for use by an eye-tracking mechanism. The position and orientation of the driver's head can be referred to collectively as the "head pose". The eye-tracking mechanism tracks the driver's eyes so as to determine an angular direction of the driver's target line of sight with respect to a fixed reference (known as "gaze direction"). In the present embodiment, the tracking data DAT1 comprises head pose data and gaze direction data. The tracking data DAT1 is output from the eye tracking apparatus 11 to the display control apparatus 1. The head pose data and the gaze direction data can be combined to form a gaze vector representing a line of sight of the driver.
https://worldwide.espacenet.com/publicationDetails/biblio?II=0&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191212&CC=US&NR=2019377534A1&KC=A1#

Ford Global 2019-12-12
Focus-Based Tagging Of Sensor Data
Data from sensors of a vehicle is captured along with data tracking a driver's gaze. The route traveled by the vehicle may also be captured. The driver's gaze is evaluated with respect to the sensor data to determine a feature the driver was focused on. A focus record is created for the feature. Focus records for many drivers may be aggregated to determine a frequency of observation of the feature. A machine learning model may be trained using the focus records to identify a region of interest for a given scenario in order to more quickly identify relevant hazards.

The controller 102 may further be coupled to one or more interior cameras 116 having an interior of the vehicle 100 in the fields of view thereof. In particular, the one or more interior cameras 116 may have a face of a driver 118 in the fields of view thereof (see FIG. 1B). In particular, multiple cameras 116 may enable determination of the three dimensional location of the passenger's eyes, particularly the pupil thereof in order to determine a gaze direction of the passenger 118. Note that various approaches are known in the art for determining the gaze of an individual. Accordingly, the cameras 116, their positioning, and logic used to determine the gaze direction of the individual may be according to any of these prior approaches. For example, modern computer vision techniques such as neural networks and optical tracking may be used to estimate the driver's attention point and identify objects viewed by the driver in a three-dimensional scene.

As described below, the gaze direction as detected using the interior cameras 116 and features detected using the exterior sensors 104 may be used to create focus records 120 that may be stored in a database 122. For example, the controller 102 may be in data communication with a server system 124. The controller 102 may communicate with the server system 124 by way of a cellular data antenna 126 coupled to a network 128 to which the server system 124 is also connected. The data in a focus record 120 may be assembled in the focus record 120 by the controller 102 or by the server system 124. For example, the controller 102 may transmit gaze tracking data and output of sensors 104 to the server system 124, which will then assemble focus records 120 as described herein below (see discussion of FIGS. 3 and 4).
https://patents.justia.com/patent/20190374151
https://worldwide.espacenet.com/publicationDetails/biblio?II=1&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191212&CC=US&NR=2019374151A1&KC=A1

Honda 2019-12-12
METHODS AND SYSTEMS FOR GENERATING ADAPTIVE INSTRUCTIONS
Systems and methods for generating adaptive instructions for a driver. The system includes a memory that stores instructions for generating adaptive instructions for a driver. The system also includes a processor configured to execute the instructions. The instructions cause the processor to: monitor one or more factors including a gaze of the driver, a physiological signal of the driver, information related to the operation of a vehicle, and information related to an external environment of the vehicle; analyze the one or more factors to determine when and how to provide the adaptive instructions to the driver; generate the adaptive instructions based on the analysis; and provide the adaptive instructions to the driver via a vehicle user interface.

DETAILED DESCRIPTION.
Additionally, in some aspects, the driver system 132 may monitor the driver's reactions, gaze, and physiological signals based on the data captured by the interior camera 125 and/or sensor 124, information related to the operation of the vehicle 102 detected by the CAN reader 123, and external scene data collected by the one or more exterior cameras 121. Using this information, the driver system 132 may determine when to provide instructions to the driver and the best mode for providing such instructions based on the trained model. After the driver system 132 determines when and how to provide the instructions, the driver system 132 may also generate the instructions, such that the instructions may be provided to the driver via the vehicle user interface. For example, the adaptive instructions may be provided to the driver using one or more of the vehicle user interface systems, such as the vehicle audio systems, vehicle video systems, vehicle infotainment systems (such as a heads-up display (HUD) or center display), vehicle telephone systems, and the like.
https://patents.justia.com/patent/20190375426
https://worldwide.espacenet.com/publicationDetails/biblio?II=1&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191212&CC=US&NR=2019377336A1&KC=A1#

Hyundai 12-12-2019
AROUND VIEW MONITORING SYSTEM AND OPERATING METHOD THEREOF 2019-12-12
A method of operating an Around View Monitoring (AVM) system according to the present invention includes: obtaining image data from image sensors disposed at different locations of a vehicle; performing image processing on the image data for improving an image; performing lens distortion correction on the processed image data; detecting an overlapping region from the corrected image data; selecting an image display method so as to prevent the detected overlapping region from overlapping when the image is synthesized, and synthesizing the image data according to the selected image display method; and outputting the synthesized image data.
https://patents.justia.com/patent/20190375330

GM 2019-12-12
METHOD AND SYSTEM FOR DISTRIBUTED LEDGER TECHNOLOGY COMMUNICATIONS FOR VEHICLES
In various embodiments, methods and systems are provided for vehicles for utilizing distributed ledger technology communications for vehicles. In certain embodiments, one or more sensors are disposed onboard the vehicle and are configured to provide sensor data pertaining to operation of the vehicle. A transceiver is disposed onboard the vehicle, and is configured to receive, using distributed ledger technology (DLT), peer network data from a peer network having, as actors, the vehicle as well as a plurality of other actors that are disposed remote from the vehicle, and that together form the peer network. The processor is disposed onboard the vehicle, and is configured to provide instructions for taking a vehicle action for the vehicle using the peer network data and the sensor data.
https://patents.justia.com/patent/20190377336
https://worldwide.espacenet.com/publicationDetails/biblio?II=1&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191212&CC=US&NR=2019377336A1&KC=A1#

Toyota 2019-12-12
SYSTEMS AND METHODS FOR LOCALIZING A VEHICLE USING AN ACCURACY SPECIFICATION
https://patents.justia.com/patent/20190376797
DETAILED DESCRIPTION
The embodiments disclosed herein include systems and methods for localizing a vehicle in an environment using accuracy specifications. Localizing a vehicle in an environment is an important feature for many autonomous driving systems. In general, global positioning systems (GPS) are implemented to localize a vehicle. However, GPS can be error prone, for example in city environments, or lack the granularity required to provide precise vehicle control. To enhance localization of a vehicle, many autonomous driving systems, which may include systems from collision avoidance and driver assistance systems to a full autonomously controlled vehicle, utilize numerous sensors and numerous types of sensors positioned to sense features within an environment around the vehicle. The sensed features may be analyzed, categorized, and/or compared to known features from map data to provide a high level of accuracy and precision in localizing a vehicle in an environment. To achieve these high levels of accuracy and precision, complex and computationally intense systems may be needed. However, autonomous vehicles do not always require the same high level of accuracy and precision for localizing the vehicle in all environments. For example, when an autonomous vehicle is traveling along a highway there may be less of a need to precisely localize the vehicle's longitudinal progress, but there may be a need to accurately and precisely localize the vehicle's lateral activity, so that the vehicle does not depart from its lane of travel. As such, the systems and methods described and shown herein provide systems and methods that can better utilize and reduce consumption of the computational resources for localizing a vehicle by implementing accuracy specifications for vehicles that may be traversing specific types of road, performing specific driving actions, or the like. By better utilizing and reducing the consumption of computational resources for localizing a vehicle as defined in the accuracy specifications, system resources may be utilized for other tasks or even eliminated thereby reducing the cost and complexity of an autonomous driving system.
https://worldwide.espacenet.com/publicationDetails/biblio?II=2&ND=3&adjacent=true&locale=en_EP&FT=D&date=20191212&CC=US&NR=2019376797A1&KC=A1#