Member Info for johnchucka

METHOD 4 ASSIST OPERATION/traffic/Participants
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019287397A1&KC=A1&FT=D&ND=3&date=20190919&DB=&locale=en_EP#

[0065] For the drivers of other cars as other traffic participants 11, an estimation of perceived objects can be obtained using a combination of on-board eye-/gaze-tracking and methods for scene and object recognition as described for the first module. This however would require special on-board technology which may not be available for all vehicles. In addition, indicators of perception are therefore also taken from internally measurable and/or externally observable vehicle behavior such as characteristic speed changes, avoidance behavior or a lack of response towards the ego-vehicle's actions. The required sensors may thereby be installed in any sufficiently nearby vehicle (including the ego-vehicle) or infrastructure elements.

[0071] Various indicators for the perception of a traffic participant exist in case that the other traffic participant 11 is either a person (for example a pedestrian) or is operated by a person (vehicle driven by a person, for example car, bicycle, or any other mobile entity). One candidate feature comes from ocular behavior such as eye movements and fixations, which can provide information about the current focus of visual attention and/or head orientations, which can serve as a coarse estimation of the same. In both cases, an estimate of the current field of view can provide information about what has been visible and in the area of focus in that instance and consequently potentially be perceived. The indicators are derived from the sensor signals, for example, a camera 2.3 capturing images of a person.

[0073] Measurements of the states of possible indicators for the perception may be obtained through various means already available: like sensors 2 already mounted in the ego-vehicle 1 for other purposes. Head- and eye-tracking devices employed within a vehicle 1 could be used to infer the regions or even points of focus. Additionally, steering wheel and pedal behavior could be monitored and mapped to environmental information for measuring domain-specific behavioral indicators such as making room for another vehicle.

[0078] In the case of machine/artificial operators, the functionality for an automatic estimate about whether or not an entity has been recognized corresponds partially to the functionality required for identifying traffic participants to whom information about recognition of their presence should be communicated. This could for instance be achieved using, radar-, lidar- or camera-based systems in combination with algorithms for object identification and classification. Alternatively, measurements may also be obtained using sensors employed in infrastructure elements 10 in which all recognized vehicles 1, 11, 12 that are connected to that infrastructure element 10 are informed about the presence of recognized ve

WAYMO LLC 2019-08-02
Systems and methods for transitioning control of autonomous vehicle to driver.
In one example, the computing device may be configured to perform a check on the driver before transferring control of the vehicle from the computing device to the driver. For example, a computing device may use an image detection device to determine whether the driver is in the correct position to assume control of the vehicle. In addition, the computing device may determine whether the driver is in a state capable of controlling the vehicle. For example, the computing device may use sensors or eye detection features to determine that the driver is alert. Similarly, computing devices may utilize sensors to ensure that the driver has both hands on the steering wheel before transferring control of the vehicle to the driver. The computing device may additionally require that other security factors be met before transferring control from autonomous mode to manual mode.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=CN&NR=110077407A&KC=A&FT=D&ND=3&date=20190802&DB=&locale=en_EP#

https://worldwide.espacenet.com/?locale=en_ep
The vehicle 102 may include a front camera 120, a roof camera 121, and a rear camera 122 (collectively referred to as cameras 120-122). Each of the cameras 120-122 may be a digital camera capable of capturing one or more images or image streams, or may be another image capturing device, such as a scanner. In some aspects, the front camera 120 may be a dashboard camera configured to capture an image of an environment directly in front of the vehicle 102, the roof camera 121 may be a camera configured to broader view of the environment in front of the vehicle 102, and the rear camera 122 may be configured to capture an image of an environment directly in behind of the vehicle 102. The cameras 120-122 may also provide the image to a driving system 132 and/or a remote system. Additionally, the vehicle 102 may collect data from one or more wearable device (not shown), such as an eye tracking device, worn by a driver of the vehicle 102. In some aspects, the wearable device may provide both the driver's perspective view and the direction in which the driver is looking within the frame of reference. In turn, the vehicle 102 may provide the data collected by the wearable device to the driving system 132 and/or the remote system.

Additionally, the manufacturer system 230 may display the received image data on a display unit, as described in further detail with respect to FIG. 3, and receive annotations from a plurality of users via an input device. In some aspects, the annotations may be markings on the image data identifying one or more objects in the image data. For example, as illustrated in FIGS. 5A-B, the annotations may be a dot or a box surrounding a vehicle in an image. In some aspects, the annotations may be directed to potential objects of fixation in the image data. Alternatively, or in addition, the manufacturer system 230 may transmit the received image data to the remote computing device, such that the plurality of users may annotate the image data at a different location than the manufacturer system 230.

In some aspects, the annotations from each of the plurality of users may be received independent of one another, and each user may have access to a synchronized view of a driver perspective and a fixed car perspective with instantaneous gaze overlaid on the driver perspective.

In some aspect, the annotations may include a bounding box around an object of fixation for the entire duration of the fixation. In this way, the annotations may be used to mark the beginning and end of a fixation and to label the object of fixation within the marked fixation window.

Further Info and SuperCruise mention

In various embodiments, the computer vision system 74 synthesizes and processes sensor data and predicts the presence, location, classification, and/or path of objects and features of the environment of the vehicle 10. In various embodiments, the computer vision system 74 can incorporate information from multiple sensors, including but not limited to cameras, lidars, radars, and/or any number of other types of sensors. The positioning system 76 processes sensor data along with other data to determine a position (e.g., a local position relative to a map, an exact position relative to lane of a road, vehicle heading, velocity, etc.) of the vehicle 10 relative to the environment. The guidance system 78 processes sensor data along with other data to determine a path for the vehicle 10 to follow. The vehicle control system 80 generates control signals 72 for controlling the vehicle 10 according to the determined path.

Although FIG. 4 illustrates five non-limiting examples of sensorimotor primitive modules: SuperCruise, collision imminent brake/collision imminent steering (CIB/CIS), Lane Change, Construction Zone Handling, and Intersection Handling it should be noted that this depiction is for illustration purposes only. SuperCruise is a product of GM Cadillac CT6 that describes level-2 autonomous driving within single lane of a freeway. CIB/CIS are examples of reactive collision avoidance maneuvers or primitives. While only five examples of sensorimotor primitive modules are illustrated in FIG. 4, it should be appreciated that the memory 140 can include any number of sensorimotor primitive modules. For instance, a few other non-limiting examples of sensorimotor primitive modules can include, but are not limited to, collision mitigation brake (CMB), adaptive cruise control (ACC), lane following, intersection right turn, intersection left turn, Michigan left turn, “U” turn, highway merging, highway exiting, yielding, parking, roundabout handling, shopping mall parking lot handling, exit shopping plaza, entering shopping plaza, etc. CMB is a primitive module that is applied when a collision is unavoidable to automatically send a harsh brake command to the brake actuator to reduce impact energy. ACC is a convenience feature that provides longitudinal vehicle control such that the host vehicle will maintain a constant headway from the preceding vehicle.

Fig4 SuperCruise
https://worldwide.espacenet.com/publicationDetails/mosaics?CC=US&NR=2019361454A1&KC=A1&FT=D&ND=3&date=20191128&DB=&locale=en_EP#
https://worldwide.espacenet.com/?locale=en_EP

Terry,
I'll have a look. I'll have to work out how to spell his Sir name first.

2
The system can the select whatever image was sampled/obtained closest to the point in time during which the lidar data was acquired such that only images that were captured near a certain target time (i.e., when the lidar device is looking at the same region that a camera is pointing) will be processed. As a result, camera-lidar pairs with excellent alignment can be determined. This gives lidar data at a certain heading/orientation along with corresponding image data for the scene/environment at that heading/orientation.

Lidar data of the lidar point clouds acquired by the lidar devices can be fused into a single lidar point cloud. Three-dimensional point sampling can then be performed to pre-process the lidar data (of the single lidar point cloud) to generate a set of three-dimensional points, which can then be segmented by an object segmentation module into objects that can be classified and tracked. For instance, an object classification module can include multiple classifiers that classify the objects to generate object classification data. An object tracking module can track the objects. Tracking information can then be used along with the radar tracking information and the object classification data to generate object tracking information (e.g., temporal tracking information for objects such as location, geometry, speed, etc. of objects in the environment).
https://patents.justia.com/patent/20190361454

GM 2019-11-28
CONTROL SYSTEMS, CONTROL METHODS AND CONTROLLERS FOR AN AUTONOMOUS VEHICLE
Detailed Description
The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the exterior environment and/or the interior environment of the autonomous vehicle 10. The sensing devices 40a-40n can include, but are not limited to, radars, lidars, optical cameras, thermal cameras, imager sensors, ultrasonic sensors, inertial measurement units, global positioning systems, navigation systems, and/or other sensors.

Cameras (or image sensors) can be spaced to provide three-hundred and sixty (360) degree image coverage of the environment surrounding the vehicle 10. The cameras capture images (e.g., image frames) and output image data (e.g., a distorted, YUV format image), which can then be processed to generate rectified (or undistorted) camera images. An image preprocessing module can process the image data by undistorting/rectifying it, preprocessing the rectified image data (e.g., image resizing and mean subtraction), and converting the rectified, pre-processed image data into rectified camera images (e.g., having a normal RGB format) that a neural network of an image classification module can classify. The image data can be rectified to correct distortions in the image can cause lines that are straight (in reality) to appear curved, for example, if point clouds in 3D space were projected onto the unrectified image data, they might actually be in the wrong place in the image because of the distortions. By rectifying the image, the projections from 3D space correspond to the correct parts of the image. The rectified camera images can then be sent to an image classification module along with other inputs including three-dimensional locations of objects from an object tracking module, and processed to generate the image classification data that can be provided to an object classification module and used to generate object classification data, which can then be sent to an object tracking module that processes the objects, the radar tracking information, and object classification data to generate object tracking information.

Cameras can be run at their maximum frame rate, and the refresh rate of the cameras is usually much higher than the lidar devices. As lidar spins clockwise from the back of the vehicle, each camera captures images in a clockwise order during the lidar device's rotation. An extrinsic calibration procedure can provide information regarding where the cameras are pointing. The lidar devices are phase locked (i.e., scheduled to be in certain rotational positions at certain times) so it is known when the lidar devices scan certain parts of their cycle. For analysis of a scene, the system can determine which imager/camera is aligned at a point in time when certain lidar data was acquired.

24.20
https://www.youtube.com/watch?v=o0F9Uktpgtk

Thanks TLS for all the feedback. It's great to have your insights. It will be interesting to see how future Patents feed in to this evolving picture.

S2020. Whoever's Tech is in the Ford Mach E they will need to be good in the V2V space as the fella @ 24.20 excitedly announces 'it's unlike anything you've ever seen' . Lets hope the AVCC baseline V2V tech knows about Fords V2V system otherwise some form of interpreter will be needed.

S2020,
I'm throwing the idea out there (to people who have deeper understanding/ knowledge on these issues) to be supported or trashed is that the system SEE/ Conti / AVCC partners maybe creating, improves on Mobileye's offering because it offers/leverages multi inward camera /multi outward alignment and calibration to support V2V safety/data architecture as well as our own Guardian system. The comments regarding Mobileye's power usage Colin Semicast made at the CMD and that Toyota 'resolutely refuses to adopt Mobileye's technology ' support such a move imo.

TLS,
My hunch is Continental are providing the hardware cameras and SEE are designing the multi camera algorithms to calibrate and align the system and the AVCC partners are working on collating that Data into a V2V system.

Seeing machines.
FORWARD FACING CAMERA IN A VEHICLE 2019-08-29
System Overview
[0031] The embodiments of the present disclosure described herein relate to determining a camera pose of a forward facing camera in a multi-camera vehicle monitoring system. In these embodiments, the scene to be imaged includes a driver of a vehicle, the interior of the vehicle/****pit, the forward road scene of the vehicle and optionally side and rear views from the vehicle. The vehicle may represent a commercial automobile, truck, earthmoving machine, airplane, jet or helicopter. However, it will be appreciated that the disclosure is applicable to other multi-camera monitoring systems.

[0069] At stage 504, the statistical data are analyzed to identify one or more peaks in the frequency distribution. Identified peaks are associated with corresponding reference points in the images of the forward road scene from forward facing camera 101. This association allows the determination of reference gaze positions in the vehicle reference frame. For mapping the gaze onto images of the forward facing camera 101, the reference points must lie within the field of view of camera 101 to be useful.
(070)

AN IMAGING DEVICE FOR MOTOR VEHICLE Conti 2019-07-11
In theory, to prevent optical aberration, the optical assembly 102 and the image sensor 110 are aligned as accurately as possible, to ensure quality of images captured. However, during manufacturing process, a perfect alignment is not possible due to tolerance in the optical assembly 102 and image sensor 110. In order to maximise the performance and accuracy of the imaging device, an alignment process is required to calibrate the optical assembly 102, i.e. the plurality of lenses 104, 106, 108, over the imaging sensor 110. Such optical alignment process may be passive or active. In a preferred embodiment, an active alignment process is applied to align the plurality of lenses 104, 106 and 108 to the image sensor 110 during mounting process. An active alignment process may be done by a machine or by an operator, to adjust the optical assembly 102 and the image sensor 110, to maximise imaging device performance, i.e. sharpest image possible. This alignment is done by moving the optical assembly 102 in each of the Cartesian coordinate axis, i.e. x, y and z-axis and also rotating about each of the aforesaid axis until a sharpest image is achieved, to determine a position of the optical assembly 102 relative to the image sensor 110.

Looks very familiar.
https://www.youtube.com/watch?v=DwQbta914no

Ford Mach-E Interior Shot On Spectacles
https://www.youtube.com/watch?v=20b2sagKVeA

Lewbo,
Why are they bothering? sounds like a coffee cup warning and the competition is so far ahead .

TLS,
It maybe just coincidence that a SEE, Conti and Toyota patent (mentioned in my 08.30 post) sounds to me like it's working on a more sophisticated multi camera system but what adds interest to this theory is the article below discussing the Toyota and Mobileye relationship and for added intrigue Renesas a AVCC partner get a mention. Also what does this say if anything about the Mobileye relationship with the AVCC

Toyota teams with Israeli co Cortica in race for autonomous car.
Cortica's AI technology combined with Renesas's chip could help Toyota break Mobileye's stranglehold on the advanced driver-assistance systems market.

'History shows that market leaders cannot forever ignore their competitors' power (ask Microsoft about Google). In this case, the elephant in the room is Toyota, the world's largest auto manufacturer, which sells over 11 million cars a year. Toyota's budget and technology resources are inexhaustible, and it resolutely refuses to adopt Mobileye's technology. Toyota has always claimed that it could achieve better results independently than those of Mobileye, and therefore has no need to tie itself to the Israeli company's closed system. Since Toyota has close ties with a number of Japanese tier-1 suppliers, it can mobilize the awesome development resources of the entire Japanese industry for its needs.'
https://en.globes.co.il/en/article-toyota-teams-with-israeli-co-cortica-in-race-for-autonomous-car-1001273221
https://www.renesas.com/eu/en/about/press-center/news/2019/news20191030.html

Maplinman,
My impression is that the news feed form all those Connect Source types reporting trucker companies signing up for Guardian has gone dark since were too good at adding up. Someone really doesn't want us to know how many Guardian units are being fitted imo. They also want us to believe the R&D is being dialled back and in my more surreal moments I imagine Alastair Protheroe was assigned to me at the CMD to stop me asking Patent questions of others while Sebs ate my share of the buffet, but that's no doubt a paranoid step too far.

Klick,
Maybe Mazda?? Maybe Harish , Maybe not . similar description.

CONTINENTAL AUTOMOTIVE GMBH [DE]; CONTINENTAL AUTOMOTIVE SINGAPORE PTE LTD
DISPLAY SYSTEM IN A VEHICLE 2019-04-11.
Description.
'The disclosed system comprises one or more sensors configured to determine a user's line of sight. The sensor assists to track where the user is looking at or focussing on. The sensor may further assist to distinguish facial gestures or expressions, such as squinting of eyes, frowning, raised eyebrow(s), or other gestures as disclosed herein. Sensors suitable for the present purpose include those that track movement of the user' s eyes or head or body, such as interior camera, head movement sensor or body movement sensor'.

L7 Why bother performing?
If as some say PM is generously gifted in the height and Ego department then for him it's all about his personal reputation and performance which hopefully will be good for shareholders too.

HONDA MOTOR CO LTD 2019-11-28
SYSTEMS AND METHODS FOR ESTIMATING AN OBJECT OF FIXATION
Systems and methods for estimating an object of fixation from a gaze of a driver. The method includes: receiving image data from a plurality of input devices; processing the image data from a first one of the plurality of input devices to identify an object track; analyzing the image data from a second one of the plurality of input devices and the image data from the first one of the plurality of input devices to determine a projected gaze of a driver; analyzing the object track and the projected gaze to identify a plurality of objects in the gaze of the driver; performing a probability analysis to estimate the object of fixation from among the plurality of objects; and generating an output image identifying the estimated object of fixation.
https://worldwide.espacenet.com/?locale=en_EP

TLS,
I'm trying to get a feel for the SEE / Mobileye overlap here .I'd be interested on your thinking around the development of what looks to me like a Inward/ multi-360 outward camera system that on one level looks to be mirroring what Mobileye are offering but with the great advantage of added inward driver fusion. My initial thoughts are that this could replace Mobileye's system because it offers (within tight power constraints , as mentioned by Colin Semicast @CMD)to the data cloud, additional Driver data and object detection and sophisticated Driver to driver recognition like 'another vehicle occupant gaze direction recognition' etc described @31 in the Honda patent @VEHICLE CONTROL DEVICE, VEHICLE CONTROL METHOD, AND PROGRAM https://worldwide.espacenet.com/publicationDetails/description?CC=WO&NR=2019130483A1&KC=A1&FT=D&ND=3&date=20190704&DB=&locale=en_EP#

Mobileye CEO Explains Camera-Centric Approach to Self-Driving Cars
https://www.electronicdesign.com/automotive/mobileye-ceo-explains-camera-centric-approach-self-driving-cars.

SEE 2019-08-29
SYSTEM AND METHOD FOR IDENTIFYING A CAMERA POSE OF A FORWARD FACING CAMERA IN A VEHICLE
https://worldwide.espacenet.com/publicationDetails/biblio?II=1&ND=3&adjacent=true&locale=en_EP&FT=D&date=20190829&CC=US&NR=2019266751A1&KC=A1#

CONTINENTAL AUTOMOTIVE GMBH
Method and Apparatus for Estimating a Range of a Moving Object
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019362512A1&KC=A1&FT=D&ND=3&date=20191128&DB=&locale=en_EP#

Toyota Patent 2019-11-21
SYSTEMS AND METHODS OF PROCESSING AN IMAGE
https://worldwide.espacenet.com/publicationDetails/originalDocument?CC=US&NR=2019354780A1&KC=A1&FT=D&ND=3&date=20191121&DB=&locale=en_EP#