Roundtable Discussion; The Future of Mineral Sands. Watch the video here.
London South East prides itself on its community spirit, and in order to keep the chat section problem free, we ask all members to follow these simple rules. In these rules, we refer to ourselves as "we", "us", "our". The user of the website is referred to as "you" and "your".
By posting on our share chat boards you are agreeing to the following:
The IP address of all posts is recorded to aid in enforcing these conditions. As a user you agree to any information you have entered being stored in a database. You agree that we have the right to remove, edit, move or close any topic or board at any time should we see fit. You agree that we have the right to remove any post without notice. You agree that we have the right to suspend your account without notice.
Please note some users may not behave properly and may post content that is misleading, untrue or offensive.
It is not possible for us to fully monitor all content all of the time but where we have actually received notice of any content that is potentially misleading, untrue, offensive, unlawful, infringes third party rights or is potentially in breach of these terms and conditions, then we will review such content, decide whether to remove it from this website and act accordingly.
Premium Members are members that have a premium subscription with London South East. You can subscribe here.
London South East does not endorse such members, and posts should not be construed as advice and represent the opinions of the authors, not those of London South East Ltd, or its affiliates.
TLS, So Lets take Toyota and GM out of my picture. Could we sensibly postulate that SEE maybe developing an expanded Inward and Outward camera after market Guardian product ? , G4 we could call it. The Data from that would be valuable.
Yes JC, In fact they even covered them self with the very modular option of the power source being a battery in the device which would be charged periodically - so effectively a mobile phone type device with 2 cameras.
Going back to "our" forward and rear facing video clips being used for training. on what the driver is looking at. I always thought that we would limit ourselves to internal and just pass the eye gaze vectors up to the car vision systems. But this would put more value and brains in our system.
I presumed that you wouldn't have two supervisory systems viewing the outside world, but now with the delay in L3/4/5 we will be in L2/L2+ for far longer, so now we need to provide the supervisory vision system for the outside world. So I can see why SEE are going there now.
TLS, further to your point regarding Guardian reference point 34 and Fig 3 make reference to it being offered or 'manufactured as an after-market product' and point 35 note that it processor 302 'may include system-on-chip technology'.
This is of interest as although I may be way off target one might optimistically imagine that IF this is all tied in to Toyota and AVCC and 'unprecedented collaboration' etc then what these patents are describing is the development of Guardian For all. I'm no doubt way of the mark but it's my working assumption at the present time.
[0034] Referring now to FIG. 3, there is illustrated a system level diagram of system 100. System 100 includes a central processing unit 300 including a processor 302, memory 304, a power source 306, a network interface 308 and a user input device 310. In the embodiments of a vehicle monitoring system, central processing unit 300 is preferably mounted within the vehicle dash or center console and can be integrated with an onboard vehicle computer system during manufacture. However, central processing unit 300 and system 100 as a whole may be manufactured as an after-market product and subsequently installed into vehicle in a modular manner.
[0035] Processor 302 may represent a conventional microprocessor or personal computer having hardware and/or software configured for processing image streams received from multiple cameras. By way of example, processor 302 may include system-on-chip technology and include a video processing pipeline for processing the stream of images from cameras 101-104. In one embodiment, processor 302 is integral with or in communication with a processor of an onboard vehicle computer system.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019266751A1&KC=A1&FT=D&ND=3&date=20190829&DB=&locale=en_EP#
Thanks TLS . If the Guardian data has more combined driver facing / forward facing applications uses like you suggest then I assume it's value to the company and leverage it offers is pretty significant. However, the patent linkage I was making by referring to SEE patent 'SYSTEM AND METHOD FOR IDENTIFYING A CAMERA POSE OF A FORWARD FACING CAMERA IN A VEHICLE' looks to me to be more sophisticated, involving multiple driver facing cameras fusion/aligned with for example as point 033 describes;
' the forward road scene or other views in or around the vehicle. Other exemplary camera locations of cameras include a rearview mirror, rear bumper, front bumper, vehicle roof and bonnet/hood',
This is why I was making the reference to the Toyota and GM patents that appear to me imo to be on a technically parallel plane to SEE in relation to patent evolution.
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019266751A1&KC=A1&FT=D&ND=3&date=20190829&DB=&locale=en_EP#
ah the tyranny of word count.
Anyway, Is the thing that the eye is looking at the vehicle in front (should be the same vehicle for many frames/minutes. Is the eye looking at a road lane with nice easy to see edges. Is it reading a sign - it needs to track as the sign approaches. Is it drawn to lateral movements as cars change lanes or person/child approaches the road/lane from the side. Is it following an object and still watching through from the side windows - was it a sign, a vehicle or a pretty girl - (sorry just classify that as a person we don't want to get in to trouble - of course it could be like Amazon and prompt that you normally look at X just in case you missed it!)
JC, I have been thinking.
Apologies, my first thoughts on labelling was that there is a whole world outside of the vehicle and that is a lot to label - is it a bike, a kangaroo, a human with a bike or an umbrella or shopping bags. Walking with a child a dog or horse etc. Having a single researcher driving to generate data didn't seem to make sense, but that would only be part of the job. It could be 2 weeks on a test site with volunteers, but some driving would be required. those tests would require labelling.
I then read the See patent and started to get more excited as I realised that we have 3 billion km of driving data - yes most is of drowsy drivers, but here is the key part that I forgot about: many or even most gen 2 Guardian's have forward facing cameras. Sure they only record for the seconds before and after an event, but we have road data. Some is triggered by the driver pressing the button to capture "the idiot in front" others will be of drivers not looking, but given the vast volume of data a reasonable amount of snippets should be usable with a driver looking ahead and the front camera seeing the view in front.
Ok, here comes the best bit. Guardian has the front and driver facing cameras mounted in the same unit with fixed geometry - far better than a car where the cameras could be separated and be at different angles in different vehicles. In Guardian, they are effectively on the same height (y axis) same forward distance (z axis) and on the X axis they are only separated by a few fixed cm (parallel to the axle) SEE also know the relative angles (say ~180 degrees apart), but again it is fixed.
Now we have a nice data set to work with. Now as discussed in the Patent, they still have to find the driver and the eyes and determine the gaze direction (this is SEE after all). The eye positions in the 2-d picture are done, then you need the distance to the eyes well we know the size of an eye, so that is another triangle and you have the distance. Now take the gaze from the eyes and draw that forward and work out where it lands on the 2-d forward facing image.
This bit is a bit trickier, so we use a bit of cunning - roads are "generally" horizontal (they mentioned measuring a slope) but I was a physicist, so I will assume an infinite horizontal surface for now. Lets say that the truck driver's eyes are 6 foot up and the guardian is at roughly the same height (only 1 foot above or below). Now you can intersect the gaze with the perfect horizontal road. Re do the maths for different heights and separations of the guardian and the fixed point stretches from a circle to an ellipse. Maybe 2-5 metres long and 1 metre wide. But on the 2-d camera image that corresponds to a more circular blob. Now assume that there was a truck in the way with a nice vertical back. The maths still works out the with the same circular blob location.
Now just need to label roads or trucks. Add in rapid movement laterally and see if eye follow
If I wanted to make a case that the AVCC partners could be collaborating in developing a complimentary technological baseline system platform I could add this Conti patent (which comes around a week after the Toyota patent) but I'm conscious that I'm being extremely selective and guilty of AVCC confirmation Bias.
CONTINENTAL AUTOMOTIVE GMBH [DE]; CONTINENTAL AUTOMOTIVE SINGAPORE PTE LTD 2019-04-11
The invention relates to a display system in a vehicle, the system comprising: one
or more displays;at least one sensor configured to determine a user's line of sight;a controller configured to determine an active zone and a non-active zone of the one or more displays based on the user's line of sight,wherein the one or more displays are configured to operate the active zone at an enhanced level as compared to the non-active zone.There is further provided a controller configured to rank content displayed on one or more displays in a vehicle according to an attention score determined by at least one sensor configured to measure gaze of a user based on the user's line of sight; and a display system comprising the controller,wherein one of the displays is selected to display the highest ranked content.There is also provided a method of operating one or more displays in a vehicle using the disclosed display systems.
https://worldwide.espacenet.com/publicationDetails/biblio?CC=WO&NR=2019068754A1&KC=A1&FT=D&ND=3&date=20190411&DB=&locale=en_EP#
Possible SEE/Toyota/GM Joint technological/patent development within a two month timeframe.
Example
SEE 2019-08-29
SYSTEM AND METHOD FOR IDENTIFYING A CAMERA POSE OF A FORWARD FACING CAMERA IN A VEHICLE
'A method includes capturing images of a vehicle driver's face from a driver facing camera and images of a forward road scene from a forward facing camera. The images are analyzed to derive gaze direction data in a vehicle frame of reference. The gaze direction data are statistically collated to determine a frequency distribution of gaze angles. One or more peaks in the frequency distribution are identified and associated with corresponding reference points in the images to determine one or more reference gaze positions in the vehicle frame of reference. The one or more reference gaze positions are correlated with a position of the reference points in a forward facing camera frame of reference to to determine a camera pose of a forward facing camera in the vehicle frame of reference'.
GM 2019-10-17
METHODS AND SYSTEMS FOR PROCESSING DRIVER ATTENTION DATA
Methods and systems are provided for processing attention data. In one embodiment, a method includes: receiving, by a processor, object data associated with at least one object of an exterior environment of the vehicle; receiving upcoming behavior data determined from a planned route of the vehicle; receiving gaze data sensed from an occupant of the vehicle; processing, by the processor, the object data, the upcoming behavior data, and the gaze data to determine an attention score associated with an attention of the occupant of the vehicle; and selectively generating, by the processor, signals to at least one of notify the occupant and control the vehicle based on the attention score.
Toyota 2019-10-24
SYSTEM AND METHOD FOR DETECTING HUMAN GAZE AND GESTURE IN UNCONSTRAINED ENVIRONMENTS
'A system for gaze and gesture detection in unconstrained environments includes a 360-degree (omnidirectional) camera system, one or more depth sensors, and associated memory, processors and programming instructions to determine an object of a human user's attention in the unconstrained environment. The illustrative system may identify the object using eye gaze, gesture detection, and/or speech recognition. The system may generate a saliency map and identify areas of interest. A directionality vector may be projected on the saliency map to find intersecting areas of interest. The system may identify the object of attention once the object of attention is located'.
Possible SEE/Toyota Joint technological/patent development Example
SEE 2019-08-29
SYSTEM AND METHOD FOR IDENTIFYING A CAMERA POSE OF A FORWARD FACING CAMERA IN A VEHICLE
'A method includes capturing images of a vehicle driver's face from a driver facing camera and images of a forward road scene from a forward facing camera. The images are analyzed to derive gaze direction data in a vehicle frame of reference. The gaze direction data are statistically collated to determine a frequency distribution of gaze angles. One or more peaks in the frequency distribution are identified and associated with corresponding reference points in the images to determine one or more reference gaze positions in the vehicle frame of reference. The one or more reference gaze positions are correlated with a position of the reference points in a forward facing camera frame of reference to to determine a camera pose of a forward facing camera in the vehicle frame of reference'.
Toyota 2019-10-24
SYSTEM AND METHOD FOR DETECTING HUMAN GAZE AND GESTURE IN UNCONSTRAINED ENVIRONMENTS
'A system for gaze and gesture detection in unconstrained environments includes a 360-degree (omnidirectional) camera system, one or more depth sensors, and associated memory, processors and programming instructions to determine an object of a human user's attention in the unconstrained environment. The illustrative system may identify the object using eye gaze, gesture detection, and/or speech recognition. The system may generate a saliency map and identify areas of interest. A directionality vector may be projected on the saliency map to find intersecting areas of interest. The system may identify the object of attention once the object of attention is located'.
This SEE Patent refers to driver gaze and road scene object / driver facing and forward facing camera calibration (See Description points 70/71/72/73/74/75 for detail) and imo look to share common themes of technical progress of the possible parallel developments that Toyota and GM patents have recently described and perhaps are related to the Data Acquisition Specialist role advertised yesterday.
SYSTEM AND METHOD FOR IDENTIFYING A CAMERA POSE OF A FORWARD FACING CAMERA IN A VEHICLE
https://worldwide.espacenet.com/publicationDetails/description?CC=US&NR=2019266751A1&KC=A1&FT=D&ND=3&date=20190829&DB=&locale=en_EP#
TLS,
This sentence I believe supports my view that it's external objects they will be 'labelling' .
'a current driver's license and confidence to drive on public roads in order to assist in data collection activities'.
Seeingtom, I get the impression that the job is a little more involved than you suggest;)
About the opportunity
The Seeing Machines Data Acquisition team works alongside key business units, marking up video datasets and preparing the data for machine learning algorithm validation and training.
As a Data Acquisition Specialist, you will:
gather data collection requirements through consultation with algorithm scientists and algorithm performance engineers
develop data collection protocols by soliciting and sharing technical feedback from all relevant stakeholders
prepare, execute and supervise data collection, data annotation and dataset curation activities in accordance with the approved data collection protocols and relevant guidelines
monitor progress against milestones and administer review processes to ensure quality benchmarks are met
support the Data Acquisition Supervisor by providing performance feedback on data acquisition assistants, and promote a continuous learning culture within the team
About you
As a Data Acquisition Specialist, you will provide critical support to the R&D team - as they apply machine learning technology to continued development of our class leading algorithms and technology.
We are seeking people with the following attributes:
strong organisational and management skills, with an ability to operate in a rapidly changing environment whilst maintaining high work quality and attention to detail
good communication and interpersonal skills, and the ability to hold effective technical discussions with engineers
ability to work with incomplete technical information and possess good problem-solving, analytical and critical thinking skills
experience in writing technical documents and reports
strong computer literacy, and experience with Microsoft Office
experience in developing scripts (e.g. in Python) to process data
experience in a technical supervisory role, especially in video image data collecting, processing and annotation (labeling) is highly desirable.
a current driver's license and confidence to drive on public roads in order to assist in data collection activities.
My dog's sat in the corner staring at me wondering 'why is that 'unpaid fool' laughing so much'? (he's a clever dog so he understands that I am laughing human )
'unpaid fools like you and me'...pmsl !! I'm on my back here, nearly pist miself
Adrein Gaiden explains what 'labelling' means to Toyota here @5.50m
Interview with Adrien Gaidon, Machine Learning Engineer, Toyota Research
https://www.youtube.com/watch?v=jBIYRoAQJuo
The actual job of labeling is normally done by unpaid fools like you and me. For example, when you have to do the "I am not a robot checks" when purchasing online. "Click all the squares that have traffic lights in them" they say, and then you do their labeling for them. I would expect a professional with labeling in their title to be managing a project not actually physically labeling, picture by picture.
TLC,
Could be but interior objects are not going to take that long I would imagine, it's pretty limited, not exactly a career move, where as Objects in the wild is the sort of thing Simon Stent of TRI could talk for days about.
https://arxiv.org/abs/1910.10088
What does it mean to annotate data?
Data annotation is the process of labeling data to make it usable for machine learning. Data can be almost any kind of data that a human might understand. This includes: ... timeseries data. LIDAR data, RADAR data, or data from other sensors.
JC I would think that sunglasses, mask, hat, are more likely. Perhaps, behaviours too: turning, smoking, eating
Possibly, counting occupancy, babies, handbags etc
street scene objects perhaps ??
Data Acquisition Specialist.
About you
As a Data Acquisition Specialist, you will provide critical support to the R&D team - as they apply machine learning technology to continued development of our class leading algorithms and technology.
'experience in a technical supervisory role, especially in video image data collecting, processing and annotation (labeling) is highly desirable'.
https://seeingmachines.springboard.com.au/jobtools/jncustomsearch.searchResults?in_organid=18900&in_jobDate=All