In this keynote held at the 2024 International Conference on Computational Photography, Prof. Davide Scaramuzza from the University of Zurich presents a visionary keynote on event cameras, which are bio-inspired vision sensors that outperform conventional cameras with ultra-low latency, high dynamic range, and minimal power consumption. He dives into the motivation behind event-based cameras, explains how these sensors work, and explores their mathematical modeling and processing frameworks. He highlights cutting-edge applications across computer vision, robotics, autonomous vehicles, virtual reality, and mobile devices while also addressing the open challenges and future directions shaping this exciting field.
00:00 – Why event cameras matter to robotics and computer vision
07:24 – Bandwidth-latency tradeoff
08:24 – Working principle of the event camera
10:50 – Who sells event cameras
12:27 – Relation between event cameras and the biological eye
13:19 – Mathematical model of the event camera
15:35 – Image reconstruction from events
18:32 – A simple optical-flow algorithm
20:20 – How to process events in general
21:28 – 1st order approximation of the event generation model
23:56 – Application 1: Event-based feature tracking
25:03 – Application 2: Ultimate SLAM
26:30 – Application 3: Autonomous navigation in low light
27:38 – Application 4: Keeping drones fly when a rotor fails
31:06 – Contrast maximization for event cameras
34:14 – Application 1: Video stabilization
35:16 – Application 2: Motion segmentation
36:32 – Application 3: Dodging dynamic objects
38:57 – Application 4: Catching dynamic objects
39:41 – Application 5: High-speed inspection at Boeing and Strata
41:33 – Combining events and RGB cameras and how to apply deep learning
45:18 – Application 1: Slow-motion video
48:34 – Application 2: Video deblurring
49:45 – Application 3: Advanced Driving Assistant Systems
56:34 – History and future of event cameras
58:42 – Reading material and Q&A
Other resources:
Our research page on event-based vision: http://rpg.ifi.uzh.ch/research_dvs.html
All worldwide resources on event cameras (publications, software, drivers, datasets, simulators, where to buy, etc.):
https://github.com/uzh-rpg/event-based_vision_resources
For a course on event-based robot vision:
1. Slides: https://sites.google.com/view/guillermogallego/teaching/event-based-robot-vision?authuser=0
2. Video recordings: https://www.youtube.com/watch?v=cXSdPdhH4RI&list=PL03Gm3nZjVgUFYUh3v5x8jVonjrGfcal8
For a tutorial on noise modeling of event cameras: https://www.youtube.com/watch?v=YY31GaiOkNM
Our key event-camera datasets:
1. https://dsec.ifi.uzh.ch/
2. https://rpg.ifi.uzh.ch/davis_data.html
3. https://github.com/uzh-rpg/event-based_vision_resources#datasets
Our key event camera simulator: http://rpg.ifi.uzh.ch/esim
Survey paper on event cameras:
https://rpg.ifi.uzh.ch/docs/EventVisionSurvey.pdf
Affiliation:
Davide Scaramuzza is with the Robotics and Perception Group, Dept. of Informatics, University of Zurich, and Dept. of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland https://rpg.ifi.uzh.ch/
