Transcript
A Sensitive Dynamic and Active Pixel Vision Sensor for Color or Neural Imaging Applications Diederik Paul Moeys, Institute of Neuroinformatics, University of Zürich and ETH Zürich, Switzerland
The work presented in this abstract is a summary of the recently defended Ph.D. investigation titled “Analog and digital implementations of retinal ganglion cells”. This was conducted in the Sensors group of the Inst. of Neuroinformatics of ETH Zürich, led by Prof. Tobi Delbruck and Shih‐ Chii Liu, in the field of Neuromorphic Engineering (NE). NE consists of trying to understand the underlying principles of the brain and reproducing them in hardware, in order to achieve fast and intelligent computations which diverge from standard approaches. In particular, the Ph.D. work focused on Dynamic Vision Sensors (DVS) and their applications. The DVS is a vision sensor, which goes beyond the frame‐based approach of normal digital cameras by imitating the biological eye. Each pixel of the DVS sensor is a single independent processing unit on its own, capable of telling if there has been a positive (ON event) or negative (OFF event) logarithmic temporal change in light intensity. The coordinates of ON and OFF events and their timestamps are thereby generated and communicated off chip. Through the sensor’s inherent preprocessing, only brightness changes from the scene are sent for further computations. This compression allows a lower data‐rate that is dependent on the activity in the visual field, which in turn means a lower power consumption and sub millisecond latencies. The logarithmic compression also allows high dynamic range (>100 dB). The detection of changes by the DVS is particularly useful in the context of tracking, since only moving objects’ edges are visible (Fig. 1). Applications such as bolometry, fluorescence microscopy and fluid dynamics, requiring the detection of small visual contrast, need high sensitivity. DVS sensors provide higher dynamic range and reduce data rate and latency, but most of these sensors have limited sensitivity. During this Ph.D., a 200x192 CMOS vision sensor in 180 nm technology, called SDAVIS, was developed for such applications. The sensor outputs DVS events as well as conventional frames. The SDAVIS improves on previous DVS sensors with higher sensitivity for temporal contrast (down to 1%). The achievement is possible through the adoption of an in‐pixel preamplification stage. This in‐ pixel preamplifier reduces the effective intrascene dynamic range of the sensor (70 dB), but an automated operating region control allows to extend it up to at least 110 dB. The sensor was developed in collaboration with Towerjazz and IMEC research institute (Interuniversity Microelectronics Centre), Leuven, Belgium, under the EU‐funded project SEEBETTER and VISUALISE. The collaboration with IMEC allowed access to state‐of‐the‐art vision sensor technology, and high‐precision characterization instrumentation. The output of SDAVIS is shown in Fig. 1, in comparison to another recent DVS sensor that was not designed for high sensitivity. As can be seen, the 10‐15X higher sensitivity of SDAVIS is noticeable by eye. Shadows do not spoil the image quality and tiny details are well perceived (both sensors have similar resolution; only the sensitivity changes).
A
B
C
Fig. 1 A: Moving hand stimulus; B: Previous DVS sensor raw output with Full Color Scale (FCS) of 1; C: SDAVIS raw output with FCS of 10 events. Both B and C show 30 ms DVS time slices. ON events are represented in white and OFF events in black.
The higher sensitivity of SDAVIS make this sensor potentially useful for neuronal calcium imaging, as shown in Fig. 2. Recent experiments in collaboration with the Brain Research Institute of the University of Zurich show in fact that SDAVIS has the potential of substituting the expensive current CCD or CMOS frame‐based imaging systems which have low frame rates when recording at full resolution. The output of a minute of recording of one of these systems can fill over 40 GB of storage. SDAVIS can reduce redundancy and increase time resolution because it responds only to fluorescence changes which streamline recordings from large neuron populations.
A
B
Fig. 2 A: State‐of‐the‐art digital camera frame compared to (B) the corresponding 100 ms DVS time slice of SDAVIS which only shows real‐time activity continuously. Both cameras image mouse neurons expressing calcium‐sensitive green fluorescent protein.
SDAVIS is also the first DVS sensor to produce color DVS events and frames, as shown in Fig. 3.
A
B
C
Fig. 3 A: ground truth (cellphone camera); B: RGBW events separate by color (RGBW ON events are represented in RGBW respectively and OFF events in black); C: interpolated RGBW frame.
The SDAVIS is being turned into an R&D USB camera product by the Swiss company IniLabs, which helped with the development on the software side. This work was also partially funded by Samsung and by the Swiss National Science Foundation.
