Revolucionando la visión artificial:cómo los sensores de eventos impulsan un procesamiento de imágenes más rápido e inteligente

The surge in content generated by cameras, across consumer and industrial sectors, has placed a burden on machines’ capacity to effectively acquire, process, and utilize visual data in a practical and efficient manner. Los desafíos actuales incluyen:la recopilación de una cantidad abrumadora de datos (muchos de los cuales son irrelevantes para las máquinas); capacidades de procesamiento insuficientes (especialmente en aplicaciones limitadas por el tamaño y la potencia):y la demanda de procesamiento en tiempo real. Consequently, developers of vision-enabled systems — spanning from smartphones, wearables, smart homes, IoT, systems, automotive technologies to industrial automation equipment — are seeking ways to transform the traditional approach to vision sensing and data acquisition.

Having originated in providing images for human consumption, camera technology’s progress over its history — primarily relying on frame-based methods — is proving inadequate in meeting the requirements of modern machine vision. For years, machine vision has been reliant on visual information acquired and structured for human interpretation:video streams composed of sequential images captured by an image sensor. Cada imagen representa una instantánea estática en un momento particular que carece de información dinámica. This method of gathering visual data is prevalent in most machine vision systems designed for monitoring changes and movements within dynamic environments.

Utilizing neuromorphic techniques inspired by the human vision system, event-based vision approach seeks to enhance efficiency and performance in various vision-enabled systems across consumer, industrial, automotive, and other sectors to elevate safety, productivity, and user experience. (Imagen:Profetizar)

The predominant challenge arises when there is movement or change in a scene, which is common in most machine vision applications, and the inherent limitations of visual frame acquisition become apparent. Independientemente de la velocidad de fotogramas establecida, si una cámara intenta capturar una escena en movimiento, siempre será inexacta. Since different parts of a scene typically exhibit varying dynamics simultaneously, employing a single sampling rate to regulate pixel exposure across an imaging array inevitably results in inadequate capture of these diverse scene dynamics occurring concurrently.

Menos es más al detectar eventos

Compounding this challenge, the issues with traditional image sensors are they are slow and energy-intensive while producing excessive redundant data and have limited dynamic range, which makes them ill-suited for machine vision tasks, particularly those in demanding operating environments. Consequently, biologically inspired “neuromorphic” event-based vision systems are now emerging as alternatives that offer enhanced speed, minimal latency, better power efficiency, and broader dynamic range that cater well to various machine vision applications.

La visión basada en eventos marca un cambio de paradigma en cómo se adquiere y procesa la información visual para los usos modernos de la visión artificial. Utilizing neuromorphic techniques inspired by the human vision system, this approach seeks to enhance efficiency and performance in various vision-enabled systems across consumer, industrial, automotive, and other sectors to elevate safety, productivity, and user experience.

La visión basada en eventos funciona de manera diferente a las cámaras tradicionales, ya que se aleja de una tasa de adquisición uniforme para todos los píxeles. En cambio, cada píxel determina de forma independiente su tiempo de muestreo en función de los cambios de luz incidente gracias a la inteligencia dedicada por píxel. La información de detección de contraste se encapsula en "eventos", que comprenden las coordenadas x,y del píxel y el tiempo preciso de generación del evento. With Prophesee’s patented event-based sensors, for example, pixels activate intelligently upon detecting contrast changes (motion), facilitating the continuous capture of essential motion details at the pixel level.

La diferencia al pasar de velocidades de cuadros fijas es cómo cada píxel puede ajustar su frecuencia de muestreo de acuerdo con su entrada visual. Este enfoque personalizado permite que cada píxel determine sus puntos de muestreo reaccionando a las variaciones en los niveles de luz incidente. Consequently, the sampling process is no longer dictated by an artificial timing source but rather by the signal itself or specifically by temporal signal amplitude fluctuations. The outcome produced by such cameras evolves from image sequences into a continual stream of individual pixel data generated conditionally based on scene dynamics.

Event sensors offer several advantages, including high-speed operation (equivalent to 10,000 fps), highly efficient power consumption (down to the microwatt range), low latency for quicker response times, reduced data processing needs (10-10,000x less than frame-based systems), and a high dynamic range of up to 120dB. Estas características hacen que los sensores de eventos sean adecuados para diversas aplicaciones y productos.

Aplicar una visión basada en eventos

Neuromorphic-enabled event sensors can be used for a variety of industrial automation tasks, helping improve productivity, quality, safety, security, and preventative maintenance. (Imagen:Profetizar)

Initially, neuromorphic event sensors found commercial use not in machines but for humans, for vision restoration in visually impaired individuals. Esto condujo a casos de uso en automatización industrial y monitoreo de procesos. These uses demonstrated the benefits of event sensors to numerous vision tasks, especially those involving fast-moving and changing elements, unpredictable ambient lighting conditions, and limited resources. Subsequent generations of event-based systems have been applied in industrial settings for tasks like high-speed counting, preventative maintenance (e.g., vibration monitoring), enhancing robotic efficiency and safety, eye-tracking or gesture tracking for AR/VR as well as various logistics and safety/security applications.

Estas ventajas inherentes hacen que los sensores de eventos sean ideales para aplicaciones de IoT. El consumo de energía juega un papel fundamental en los dispositivos de IoT, particularmente aquellos que dependen de baterías. La visión basada en eventos es muy adecuada para tales escenarios, ya que funciona a niveles de potencia significativamente más bajos en comparación con los sistemas de cámaras basados ​​en marcos. Moreover, event-based cameras excel in challenging lighting conditions common in many IoT applications due to their light-independent information processing. Their high dynamic range allows them to capture a wide range of light intensities within a single frame, making them perfect for environments with varying lighting conditions like outdoor scenes with bright sunlight or nighttime settings.

With a dynamic range exceeding 120dB, event-based cameras can function effectively even in environments where traditional cameras struggle with varying lighting conditions — be it extremely bright settings like public spaces or vehicles during the day or dimly lit scenarios such as nighttime operations or dark factory settings where event sensors can be used for preventative maintenance and safety monitoring tasks. (Imagen:Profetizar)

With a dynamic range exceeding 120dB, event-based cameras can function effectively even in environments where traditional cameras struggle with varying lighting conditions — be it extremely bright settings like public spaces or vehicles during the day or dimly lit scenarios such as nighttime operations or dark factory settings. Además, estas cámaras ofrecen una latencia mínima al transmitir información solo cuando hay un cambio en el brillo dentro de la escena. La respuesta en tiempo real resulta ventajosa en situaciones de iluminación que cambian rápidamente, como cambios abruptos de claro a oscuro o viceversa. Event-based cameras, which detect individual alterations in light intensity, are less prone to motion blur compared to conventional frame-based cameras.

Esta característica es particularmente valiosa en escenarios que involucran movimientos rápidos, asegurando una calidad de imagen nítida. New uses that take advantage of this benefit are being developed for cameras in smart-phones, for example Prophesee’s partnership with Qualcomm to integrate its event-based technology with the popular Snapdragon platform.

Further development in event sensors for IoT involves adapting them for edge vision tasks with limited onboard computing capabilities due to acquiring sparse data. Sin embargo, desafíos como formatos de datos no convencionales, velocidades de datos variables e interfaces no estándar han obstaculizado una adopción más amplia. To address this issue, the latest generation of event sensors, exemplified by Prophesee’s GenX320, aims to enhance integration and usability in embedded edge vision systems by incorporating features like event data pre-processing and formatting, compatible data interfaces, and low-latency connectivity with various processing platforms including energy-efficient neuromorphic processors. For instance, the GenX320 offers multiple pre-processing functions, adaptable interfaces, and power management options to cater to power-sensitive vision applications efficiently.

A pesar de su eficiencia operativa, la optimización de los sensores de eventos para un uso de bajo consumo adecuado para configuraciones de IoT sigue siendo fundamental. Implementing a range of power modes and application-specific operation modes can enhance energy efficiency significantly for ’always on’ applications. La utilización de estrategias y mecanismos inteligentes de gestión de energía en el chip puede perfeccionar aún más la flexibilidad y usabilidad del sensor; Las soluciones de Prophesee han demostrado un consumo de energía reducido hasta 36uW con la funcionalidad de activación inteligente por eventos habilitada. Además, puede resultar beneficioso admitir los modos de suspensión profunda y de espera.

Specific considerations for an event sensor targeting IoT applications include achieving microsecond resolution time-stamping of events with minimal latency along with seamless interfacing capabilities with standard SoCs through integrated event data preprocessing functions. Leveraging MIPI or CPI output interfaces ensures swift connectivity with embedded processing platforms such as low-power microcontrollers and modern neuromorphic processor architectures. La privacidad a nivel de sensor se garantiza a través de los escasos datos de eventos sin marco de los sensores de eventos e incluye la eliminación de escenas estáticas.

Los sensores basados en eventos continúan evolucionando para satisfacer las necesidades de una gama más amplia de aplicaciones. Prophesee’s most recent sensor, the Genx320, makes it well suited to the demands of many IoT use cases that must operate in lower power and small form factor systems. (Imagen:Profetizar)

Los sensores basados en eventos ahora se utilizan en una gama más amplia de aplicaciones. Al integrar estos sensores con plataformas de IoT, los desarrolladores de productos satisfacen necesidades específicas del mercado relacionadas con el tamaño y el consumo de energía. Use cases include foveated rendering for enhanced AR/VR experiences, eye tracking for human-machine interfaces and safety applications like driver monitoring systems and emotion detection. They also support always-on capabilities for security purposes such as fall detection cameras and gesture/ hand tracking for immersive interfaces. In the AR/VR domain, applications like inside-out tracking and constellation tracking based on flickering LCDs enable precise object or controller tracking.

Further new use cases, enabled by enhancement in silicon technology, are under development, including high-speed structured light 3D technology that enables point cloud generation at kilohertz repetition rates for industrial applications. Privacy-conscious smart home systems like fall detection units are also proliferating more broadly as the vision technology address privacy concerns by not capturing or transmitting images.

Event-based vision is well on its way to establishing itself as a paradigm that will create a new standard in many markets requiring efficiency in how machines can see. En los últimos años, ha evolucionado con éxito para satisfacer una gama más amplia de usos. Y si continuamos adaptándonos y abordando los requisitos de muchas aplicaciones, veremos más cámaras basadas en eventos a nuestro alrededor.

Este artículo fue escrito por Luca Verre, director ejecutivo y cofundador de Prophesee (París, Francia). Para obtener más información, visita aquí  .