A new neural network, designed to align event-camera data with conventional red-green-blue (RGB) images, improves semantic segmentation on two public benchmarks. The network outperforms MambaSeg on both datasets, achieving the largest improvement of 2.26 percentage points in mean intersection over union, a standard metric for evaluating segmentation accuracy.

Developed by researchers from the Changchun Institute of Optics, Fine Mechanics and Physics of the Chinese Academy of Sciences and the University of Toronto, the network was named EGHF-Net and published in Expert Systems with Applications.

Semantic segmentation assigns a category to every pixel in an image, enabling a visual system to distinguish between roads, vehicles, pedestrians, signs, and other elements in a scene. Conventional RGB cameras capture color and texture; however, fast motion can blur object boundaries, and low-light conditions can obscure visual details. Event cameras, by contrast, record changes in brightness with high temporal resolution, preserving motion information that conventional cameras may miss.

The two sensors therefore provide complementary evidence. However, their signals do not automatically align because the cameras observe a scene from different positions and sample it in different ways. A direct merger can duplicate structures or blur boundaries, particularly around nearby objects where parallax is stronger.

EGHF-Net addresses this mismatch before the final prediction is made. Its enhanced event-input stage reorganizes the sparse event stream so that spatial and temporal attention can reduce isolated noise while retaining short edges and longer motion traces. Then the geometric fusion stage uses deformable cross-attention to estimate where event features should be sampled to match structures in the conventional image. Contextual information guides this process, while confidence-based residual fusion incorporates event features only when they correspond reliably with image features.

By combining alignment and fusion in one process, the network avoids a separate optical-flow step and keeps the event branch focused on information that adds value.

Experiments show why the geometric step matters. The largest improvements appeared around nearby objects and narrow boundaries, where sensor parallax creates the clearest displacement between the two image streams. The network also maintained stronger segmentation under weak illumination and fast motion, the conditions in which conventional frames lose detail or become blurred. Visual comparisons on both public benchmarks showed cleaner outlines around vehicles, poles, pedestrians, and other small structures. These results link the overall benchmark gains to observable changes in the output rather than to a single aggregate score.

Ablation tests confirmed that the two stages solve different parts of the problem. Event-input enhancement improved the quality of sparse motion features, and geometric fusion contributed more to boundary alignment. Using both produced the most accurate overall segmentation.

The researchers also used an asymmetric encoder design to accommodate the different characteristics of the two inputs. A larger branch processes dense color images, and a smaller branch handles the sparse event stream. This arrangement reduced computation compared to MambaSeg, while maintaining the accuracy advantage. Therefore, the evidence supports the architecture as a coordinated alignment system rather than simply a larger feature extractor.

The researchers also identified a limitation. EGHF-Net still relies on conventional images as its main source of semantic information. In a controlled test where those images were removed, the network remained stable but could not maintain segmentation quality because it suppressed event features instead of using them as an independent fallback.

Future work will require dynamic weighting that shifts responsibility between sensors when one input degrades. Additionally, the researchers pointed to lighter models for edge hardware and training strategies requiring fewer pixel-level annotations. These improvements could make the method easier to adapt to new devices and datasets.