The annotations such as light source, shooting location, and human presence are related to the restoration performance. It is necessary to recognize the annotation type as a significant factor affecting PSNR, SSIM, and LPIPS values. The experimental results below show the critical factors in restoring UDC-degraded videos. Human motions exist in the UDC-VIT dataset, which can contribute to face recognition research in the UDC setting; it is crucial to understand the critical factors in restoring humans to face recognition.
One significant observation is that the benchmark models (e.g., DDRNet pretrained model) do not produce flickers when restoring the VidUDC33K dataset. However, the benchmark models generate flickers in nearly every video in the UDC-VIT dataset due to varying degrees of degradation, such as flares and reduced transmittance. This finding underscores the importance of training on datasets that accurately reflect real-world degradation, as it can significantly impact the restoration results.
| VidUDC33K (GT) |
UDC-VIT (GT) |
|
|
|
| VidUDC33K (Input) |
UDC-VIT (Input) |
|
|
|
| VidUDC33K (Restored by DDRNet): w/o Flicker |
UDC-VIT (Restored by DDRNet): w/ Flicker |
|
|
|
Flare presence and light source
All benchmark models struggle to restore scenes with flare compared to those without flare. The severity of degradation within flare-present scenes varies based on the light source (e.g., natural sunlight, artificial light, or both). Intense sunlight can oversaturate pixel values, making objects around the flares less visible. Consequently, the benchmark models have more difficulty restoring videos with flares caused by natural sunlight than those caused by artificial light. This distinction in the impact of different light sources on restoration performance is a significant finding of this research.
Shooting location
The benchmark models struggle more with restoring outdoor scenes than indoor scenes. Approximately 33.3% of outdoor and 15.4% of indoor scenes include flares caused by natural sunlight in the UDC-VIT dataset. Moreover, sunlight-induced flares tend to be less severe indoors. For example, outdoor scenes with natural sunlight flares show severe flare, whereas indoor scenes with the same type of flare tend to be less severe. Notably, the flare on the glass window is subtle, caused by sunlight scattering off the glass rather than directly entering the camera. This understanding is crucial as it highlights the unique challenges of restoring outdoor scenes where direct sunlight is a significant factor. Consequently, all models face more significant difficulties restoring outdoor scenes than indoor scenes.
| UDC-VIT (GT) |
UDC-VIT (Input) |
|
|
|
| UDC-VIT (Restored by DISCNet) |
UDC-VIT (Restored by UDC-UNet) |
|
|
|
| UDC-VIT (Restored by FastDVDNet) |
UDC-VIT (Restored by EDVR) |
|
|
|
| UDC-VIT (Restored by ESTRNN) |
UDC-VIT (Restored by DDRNet) |
|
|
|
Human
The presence of humans alone does not pose a significant challenge to restoration. Instead, the restoration difficulty hinges on how UDC degradations, such as noise, blur, transmittance decrease, and flare, impact humans. For example, despite the presence of flares in the frames, they do not affect humans. However, the reflection of fluorescent light on the person’s glasses poses challenges for restoring fine details around the eyes. Sometimes, human faces appear reddish in the input frames compared to the ground-truth frames due to UDC-induced diffraction occurring differently across RGB channels. Moreover, the restored facial colors vary among models. In applications like face recognition for smartphone unlocking, financial authentication, and video conferencing, it is crucial to consider these diverse UDC degradations for accurate human restoration since facial color is crucial in images or videos.
| UDC-VIT (GT) |
UDC-VIT (Input) |
|
|
|
| UDC-VIT (Restored by DISCNet) |
UDC-VIT (Restored by UDC-UNet) |
|
|
|
| UDC-VIT (Restored by FastDVDNet) |
UDC-VIT (Restored by EDVR) |
|
|
|
| UDC-VIT (Restored by ESTRNN) |
UDC-VIT (Restored by DDRNet) |
|
|
|
Flicker
Despite some successful restoration results of a frame, multiple frames in the video often exhibit flickering across all models. This flickering may result from varying degradations between consecutive frames, such as transmittance decreases and flares.
| UDC-VIT (GT) |
UDC-VIT (Input) |
|
|
|
| UDC-VIT (Restored by DISCNet) |
UDC-VIT (Restored by UDC-UNet) |
|
|
|
| UDC-VIT (Restored by FastDVDNet) |
UDC-VIT (Restored by EDVR) |
|
|
|
| UDC-VIT (Restored by ESTRNN) |
UDC-VIT (Restored by DDRNet) |
|
|
|