Hidden state refinement for optical flow forecasting

Abstract

In recent years the topic of optical flow has become well-spread due to computation power support and optical flow estimation applications used on mobile phones and edge devices: video editors, frame stabilizations, and autonomous driving feature providers. This work analyzes multiple approaches to optical flow estimation and finds the main problems of the optical flow methods: slow convergence and long execution of the prediction algorithm. We propose to solve the slow convergence and long execution time with hidden state refinement to provide the initialization for optical flow estimation based on several previous frames and their hidden state transformations, which imitates the pixel movement at the hidden state level. The proposed method uses CNN, LSTM, and Transformer blocks which help to achieve the optical flow estimation and hidden state refinement to speed up the system. We used Sintel, KITTY-15, FlyingChairs, FlyingThings, HD1K, DAVIS, and YouTubeVOS datasets for our experiment.