Sketch based image and video retrieval

Rui Hu, John Collomosse
(University of Surrey)

Our work addresses the sketch based visual information retrieval. Compared with other query types, free-hand sketch is a more flexible and convenient query mechanism for visual information retrieval. It can represent the users\' intuitive thought of the shape and structure of an object or scene in their mind when searching for images, motion vector can also be considered in the case of retrieving related video clips. However, the challenge is the gap between the freehand sketch and the content of image/video dataset; the free style of sketches makes the semantic gap a more challenging problem.
We introduce Gradient-Field-HOG (GF-HOG) as a depiction invariant image descriptor, encapsulating local spatial structure in the sketch and facilitating efficient codebook based image retrieval driven by freehand sketched queries. We also explore techniques for extracting motion trajectories and the according appearance information from videos and matching to the free-hand sketched queries.
Our recent work also investigates approaches to bridge the semantic gap of sketch based image and video retrieval.

Poster presented at the BMVW.