Data-driven Segmentation and Labeling of Freehand Sketches

SIGGRAPH Asia 2014

Zhe Huang   Hongbo Fu   Rynson W. H. Lau

City University of Hong Kong


Our method performs simultaneous part-level segmentation and labeling of input sketches (a), using database components and their interrelations. It first produces many local interpretations (b), which are optimized into a global interpretation (c) that fits the input sketches as well as forming plausible structures, with which the input sketches can be appropriately labeled (d).

We present a data-driven approach to derive part-level segmentation and labeling of free-hand sketches, which depict single objects with multiple parts. Our method performs segmentation and labeling simultaneously, by inferring a structure that best fits the input sketch, through selecting and connecting 3D components in the database. The problem is formulated using Mixed Integer Programming, which optimizes over both the local fitness of the selected components and the global plausibility of the connected structure. Evaluations show that our algorithm is significantly better than the straightforward approaches based on direct retrieval or part assembly, and can effectively handle challenging variations in the sketch.
Source Code
The source code for global interpretation, which is the core of our algorithm.


 Sketch dataset (3.8M)

 Note: please cite our paper if you use our dataset.

More Results
author = {Zhe Huang and Hongbo Fu and Rynson W. H. Lau},
title = {Data-driven Segmentation and Labeling of Freehand Sketches},
journal = {ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2014)},
year = {2014}


We thank the reviewers for their constructive comments. This work was partially supported by three grants from RGC of Hong Kong (Project No.: CityU 113513, CityU 11204014 and CityU 115112).