Widget detection on screenshots using machine learning algorithms

authors: Radzikowski, Kusmirek, Forc, Lepak, Chęciński, Twardowski, Wawrzynski, Nowak

In this paper we consider the problem of detecting and recognizing widgets in screenshots of computer programs' graphical user interface (GUI). That problem is fundamental in business process automation. The solution we propose here is based on detecting GUI elements with Canny edge operator, and recognizing already detected GUI elements with classifiers: neural networks, random forests, XGBoost, and others.