Agentic Vision-Language AI System for Autonomous Desktop Application Testing

Abstract

This work presents TestAny, a desktop User Interface (UI) testing application that reduces the effort involved in manual testing and test creation. The software combines multiple technologies, including computer vision, optical character recognition (OCR), and retrieval-augmented generation (RAG), to generate test cases from the test application’s official user manuals. The application to be tested is embedded in our system, and its interface is captured through screenshots. From the captured screenshots, the system attempts to find the clickable elements and read any visible labels on the screen. The extracted information is then matched with the relevant portions of the provided user manual, which helps guide the generation of tests to be executed. This combined use of visual and textual references will help us produce more meaningful test cases than relying on a single source of information. Another advantage is that the workflow reduces the amount of manual scripting required and keeps the generated tests traceable to the user manual content. Since this method avoids strict UI selectors, it is also less sensitive to minor UI layout changes. In initial experiments with the sample test application, the UI elements were detected with reasonable accuracy, and OCR correctly identified the labels in most cases. The generated test plans also closely followed the intent of the documented procedures. This paper outlines the system design, main implementation, and overall integration flow, and lays the groundwork for a more detailed evaluation in future work.