Background and Goal of the Thesis
When humans interact with unfamiliar UIs, they rarely succeed to accomplish a task immediately. Instead, they explore, make mistakes, and gradually build an understanding of how tasks can be completed efficiently in the UI. This allows them to navigate UIs more directly over time, and avoid previous errors. In contrast, LLM-based agents that operate UIs, yet lack the ability to systematically accumulate and reuse knowledge gained from previous interactions and hence improve over time. As a result, they keep repeating inefficient behaviors and fail to improve over time. The goal of this thesis is to enable UI Agents to learn from their own past interactions by designing a continuously updated knowledge base. The agent should store and refine knowledge about UI structures, action outcomes, navigation paths, and discovered functionality, and leverage this knowledge in future interactions. This includes remembering efficient workflows for recurring tasks as well as observations that may become relevant in different contexts later on.

Your Responsibilities

• conduct a comprehensive literature review on existing work and state-of-the-art methods
• explore and design novel approaches to the problem
• develop and implement prototypes or experimental systems to validate proposed ideas
• define and conduct experiments and evaluations to assess performance, robustness, and limitations
• continuously present your work in team presentations and written reports
• collaborate closely with researchers and engineers to align research outcomes with real-world applications