Facts
Duration: 01.02.2025 - 31.01.2028
Funder: Federal Ministry of Research, Technology and Space (BMFTR)
Funding volume: 664,000 EUR for the University of Rostock
Project partners: Ramblr GmbH, neoBIM GmbH, TU Clausthal
Show2Instruct
Generating machine-processable control commands from natural-language interaction using object references in the visual system context
Major advances in foundation models, especially semantic image analysis and large language models, enable natural-language and context-specific interaction between physical systems and humans. However, the development of AI-based interaction mechanisms that integrate foundation models from computer vision and large language models is still at an early stage.
For example, during a construction-site inspection, context-specific natural-language queries could be evaluated: “Do all windows and doors in this room match the specification in the BIM system, and have all accessibility requirements been met?” This research field is only beginning to emerge, but it will play an important role for future context-aware voice interaction systems.
This project develops a generative AI technology basis for human-machine interfaces. The goal is not only natural-language operation of software and machines through large language models, but also the ability to include visually recognized objects from the local system context in prompts.
Publications
Nithyanantham, B. K., Sesterhenn, T., Nedungadi, A., Garijo, S. P., Zenkner, J., Bartelt, C., & Lüdtke, S. (2025). MCP4IFC: IFC-Based Building Design Using Large Language Models. arXiv preprint arXiv:2511.05533.
Contact
Bharathi Kannan Nithyanantham bharathikannan.nithyanantham@uni-rostock.de