Jiayang Song

Avenida Wai Long

Taipa, Macao SAR

About Me

I am currently an Assistant Professor in the School of Computer Science and Engineering at Macau University of Science and Technology. I received my Ph.D. (Software Engineering and Intelligent System) at the University of Alberta in 2025, M.Eng at the University of Toronto in 2021, and B.Eng at Western University in 2019.

Research Interest

My primary research direction revolves around Quality Assurance for Trustworthy AI systems such as AI-enabled Cyber-Physical systems (AI-CPS), Large Language Models (LLMs), and Multimodal AI Agents.

My research is devoted to exploring these directions in three stages: (1) testing and evaluating the performance and reliability of generative AI in dynamic, real-world environments, (2) developing methods to enhance the safety and resilience of complex AI systems, and (3) investigating the potential of multimodal foundation models to revolutionize the capabilities of AI applications.

My long-term vision is to create a cohesive research program that pushes the boundaries of AI systems by combining foundational models and cutting-edge quality assurance techniques.

Prospective Students

I am looking for sel-motivated Ph.D. and MSc. students with strong programming skills and relevant research experience. Please send me an email with your CV and transcripts if you are interested.

News

May 08, 2025	Our paper Towards Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation: An Empirical Study is accepted at The ACM International Conference on the Foundations of Software Engineering (FSE).
Jan 24, 2025	Our paper Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixtures is accepted at NAACL 2025 Findings. This paper introduces Multilingual Blending, a mixed-language query-response scheme designed to evaluate the safety alignment of SOTA LLMs under sophisticated, code-switching conditions.

Selected Publications

LLM

Look before you leap: An exploratory study of uncertainty measurement for large language models

Yuheng Huang, Jiayang Song, Zhijie Wang, and 4 more authors

IEEE Transactions on Software Engineering., 2024

Bib PDF

@article{huang2023look,
  title = {Look before you leap: An exploratory study of uncertainty measurement for large language models},
  author = {Huang, Yuheng and Song, Jiayang and Wang, Zhijie and Zhao, Shengming and Chen, Huaming and Juefei-Xu, Felix and Ma, Lei},
  journal = {IEEE Transactions on Software Engineering.},
  year = {2024},
  organization = {IEEE},
}

AI-CPS

When cyber-physical systems meet AI: A benchmark, an evaluation, and a way forward

Jiayang Song, Deyun Lyu, Zhenya Zhang, and 3 more authors

In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice, 2022

Bib PDF

@inproceedings{song2022cyber,
  title = {When cyber-physical systems meet AI: A benchmark, an evaluation, and a way forward},
  author = {Song, Jiayang and Lyu, Deyun and Zhang, Zhenya and Wang, Zhijie and Zhang, Tianyi and Ma, Lei},
  booktitle = {Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice},
  pages = {343--352},
  year = {2022},
}

AI-CPS

Towards building AI-CPS with NVIDIA Isaac sim: An industrial benchmark and case study for robotics manipulation

Zhehua Zhou, Jiayang Song, Xuan Xie, and 5 more authors

In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice, 2024

Bib PDF

@inproceedings{zhou2024towards,
  title = {Towards building AI-CPS with NVIDIA Isaac sim: An industrial benchmark and case study for robotics manipulation},
  author = {Zhou, Zhehua and Song, Jiayang and Xie, Xuan and Shu, Zhan and Ma, Lei and Liu, Dikai and Yin, Jianxiong and See, Simon},
  booktitle = {Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice},
  pages = {263--274},
  year = {2024},
}

AI-CPS

SIEGE: A Semantics-Guided Safety Enhancement Framework for AI-Enabled Cyber-Physical Systems

Jiayang Song, Xuan Xie, and Lei Ma

IEEE Transactions on Software Engineering, 2023

Bib PDF

@article{song2023mathtt,
  title = {SIEGE: A Semantics-Guided Safety Enhancement Framework for AI-Enabled Cyber-Physical Systems},
  author = {Song, Jiayang and Xie, Xuan and Ma, Lei},
  journal = {IEEE Transactions on Software Engineering},
  volume = {49},
  number = {8},
  pages = {4058--4080},
  year = {2023},
  publisher = {IEEE},
}

LLM

LUNA: A Model-Based Universal Analysis Framework for Large Language Models

Da Song, Xuan Xie, Jiayang Song, and 4 more authors

IEEE Transactions on Software Engineering, 2024

Bib PDF

@article{song2024luna,
  title = {LUNA: A Model-Based Universal Analysis Framework for Large Language Models},
  author = {Song, Da and Xie, Xuan and Song, Jiayang and Zhu, Derui and Huang, Yuheng and Juefei-Xu, Felix and Ma, Lei},
  journal = {IEEE Transactions on Software Engineering},
  year = {2024},
  publisher = {IEEE},
}

Robotics-RL

GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model

Zhehua Zhou, Xuan Xie, Jiayang Song, and 2 more authors

IEEE Transactions on Neural Networks and Learning Systems, 2024

Bib PDF

@article{10766903,
  author = {Zhou, Zhehua and Xie, Xuan and Song, Jiayang and Shu, Zhan and Ma, Lei},
  journal = {IEEE Transactions on Neural Networks and Learning Systems},
  title = {GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model},
  year = {2024},
  publisher = {IEEE},
  pages = {1-15},
}