NSF CAREER awardee protects AI users from harmful content
Assistant professor Xiaolong Ma’s artificial intelligence tool weeds out explicit content without replacing a model’s data set.
Xiaolong Ma, an assistant professor in the School of Electrical, Computing, and Software Engineering, says the growing availability of generative AI models demands stronger guardrails.
Generative artificial intelligence models turn user text prompts into images, video, audio and other media – including harmful material.
Xiaolong Ma, assistant professor in the School of Electrical, Computing, and Software Engineering, won a five-year, $500,000 National Science Foundation CAREER award to create safety protocols for generative AI models.
“Xiaolong’s outstanding work on scalable, efficient, and trustworthy artificial intelligence addresses a critical frontier in advancing AI, helping make increasingly powerful AI systems more practical, reliable and responsible,” said Michael Wu, Thomas R. Brown Leadership Chair and head of ECSE. “As one of the school’s AI experts, his success also highlights the growing strength and breadth of ECSE’s expertise in AI.”
The NSF CAREER grant, one of the agency’s most prestigious awards, honors early-career researchers who serve as academic role models and show leadership potential.
Ma has devoted much of his research to developing privacy-preserving, safe and controllable learning methods for generative models.
“This project represents an important step toward my long-term vision of building trustworthy and controllable generative AI,” he said.
His next tool will train AI algorithms to exclude harmful content without degrading performance.
“It focuses on developing machine unlearning techniques for diffusion models, enabling AI systems to selectively forget unwanted knowledge,” said Ma.
Diffusion models are a powerful family of AI generators that can create clear and highly realistic content. However, some models are trained on unvetted data sets that allow the model to output – by design or inadvertently – content that includes violence or inappropriate themes.
“The problem is you can’t delete this content one by one because it is costly. You would have to remove all the harmful information from the data set and train the model again, which is extremely expensive,” Ma said. “We want to do it all at once, using one run to delete the capacity of the model to serve up harmful material.”
During the run, several mechanisms merge with the model to retrain it. Gradient masking hides harmful base material, while a concept-aware loss function retains overall quality. Ma’s tool also uses explainable AI to locate areas of the algorithm that propagate harmful information.
“This allows us to see the parts of the model, the neurons, that are active during sensitive content generation,” Ma said, adding that the tool can then tweak the neurons responsible for harmful outputs, effectively disabling them.
But he will need to strike a balance. Deleting all the affected neurons could compromise the algorithm since some might generate harmless content, such as images or video of models at a fashion show.
“Say I want to delete two concepts – meaning two types of harmful material – I might find an intersection of neurons that are used for both concepts, like in a Venn diagram,” Ma explained. “We would define those as critical neurons that are likely responsible for many other concepts, and so we would not touch them.”
Ma’s tool, when complete, will be available to generative AI model builders to efficiently sweep their algorithms before releasing them to the public.
“Anyone can find these kinds of models freely on the internet,” he said. “We want to make them safe and protect people, especially young adults and teenagers, from harmful generative models.”
