People of ACM - Tim Menzies

March 18, 2025

You have been recognized for your work in developing the PROMISE data repository , which continues to be a vital resource for the field. What are the goals of this project?

One night in 2004, walking around Chicago’s Grant Park, Jelber Sayyad and I were lamenting the sad state of machine learning in software engineering (SE). “We must do better,” we said. “Why don’t we make conclusions reproducible? Make authors promise to publish their data with their papers?” In 2025 it is hard to believe that this idea of “reproducible SE” was a radical idea. But at that time, there was little sharing—so much so that in 2006 Lionel Briand predicted it would not work, famously saying “no one will give you data.” At first, the PROMISE series got off to a shaky start. But once Gary Boetticher, Elaine Weyuker, Thomas Ostran d, and Guenther Ruhe joined the steering committee, the meeting earned the prestige needed for future growth. This PROMISE-style of research lead to many successful papers. By 2018, 20% of the articles listed in Google Scholar Software Metrics for IEEE Transactions on SE used data sets from that first decade of PROMISE.

How can industry and academia better coordinate their work?
Industry can get any research it wants just by publishing a baseline result along with the data and scripts need to reproduce that work. For instance, my 2007 IEEE Transactions on SE paper "Data Mining Static Code Attributes to Learn Defect Predictors” presented such a baseline, using static code attributes from NASA projects. Those results were enthusiastically embraced by a software engineering research community that was hungry for data. To support that, the paper shared all its scripts and data. As such, it became a handy “go away and try this!” document that a hundred supervisors could give to a thousand graduate students. This perhaps explains the popularity of the paper. At its peak in 2016, this work was the most cited (per month) paper in software engineering. At the time of this writing, that 2007 paper and the PROMISE data repository it created have 1,924 and 1,242 citations (respectively) in Google Scholar.

What is one important way automated software engineering (ASE) has changed since you began your career.

I’ll give you three. One is the nature of the ASE. In the early 2000s the work was often about different kinds of verification tools (model checkers, theorem provers, etc.) and now we deal with a much broader range of issues—e.g., these days inferring the license type from hints in the source code is a valid ASE problem, as is understanding how humans and automation interact.

The second big change is the rise of AI. Starting with analytics and then the more recent explosion in generative systems.

But the best change is the repeatability of our research (and I’m egotistical enough to claim credit, at least in some small part, for that). Decades ago, no one shared their data or scripts or tools. But these days, reviews tend to reject papers without a reproduction package. Now, to be sure, some reproduction packages are better than others—but it is an amazing world when, in 2025, I can tell all my grad students to “go find the current state-of-the-art, get it running, collect baseline data, now let’s see if we can do it better”.

In the recent ACM TOSEM article, “Learning from Very Little Data: On the Value of Landscape Analysis for Predicting Software Project Health,” you and co-author Andre Lustosa addressed how having insufficient data can lead to software errors. Will you tell us what landscape analysis is and how it overcomes this challenge? How can this technique be applied to other areas?

For years I was all about data mining and that work assumed that analysts can access a large number of good quality labels. Increasingly, I have been growing more and more suspicious of that assumption. These days, my research focus is on how much can be achieved in software engineering using as little data as possible. This work explores methods such as:

• Landscape analysis that carefully surveys the space of the independent variables before deciding where to probe next
• Surrogate learning that uses small amounts of data to build a predictive model, then it generates all other labels from that surrogate
• Active learning that reflects over what the surrogate is telling us then steering the inference to regions that are most informative
• Semi-supervised learning where learners, or groups of learners take hints from the data (or other learners), to find labels all over the data

Do you think generative AI tools will be able to replace human software developers in the near future?

Goodness me, no. I’m predicting a software engineering gold rush where more and more human engineers will have to be hired to clean up the train wrecks left behind by the next few years of vibe programming (vibe programming: driven by large language models (LLMs) with hints from humans, where no one reads or checks the hastily generated code).

What will the relationship between AI and computer programming look like ten years from now?

What I think is more interesting, apart from the current LLM lemming rush over a cliff, is what we are learning about decision making using limited data. I can see true AI/human partnerships arising where AIs accept human direction to derive solutions, and those solutions are simple enough for humans to read / understand / critique and improve on them. I am not talking about low-level programming so much as requirements decisions or very large scale decision making, driven by understandable and explainable inputs from computers.

What’s another emerging trend in AI or software engineering that has great potential?
All the human factors research is getting very, very interesting. As a side effect of instrumentation systems for the purpose of learning more about automated reasoning, we are also learning more about how humans interact to explore ideas and find new knowledge. I can see us replacing “software engineering” in the very near figure with “knowledge engineering.”