People of ACM - Jun Kato

April 25, 2023

What does your role at AIST involve?

AIST is a Japanese national research institute, and researchers can choose a wide range of ways to contribute, from fundamental research of writing academic papers to applied research that aids industry and technology transfer aimed at implementing systems for real users.

Since I joined AIST in 2014, I have enjoyed all of these, one by one. The transition has also been comfortable as I became a father and wanted a twist in my working style. Having a baby makes focusing for a long time challenging, so I shifted to software development. These days, the happy babysitting time has settled down a bit, and I am returning to pure research.

Right now, I am leading an internal research initiative exploring how Human-Computer Interaction would (should/could) look in the future. We believe promoting HCI for encouraging and unleashing people’s creativity is the key. Not only those “big” creative projects (so-called “Big-C” creativity) aimed at major productions of scientific or artistic importance, but also smaller personal projects (so-called “mini-c” creativity) that can enhance experiences in everyday life.

How did you become interested in designing user interfaces and integrated environments for creativity support? How has your approach been different from the existing state-of-the art?

I dedicated my youth to programming and design. Nowadays, all kinds of artistic activities use the computer as a tool. While the act of programming those tools initially grabbed my attention, I eventually came to see programming as artistic and creative in its own right. This approach led me to create TextAlive, which won an Honorable Mention Award at CHI 2015. It implements a novel interaction design that allows both video authoring and algorithm development to render video frames.

Later, as we continued to develop TextAlive as a public web service, we came to think of a novel interactive media format that shows lyrics and motion graphics synchronized with music playback in response to user inputs. To aid the development of such “lyric apps,” we proposed a Lyric App Framework, which won the Honorable Mention Award at CHI 2023.

The common thread in all our work is to propose novel interactions that support people’s creativity with the underlying technology of programming systems. While there are now other HCI researchers who are following this approach—which is quite welcome—what I believe still makes our approach unique is that we don’t confine our creativity support tools to academic publications, but make them publicly available.

Why do you think your Programming Experience project will be widely adopted?

My Programming Experience (PX) project has focused not only on artistic applications but also on other creative domains including GUI programming and physical computing , and I believe our approach is general enough to be adopted in many domains. Researchers in HCI, programming languages, software engineering, and computer science education have become more interested in a holistic view of programming activities, as seen in the Dagstuhl Seminar on Theories of Programming .

There seem to be various reasons for this; from a technical perspective, using open-source building blocks to build a programming environment make it more accessible—thus, there is more focus on the environment than a single tool; and from a societal perspective, computer science education has become popular, and the concept of an “end-user” or “programmer” has become somewhat vague or obsolete—thus, there is more focus on the activities than the users.

What is an example of how new AI tools are impacting multimedia generation?

My colleagues develop wonderful music understanding technologies that automatically detect musical elements such as beats, chorus segments, the timing of lyrics, text being vocalized, chord progressions, and vocal amplitudes. TextAlive is a happy marriage of such technologies (which could be classified as AI tools) with my programming infrastructure. The video authors do not need to worry about synchronizing motion graphics with beats and lyric animations with their vocalization timing, and can focus on exploring desirable visual styles.

What is an advance in your field you would like to see come to fruition in the near future?

The more AI tools can “understand” content, I predict that people will become accustomed to a more semantic authoring process. I would not call it “content generation,” but more like an interactive exploration through dialogues using a shared vocabulary between AI tools and users. This could potentially happen in every application in creativity support, including programming and multimedia authoring.

One thing to note is that building such a shared vocabulary requires deep domain-specific knowledge of the creative process. My latest work at Arch Inc. studies and supports the storyboarding process of Japanese animation directors, and I feel that such a sociocultural background will become more important in creativity support research in the future. We recently held a Special Interest Group meeting on Creativity and Cultures in Computing (SIGCCC) at CHI 2023, and I encourage interested readers to join our community!