Jazz Artist Stefon Harris Invents Music App that Programs the Unpredictable

After 13 years of research, internationally acclaimed jazz artist Stefon Harris did something no one’s done before: create an app that makes music as unpredictable as a jazz solo.

Harris, a Rutgers-Newark professor of music, invented Harmony Cloud, an AI-powered software platform he developed to help musicians practice the art of not knowing what comes next.

“It took years and years of trying to figure out how to create an algorithm that surprises,” Harris said. “It surprises me—and I created it.”

Harmony Cloud can generate any number of sounds and combinations–from one note to hundreds–with which a musician can play along. It was designed to help develop the art of improvisation. 

Wach a video of Harris using Harmony Cloud with students here.

“It’s like having another person in the room who will challenge you,’’ said Harris, a Grammy-nominated vibraphonist.

He describes Harmony Cloud as an “ear training class,’’ one that moves beyond practicing scales and rehearsing notes. He also believes it helps cultivate skills that transcend music: the ability to navigate uncertainty, remain in the moment, and be attuned to others.

“None of us know what’s coming next. We’ve got to get comfortable with not knowing and see what it means to be human. Really good musicians spend a lot of their time developing their ability to listen to other people,’’ said Harris. “That’s the point of empathy in music—it breeds connection.”

Harmony Cloud’s creation took years of self-funded and self-directed research. Harris collaborated with engineers—including one who worked for Microsoft—and produced thousands of pages of development notes. The app launched in January after more than a year of beta testing.

Although it’s geared toward musicians–aspiring and professional, from grader schoolers on up–for non-musicians, it can foster a deeper appreciation for music.

In his Harmony Hang sessions, Harris asks students to envision a scene inspired by the app’s music and the emotion it conveys. “What’s the weather like in this chord? If this sound were a person, what would they be wearing?’’ he asks. He has students close their eyes and observe how their bodies respond.

“People have this incredible visceral reaction to sound that we’re just not cognizant of,” said Harris, who calls his teaching method the Melodic Progression System.

Harris’s invention focuses on harmony, an area that has long lacked practical tools. Metronomes address rhythm, and many apps focus on melody. “But there’s no tool that addresses both together,” he said.

He envisions Harmony Cloud becoming a fundamental musical utility, one that will eventual be as ubiquitous as a metronome.

One of Harmony Cloud’s most distinctive features is that it does not train musicians within a specific style. It encourages them to focus on the notes themselves and the feelings they evoke rather than remain locked into a familiar genre.

It can enable them to improvise with music from any culture without ever playing or even listening to it before, said Harris.

“The promise of the app is that if you train your ability to perceive the world on a deep level, you should be able to go anywhere in the world and connect with anyone,” he said.

Harris also sees Harmony Cloud as a way to democratize access to elite musical training. The app is designed for serious learners at all levels and includes video lessons and curriculum elements comparable to what students might receive in a conservatory program.

The platform tracks a user’s progress, categorizes chords by emotional intensity, and builds from basic harmonies to lush orchestrations.

“What would cost $70,000 at another institution, I’m putting inside the app,” he said.

It aligns with his mission at RU-N, where Harris works with students from all backgrounds, including Biology majors, accountants and people who have never taken a music class before.

“I can explore what it means to put the power of the arts in the hands of as many people as possible,” he said.

Despite being an AI-driven platform, Harris insists that Harmony Cloud is not meant to replace human interaction or artistry.
.
“It’s a tool,” he said. “A hammer doesn’t replace the vision of the person who is building a home.”

Instead, he sees the technology as illuminating what it means to be human, especially in a world increasingly shaped by artificial intelligence.

“It doesn’t replace your desire to talk to another human being,” Harris said. “It helps prepare you to form deep
connections.”