Joel Lehman, OpenAI: On evolving intelligence, open-endedness, and reinforcement learning

RSS · Spotify · Apple Podcasts · Pocket Casts

Our fourth episode features Joel Lehman (Google Scholar), previously a founding member at Uber AI Labs and assistant professor at the IT University of Copenhagen. He's now a research scientist at OpenAI, where he focuses on open-endedness, reinforcement learning, and AI safety.

Joel’s PhD dissertation introduced the novelty search algorithm. That work inspired him to write the popular science book, “Why Greatness Cannot Be Planned”, with his PhD advisor Ken Stanley, which discusses what evolutionary algorithms imply for how individuals and society should think about objectives.

Highlights from our conversation:

• How discovering novelty search totally changed Joel’s philosophy of life
• Sometimes, can you reach your objective more quickly by not trying to reach it?
• Better ways to evolve intelligence
• Why reinforcement learning is a natural framework for open-endedness

Below is the full transcript. As always, please feel free to reach out with feedback, ideas, and questions!

Some quotes we loved:

“Novelty search is a way of looking at the question of how ambitious objectives are reached: how do you accomplish ambitious things?”

“What novelty search did was ask a seemingly zen-like question, which is, “Sometimes, can you reach your objective more quickly by not trying to reach it?””

“The easiest way to talk about open-endedness is to point to some processes that we’re familiar with that are open-ended. Biological evolution, for example, is an algorithm that’s run for billions of years and one run of biological evolution has produced all the amazing diversity of life, including ourselves. And so, it’s amazing that a volitionless process produced volition. That’s what I mean by open-ended. It’s continuing to create new things, diverse things, over time.”

“Imagine this SAT analogy problem, which is bird is to jet as evolution is to open-endedness. The idea is that birds’ flight is really interesting, it shows it’s possible to fly, and maybe we even tried to mimic that with ornithopter and those kinds of machines. But really, the history of flight took off when we had some hypothesis about the core principles of flight, that enabled us to engineer things that could more efficiently do things maybe that weren’t even possible for biology. You can’t create a bird that carries tons and tons of cargo across the ocean. Similarly, while we might take inspiration from biological evolution, what are the core principles of open-ended creativity? If you could bottle creativity into a jar, into an algorithm, that could efficiently instantiate an open-ended process that is aimed towards whatever domain we want, that could do so really efficiently and in a way that’s not like biological evolution, but actually has capabilities that are much more impressive than biological evolution?”