Gedas Bertasius uses basketball to train computers

Today, the same instincts that once helped Gedas Bertasius scan a basketball court — where the pass is headed, how the defense might collapse — shape the questions he asks as a researcher. How do you teach a computer to notice the right details at the right time for the right learner? How do you turn hours of raw footage into meaningful insight?

Bertasius played basketball on the Lithuanian National Team and for Dartmouth University before becoming an assistant professor in the UNC College of Arts and Sciences’ computer science department. Now he wants to create a ChatGPT-like program that can offer real-time commentary on sports videos.

This technology could also help scouts and coaches predict an athlete’s career trajectory — insight that could shape draft decisions, training plans or even how a team structures its long-term strategy. Machine learning models can scan hundreds of hours of footage in minutes, surfacing patterns a human reviewer could easily miss.

“We’re starting with sports, but I think this sort of decision-making capability goes way beyond that,” Bertasius says. “The goal is to build an AI technology that can perform complex video analysis similar to how humans do it, whether it’s a sports analyst or a medical researcher who needs to navigate multiple information sources to find answers to complex questions.”

That broader vision is why he’s developing a virtual personal coach for anyone learning a new skill. Users wear camera-equipped glasses while performing a task — chopping onions, shooting free throws, playing guitar — and the AI analyzes their movements in real time, offering tailored feedback on how to improve.

Bertasius is also exploring how this technology could support classroom learning. He’s fed recorded lectures into these models and then asked them questions about the material, testing how well the AI can interpret, summarize and reason through complex instruction.

In 2025, this technology won the Multi-Discipline Lecture Video Understanding Challenge at a global AI conference, outperforming models from OpenAI, Google and Anthropic.

“We can use this technology to scale office hours by giving every student instant, personalized help,” Bertasius says.

When asked what ties these goals together, his answer is simple.

“I think it’s about empowering people,” he says. “Whether it’s an AI companion that helps you understand a basketball game or a virtual coach that helps you learn new skills, democratizing this expertise is the most impactful thing I could accomplish with my research.”

The work to make this happen is painstaking. Bertasius and his lab work with coaches to get game footage, pull content from YouTube and record their own videos at basketball gyms, soccer fields and more. They then label everything, teaching their model how to recognize what’s happening — using terms like “pick and roll” or “lateral shuffle” to describe the actions unfolding on the screen.

“Honestly, it’s not glamorous,” Bertasius says. “We’re just setting a lot of different values and seeing how the model learns under certain conditions.”

Processing these high-action videos in real time is no small task. Players often look similar, their movements are fast and layered, and bodies constantly block one another from view. The AI must be trained on exceptionally detailed descriptions, and the final system has to map each step in sequence to the larger goal, understanding which actions actually matter at any given moment.

But the outcome could be a game changer — in multiple ways.

Read more about Gedas Bertasius.