Revolutionary Video Analysis Ai Set To Unlock New Era Of Content Search

Twelve Labs is revolutionizing video content interaction by developing AI models that analyze and search through videos. The company’s co-founder, Jae Lee, believes these models unlock powerful new applications but acknowledges video search has been a challenging task.

“We’ve struggled with basic search for years,” Lee said. “Keyword searches can pull up titles, tags, and descriptions, but they can’t get at the actual content of clips.” This is why Twelve Labs developed its approach to video analysis, which involves training models to map text to what’s happening inside a video, including actions, objects, and background sounds.

Unlike other companies like Google and Microsoft, which offer general-purpose multimodal models for search through footage, Twelve Labs’ products are optimized specifically for video. Lee argues that this focus is key to unlocking new applications, from ad insertion and content moderation to auto-generating highlight reels from clips.

One of the key benefits of Twelve Labs’ models is their customization options. Customers can tailor these models using their own data, allowing them to address specific use cases and challenges. For example, a city might use Twelve Labs’ technology to detect threats in real-time and enhance emergency response times.

Lee also acknowledges that bias is a significant risk factor when it comes to AI models. A 2021 study found that training a video understanding model on clips of local news could cause the model to learn racist patterns. To address this issue, Twelve Labs conducts bias tests on all its models prior to release and plans to release formal bias benchmarks soon.

The company is expanding into new areas like “any-to-any” search and multimodal embeddings, which allow users to search across images and audio in addition to video, making it possible to find specific moments or angles in clips more efficiently.

Twelve Labs has attracted big-name backers, including Nvidia, Samsung, and Intel, with a total of $30 million in new funding. This brings the company’s total raised to $107.1 million, used for product development and hiring.

The company is making significant hires, including Yoon Kim as its president and chief strategy officer. Kim, former CTO of SK Telecom and key architect behind Apple’s Siri, will split time between Twelve Labs’ San Francisco HQ and its offices in Seoul. Lee says Kim will be instrumental in driving future growth with key acquisitions, expanding global presence, and aligning teams toward ambitious goals.

The investment from In-Q-Tel reflects the versatility and potential of Twelve Labs’ technology across many sectors, including national security. Lee says the company is always open to exploring opportunities where its technology can have a positive, meaningful, and responsible impact that aligns with its ethical guidelines.