Just two episodes in and Love Next Door, the new Netflix Korean rom-com of childhood friends, sharing a complex history, promising to make you laugh and cry has already garnered a huge fan following (while not being a part of a debate, we can easily say it might rank just below our latest favourite Queen of Tears). But wait, what if you hear that you can watch Queen of Tears again without the need for subtitles? Imagine our very own Baek Hyun-Woo (Kim Soo-Hyun) and Hong Hae-In (Kim Ji-Won) bantering in Hindi and their lips syncing with the same language too, creating the perfect symphony between visual expression & the dubbed audio.

Or, let’s take RRR, which although has been dubbed in several languages; the flow between the dubbed audio and the actor’s lip and jaw movement can often destroy the visual experience of the whole movie. But, what if, alongside watching RRR dubbed in Hindi, you observe Ram Charan’s lip and jaw movement happening in a similar language as if the whole movie was shot in Hindi and not in Telugu?

This is exactly what a Bangalore-based generative AI company, NeuralGarage has been able to achieve. While AI dubbing has been part and parcel of consuming content in diverse languages for a long time now, helping automate the replacement of the original audio with the translated language, what NeuralGarage has come up with is completely innovative. The viewing experience of any individual till now always included a mismatched concurrence of the actor’s lip movement and the dubbed audio, detaching and distracting non-native viewers.

In an interview to The Statesman, Rajesh Mishra, executive director and group CEO of UFO Moviez, talked about the drawbacks and the limitations the entertainment industry has faced in film dubbing, stating, “When dialogues are dubbed in another language and the audio track superimposed on original image, there is an inherent mismatch in the dialogue track and the lip movements due to speed of speaking, usage of different words due to language, style of speaking of the original vs the dubbing artists etc. This creates a visual track which is not in sync with the audio track. This takes away the pleasure of viewing from the audience. No matter what quality of dubbing or voice-over artists are used, there is always a difference which is visible. The entertainment industry constantly struggles to eliminate this out-of-sync phenomenon but rarely succeeds completely.”

What is VisualDub? 

According to Mandar Natekar, co-founder and CEO of NeuralGarage, since the time from when content has been dubbed to make it more viable for a larger audience, the problem of dubbing and mismatched visuals has been persistent, and it is impossible for humans to solve it. According to him, this is where technology comes in. NeuralGarage is a brainchild of  IIT Kanpur alumni Subhabrata Debnath, Subhashish Saha, Anjan Banerjee, and media &  entertainment veteran Mandar Natekar, with VisualDub being their flagship technology that has been designed to address the major issue that the entertainment sector, OOT platforms and advertisements agencies are facing regarding the asynchronism between the visual and audio cues.

According to him, VisualDub will be a revolutionary technology because “Our technology actually comes in after the dubbing is done. This is something that has been completely created in India, and our technology has already been incorporated by brands like Amazon, Britannia, Coca Cola and others.” VisualDub, through NeuralGarage’s proprietary technology, creates magic by synching the lip movements of the actors with the syllables, their smile lines and micro muscles with the audio in which the content has been dubbed, ensuring that the visuals feel authentic, giving viewers an exceptional experience of a satisfactory sensory event as if the content has been shot in the language in which it has been dubbed.

“No language is a challenge or a barrier, our technology has already been tested on over 35 languages, and the results have been outstanding, making this technology unique and distinctive from any other technology that is present in India. Being able to convert any content in so many languages helps the viewers feel even more connected with the content.” adds Mandar Natekar.

In a recent collaboration, NeuralGarage had joined hands with UFO Moviez, which according to Natekar, would allow the entertainment industry to create native and authentic content in every language now by only using AI, making creating content time and cost-effective. Rajesh Mishra elaborates further by adding how, during the Covid situation, content being scarce, they realised the gap and started pushing South language content to North India; however, the viewer experience was often marred due to audio-visual discord, “VisualDub solution seamlessly syncs the visual to the dialogue track making it totally natural for the viewer. Additionally, for films which were shot in multiple languages to avoid the lip sync issues, this could be a huge cost saver as they could just shoot once and handle any dubbing-related issues using the VisualDub process. This initiative can help all films to potentially release pan India in local languages.”