On May 17, a trio of Penn State graduates traveled to Paris to pitch their ‘smart mirror’ prototype at the 2018 L'Oréal Brandstorm competition.

The members of team SalonAI, Vincent Trost, Vamshi Voruganti, and Yuya Ong, have spent several years exploring new applications of data sciences through their work at Nittany Data Labs. Recognizing that salon interactions are most often based around a mirror, SalonAI aims to bring the mirror into the 21^st century using computer vision technology, machine learning algorithms, and data collection infrastructure.

To create the mirror, Ong programmed facial recognition software that identified key points on the user's face — such as the distance between the eyes and the curve of the jawline — to determine face shape.

Trost used an algorithm to determine which hairstyles were most often seen on models with different face shapes. By combining the two, the mirror can recommend and simulate hairstyles and cosmetic products that are popular among others with similar features.

"With this system, we can better analyze many features including facial shape, age, skin tone, and other critical information about a customer in real time," said Ong. "By leveraging this data we can better deliver a higher quality of services than ever before and will generate tremendous brand loyalty."

Ong built the mirror in his dorm room over two days using an old computer monitor and spare parts he found online. He then created the entire facial tracking algorithm in 24 hours during HackPSU.

The mirror also serves as a pipeline for data collection, which the group views as the most valuable element of their product. For example, by indexing individual preferences, companies can anticipate and develop products that respond to trends, and stylists can improve performance.

"The entire salon experience is seen in the mirror, which allows us to capture the subtler points," said Ong. "This lets us solicit real-time feedback using facial reactions to see whether the consumer likes the style or not. The stylist can record the user's preferences and get feedback on how well they performed the services they provided."

"We are not trying to replace the stylists," said Trost. "They are the reason it works, but we are trying to enhance the experience and make it more collaborative."

Creating a shared salon experience

L'Oréal's internal research has shown that 97 percent of stylists believe they provide consultations to their clients, but only 7 percent of clients feel like they receive one. This disconnect helped the group members hone their proposal.

The team also surveyed more than 100 women to gather feedback about the salon experience. Their research showed that most people either asked the stylist to recreate a hairstyle based on a photo from their smartphone or simply let the stylist choose which style would look best. But neither is ideal.

"When you get a haircut, you spend the whole time staring in the mirror," explained Trost. "Instead of trying to describe it or zoom in on a photo on your phone, why not just display the style on the mirror? Then you and the stylist are working together, identifying what you do and don't like, simulating it on you in real time, and everyone is happy because you can see exactly what you're going to get before you get it."

Added Ong, "This isn't automated intelligence or artificial intelligence; it's augmented intelligence."

Reflecting on an idea

To make it to the world finals, more than 1,000 US registrants submitted video pitches, and 10 teams were selected to present their ideas at the national finals in New York City. The top two teams from that national competition — including SalonAI — were selected to participate in the final competition in Paris.

The team believes much of their success so far has come from their ability to capitalize on individual strengths.

"To run a successful startup, you need three people," said Trost, recalling advice given to him by Steven Haynes, associate teaching professor of IST. "One should be completely devoted to product development — that's Yuya; one should be completely devoted to marketing the product and securing funding — that's Vamshi; and one should be focused on making things run smoothly — which has been my role."

Along the way, the team received hands-on coaching and regular feedback from L'Oréal executives, industry experts, and Penn State faculty.

And while they're excited to be in the finals, the group is focusing their attention on turning their idea into an entrepreneurial business. Noticing the potential for smart mirrors in industries like high-end retail, fitness and personal health, the team is hoping additional feedback from judges will help them explore more potential opportunities.

"We're ultimately trying to start a business," said Trost. "If we lose this competition we lose nothing, but we've had the opportunity to pitch our idea to CEOs of Fortune 500 companies, refine it, and make it better."