- Computers have difficulties understanding the nuances of language.
- Using six dictionaries and Wikipedia, researchers ‘trained’ an artificial neural network system to understand words, phrases, and sentences.
- Available freely online, it acts as a reverse dictionary and can answer crossword puzzles more efficiently than commercial products.
A team of researchers from the UK, US, and Canada have designed a web-based artificial intelligence system that is a step forward in enabling computers to ‘learn’ language — plus it can answer crosswords and act as a reverse dictionary.
The research, which was published in Transactions of the Association for Computational Linguistics, used a relatively new approach in language technology known as artificial neural networks or 'deep learning'. The model was trained for almost four days on the data (hundreds of thousands of definitions) from six dictionaries and Wikipedia.
“Our system can’t go too far beyond the dictionary data on which it was trained, but the ways in which it can are interesting, and make it a surprisingly robust question and answer system — and quite good at solving crossword puzzles,” says Felix Hill from the University of Cambridge who worked on the project.
Beyond helping you answer that pesky clue for 16 down, the artificial neural network enables language understanding not only more quickly but also more efficiently. Computers have traditionally had problems learning language due to researchers’ struggle to recreate the rich and diverse information sources available to humans when they learn to speak and read.
The approach of using multiple dictionaries is key to the success of this system. Even though they contain definitions for many of the same words, the definitions will be written slightly differently. This exposes the model to the natural subtle variation of language, making it more robust and bridging the gap between lexical and phrasal semantics.
Linguists are constantly debating the principles of language, which makes it difficult to encode rules in a language system. This has been tried previously, but with limited success. Using neural networks instead allows for a more 'organic' way of learning: directly from raw data, with comparatively little input from humans, explains Hill.
The model builds up meanings as it learns and then works out how to combine those words’ meanings to interpret phrases and sentences. When the model was inspected, the meanings it had built matched many human intuitions about word meanings.
Even if you give the model a description not in the dictionaries used for training it will return words that could be a match to that definition. The fact that the model can answer crossword questions tells us that it is good at generalizing away from the precise task on which it is trained.
“This research tells us that dictionaries can be useful resources for training neural language models,” says Hill. “I hope it will make other researchers take our system and improve it, which will, in turn, speed up the overall pace of progress.”
Hill says he would like to build a model that, after initial training using dictionaries or books, continues to learn via interactions with humans. “In the long term, I hope to make the way in which these sorts of models learn much more similar to how humans learn language.”