" The great enemy of foreign language learning is a sense of shame, an inability or unwillingness to become like a child again and let one’s inadequacies show "
(Herbert A. Simon)
The internet abounds with language learning apps and translation technologies for improving second language skills. One no longer needs to set foot in a language classroom or a foreign country to learn a new language. Some of these apps allow users to answer written vocabulary and grammar questions but seldom give them the opportunity to converse using their language skills. However, speaking and listening skills are an essential component of second language acquisition and long-term memory retention.
Many apps and web-based programs such as 'Duolingo', 'Google Translate' and 'Memrise' are currently available and can be used just about anywhere with internet access. With the growing ubiquity of smart home speakers such as 'Google Home', 'Amazon Echo' and 'Apple HomePod', we are now seeing these apps shift to voice-based user interface design. Voice User Interface Design is a growing field in design and is disrupting the digital interface.
This research followed the principles of User Experience design. Literature review, user surveys, expert interviews and journey mapping were used to develop and refine the ‘Hanashi’ speaking app. ‘Hanashi’ is a tool that language learners can use on their smart home speakers and mobile devices to engage in foreign language conversation within the home. The tool was named after the Japanese word for ‘talk’ or ‘story’.
In the language of their choice, the 'Hanashi' voice assistant prompts users to recall vocabulary and pronounce different objects in their surroundings. The user is then asked follow-up questions to prompt further spontaneous conversation. ‘Hanashi’ is a contextual mnemonic device that aims to help users retain their language skills and supports visual and auditory learning styles in the home environment.
The theory of speech production shows, ‘The more they (language learners) repeat words, the more these are consolidated in long-term memory...'
Features of existing digital language learning tools
This was an insightful way to identify potential opportunities within this field. It also showed that it is possible to learn a language with no prior knowledge with apps such as Duolingo and Rosetta Stone. It is also possible to have a cross-cultural conversation through translation apps like Google Translate and Say Hi without needing to learn a new language! These incredible tools are giving language learners more opportunities to engage in ‘informal’ language studies beyond the classroom.
Understanding the Problem and Identifying Stakeholders
A brainstorming session was done at the beginning of the research phase to understand and map out why people learn languages, what are the benefits of learning languages and why it is so difficult to study and practise speaking a new language. This brainstorm raised a lot of questions but needed further clarification with a user survey.
A stakeholders map was used to identify the companies and professionals who would hold an interest in this research and who the research will affect. This method of identifying who is at stake has placed the users as the key stakeholders. As this is a language learning tool, the users are the key people who will determine the success or failure of the outcome.
The process of producing speech in our first language is not something we often need to think about; it occurs very naturally. However, when trying to speak a foreign language, we become more aware of our brains searching for the right thing to say with the correct pronunciation.
I sent out a survey as a way to collect preliminary research data for the Hanashi project. It was distributed to fifty current university students and recent graduates via Facebook. The survey aimed to uncover if people studied a language in high school, if people continued their language study after high school and why, methods for maintaining languages after high school and how likely they would be to use a product that enabled them to maintain a second language should one be available.
Why did/didn’t you choose to continue your second language study after high school?
Reasons people didn’t continue their second language studies
High school ruined it for me, I was no longer interested in studying it
Couldn’t be bothered. Too much effort
Because I didn’t even learn anything
Already could speak French, took it because I was lazy; didn’t want to continue Japanese
Wasn’t too interested in Japanese
University didn’t allow for it with the degree I am doing
I didn’t see a need for knowing Spanish in my community, and I wanted to focus on German, my other first language, or Korean instead since I was going to study abroad there
Too difficult and not what I wanted to do
I was no longer invested and it became too challenging/complicated
Decided it wasn’t a language I was interested in learning, other languages I want to learn more
Reasons people did continue their second language after high school
Languages are interesting
It would have been a waste otherwise, and I would like to be able to understand and communicate with people from a dissimilar cultures.
My uni asked to me keep learning
Travelling, in a relationship with a French person, working in Mexico
It is very useful for my career aspirations (teacher)
I was just interested in learning English
I wanted to study Spanish during my travels and instead picked up Mandarin again living in China.
Wanted to keep up vocabulary and writing skills
My parents made me, as Dutch people generally speak at least 3 languages.
To maintain my ability and also want to major in the language
It’s part of the curriculum to finish my Bachelor’s Degree.
I did a persona exercise to try and refine the target audience for the Hanashi project. The characteristics, motivations and desires of the Personas were crafted from the outcomes of the aforementioned research methods.
Initial Prototypes and Testing
Initially, I made a collection of objects to represent a language learning speaker that you could place around your home in various places. This way, users could practice speaking in their second language wherever they went in their home. Ultimately, after experimenting with this idea, I decided to move in the direction of AI Smart Home Voice Assistants. Because voice assistants were becoming more ubiquitous in households, it was more likely that this type of technology would be easier to adopt and therefore give users a better chance to practice their language skills.
Conversation Design and Voice User Interface Design
After a failed attempt to redesign and digitalise random objects in the house, I began experimenting with Google’s ‘Dialogflow’. Learning how to use Dialogflow created a paradigm shift in this design process. It proved my suspicions that all roads really do lead to Google. Dialogflow became my way to create a tool for users to engage in basic two-way second language conversation in the home. Instead of placing strange talking robot objects all over the home, users would now be able to interact with this second language speaking tool using the products that they may already have in the home (including smart home speakers, smartphones and computers). However, with Dialogflow came another set of challenges. One of the challenges was designing an interface... with only the voice.
Voice User Interface Design (VUI Design) is a form of interface design that uses audio and speaking as the main outputs and inputs. Conversation Design is the design of the flow of conversation that users have with these voice-based interfaces. This research aimed to uncover an approach to conversation design that considered users interacting in their non-native speaking language.
Testing my Prototype
After playing around with Dialogflow and writing a lot of dialogue, I created a testing website to try and get people to interact with my language learning voice assistant. You can view the testing site at the link below.
The final outcome of this research is a voice-based language learning tool that can currently be used on Google Home, Google Assistant as well as online. The final release of the app is currently pending. Once it is released to the public, Ms. Hanashi can be accessed by the user commanding Google with, ‘Hey Google, talk to Ms. Hanashi’. From there, Hanashi greets the user and gives the user the option to select a language to converse in.
The version made for this project was only available in English. Ms. Hanashi has been programmed to respond to the names of different household objects and other abstract concepts. The responses prompt further discussion about the object or idea that the user has input. This leads to a natural flow of conversations that gives the user the chance to engage in a spontaneous, unpredictable conversation, much like they would if conversing with a native speaker of a foreign language. Each follow up question that is prompted by the user’s input has been designed to expose users to a variety of grammar structures, tenses and question words. Often, Ms. Hanashi will ask the user to count how many of a particular object there is, describe the objects next to a particular object, discuss what they do in a particular room and so on.
User Scenario Video
Mini User Journey Map
This project was completed in 2018 as part of my Industrial Design honours project under the supervision of Pierre Proske.