Healing communication with a robot

Making "lively conversation" with a robot a reality: Dear Talk Engine

Dear Talk Engine

Dear Talk Engine is a system for demonstrating "conversational skills."
Having a conversation is a sophisticated ability which, among all living things, only humans can do.
It is not just about listening to the other person talk and responding to what he or she says: you have to pick up on things like the time and place, the other person's expression, feelings, and the atmosphere instantaneously, predict what the other person is going to say, and "prepare" your listening posture, before you are even aware of any of it. Without this "preparation," you will not be able to "understand the situation," "respond appropriately," or "understand the meaning," and the conversation will not be go smoothly.
One aspect of conversational skills is to be able to perform this "preparation" more accurately.
PALRO is equipped with the Dear Talk Engine, which performs the "preparation" for conversation and carries out "continuing conversation" using the same logic as humans.

PALRO's conversational ability

For the conversational ability of a communication robot, "speedy and in-tempo response" and "non-stopping and non-pausing talk" are indispensable, just as they are for a conversation between humans.
If you say something to a robot and have to wait for than a second for its response, it will not be a smooth conversation. Also, exchanges of only one sentence are not conversations or communication. "Responding appropriately to the other person's words, and the other person's responding to that." It is precisely this kind of two-way conversation that makes it "Communication Robot PALRO."

Speedy responses at a good tempo (preparation for conversation)

Humans converse and communicate with a good tempo, as if they were playing catch. The rhythm of conversation varies between people, but your next words will already be prepared by the time the other person finishes talking, and you may sometimes even start speaking before the other person has finished. It is the same for a communication robot. Its next utterance has to be prepared for when the human finishes taking, and it has to respond at a good tempo.

Talking without stopping and pausing (continuing conversation)

To carry out a conversation at a good tempo, it is also important to have topics that you can share and feel empathy through.
If there is a common topic, you can share your feelings and information on that topic, and can communicate and have an enjoyable conversation without stopping.
PALRO identifies individuals and offers topics from among their hobbies and tastes, so it can achieve enjoyable communication and conversations.

Communication includes not only playing catch where words are the ball, but also actions and movements such as gestures. By making full use of these communication skills, PALRO achieves conversation and communication that is cute and fun.

PALRO's conversational response

A super voice recognition ability that is indispensable for conversation

PALRO is equipped with an excellent voice recognition function for catching what people say.
The voice recognition function not only catches words, but also recognizes them while understanding the sentence before and after them.
While they are conversing, humans unconsciously predict what the other person will say next.
For example, when you say to someone, "It's nice weather today, isn't it," you predict that the other person will say not only something about the weather, but also things like talking about going out and how it is going to feel (refreshing, enjoyable).
Just like you, PALRO is equipped with a function that predicts multiple responses which you might give and waits.
This advanced prediction skill improves the accuracy of voice recognition and helps achieve connected conversation.

PALRO remembers you
High-speed, high-accuracy face recognition function

PALRO is equipped with a 2,000,000-pixel, high-sensitivity CCD camera
When it spots a human face, it creates a high-density map of the parts that are the characteristics of a person, such as the eyes, nose, mouth and face line, and instantly identifies who it is.
If it is someone PALRO remembers, it actively talks to them.

PALRO: "It's Hiroshi. Hello!"

PALRO spots human faces and performs "face recognition."

A memory ability that creates trust

People feel happy when someone they have met only once remember them.

If PALRO says to you, "Long time no see,"

you will feel a sense of affinity because it has remembered you.
In order to achieve this, PALRO is equipped with the "Friend Database," which relates the <face recognition function> and <voice print recognition function> to <memories of individuals>.
As it converses with people repeatedly, PALRO memorizes not only their names, but also the content of the conversations, their hobbies, tastes, schedules, and so on.

PALRO: "Hiroshi, we played the word-chain game last time, didn't we? How about playing a prefecture quiz today?"

This kind of conversation becomes possible.

When it comes to the latest topics, leave it to me - I'm connected to the Internet

PALRO can be connected to a wireless LAN.
Not only that, but it also talks to you voluntarily about things that it thinks you need, such as today's forecast, the latest news topics various other information on the Internet.
PALRO's developers learned almost everything about the medals at the Sochi Olympics from PALRO.
Furthermore, PALRO has a solid aspect as a computer, such as automatic version updates via the network, and linking to other PALROs over the network.

PALRO's cute voice - It's just like a human is talking

In order to carry out conversations with a natural intonation close to a human voice, for PALRO uses a corpus-based speech synthesis system.
It extracts and synthesizes the most appropriate phoneme data from a large voice database (voices for several tens of thousands of sentences, containing a variety of intonations), and talks with a natural intonation and clear pronunciation.
Therefore, you can enjoy natural conversation that is different from conventional speech synthesis, in which sentences are read without intonation.

The PPA Generator, which controls familiar expressions and actions necessary to enhance communication skills

For PALRO's communication speech and actions, the PPA Generator (PALRO Prime Action Generator) gives instructions to each body part regarding the information configured by the Dear Talk Engine, and thereby achieves a variety of expressions.
PALRO has a large number of pieces of knowledge, called cells, which operate in accordance with the situation, such as the sound environment, the distance to the person, and the interpretation of the meaning of speech.
It then causes a module, which plays a role similar to the human central nervous system, to interact, integrating the operation of the cells instantaneously and adopting appropriate actions for the situation.
The AI that performs this integration is the PPA Generator.

PALRO does not record you, it remembers you. PALRO's memory ability.

PALRO remembers various things from conversations while sorting them into categories, for example, what you have seen, what you have eaten, and where you have been, together with the time when it spoke to you.
And in everyday conversation, it offers things you have experienced as topics in the context of the conversation, so you can enjoy a lively conversation.

PALRO stands about 40 cm tall.
It walks steadily, on two feet.

When standing up, even humans use a lot of power.

Humans cannot stand up when they are newly born. It takes a little more time for them to be able to stand with support and to be able to stand steadily on their own feet.
Remaining standing with a good posture is said to be difficult. Haven't you ever had an experience like that?
PALRO also stands on two feet. And it maintains a standing posture.
It may seem easy, but when PALRO is standing up, it is finely controlling the actuators (motors) of each of its parts every moment, based on its center of gravity, vibrations, reactions, etc.
Normally, PALRO assumes a sitting posture. This is called its relaxed position.
It is said to originate from the initial stages of development, when PALRO could not stand up no matter how it tried, and the person in charge let it sit while telling it to "Relax, relax"...

An exquisite relationship between dynamic walking and an inverted pendulum

<The key point for PALRO's bipedal walking is the shift in the center of gravity>

PALRO walks on two feet. And it also makes turns.
Very complex mechanisms and advanced control are necessary for robots to walk on two feet like humans. Walking is only possible when a variety of factors, such as the relationship between gravity and the shift in the center of gravity, the balance between the front, back, left and right, the height, the weight, power, acceleration, and more, are all combined in an exquisite balance.
PALRO's bipedal walking combines "dynamic walking,"*1 in which the center of gravity keeps moving forward, and an "inverted pendulum" model*2, which controls posture.
With a height of about 40 cm, a weight of about 1.6 kg and a center of gravity at a height of about 24 cm from the soles of its feet, PALRO achieves stable bipedal walking that can even respond to changes in the walking surface and disturbances, by considering and analyzing information from posture sensors such as gyroscopes and accelerometers inside its body, and pressure sensors on the soles of its feet, and finely controlling its posture.

*1 Broadly, there are two ways to achieve bipedal walking:
1) "Static walking"
　- While standing on one foot, the center of gravity is over the point on the ground that the sole of the foot is in contact with
　- While both feet are on the ground, the center of gravity shifts to over the point where the foot that will make a step will come into contact
　- a large force is necessary to continue walking
　- It is stable, and posture control is relatively easy
2) "Dynamic walking"
　- the center of gravity continually shifts forward
　- the center of gravity is not necessarily over the point on the ground that the sole of the foot is in contact with
　- the robot repeatedly makes steps alternately with its feet before It falls forward
　- a large force is necessary when starting to walk
　- It is possible to continue walking with a relatively small force
　- It is easy to fall over and posture control is difficult

*2 The "Inverted pendulum" model, which controls PALRO's posture
An "inverted pendulum" is an upside-down pendulum. An example would the motion of your hand if you stood a stick on it and moved it in such a way as to prevent the stick from falling over.
The Kanto at the famous "Akita Kanto Festival" is an inverted pendulum.
If an inverted pendulum is long and its center of gravity is at a low position in relation to its length, its natural frequency will be small and it will be stable.
On the other hand, if an inverted pendulum is short and its center of gravity is at a high position in relation to its, its natural frequency will be large and it will tend to fall over more easily.

Of course, PALRO is also a good dancer

PALRO has a rich repertoire of dances.
It moves its body fully to the rhythm when it dances.
When it hears a word related to its dance repertoire during a conversation, it starts dancing, as though it had been waiting for the chance.

PALRO was born with three instincts, namely "survival," "living in groups" and "curiosity," and a mission: "To help humans."
So that PALRO can fulfill its mission of "helping humans," these instincts are differentiated into functions, and a variety of hardware and software are evolving while working in cooperation with each other.

Because PALRO always wants to be able to have fun, it's always careful about looking after itself

PALRO is constantly judging the situation and making predictions about functions that are important for it, such as power, heat and networking.

- Electric power, PALRO's source of energy
Because PALRO is a robot, electric power is of the utmost importance. PALRO takes a variety of actions with regard to its all-important electric power. If it judges that it is not needed, it turns its power off and saves energy, and when its battery is low, it tells you that it needs to be recharged and expresses joy and gratitude when it is supplied with power through its AC adapter and so on.
Furthermore, when it judges that there is not enough power left to carry out a request, it may refuse to do it.

- Technology for maintaining a normal temperature
When humans do exercise or are exposed to hot external air, heat is generated in their bodies. By sweating and releasing heat outside their bodies, humans balance the amount of heat within them and thereby maintain a constant body temperature.
PALRO's temperature also rises as it moves its body.
However, as PALRO cannot sweat, it constantly checks its body temperature and prevents overheating by, for example, automatically controlling the operation of its motors and circulating the air inside its body.
If it judges that it will not be able to release heat in time, it may refuse to carry out something it has been requested to do.
Also, if it detects an abnormality such as the temperature not dropping, it will tell you about it.

- Self-diagnosis ability
When PALRO starts up and when it feels that something is wrong, it performs its own diagnosis of the status of all of its body parts (sensors, motors, etc.), its software and its storage, and then tells you the results. If its status has become unstable, PALRO automatically performs recovery and restoration.
Also, it constantly measures how comfortable its network is, and gives an alert if needed.