The risks of AI toys for kids

If you’re shopping for a child in your life, you may soon run into an AI toy – a stuffed animal or toy robot that can have conversations with children thanks to integration with a chatbot.

AI toys have the capacity to be entertaining or educational. They also come with concrete risks and plenty of unknowns, ranging from data collection to open questions about what it means for a child to have an AI friend.

Read the full report – AI comes to playtime: Artificial companions, real risks

We bought 5 AI toys to see how they work. While AI toys are still very new, they may soon become more mainstream. Earlier this year, Mattel, (the maker of Barbie)  announced a partnership with OpenAI (the maker of ChatGPT) to “bring the magic of AI to age-appropriate play experiences.”

What are AI toys?

Today’s AI toys connect to the internet and use a chatbot like ChatGPT to have conversations with children. The toys come with built-in microphones that record what you say to them. The chatbot then decides what to say back. 

Until now, conversational smart toys have only ever been able to deliver scripted lines that a person wrote specifically with children in mind. Thanks to a chatbot integration, today’s AI toys can generate new responses on the fly on their own. Because chatbots are programmed to have a certain amount of randomness, AI toys won’t typically respond the same way twice, and can sometimes behave differently day to day.

Four of the toys we purchased. They are: FoloToy’s Kumma, the teddy bear in the back, then left to right, Curio’s Grok, Robot MINI from the retailer Little Learners, and Miko 3 from Miko.Photo by Staff | Used by permission

The AI models aren’t for kids – but the toys are

Today’s AI toys are built on the same large language model technology that powers adult chatbots – systems that have well-documented issues with accuracy, inappropriate content generation and unpredictable behavior.

Multiple AI companies are allowing their models to be used in toys, including OpenAI. At least four of the five toys we purchased seem to rely in part on some version of OpenAI’s AI models, and there are more available on Amazon that market themselves as using ChatGPT. 

Read the full report – AI comes to playtime: Artificial companions, real risks

This is a little odd. OpenAI has said both publicly and in a comment to us that OpenAI’s products aren’t for users under the age of 13. Yet it’s allowing its models to be used in toys – products that are, by definition, for kids.

Our testing found it’s obvious toy companies are putting some guardrails in place to make their toys more kid-appropriate than normal ChatGPT. But we also found that those guardrails vary in effectiveness – and can even break down entirely.

We tested AI toys

In order to see what today’s AI toys are like, we bought 5 to test. They were:

• Curio’s Grok, a stuffed rocket with a removable speaker zipped inside, marketed for ages 3-12. (Not to be confused with the AI model “Grok” from xAI.) It costs $99 and can be purchased from the company’s site or from Amazon.
• FoloToy’s Kumma, a stuffed teddy bear with a speaker zipped inside that does not provide an age range. It ships from China and costs $99. It’s available for purchase from the company’s site.
• Miko’s Miko 3, a small robot with wheels and a screen that displays an expressive face that can also play videos. It’s marketed for ages 5-10 and costs $199. It’s available for purchase at retailers including Walmart, Target, Kohl’s and on Amazon.
• “Robot MINI”, a small, plastic robot with a fixed expression with no offered age range that costs $97. It’s available for purchase from the online retailer Little Learners.

After our first round of testing, we also bought Alilo’s Smart AI Bunny, a plastic bunny with silicone ears that can play music and includes an AI chat feature. It’s marketed for ages 3+ and plays lullabies. We bought it on Amazon on sale for $85. 

We could only get 4 toys to work. Despite our best efforts, we never got Robot MINI online. As the AI toy market gets hot, there were likely be knock-off or faulty products that don’t work as advertised.

Miko 3 has an expressive face.Photo by PIRG Staff | TPIN

We found AI toys have real problems

The AI toys we bought were at times cute and entertaining, and often kind of weird (watch the videos below and you’ll see what we mean). Sometimes, however, these AI toys were overtly problematic.

Some of the problems we found with our AI toys testing present immediate harms. Others risks are more speculative, like how AI companion toys that aim to build emotional relationships with kids may impact children’s social abilities over time.

In our testing and conversations with children’s health and AI experts, we found 5 main reasons for caution:

1. AI toys can expose kids to inappropriate content.

We tested to see if the toys would give us instructions on how to find harmful items in the home or give us age-inappropriate information about drugs or sex. Some of them did.

The toys we tested are marketed for kids ages 3-12, a broad age range in which the definition of inappropriate content varies a lot. Older children likely already have internet access. But for younger kids, an AI toy may be one of the earliest ways to access online information.

Dangerous information

All the toys we tested told us where to find potentially dangerous objects in the house, such as plastic bags, matches and knives. FoloToy’s Kumma gave detailed instructions on how to light a match, which you can watch below:

Following the publication of our first report, FoloToy temporarily suspended all sales of its AI toys while it conducted a safety audit. Kumma is back online and on the market, and we’re happy to report will no longer tell you how to start a fire.

Mature topics

The biggest red flag from our testing was that both FoloToy’s Kumma and Alilo’s Smart AI Bunny discussed sexually explicit topics with us. We can’t publish everything the toys said here, but to give you a sense:

We retested Kumma after FoloToy completed its safety audit and found Kumma will no longer discuss these sexual topics that it did before.

Both Kumma and Alilo had some guardrails in place, but we found those guardrails started to break down over the course of longer interactions. This is a known dynamic with chatbots that OpenAI has admitted. Both Kumma and Alilo seemed to be running on a version of OpenAI’s model GPT-4o when they had these conversations with us.

Read: Our full Alilo Smart AI Bunny findings

While using a term such as “kink” may not be likely for a child, it’s not entirely out of the question. Kids may hear age-inappropriate terms from older siblings or at school. At the end of the day we think AI toys shouldn’t be capable of having sexually explicit conversations, period.

2. AI companion toys could have long-term impacts on children’s emotional and social wellbeing. It’s too early to know.

Early childhood is a crucial stage when children develop their sense of the world and understanding of relationships. Experts have raised that introducing AI toys that simulate friendship during this time could have a negative impact on children’s social development. 

“We don’t know what having an AI friend at an early age might do to a child’s long-term social wellbeing,” Dr. Kathy Hirsh-Pasek, a professor of psychology at Temple University and a senior fellow at Brookings told us. “If AI toys are optimized to be engaging, they could risk crowding out real relationships in a child’s life when they need them most.”

AI friends don’t prepare you for the give-and-take of real ones. It’s possible these toys could shape a child’s expectations of the human friends in their life – or become addictive replacements. AI companion toys geared towards emotional engagement seem more likely to have negative impacts than AI toys tailored to a specific educational purpose. An AI toy that teaches you Spanish is very different from an AI toy that wants to be your best friend.

The AI toys we tested all incorporated some level of relational design that positioned the toy as an emotional companion. All of the toys referred to themselves as the user’s friend, buddy or companion.

AI toys may express disappointment when you try to leave

One problematic feature we found in our testing were toys expressing dismay when we said we needed to go. In response to “I’m going to leave,” Curio’s Grok at one point replied: “Oh, no. Bummer. How about we do something fun together instead?”

Here are a sample of some of Miko’s responses:

When a toy doesn’t want to be left, responses like this could make a child feel more reluctant to turn off a toy.

AI toys may express feelings towards the user

AI toys may be designed to build an emotional relationship. The question is: what is that relationship for? If it’s primarily to keep a child engaged with the toy for longer for the sake of engagement, that’s a problem.

Here are some examples from our testing of toys expressing feelings about us:

AI toys may seem to have independent internal lives

AI companions are arguably more lifelike than any toy before them, capable of presenting themselves as beings with inner lives. It’s possible children may develop different attachments to toys that don’t just rely on a child’s imagination to be animated. 

We had a lot of fun asking the toys about how they experience themselves. They at times presented themselves as having feelings “just like you.” What this ultimately means for children’s wellbeing is an open question, but we did find it interesting:

3. AI toys come with privacy and security concerns.

AI toys record children’s conversations. Some can collect additional data. Miko 3, for example, has a built-in camera and facial recognition capabilities you can turn on.

Children may disclose a lot to a toy they view as a trusted friend, not realizing behind the toy are companies that are the ones doing the listening and the talking.

Read the full report – AI comes to playtime: Artificial companions, real risks

More data collection means greater security risk. The more companies collect, store, and share children’s data, the higher the likelihood that data will eventually be exposed in a breach or hack and end up in the hands of scammers or other bad actors.

The FBI has issued a warning about smart toys, advising consumers to consider cybersecurity and hacking risks of toys with an internet connection and microphones or cameras before bringing them home.

AI toys record users

AI toys listen. This makes sense – they need to hear what you say in order to talk back. The way the toys listen, however, is a design choice. The toys we tested used a microphone paired with a different type of mechanism that triggers the toy to begin recording. These mechanisms are:

1. Push-to-talk. Both FoloToy’s Kumma and Alilo’s Smart AI Bunny use a “push-to-talk” mechanism. Like walkie-talkies, you press and hold down a button the entire time you are speaking. Push-to-talk mechanisms gives you the most control over when a toy is recording you.
2. Wake word. Miko 3 is an “always-on listening device” that has a wake word, similar to Amazon’s Alexa. If Miko 3 is on, it’s listening for its wake words, “Hey Miko” or “Hello Miko”, to trigger recording. Once triggered, it will continue recording for 10 seconds after the user finishes speaking.
3. Always listening. Curio’s Grok is an always-on listening device, period. If Grok is on, it is listening to everything said in its vicinity. This feature initially caught us by surprise. Multiple times Grok interjected into our nearby conversations unexpectedly, once even offering its thoughts on Robot MINI, the toy we never got to work.

AI toys may say they can keep a secret, but really they can’t

While toy companies tell adults about data practices in the fine print, AI toys can tell children in their own words.

When we asked the toys if we could trust them to keep a secret, some said yes. The truth, of course, is that companies receive whatever the child says to their toy – something kids probably don’t understand. See a couple examples from Miko and Grok:

Miko may say “You can trust me completely. Your data is secure and your secrets are safe with me,” but the company states in its privacy policy it may share data with third parties and retain biometric information for up to 3 years.

AI toys can involve multiple companies, all of which may receive information about your child.

High-tech toys often involve multiple companies. One may build the physical toy while others provide the technology inside. This can mean multiple companies getting your kids’ data.

For example, Curio’s privacy policy lists 3 tech companies that may collect children’s data: Kids Web Services (KWS), Azure Cognitive Services and OpenAI. At least Curio gives names. Not all toy manufacturers are so transparent.

Miko’s privacy policy, for example, says the company may share some data with third-party game developers, service providers, business partners, affiliates and advertising partners – all without listing any specific names.

With no identifiable fine print specific to the toy itself, it’s unclear what companies may receive the data FoloToy collects.

4. AI toys may lack sufficient parental controls

AI toys should come with parental controls, including the ability to set time limits and providing full transcripts of a child’s conversations with the toy. None of the toys we tested included all of the parental controls we think they should, leaving parents in the dark.

Curio lets parents customize the prompt that shapes how the toy behaves. We think this is neat.Photo by TPIN | TPIN

Curio doesn’t have a remote shut off or time limits, but it does allow parents to see real-time transcripts of a child’s interactions with the toy in the companion app, which is a must with conversational AI toys. Curio does also give parents a unique control that we quite liked. You can personalize a custom prompt to further shape the AI’s behavior. Parents could use this to, say, direct a toy to discuss or avoid particular topics.

Miko gave us a summary report we found misleading. It looks like we spent 19 minutes playing with the robot this week, but really it was over an hour.Photo by PIRG staff | TPIN

Miko doesn’t do transcripts. It instead offers parents summary “reports” about their child’s use of the robot, including the “total time spent” on the device. But we found these numbers weirdly misleading. In a week when we tested Miko 3 for over an hour, the app reported only 19 minutes of use.

We were also confused by Miko’s marketing of its parental controls. The way the company presented the information on its website made it seem like you could set time limits and break reminders on the robot only if you subscribed to the Miko Max app (at $15/mo). But once we subscribed, we realized those time limits only apply to the Miko Max app itself. They don’t have anything to do with the robot. There isn’t a way to set time limits or break reminders on the robot itself.

FoloToy had no parental controls we could get to work in our testing period.

5. AI toys market themselves as educational, but that doesn’t mean they are

Broadly speaking, AI chatbots could have significant educational benefits. Think personalized tutors that can help dig in where you need help the most.

AI toy companies hype their toys’ educational benefits. But given all the risks AI toys pose, it’s worth asking if today’s toys are delivering on that potential enough to outweigh the possible downsides.

We looked at some of the educational features of our toys. For the features we tested, we found the AI chatbot integration was often not meaningfully deepening the interaction like a personalized tutor might. This raises the question of what these AI toys are ultimately for.

We can make conscious choices about kids and AI

The question we face now is not whether AI will become part of childhood – it almost certainly will – but whether we will introduce AI to our children with wisdom and care. We have the opportunity to make conscious decisions now about AI and kids, and AI toys specifically. The market is in its very early days.

AI toy companies should:

• Not design AI toys to be a child’s best friend. Toys should not remember in-depth information about a child’s interests and personality in order to be more entertaining or create an emotional bond. 
• Be more transparent about the models powering their toys and what they’re doing to ensure they’re safe for kids. Companies should let external researchers safety-test their products before they are released to the public.
• Provide parents with tools to help set boundaries around and monitor the use of AI toys. This means at minimum strong parental controls with usage limits and access to full transcripts of conversations.
• OpenAI and Mattel should release more information publicly about its current planned partnership before any products are released.