By Kashmir Hill and Surya Mattu, for Gizmodo
In December, I converted my one-bedroom apartment in San Francisco into a “smart home.” I connected as many of my appliances and belongings as I could to the internet: an Amazon Echo, my lights, my coffee maker, my baby monitor, my kid’s toys, my vacuum, my TV, my toothbrush, a photo frame, a sex toy, and even my bed.
“Our bed?” asked my husband, aghast. “What can it tell us?”
“Our breathing rate, heart rate, how often we toss and turn, and then it will give us a sleep report each morning,” I explained.
“Sounds creepy,” he said, as he plopped down on that bed, not bothered enough to relax instead on our non-internet-connected couch.
I soon discovered that the only thing worse than getting a bad night’s sleep is to subsequently get a report from my bed telling me I got a low score and “missed my sleep goal.” Thanks, smart bed, but I know that already. I feel like shit.
Why? Why would I do this? For convenience? Perhaps. It was appealing to imagine living like the Beast in the Disney movie, with animated objects around my home taking care of my every need and occasionally serenading me. As a result of the apartment upgrade, I could watch what was happening in the house when we weren’t there. I could use voice commands to turn on the lights, coffee maker, and music. I could exchange voice messages with our toddler (and her caregiver) through a toy. I got reminders from my toothbrush to brush and tips on how best to do it. If I got cold in the night, my bed could warm me up. And I no longer had to push a vacuum around the house, instead activating a robot to do it for me with a press of a smartphone button.
Thanks to the Internet of Things, I could live in my very own tech-mediated Downton Abbey. That’s the appeal of smart homes for most people, and why they are supposed to be a $27 billion market by 2021. But that wasn’t my primary motivation. The reason I smartened up my house was to find out whether it would betray me.
I installed internet-connected devices to serve me, but by making the otherwise inanimate objects of my home “smart” and giving them internet-connected “brains,” I was also giving them the ability to gather information about my home and the people in it. The company that sold me my internet-connected vacuum, for example, recently said that it collects a “rich map of the home” and plans to one day share it with Apple, Amazon, or Alphabet, the three companies that hope to dominate the smart home market. Once I made my home smart, what would it learn and whom would it tell?
One person I knew it would be leaking to was my colleague, Surya Mattu, because he built a special router to monitor the devices monitoring me.
Yes, I am basically Kashmir’s sentient home. Kashmir wanted to know what it would be like to live in a smart home and I wanted to find out what the digital emissions from that home would reveal about her. Cybersecurity wasn’t my focus. (I wasn’t interested in hacking her sex toy or any of her other belongings.) Privacy was. What could I tell about the patterns of her and her family’s life by passively gathering the data trails from her belongings? How often were the devices talking? Could I tell what the people inside were doing on an hourly basis based on what I saw?
Using a Raspberry Pi computer, I built a router with a Wi-Fi network called “iotea” (I’m not very good at naming things) to which Kashmir connected all of her devices, so that I could capture the smart home’s network activity. In other words, I could see every time the devices were talking to servers outside the home.
I had the same view of Kashmir’s house that her Internet Service Provider (ISP) has. After Congress voted last year to allow ISPs to spy on and sell their customers’ internet usage data, we were all warned that the ISPs could now sell our browsing activity, or records of what we do on our computers and smartphones. But in fact, they have access to more than that. If you have any smart devices in your home—a TV that connects to the internet, an Echo, a Withings scale—your ISP can see and sell information about that activity too. With my “iotea” router I was seeing what information about Kashmir and her family that Comcast, her ISP, could monitor and sell.
There was a lot to see. Since the router was set up at the beginning of December, there hasn’t been a single hour of complete silence from it, even when there was no one in the house.
After a week of living in my newly smartened home, I could tell why the Beast was always in such a bad mood: The animate objects in my home were becoming a constant source of annoyance. I thought this was going to be a story about privacy, but instead I was finding out how infuriating it is to live in a janky smart home.
Our 1970s apartment building did not offer enough electrical outlets for this 2018 smart home, so we had power strips and outlet expanders everywhere, to the point where I was worried I was going to spark a fire and burn our smart home down. (This actually might have been cathartic.)
I had to download 14 different apps to my phone to control everything, which meant creating an account for each one of those apps. (Yes, my coffeemaker has a log-in and a very long terms of service agreement.) After setting them up, I thought I’d be able to control all the devices by issuing voice commands to Alexa via the Echo—the smart speaker that we’ve been using for the last year as a glorified timer and music player— but this did not go as well as I had hoped.
It took at least two hours to get all of our Christmas lights plugged into smart plugs from WeMo and Sonoff, and then to get those plugs online with their apps, and then to get those apps to talk to the Alexa app. The first night I said, “Alexa, turn on the Christmas lights,” they all turned on in sparkly synchronicity and it was magical. But one day, Alexa stopped recognizing “Christmas lights” as a group, and I could not figure out how to fix it, so I had to ask Alexa each night to turn off the lights one-by-one. (“Turn off kitchen Christmas lights.” “Turn off living room Christmas lights.” “Turn off bookcase lights.”) This was way more annoying than turning them off manually. The fantasy of the smart home is that it will save us time and effort, but the friction involved in getting various devices from different companies to work together meant that many things took longer to do.
I could tell when the lights were being turned on and off. And even when Kashmir’s family wasn’t using them or weren’t home, those smart plugs were constantly talking to their home servers. One of them checked in erratically, around four times per day at random intervals. The other smart plug, that promised insights about how much electricity you were using, was far more chatty, pinging home almost every hour.
Smart coffee was also a world of hell. The Brewgenie Smart Coffeemaker I ordered first was “smart” in that it had Bluetooth connectivity, so I could use its custom app on my phone to make it run a pot of coffee, but it wasn’t compatible with Echo, so I couldn’t say, “Alexa, make me coffee.” To remedy this, I ordered the Behmor Connected Customizable Temperature Control Coffee Maker which promised Alexa compatibility. I came to regret this while setting up my Eight Sleep Tracker—a sensor layer that went over my mattress that could track sleep, warm the bed, and connect to “any wifi-enabled device in your house.” Its instruction manual, along with dire warnings about possibly setting my bed on fire, informed me that it could automatically brew me coffee when it sensed I was waking up … but only if I had a WeMo coffeemaker.
I refused to order a third coffee maker for my smart home.
I made do with the Behmor, which makes incredible coffee but had a prickly relationship with Alexa. Each morning, my husband and I begged Alexa to put the coffee on. She would only react to a very specific phrasing of the request and then would only do so sometimes.
“Ask Behmor (pronunciation: Be-more) to brew me coffee.”
“Behmor,” she would respond. “A passion for coffee. How can I help you?”
“Brew me coffee.”
“I don’t understand,” she would respond. This was especially aggravating for two caffeine-addicted people who had not yet had their coffee. Sometimes we would keep rephrasing the question until she got it, but more often, one of us would just get up, walk to the kitchen and press the button on the coffeemaker rather than doing it the “smart” way.
And that’s when I ran into the next issue: the Withings Home Wi-Fi Security Camera with Air Quality Sensors that I had set up in our living room. When the camera detects motion or noise, it automatically records what it’s seeing. That’s great if you’re worried about break-ins or how people treat your kid when you’re not around, but not great for protecting the intimacy of your home. The day after I set it up, it caught me walking through the living room naked, resulting in the very first nude video of me (that I know about), which was promptly sent to the cloud and saved to the Home Cam app on my phone. This appears to be a common problem for the smart home set.
The camera was constantly sending huge amounts of data, which makes sense given it’s sending video. But good news—it was all encrypted, so someone monitoring your network will not get access to what the camera sees, including nude videos.
“You set up a camera!??” my husband asked, horrified. Four clicks deep into the app, I found out it keeps everything it records by default for two days but that you have to sign up for a “premium account” if you want to save anything longer.
“It deletes the video after two days,” I told my husband.
“How comforting,” he replied, without looking up from his phone, his voice doing the eye roll for him.
When I went away to Spain for a few days for work in December, I was looking forward to being able to check in on my family via the camera. But eight hours after I left, during my layover in Toronto, I opened the app on my phone and it reported back that the camera was offline; the most recent video was from that morning, showing my daughter in my husband’s arms in the kitchen.
“The camera’s not working,” I texted my husband.
He replied that he had unplugged it.
“It was staring at me while I made coffee,” he texted back.
I told him I needed it plugged in for the story to monitor the data flows and he said he would do it. But as the days passed in Spain it remained offline and I stopped bothering him about it. (The acceptance of complete surveillance had not been part of our marriage vows.)
When I got back from Europe, I moved the camera to the nursery. It would be more useful there, and thankfully our 1-year-old daughter is too young to care about privacy invasions. The downside of this was that the move somehow screwed up the camera’s power cord, so it started randomly losing its connection to the router; when that happened, the camera would glow bright orange to inform me that it was offline. This is not a great design for a camera billed as a baby monitor, as the bright orange light in the otherwise dark nursery would then wake my daughter.
The whole episode reinforced something that was already bothering me: Getting a smart home means that everyone who lives or comes inside it is part of your personal panopticon, something which may not be obvious to them because they don’t expect everyday objects to have spying abilities. One of the gadgets—the Eight Sleep Tracker—seemed aware of this, and as a privacy-protective gesture, required the email address of the person I sleep with to request his permission to show me sleep reports from his side of the bed. But it’s weird to tell a gadget who you are having sex with as a way to protect privacy, especially when that gadget is monitoring the noise levels in your bedroom.
The Eight Sleep tracker sent its data through a nonstandard port that I wasn’t monitoring, so I wasn’t able track what was happening in the bedroom.
One of the decisions we made early in the process was not to break the encryption on the devices in Kashmir’s house. We wanted them in their normal, off-the-shelf state. We’ll leave the encryption-breaking to the growing number of computer-science academics who are building their own smart homes to try to figure out the future of corporate surveillance.
When the data streams were unencrypted, which was the case every time someone watched Hulu on the Vizio smart TV, I could see exactly what was being sent. When they were encrypted, as the majority of the data turned out to be, I could see only the metadata—the volume of data being sent and to where, which is like seeing the outside of an envelope but not being able to read the letter inside. But sometimes, metadata is the message. I know, for example, when the family wakes up, because the Amazon Echo usually starts playing songs from Spotify between 6 a.m. and 8 a.m., even if I don’t know which songs. I also know that Kashmir likes to use the Alexa Sounds app—which loops ambient sounds such as rain, oceans, and fireplaces—between 6 p.m. and 8 p.m., which is when she puts her 1-year-old daughter to sleep.
It turns out that how we interact with our computers and smartphones is very valuable information, both to intelligence agencies and the advertising industry. What websites do I visit? How long do I actually spend reading an article? How long do I spend on Instagram? What do I use maps for? The data packets that help answer these questions are the basic unit of the data economy, and many more of them will be sent by people living in a smart home.
The thing in the smart home that most fascinated me, because of its value to advertisers, was the television. In Kashmir’s house, it doesn’t get turned on every day, but when it does, it’s usually between 8pm and midnight. A typical day of TV viewing looks like this in the data:
An outlier to this trend was Christmas Day, when, it looks like, the television was on throughout the day. I couldn’t tell if it was for watching a show or streaming music, but there was a consistent stream of activity.
I didn’t like it, but the TV was tuned to basketball for most of the day on Christmas. Thanks for the reminder, Surya/smart home!
Similarly, it looks like 2017 ended with a low-key TV night:
Okay, yes, we stayed in on New Year’s Eve. Don’t shame me. We have a baby! We watched Phantom Thread on DVD. There was no internet involved. Why do you even know that?
Even though the “smart” part of the TV wasn’t being used that night, it was still sending data about its use.
When the TV is on, it’s usually tuned to Netflix or Hulu. I couldn’t see what they watched on Netflix because Netflix encrypts streams. But I discovered that Netflix doesn’t encrypt images, so I could see the shows being recommended to them, which is revealing in that it shows what Netflix thinks they should like:
Meanwhile, Hulu sends its traffic unencrypted, so I could spy on exactly what they watched, like when Netflix spied on A Christmas Prince addicts. I was able to figure out which nights they were binge-watching Difficult People on Hulu (which, based on the traffic, seems to be one of their favorite shows).
Here’s what the reward TV request looked like; it included the name of the episode in plain text:
And this is the scorecard research tracking request:
Hulu’s decision not to encrypt streams means people who watch its shows can have their viewing habits tracked by third parties. (Hulu did not respond to a request for comment about its lack of encryption.)
“Surya wants to know who is binging on Difficult People,” I told my husband, the responsible party.
“Our TV is watching us?” he asked, surprised, even though he’d been the one to connect the TV to the router surveilling us. “Wow, I forgot.”
This must be what it’s like to be in a documentary or in a reality TV show. The cameras eventually move to the periphery of your vision and then disappear altogether. If homes become sentient, and it becomes the norm that activity in them is captured, measured, and used to profile us, all of the anxiety you currently feel about being tracked online is going to move into your living room.
Talking to the human who actually got to see and analyze my smart home’s activity made me realize just how deeply uncomfortable it is to have that data pooled somewhere.
After two months of data collection, I was able to pick up a bunch of insights into the Hill household—what time they wake up, when they turn their lights on and off, when their child wakes up and falls asleep—but the weirdest one for me personally was knowing when Kashmir brushes her teeth. Her Philips Sonicare Connected toothbrush notifies the app when it’s being used, sending a distinctive digital fingerprint to the router. While not necessarily the most sensitive information, it made me imagine the next iteration of insurance incentives: Use a smart toothbrush and get dental insurance at a discount!
The larger pattern that emerged about the smart home was that all of the devices phoned home daily, even if they hadn’t been used, telling the companies that made them, “Hey. I’m still here. I’ve still got power. Have any updates for me?”
An exaggerated version of this was seen in the Echo and Echo Dot, which were in constant communication with Amazon’s servers, sending a request every couple of minutes to http://spectrum.s3.amazonaws.com/kindle-wifi/wifistub-echo.html. Even without the “Alexa” wake word, and even when the microphone is turned off, the Echo is frequently checking in with Amazon, confirming it is online and looking for updates. Amazon did not respond to an inquiry about why the Echo talks to Amazon’s servers so much more frequently than other connected devices.
The funniest “conversation” that happened over the two months was a week in January when Kashmir was out of town. I could tell the house was empty because the amount of data being sent out slowed, but her home remained active despite being empty. All of her devices, from her TV to her WeMo smart plugs, continued to send out information every day. But the Behmor Connected coffee machine seemed to miss its inhabitants because it completely freaked out. The coffee machine, which typically pings its servers a few times a day, phoned home over 2,000 times on Thursday, January 24th.
A comparison in data:
When we asked Behmor about it, they had to check with Dado Labs, the third-party that makes their devices smart. “We pretty much stay at arm’s length. We don’t even have our customers’ email addresses for log-in,” said Joe Behm from Behmor. Dado Labs reported that a server was down that day, meaning the coffee machine just kept calling and calling a line that wouldn’t pick up.
Overall, my takeaway is that the smart home is going to create a new stream of information about our daily lives that will be used to further profile and target us. The number of devices alone that are detected chattering away will be used to determine our socioeconomic status. Our homes could become like internet browsers, with unique digital fingerprints, that will be mined for profit just like our daily Web surfing is. If you have a smart home, it’s open house on your data.
I was looking forward to the end of the experiment and getting rid of all the internet-connected devices I’d accumulated, as well as freeing up the many electrical outlets they’d been hogging. The Internet of Shit Twitter account is right. Smart homes are dumb.
But the truth is that my house will remain smart, just like yours may be. Almost every TV on the market now is connected—because otherwise how do you Netflix and chill?—and over 25 million smart speakers were sold last year alone, with Apple soon to release its version, the HomePod, meaning a good percentage of American homes have or will have an internet-connected assistant waiting patiently for someone in the house to say their wake word.
This ability was turned on by default for the 11 million smart TVs Vizio had sold to consumers since 2014. Our TV had likely been tracking everything we watched until 2017, when the Federal Trade Commission and the New Jersey Attorney General sued the company for unfair and deceptive practices. Vizio wound up settling the suit for $2.2 million and turning off the tracking unless the TV owners chose to turn it on (in the pursuit of well-targeted ads?).
We may already be past the point of no return: internet functionality is a necessary component for the operation of many devices in our home, and it increasingly gets added on as a feature even when it’s not strictly necessary. So when you get a sex toy, it connects to the internet just in case your partner wants to be able to please you from afar. But once the data is going over the wires, companies can’t seem to resist peeking at it, no matter how sensitive it is. The Canadian company We-Vibe wound up paying millions of dollars to its customers because its internet-connected sex toy was collecting stats about their orgasms, for “market research.”
What our experiment told us is that all the connected devices constantly phone home to their manufacturers. You won’t be aware these conversations are happening unless you’re technically savvy and monitoring your router like we did. And even if you are, because the conversations are usually encrypted, you won’t be able to see what your belongings are saying. When you buy a smart device, it doesn’t just belong to you; you share custody with the company that made it.
That’s not just a privacy concern. It also means that those companies can change the product you bought after you buy it. So your smart speaker can suddenly become the hub of a social network, and your fancy smart scale can have one of its key features taken away in a firmware update.
There was one connected device I came to love as a result of the experiment: the iRobot Roomba 890 vacuum. (It doesn’t make a map of my home, by the way, because it’s not the newest 900 series.) It terrified my daughter, who cried and ran to me every time the low, round robot rolled in as if the clown from Stephen King’s It had just entered the room, but I loved it. The Roomba did what robots do best: easy, boring, monotonous work. But it didn’t do so independently; like my other “smart” products, it used its internet connectivity to pop up notifications on my phone. Being smart meant it could nag me:
“Roomba requires your attention: Your Roomba is stuck.”
“Roomba requires your attention: Your Roomba’s bin is full.”
“Roomba’s cleaning job was canceled.”
I thought the house would take care of me but instead everything in it now had the power to ask me to do things. Ultimately, I’m not going to warn you against making everything in your home smart because of the privacy risks, although there are quite a few. I’m going to warn you against a smart home because living in it is annoying as hell.
Data analysis contributed by Dhruv Mehrotra