Tuesday, February 2, 2016

Reverse engineer smart meter data packets (Part I)

I received an Energy Bridge from my electric utility, DTE.  DTE has a great smartphone app but I wanted a way to intercept the data from the Energy Bridge to monitor the real-time data without the smartphone.

I had previously opened up the Energy Bridge and found an XBee SMT board.  I assumed the XBee was communicating to the smart meter and gating the data to the ethernet connection.


As a first step to intercept smart meter data, I put a shared internet connection between the Energy Bridge and the Internet and recorded TCP packets using Wireshark. The ethernet route I used did not allow the Energy Bridge to connect to the Internet.  Although I could see packets to/from relaypilot.vectorform.com 208.81.180.78, I could not be sure the packets were reporting good data from the smart meter.  I have some ideas how to improve this but they will wait for Part II.
 

Since the TCP packet sniffing turned out to be slower than I had hoped, I opened up the Energy Bridge again to poke around to see if I could tap in to the serial data between the XBee board and the microprocessor that was talking to the Internet.  First, I found the schematic of the XBee board online and looked for the DOUT and DIN pins, that are pins 3 and 4.


Those pins are right next to the tricolor LED and they have traces running from them.  So, it looks like they are connected to something.



So, I added a couple of wire taps to pins 3, 4, and 13 for ground.

 

I measured the signals with an oscilloscope and confirmed that it looks like serial data should look.


I looked at the shortest pulse duration at a given state to determine the baud rate.  The shortest pulse was ~8.75us wide.  I calculated 1 bit / (8.75E-6 sec) = 114285.  That is close enough to 115200 bit/s or 115200 baud.

Once I knew the baud rate, I hooked up the wire leads to an Arduino and wrote a little script to read the data from the XBee Pro and transmit them to the PC.  I could see 8 bits in between the start bits and then assumed no parity and 1 stop bit.

The first few bytes of the data packet in hex format looked like

The 7E, 0, 7, and 8B were constant.
The 18 is a counter running from 00 to FF incrementing by 1 in each packet.
The five 0s were constant.
The 5E looks like a counter running from FF to 00.  It goes down by one in each packet.

I also manually decoded a few bytes from the oscilloscope traces to make sure the bitbang serial in the Arduino was working correctly.

The full packet is about 43 bytes long.

I was not entirely successful in intercepting and decoding the smart meter data yet but I can see the consumption data in the 43 bytes.  It moves in a similar way as the real-time data in the smartphone app.

In Part II I will make a data logger to look at the packets over time and hopefully correlate the serial data with the TCP packets.

Stay tuned!

18 comments:

  1. [your bridge ip]/instantaneousdemand

    ReplyDelete
  2. +leim9109
    Thanks for that! That's all I was trying to do and you saved me a lot of time. Can you tell where you learned about that?

    ReplyDelete
    Replies
    1. There's a Mac app called Charles. It's an HTTP proxy server that you can hook your phone up to. It also allows you to put a self-signed SSL root cert on your phone so that you can actually observe unencrypted HTTPS traffic.
      After playing around I figured out that relaypilot.vectorform.com is just a dumb proxy to your local device, and that all traffic you observe between your phone and the external URL can easily just be directed right at your local device's IP.
      Another endpoint I've found with some metadata is /status
      Still looking for the daily summation/graph plot though

      Delete
  3. just tried /history .. its downloading a 3.1MB file..

    ReplyDelete
    Replies
    1. I just tried this, and the HTTP header says the file is 3592800 bytes, yet both IE and Chrome download a file that is actually 0 bytes. Strange!

      Delete
    2. if you let it download, before completing, copy the temp file in your download dir.. there's data there

      Delete
    3. No luck here for me. I watch the network activity to my computer, and it's not downloading anything on the order of megabytes. And upon saving the file, it's still 0 bytes as before. Also tried cURL and I also get no data back.

      Delete
    4. I am able to see data with this short python code.

      import urllib

      webFile = urllib.urlopen('http://192.168.1.124/history')
      for _ in range(2000): #2000 bytes short
      readbyte = webFile.read(1)
      print('0x'+str(ord(readbyte)))

      webFile.close()

      Delete
    5. Guess I'm just out of luck... :) I changed the IP address to mine and here's what I got:
      File "../snippet.py", line 6, in
      print('0x'+str(ord(readbyte)))
      TypeError: ord() expected a character, but string of length 0 found

      Delete
  4. Really appreciate this in-depth technical article! It was a fascinating read as I'm also interested in how to extract data from the energy bridge. Looking forward to Part 2!

    ReplyDelete
  5. +Steve, I put Part 2 on the back-burner when I got the note from +leim9109 saying that the /instantaneousdemand endpoint had the information I was looking for. I wrote a short python script to get that page and collect the data from it. I could post the code, if that helps you.

    I saw the same effect with the /history endpoint. If I have time, I will have a look at the data stream to see if it seems useful.

    ReplyDelete
    Replies
    1. data looks valid, you can see the way its formatted that their is what appears to be kW data, but still trying to figure out encoding. Really wish we could get a voltage and/or current reading rather than just kW!

      Delete
  6. Replies
    1. got the info from this patent: www.google.com/patents/US8832428

      Delete
    2. This output varies wildly and is a huge number. Mine reports 147334558202.9 kWh. And the number sometimes decreases. Not sure what it's reporting, but an interesting find.

      Delete
  7. FYI, I put the data from /history into notepad++ with the HEX-Editor plugin. I then set it to 6columns wide. Immediately it was obvious that I had the correct number of columns because columns (0 based) 3 and 4 turned into a counter while column 5 was a constant. I still don't understand the entire format, but hopefully this helps someone else make some more progress reverse engineering it.

    ReplyDelete
  8. Looks like there may be a new version of the EB device that doesn't accept these IP queries. I got mine last week and it has the LED on the front edge, and the power comes in via USB jack. I can see the data via the app but the IP requests are refused or time out.

    ReplyDelete
    Replies
    1. For v2 try port 8888, endpoint /zigbee/se/instantaneousdemand

      Delete