This project intends to provide a complete description and re-implementation of the WhatsApp Web API, which will eventually lead to a custom client. WhatsApp Web internally works using WebSockets; this project does as well.
Trying it out
With Nix
There's no need to install or manage python and node versions, the file
shell.nix defines an environment with all the dependencies included to run
this project.
There's an .envrc file in root folder that is called automatically when
cding (changing directory) to project, if program direnv is installed along
with nix you should get an output like this:
If you don't use direnv or just want to manually get into the build
environment do:
nix-shell
in the project root
Bare metal
Before you can run the application, make sure that you have the following software installed:
Node.js (at least version 8, as the asyncawait syntax is used)
Python 2.7 with the following pip packages installed:
websocket-client and git+https://github.com/dpallot/simple-websocket-server.git for acting as WebSocket server and client.
curve25519-donna and pycrypto for the encryption stuff.
pyqrcode for QR code generation.
protobuf for reading and writing the binary conversation format.
Note: On Windows curve25519-donna requires Microsoft Visual C++ 9.0 and you need to copy stdint.h into C:\Users\YOUR USERNAME\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0\VC\include.
Before starting the application for the first time, run npm install -f to install all Node and pip install -r requirements.txt for all Python dependencies.
Lastly, to finally launch it, just run npm start on Linux based OS's and npm run win on Windows. Using fancy concurrently and nodemon magic, all three local components will be started after each other and when you edit a file, the changed module will automatically restart to apply the changes.
Reimplementations
JavaScript
A recent addition is a version of the decryption routine translated to in-browser JavaScript. Run node index_jsdemo.js (just needed because browsers don't allow changing HTTP headers for WebSockets), then open client/login-via-js-demo.html as a normal file in any browser. The console output should show decrypted binary messages after scanning the QR code.
adiwajshing created Baileys, a Node library that implements the WhatsApp Web API.
ndunks made a TypeScript reimplementation at WaJs.
Python
p4kl0nc4t created kyros, a Python package that implements the WhatsApp Web API.
The project is organized in the following way. Note the used ports and make sure that they are not in use elsewhere before starting the application.
Login and encryption details
WhatsApp Web encrypts the data using several different algorithms. These include AES 256 CBC, Curve25519 as Diffie-Hellman key agreement scheme, HKDF for generating the extended shared secret and HMAC with SHA256.
Starting the WhatsApp Web session happens by just connecting to one of its websocket servers at wss://w[1-8].web.whatsapp.com/ws (wss:// means that the websocket connection is secure; w[1-8] means that any number between 1 and 8 can follow the w). Also make sure that, when establishing the connection, the HTTP header Origin: https://web.whatsapp.com is set, otherwise the connection will be rejected.
Messages
When you send messages to a WhatsApp Web websocket, they need to be in a specific format. It is quite simple and looks like messageTag,JSON, e.g. 1515590796,["data",123]. Note that apparently the message tag can be anything. This application mostly uses the current timestamp as tag, just to be a bit unique. WhatsApp itself often uses message tags like s1, 1234.--0 or something like that. Obviously the message tag may not contain a comma. Additionally, JSON objects are possible as well as payload.
Logging in
To log in at an open websocket, follow these steps:
Generate your own clientId, which needs to be 16 base64-encoded bytes (i.e. 25 characters). This application just uses 16 random bytes, i.e. base64.b64encode(os.urandom(16)) in Python.
Decide for a tag for your message, which is more or less arbitrary (see above). This application uses the current timestamp (in seconds) for that. Remember this tag for later.
The message you send to the websocket looks like this: messageTag,["admin","init",[0,3,2390],["Long browser description","ShortBrowserDesc"],"clientId",true].
Obviously, you need to replace messageTag and clientId by the values you chose before
The [0,3,2390] part specifies the current WhatsApp Web version. The last value changes frequently. It should be quite backwards-compatible though.
"Long browser description" is an arbitrary string that will be shown in the WhatsApp app in the list of registered WhatsApp Web clients after you scan the QR code.
"ShortBrowserDesc" has not been observed anywhere yet but is arbitrary as well.
After a few moments, your websocket will receive a message in the specified format with the message tag you chose in step 2. The JSON object of this message has the following attributes:
status: should be 200
ref: in the application, this is treated as the server ID; important for the QR generation, see below
ttl: is 20000, maybe the time after the QR code becomes invalid
update: a boolean flag
curr: the current WhatsApp Web version, e.g. 0.2.7314
time: the timestamp the server responded at, as floating-point milliseconds, e.g. 1515592039037.0
QR code generation
Generate your own private key with Curve25519, e.g. curve25519.Private().
Get the public key from your private key, e.g. privateKey.get_public().
Obtain the string later encoded by the QR code by concatenating the following values with a comma:
the server ID, i.e. the ref attribute from step 4
the base64-encoded version of your public key, i.e. base64.b64encode(publicKey.serialize())
your client ID
Turn this string into an image (e.g. using pyqrcode) and scan it using the WhatsApp app.
Requesting new ref for QR code generation (not implemented)
You can request up to 5 new server refs when previous one expires (ttl).
Do it by sending messageTag,["admin","Conn","reref"].
The server responds with JSON with the following attributes:
status: should be 200 (other ones: 304 - reuse previous ref, 429 - new ref denied)
ref: new ref
ttl: expiration time
Update your QR code with the new ref.
After scanning the QR code
Immediately after you scan the QR code, the websocket receives several important JSON messages that build up the encryption details. These use the specified message format and have a JSON array as payload. Their message tag has no special meaning. The first entry of the JSON array has one of the following values:
Conn: array contains JSON object as second element with connection information containing the following attributes and many more:
battery: the current battery percentage of your phone
browserToken: used to logout without active WebSocket connection (not implemented yet)
clientToken: used to resuming closed sessions aka "Remember me" (not implemented yet)
phone: an object with detailed information about your phone, e.g. device_manufacturer, device_model, os_build_number, os_version
platform: your phone OS, e.g. android
pushname: the name of yours you provided WhatsApp
secret (remember this!)
serverToken: used to resuming closed sessions aka "Remember me" (not implemented yet)
wid: your phone number in the chat identification format (see below)
Stream: array has four elements in total, so the entire payload is like ["Stream","update",false,"0.2.7314"]
Props: array contains JSON object as second element with several properties like imageMaxKBytes (1024), maxParticipants (257), videoMaxEdge (960) and others
Key generation
You are now ready for generating the final encryption keys. Start by decoding the secret from Conn as base64 and storing it as secret. This decoded secret will be 144 bytes long.
Take the first 32 bytes of the decoded secret and use it as a public key. Together with your private key, generate a shared key out of it and call it sharedSecret. The application does it using privateKey.get_shared_key(curve25519.Public(secret[:32]), lambda a:a).
Extend sharedSecret to 80 bytes using HKDF. Call this value sharedSecretExpanded.
This step is optional, it validates the data provided by the server. The method is called HMAC validation. Do it by first calculating HmacSha256(sharedSecretExpanded[32:64], secret[:32] + secret[64:]). Compare this value to secret[32:64]. If they are not equal, abort the login.
You now have the encrypted keys: store sharedSecretExpanded[64:] + secret[64:] as keysEncrypted.
The encrypted keys now need to be decrypted using AES with sharedSecretExpanded[:32] as key, i.e. store AESDecrypt(sharedSecretExpanded[:32], keysEncrypted) as keysDecrypted.
The keysDecrypted variable is 64 bytes long and contains two keys, each 32 bytes long. The encKey is used for decrypting binary messages sent to you by the WhatsApp Web server or encrypting binary messages you send to the server. The macKey is needed to validate the messages sent to you:
encKey: keysDecrypted[:32]
macKey: keysDecrypted[32:64]
Restoring closed sessions (not implemented)
After sending init command, check whether you have serverToken and clientToken.
If so, send messageTag,["admin","login","clientToken","serverToken","clientId","takeover"]
The server should respond with {"status": 200}. Other statuses:
401: Unpaired from the phone
403: Access denied, check tos field in the JSON: if it equals or greater than 2, you have violated TOS
405: Already logged in
409: Logged in from another location
Resolving challenge (not implemented)
When using old or expired serverToken and clientToken, you will be challenged to confirm that you still have valid encryption keys.
The challenge looks like this messageTag,["Cmd",{"type":"challenge","challenge":"BASE_64_ENCODED_STRING=="}]
Decode challenge string from Base64, sign it with your macKey, encode it back with Base64 and send messageTag,["admin","challenge","BASE_64_ENCODED_STRING==","serverToken","clientId"]
The server should respond with {"status": 200}, but it means nothing.
After solving challenge your connection should be restored.
Logging out
When you have an active WebSocket connection, just send goodbye,,["admin","Conn","disconnect"].
When you don't have such connection (for example your session has been taken over from another location), sign your encKey with your macKey and encode it with Base64. Let's say it is your logoutToken.
Send a POST request to https://dyn.web.whatsapp.com/logout?t=browserToken&m=logoutToken
Remember to always clear your sessions, so sessions list in your phone will not grow big.
Validating and decrypting messages
Now that you have the two keys, validating and decrypting messages the server sent to you is quite easy. Note that this is only needed for binary messages, all JSON you receive stays plain. The binary messages always have 32 bytes at the beginning that specify the HMAC checksum. Both JSON and binary messages have a message tag at their very start that can be discarded, i.e. only the portion after the first comma character is significant.
Validate the message by hashing the actual message content with the macKey (here messageContent is the entire binary message): HmacSha256(macKey, messageContent[32:]). If this value is not equal to messageContent[:32], the message sent to you by the server is invalid and should be discarded.
Decrypt the message content using AES and the encKey: AESDecrypt(encKey, messageContent[32:]).
The data you get in the final step has a binary format which is described in the following. Even though it's binary, you can still see several strings in it, especially the content of messages you sent is quite obvious there.
Binary message format
Binary decoding
The Python script backend/decoder.py implements the MessageParser class. It is able to create a JSON structure out of binary data in which the data is still organized in a rather messy way. The section about Node Handling below will discuss how the nodes are reorganized afterwards.
MessageParser initially just needs some data and then processes it byte by byte, i.e. as a stream. It has a couple of constants and a lot of methods which all build on each other.
Constants
Tags with their respective integer values
LIST_EMPTY: 0
STREAM_8: 2
DICTIONARY_0: 236
DICTIONARY_1: 237
DICTIONARY_2: 238
DICTIONARY_3: 239
LIST_8: 248
LIST_16: 249
JID_PAIR: 250
HEX_8: 251
BINARY_8: 252
BINARY_20: 253
BINARY_32: 254
NIBBLE_8: 255
Tokens are a long list of 151 strings in which the indices matter:
Unpacking nibbles: Returns the ASCII representation for numbers between 0 and 9. Returns - for 10, . for 11 and \0 for 15.
Unpacking hex values: Returns the ASCII representation for numbers between 0 and 9 or letters between A and F (i.e. uppercase) for numbers between 10 and 15.
Unpacking bytes: Expects a tag as an additional parameter, namely NIBBLE_8 or HEX_8. Unpacks a nibble or hex value accordingly.
Number formats
Byte: A plain ol' byte.
Integer with N bytes: Reads N bytes and builds a number out of them. Can be little or big endian; if not specified otherwise, big endian is used. Note that no negative values are possible.
Int16: An integer with two bytes, read using Integer with N bytes.
Int20: Consumes three bytes and constructs an integer using the last four bits of the first byte and the entire second and third byte. Is therefore always big endian.
Int32: An integer with four bytes, read using Integer with N bytes.
Int64: An integer with eight bytes, read using Integer with N bytes.
Packed8: Expects a tag as an additional parameter, namely NIBBLE_8 or HEX_8. Returns a string.
First reads a byte n and does the following n&127 many times: Reads a byte l and for each nibble, adds the result of its unpacked version to the return value (using unpacking bytes with the given tag). Most significant nibble first.
If the most significant bit of n was set, removes the last character of the return value.
Helper methods
Read bytes: Reads and returns the specified number of bytes.
Check for list tag: Expects a tag as parameter and returns true if the tag is LIST_EMPTY, LIST_8 or LIST_16 (i.e. 0, 248 or 249).
Read list size: Expects a list tag as parameter. Returns 0 for LIST_EMPTY, returns a read byte for LIST_8 or a read Int16 for LIST_16.
Read a string from characters: Expects the string length as parameter, reads this many bytes and returns them as a string.
Get a token: Expects an index to the array of Tokens, and returns the respective string.
Get a double token: Expects two integers a and b and gets the token at index a*256+b.
Strings
Reading a string needs a tag as parameter. Depending on this tag, different data is read.
If the tag is between 3 and 235, the token (i.e. a string) of this tag is got. If the token is "s.whatsapp.net", "c.us" is returned instead, otherwise the token is returned as is.
If the tag is between DICTIONARY_0 and DICTIONARY_3, a double token is returned, with tag-DICTIONARY_0 as first and a read byte as second parameter.
LIST_EMPTY: Nothing is returned (e.g. None).
BINARY_8: A byte is read which is then used to read a string from characters with this length.
BINARY_20: An Int20 is read which is then used to read a string from characters with this length.
BINARY_32: An Int32 is read which is then used to read a string from characters with this length.
JID_PAIR
First, a byte is read which is then used to read a stringi with this tag.
Second, another byte is read which is then used to read a stringj with this tag.
Finally, i and j are joined together with an @ sign and the result is returned.
NIBBLE_8 or HEX_8: A Packed8 with this tag is returned.
Attribute lists
Reading an attribute list needs the number of attributes to read as parameter. An attribute list is always a JSON object. For each attribute read, the following steps are executed for getting key-value pairs (exactly in this order!):
Key: A byte is read which is then used to read a string with this tag.
Value: A byte is read which is then used to read a string with this tag.
Nodes
A node always consists of a JSON array with exactly three entries: description, attributes and content. The following steps are needed to read a node:
A list sizea is read by using a read byte as the tag. The list size 0 is invalid.
The description tag is read as a byte. The value 2 is invalid for this tag. The description string descr is then obtained by reading a string with this tag.
The attributes object attrs is read by reading an attributes object with length (a-2 + a%2) >> 1.
If a was odd, this node does not have any content, i.e. [descr, attrs, None] is returned.
For getting the node's content, first a byte, i.e. a tag is read. Depending on this tag, different types of content emerge:
If the tag is a list tag, a list is read using this tag (see below for lists).
BINARY_8: A byte is read which is then used as length for reading bytes.
BINARY_20: An Int20 is read which is then used as length for reading bytes.
BINARY_32: An Int32 is read which is then used as length for reading bytes.
If the tag is something else, a string is read using this tag.
Eventually, [descr, attrs, content] is returned.
Lists
Reading a list requires a list tag (i.e. LIST_EMPTY, LIST_8 or LIST_16). The length of the list is then obtained by reading a list size using this tag. For each list entry, a node is read.
Node Handling
After a binary message has been transformed into JSON, it is still rather hard to read. That's why, internally, WhatsApp Web completely retransforms this structure into something that can be easily processed and eventually translated into user interface content. This section will deal with this and awaits completion.
Binary conversation format
When a node has been read, the contents of messages that have been actually sent by the user (i.e. text, image, audio, video etc.) are still not directly visible or accessible via the JSON. Instead, they are kept in a protobuf message. See here for the definitions. The "wrapper" message type is WebMessageInfo.
WhatsApp Web API
WhatsApp Web itself has an interesting API as well. You can even try it out directly in your browser. Just log in at the normal https://web.whatsapp.com/, then open the browser development console. Now enter something like the following (see below for details on the chat identification):
Using the amazing Chrome developer console, you can see that window.Store.Wap contains a lot of other very interesting functions. Many of them return JavaScript promises. When you click on the Network tab and then on WS (maybe you need to reload the site first), you can look at all the communication between WhatsApp Web and its servers.
Chat identification / JID
The WhatsApp Web API uses the following formats to identify chats with individual users and groups of multiple users.
Chats: [country code][number]@c.us, e.g. [email protected] when you are from Germany and your phone number is 0123 456789.
Groups: [phone number of group creator]-[timestamp of group creation]@g.us, e.g. [email protected] for the group that [email protected] created on November 5 2017.
Broadcast Channels[timestamp of broadcast channel creation]@broadcast, e.g. 1509911919@broadcast for an broadcast channel created on November 5 2017.
WebSocket messages
There are two types of WebSocket messages that are exchanged between server and client. On the one hand, plain JSON that is rather unambiguous (especially for the API calls above), on the other hand encrypted binary messages.
Unfortunately, these binary ones cannot be looked at using the Chrome developer tools. Additionally, the Python backend, that of course also receives these messages, needs to decrypt them, as they contain encrypted data. The section about encryption details discusses how it can be decrypted.
Dealing with E2E media
Encryption
Generate your own mediaKey, which needs to be 32 bytes.
Expand it to 112 bytes using HKDF with type-specific application info (see below). Call this value mediaKeyExpanded.
Split mediaKeyExpanded into:
iv: mediaKeyExpanded[:16]
cipherKey: mediaKeyExpanded[16:48]
macKey: mediaKeyExpanded[48:80]
refKey: mediaKeyExpanded[80:] (not used)
Encrypt the file with AES-CBC using cipherKey and iv, pad it and call it enc.
Sign iv + enc with macKey using HMAC SHA-256 and store the first 10 bytes of the hash as mac.
Hash the file with SHA-256 and store it as fileSha256, hash the enc + mac with SHA-256 and store it as fileEncSha256.
Encode the fileEncSha256 with base64 and store it as fileEncSha256B64.
This step is required only for streamable media, e.g. video and audio. As CBC mode allows to decrypt a data from random offset (block-size aligned), it is possible to play and seek the media without the need to fully download it. That said, we need to generate a sidecar. Do it by signing every [n*64K, (n+1)*64K+16] chunk with macKey, truncating the result to the first 10 bytes. Then combine everything in one piece.
请发表评论