Parsing raw gossip messages using Node.js
This blog contains the technical “how-to” for converting the raw messages stored by gosssip_store in a sqlite3 database using Node.js
Storing lightning network messages
There are multiple ways to approach this problem, c-lightning as well as LightningD can be utilized for fetching the latest gossip_messages.
The approach used for the purpose of this project is to archive all the gossip_messages in a sqlite3 database using the Historian plugin created for core-lightning (https://github.com/lightningd/plugins/tree/master/historian)
Which lightning network messages are fetched?
Three types of lightning network messages are fetched using the historian plugin.
- Channel_announcement
- Channel_update
- node_announcement
The Channel_announcement gossip_message is responsible for delivering information about the ownership of a channel. It creates a link between each on-chain bitcoin key with each lightning network node key.
The Channel_update gossip_message is responsible for updating the network about each side of the channel (node). The two nodes on each end of the channel update the lightning network about their fees and the minimum expiry delta it requires to relay HTLCs through this channel.
The node_announcement gossip_message allows a node to indicate extra data associated with it, in addition to its public key.
Parsing Channel_announcement
The channel_announcement raw message is normally 432 bytes long buffer. Each byte has its own meaning and requires to be assigned to a particular field in order to get meaningful output from the raw message.
The conversion of the raw messages into JSON can be done using Node.js or JavaScript. To understand more about the fields that are related to channel_announcement you can check out the Lightning network specifications at https://github.com/lightning/bolts/blob/master/07-routing-gossip.md#the-node_announcement-message
The first step in order to parse the raw message is to collect the raw message, it can be done using the SCID of the channel or if you are parsing all the channel_announcement messages, we can simply iterate over them.
Node.js has an inbuilt buffer function that can be utilized for reading buffer streams but we can also simply iterate over the buffer using loops. The inbuilt buffer function of Node.js has a few memory-related problems that we can avoid by using loops.
The first 2 bytes of the channel_announcement buffer contain information about the version.
The next 64 bytes contain the node signature of the first node. We can parse them using the below method:
In the above method, I created a curr_index variable which makes iteration over a buffer much easier compared to calculating the starting and ending point index-wise. The above code helps us store the hex value of the node_signature_1
The next 64 bytes contain the node signature of the second node, the next 64 bytes contain the bitcoin signature of the first node, and the 64 bytes after that contain the bitcoin signature of the second node.
This is how the output looks after parsing the bitcoin and node signature of both the nodes.
Now the next 2 bytes are flen bytes, we collect and parse these flen bytes into integer format.
After parsing we get to know the exact number of bytes we need to iterate over in order to get the feature variable.
After getting the feature variable we parse the next 32 bytes in order to get the chain_hash
The next 8 bytes contain the short_channel_id, the short_channel_id helps us in verifying a channel with the on-chain bitcoin blockchain as it contains the block number, transaction_id, and output_index of the channel. The first 3 bytes of these 8 bytes contain the block number, the next 3 bytes contain the tx_id and the last 2 bytes contain the output_index. We also need to parse the hexadecimal values into integer values in order to make them verifiable.
The short_channel_id upon parsing should look something like the above output, we can further verify this channel using this short_channel_id.
Further, we parse the node_id_1 by iterating over the next 33 bytes followed by another 33 bytes for node_id_2. In the same manner, we read the bitcoin_key_1 by reading the next 33 bytes and bitcoin_key_2 by reading the last remaining 33 bytes.
Parsing Channel_update
The channel_update raw message is normally a 138 bytes long buffer.
The first 2 bytes of the channel_update buffer contain information about the version.
The first 64 bytes contain the signature, the next 32 bytes contain the chain_hash. The unique fact with chain_hash is that it should be equivalent to “000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f”, this value can be used for verifying if you are able to parse the chain_hash correctly or not.
The next 8 bytes contain the short_channel_id, parsing them is the same as parsing the short_channel_id of the channel_annoucement message.
The next 4 bytes contain the timestamp, we further need to parse this timestamp from hexadecimal to decimal.
The next 1 byte contains the message_flags, the next 1 byte contains the channel_flags, the next 2 bytes contain the cltv_expiry_delta, the next 8 bytes contain the htlc_minimum_msat, the next 4 bytes contain the fee_base_msat, the next 4 bytes contain the fee_proportional_millionths.
Now, if the buffer length is equal to 138, we can read the remaining 8 bytes for the htlc_maximum_msat
Parsing node_announcement
The node_annoucement buffer is 161 bytes long.
The first 2 bytes of the node_annoucement buffer contain information about the version.
The next 64 bytes contain the signature of the node.
Now the next 2 bytes are flen bytes, we collect and parse these flen bytes into integer format.
After parsing we get to know the exact number of bytes we need to iterate over in order to get the feature variable. (Same as channel_announcement)
The next 4 bytes contain the timestamp, we parse the timestamp from hexadecimal to decimal.
The next 33 bytes contain the parsed_node_id, the next 3 bytes contain the RGB color of the node, and the next 32 bytes contain the alias.
Same as flen bytes the next 2 bytes of the node_announcement buffer contain abytes, we parse these 2 bytes from hexadecimal to decimal in order to get the number of bytes we need to read in order to fetch the address.
The first abyte contains the type of address, depending upon the address we get to know the number of bytes we have to read in order to fetch the address. The address can be of the following type:
- ipv4 (we read 4 bytes)
- ipv6 (we read 16bytes)
- Tor v2 onion services (we read 10 bytes)
- Tor v3 onion service (we read 35bytes)
further in all the above 4 types, we have to read the remaining two bytes for fetching the port number.
The final address should look like the above JSON object, the address in the above picture represents an ipv6 type address.
I have created a GitHub repository https://github.com/vinayaksh42/clightning-node-server with the help of which you can easily deploy your own Node.js server for parsing gossip_messages. In the above repository, you will have to copy the content inside the config/example.default.json file and create a config/default.json file which will point towards your own deployment of sqlite3 database.
I hope this blog helped you parse your gossip_messages!
