Getting Started with the HoneyDB Payload History API (Part 1)
A deeper dive on HoneyDB's Payload History API Endpoints
HoneyDB collects and provides invaluable insights into malicious activity that occurs on the Internet. In the face of ever-evolving cyber threats, understanding and analyzing malicious activity historical and new trends is crucial for any organization seeking to protect its digital assets. Through HoneyDB’s Payload History API, users can analyze historical interaction data to enhance their security posture. This API delivers details on attributes of payloads dating back to 2014, enabling you to monitor trends and enrich security telemetry.
In this post, we’ll explore how to retrieve payload attribute data by year or year-month and delve into understanding the response data. We’ll also show how to track the history of a specific payload attribute by its hash. This will be the first of three posts, where subsequent posts will cover the remainder of HoneyDB’s Payload History API endpoints.
Why are you posting on the HoneyDB Payload History API again?
The previous post was a brief introduction. This three-part series will offer a more comprehensive look at HoneyDB Payload History by covering all of the API endpoints.
What Are Payload Attributes?
Payload attributes, within the context of HoneyDB's honeypot interactions, are specific elements extracted from the data or commands sent to these honeypots by potentially malicious actors. These attributes provide granular insights into the nature of the attempted attacks. Examples of payload attributes include domains embedded in the payload, HTTP headers, usernames and passwords used in login attempts, and specific commands issued to the honeypot system. By analyzing these attributes, security researchers and analysts can identify patterns, track attack trends, to help develop more effective defense strategies against evolving threats.
Retrieve Payload Attributes by Year or Year-Month
Let’s get started with the first set of API endpoints. These endpoints list all attribute data for a given year or month. Due to the large volume of data collected over time, results are condensed to one record per day, service, and attribute.
Query by year example:
Endpoint: /api/payload-history/YYYY
curl --header "X-HoneyDb-ApiId: <api_id>" \
--header "X-HoneyDb-ApiKey: <api_key>" \
https://honeydb.io/api/payload-history/2014The last parameter in the path, "2014", specifies the year for which the data should be fetched. Below is an example of data returned.
[
{
"date": "2014-01-04",
"service": "HTTP.ALT",
"attribute": "method",
"hash": "e15e216fc1c639f787b1231ecdfa1bf8",
"value": "HEAD"
},
{
"date": "2014-01-04",
"service": "HTTP.ALT",
"attribute": "path",
"hash": "f9360b58573c95bf244926c5e4a4ebbb",
"value": "/HNAP1/"
},
{
"date": "2014-01-04",
"service": "VNC",
"attribute": "remote-frame-buffer",
"hash": "9006253b0c51e41cd769c06c3dee8938",
"value": "RFB 003.004"
},
{
"date": "2014-01-05",
"service": "HTTP.ALT",
"attribute": "header",
"hash": "00c78fbf3eeb2d017e45bc2dd94f3e18",
"value": "Accept-Encoding"
}
]We’ll go over the response data in more detail further down in this post.
In the example above we queried all attribute data for 2014, which has a relatively smaller set of data in comparison to subsequent years. As the HoneyDB honeypot network has grown over the years, so has the volume of data collected. As a result subsequent years will have very large response sizes (hundreds of megabytes compressed). Make sure your API client script or application is able to handle these large response sizes. However, it may be more efficient to download smaller chunks of the data, by month instead of by year, which we’ll now show in the next endpoint example.
Query by month example:
Endpoint: /api/payload-history/YYYY/MM
curl --header "X-HoneyDb-ApiId: <api_id>" \
--header "X-HoneyDb-ApiKey: <api_key>" \
https://honeydb.io/api/payload-history/2014/05This example query is just like the previous except we add the desired month at the end. As you’d expect, the results consist of the same data, only filtered for the year-month specified in the query.
Keep in mind that querying the API by year will return a large dataset encompassing all payload attribute data for that specific year. This may result in a very large response that could take a significant amount of time to download. Depending on your specific requirements and use case, it might be more efficient to utilize the year-month endpoint. This alternative approach allows you to retrieve data for a specific month within a given year, leading to smaller and more manageable responses that can be processed more quickly.
Understanding the Response Data
The API returns data in a JSON array. This array contains numerous elements, where each element represents a date the attribute was seen for a given service. This is where the distilled data is helpful. If you were analyzing the raw interaction data you’d have to deduplicate hundreds or thousands of instances for a given date to uncover all the unique values. Each interaction element within the array includes the following key-value pairs:
date: This field contains the specific date on which the interaction took place.
service: This field specifies the particular service with which the interaction occurred.
attribute: This field contains the name of the attribute that was extracted from interaction data.
hash: This field contains the MD5 hash value of the payload value.
value: This field contains the actual attribute value extracted from interaction data.
By providing this structured data, the API allows developers and analysts to easily access and analyze the details of each interaction, enabling them to gain insights into user behavior, track service usage, and identify trends or patterns.
Retrieve Payload Attribute History by Hash
Now that you're familiar with querying payload attributes by year and month, and understand the returned data, you can use the next endpoint to retrieve the complete history of a specific attribute by its hash. This endpoint takes the attribute hash value as a parameter and returns the distilled history (one entry per day) for that attribute.
Query by hash example:
Endpoint: /api/payload-history/HASH
curl --header "X-HoneyDb-ApiId: <api_id>" \
--header "X-HoneyDb-ApiKey: <api_key>" \
https://honeydb.io/api/payload-history/f9360b58573c95bf244926c5e4a4ebbb
The response data has the same structure and fields previously described, only it is now filtered by the specified hash. The end result is effectively a historical timeline of the attribute.
Analysis of the data backing HoneyDB Payload History API endpoints will continue to improve and expand, which means the data will only get better over time. This includes improved parsing of attributes values, additional attributes produced, and more services covered.
Conclusion
In this post, we explored the initial steps for working with HoneyDB's Payload History API.
We delved into retrieving payload attribute data based on specific time frames (year or year-month) and by attribute hash.
Understanding these fundamental endpoints and the structure of the returned data is crucial for effectively leveraging the API's capabilities.
By analyzing historical threat data, you can identify trends and patterns in malicious activity, enabling you to proactively mitigate risks and strengthen your security posture.
Additionally, the API's data can be used to enrich security telemetry, providing valuable context and insights that can further enhance your cyber defense measures.
In subsequent posts, we will continue our exploration by diving into the next set of Payload History API endpoints for services and attributes.
To get started with the HoneyDB Payload History API, register for a community tier account at honeydb.io and generate your API keys. Note that the community tier is intended for personal and enterprise network defense (non-commercial use). If you have a commercial use case for this data as part of a product or service, commercial plans are available at https://honeydb.io/plans.
Stay tuned for the next installment to further enhance your understanding and proficiency in utilizing the HoneyDB Payload History API.