CVE Exploitation Dashboard
My approach on developing an end to end Machine Learning project to detect CVE vulnerabilities and their exploitabilities in KEV.
Inspiration
My motivation for this project purely came from a vulnerability I was affected by, which made me to look more into CVEs and their exploitabilities. The CISA KEV catalog lists vulnerabilities that are known to be actively exploited in the wild. I was curious about the number of vulnerabilities that have been added to the catalog over time and the rate at which new vulnerabilities are being added.
What it does
This project gave me a practical exposure to using HDFS and Spark. It has two main parts. The first is to develop a way to detect CVEs and their exploitabilities from CVE databases. The second is to develop a dashboard to visualize the CVEs and their exploitabilities. Hadoop and its infrastructure was something that was completely new to me, which required lots of learning and exploring. What started off as a simple attempt to gain understanding of the higher level design of HDFS, MapReduce, Spark, turned into a full-fledged project in a matter of weeks
How I built it
I remember for my Machine Learning course, the final project I worked on had us work with CVE data. The project and its tasks were simple, it consisted of basic ML pipeline: data collection and preparation, model training and evaluation. There were no strict requirements on using particular technology stack, so I used Python and ML library like scikit-learn to implement my ML pipeline. By utilizing the work already done from the project, I was off to a good start and all I really had to worry about was migrating the pipeline to Spark and HDFS ecosystem.
Challenges I ran into
Biggest challenge, simply put, messy semi structured data. NVD has so much data and its not structured in a way that is easy to digest. Most of the records, from the early 90s, came in JSON format, with lots of null values, inconsistencies, and variations in structure. Over time, it got better, but it still took time to clean the data and get it into a usable format. The second challenge was in orchestrating Spark jobs. Kubernetes was something that came in handy, a tool that I have minimal experience with.
Accomplishments I'm proud of
Building a real pipeline (and feeling Like a rockstar). This project is what made me realize the importance of building a pipeline that captures data, is scalable and has industry standards met. It was definitely not easy, but definitely not something that I would skip adding to my Github and portfolio site
What I learned
I learned a lot about workflow automation as well as data engineering practices. Before this, my idea of a pipeline was simple: train model, save model, done. But I soon realized that in a real world scenario, there is so much more that goes into it. From data collection and cleaning to model deployment and monitoring, there are so many moving parts. It was also here that I got a practical exposure to Docker and Kubernetes, tools that I have minimal experience with.
What's next
My first ever data related project was working with Rate My Professor's data and analyzing patterns. It had some machine learning involved, with sentiment analysis. My goal is to apply whatever I learned throughout this project and apply it there as well
One thing I'd redo
I would definitely take the documentation part more seriously. From time to time, I would forget details about the project and had to struggle to recall them. For example, I forgot why I used client manager over cluster manager. How I had Spark server running over limited work nodes etc. There were bugs, confusion errors, performance issues, but I am certain these could have been solved in an easier manner if I had proper documentation from the start