Automated Vulnerability Scrapper for Real-Time Threat Intelligence in OEM Environments

Automated Vulnerability Scrapper for Real-Time Threat Intelligence in OEM Environments

1 Daksh Patil,2 Shubham Patil,3 Darshan Rana,4 Ranjit Sahu,5 Chaitali Mhatre.

Department of Computer Engineering, Universal College of Engineering/Mumbai University, Vasai, India Department of Computer Engineering, Universal College of Engineering/Mumbai University, Vasai, India Department of Computer Engineering, Universal College of Engineering/Mumbai University, Vasai, India Department of Computer Engineering, Universal College of Engineering/Mumbai University, Vasai, India Department of Computer Engineering, Universal College of Engineering/Mumbai University, Vasai, India

Email:1 dakshpatilhere@gmail.com  ,2 shubham.apatil1234@gmail.com ,

3 darshanrana1010@gmail.com , 4 ranjitsahu2145@gmail.com ,

5 chaitali.mhatre@universal.edu.in

Abstract- The increasing reliance on Information Technology (IT) and Operational Technology (OT) equipment from Original Equipment Manufacturers (OEMs) across critical sectors has created an urgent need for timely vulnerability monitoring. Traditional vulnerability databases like the National Vulnerability Database (NVD) often suffer from significant reporting delays [6], potentially leaving organizations exposed to critical threats. Gathering real-time threat intelligence can help organizations patch systems and improve the chances of mitigating cyber attacks before exploitation occurs.

Recent advancements in data mining, web automation, and Open Source Intelligence (OSINT) have made it possible to develop intelligent systems that assist security analysts in tracking vulnerabilities more efficiently. These systems can extract unstructured data from various sources and normalize it into actionable, structured threat intelligence.

This research focuses on developing a Vulnerability Scrapper that can monitor OEM websites and relevant  security  platforms  for  critical  and high-severity vulnerabilities in real-time [5]. The system utilizes Python-based web scraping technologies including Requests and BeautifulSoup4 to extract data, and Streamlit to create a comprehensive vulnerability monitoring platform. Furthermore, the system incorporates automated data cleaning pipelines to resolve formatting inconsistencies across heterogeneous vendor formats [14].

To improve system performance, several core operational steps are performed: automated data acquisition, data cleaning through regular expressions, severity normalization mapping to the Common Vulnerability Scoring System (CVSS), and database storage using SQLite. The system processes the scraped data and predicts whether immediate administrative alerting is required based on predefined severity thresholds.

The experimental results show that the proposed system is highly effective in tracking vulnerabilities and triggering real-time email alerts. The system demonstrates a 60% reduction in manual vulnerability tracking time and significantly improves the overall efficiency of an organization's security posture management by minimizing the gap between zero-day disclosures and administrative awareness [10].

Keywords— Web scraping, vulnerability monitoring, OEM security, threat intelligence, Streamlit, real-time alerts, CVSS, NVD lag, Open Source Intelligence.