Abstract

Gathering and analyzing enormous volumes of data is known as "big data," and it includes information from users, sensors, healthcare providers, and companies. Using the Hadoop framework, large amounts of data are stored, managed, and dispersed across multiple server nodes. Big Data issues, including security holes in the Hadoop Distributed File System (HDFS), the architecture's core layer, are highlighted in this article. The methodology includes setting up a Hadoop environment, integrating Kerberos for authentication, enabling HDFS encryption zones, implementing SSL/TLS for data in transit, and utilizing Apache Ranger and Apache Knox for access control and perimeter security, respectively. The results demonstrate the successful implementation of all planned security measures, achieving a robust security framework for the Hadoop cluster. Performance testing indicates a 10% reduction in processing speed due to the security features, a trade-off deemed acceptable given the significant enhancement in data protection. Compliance testing confirms adherence to GDPR and CCPA regulations, ensuring legal and secure data management. Overall, the study underscores the feasibility of integrating comprehensive security measures within a Hadoop environment, balancing the need for robust data protection with minimal performance impact. Future work includes optimizing security configurations to further mitigate performance degradation and exploring advanced security measures for enhanced threat detection and response. This methodology provides a scalable and secure solution for managing large datasets in compliance with global data protection standards.