A Survey of Serverless Data Pipelines on AWS: Automating CSV Processing with Scalable and Secure Architecture

Abstract

Serverless architecture has emerged as a transformative approach for building scalable, secure, and cost-efficient data pipelines in cloud environments. This paper presents a comprehensive survey of recent advancements in AWS-based serverless data processing, with a focus on automating CSV ingestion, transformation, and visualization. The proposed architecture leverages Amazon S3 for multi-zone data storage, AWS Lambda for real-time cleaning, AWS Glue and Glue Crawler for schema-aware transformation, and Amazon QuickSight for interactive dashboards. Designed to operate without Athena or Step Functions, the pipeline demonstrates significant improvements in latency, modular scalability, and governance. Performance benchmarks show up to 25% reduction in transformation latency and enhanced schema adaptability across evolving datasets. A comparative analysis of state-of-the-art serverless frameworks is included, identifying critical gaps in orchestration-free automation, cost-performance optimization, and visualization accessibility. Finally, the paper outlines future research directions to develop lightweight, secure, and domain-adaptable serverless pipelines for enterprise and IoT analytics.