Senior Data Engineer

Position Summary   We are seeking a Senior Data Engineer with specialized expertise in data streaming technologies to join our data team. This role focuses on building and maintaining high-performance data streaming architectures that enable real-time data processing and analytics. The ideal candidate will have deep experience with Apache Kafka, AWS Managed Streaming for Apache Kafka (MSK), Amazon Kinesis, and related streaming technologies in cloud environments.   Role Focus   A Senior Data Engineer at Effectual is primarily responsible for building and maintaining the streaming data architecture that enables real-time data processing and analytics. This involves constructing robust data streaming pipelines that transform and transport data from various sources in real-time, ensuring data flows efficiently through streaming systems for immediate analysis and operational decision-making. You will focus on the efficient and secure management of streaming data systems, ensuring that data is processed, stored, and made available for real-time analytics and downstream applications.   Essential Duties and Responsibilities    Streaming Data Architecture & Pipeline Development   Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis   Develop real-time data pipelines that handle high-volume, high-velocity data streams   Implement event-driven architectures and microservices patterns for streaming data processing   Create and optimize data streaming topologies for complex event processing scenarios   Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms   Kafka & MSK Management   Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments   Implement Kafka Connect pipelines for streaming data integration   Design optimal Kafka topic partitioning strategies and replication configurations   Monitor and optimize Kafka cluster performance, throughput, and latency   Implement Kafka security configurations including SSL/TLS, SASL, and ACLs   Manage Kafka Schema Registry for data serialization and evolution   Kinesis & AWS Streaming Services   Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions   Configure Kinesis Analytics applications for real-time stream processing   Optimize Kinesis shard management and auto-scaling configurations   Implement Kinesis data retention and archival strategies   Integrate Kinesis with other AWS services for comprehensive streaming solutions   Data Processing & Analytics   Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda   Implement complex event processing (CEP) patterns for real-time analytics   Build streaming ETL pipelines that transform data in motion   Create real-time aggregations, windowing operations, and stateful stream processing   Optimize streaming query performance and resource utilization   Integration & Data Flow Management   Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases   Implement data lineage and monitoring for streaming data pipelines   Create automated data quality checks and validation for streaming data   Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution   Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements   DevOps & Infrastructure Management   Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation   Automate deployment and management of streaming infrastructure through CI/CD pipelines   Monitor streaming system health, performance metrics, and alerting   Implement disaster recovery and high availability strategies for streaming systems   Stay current with emerging trends in streaming technologies and cloud-native solutions   Team Collaboration and Project Management   Collaborate with data architects, data scientists, and application teams on streaming data requirements   Support rigorous project governance through daily progress reviews and time tracking   Provide technical leadership and mentorship to junior data engineers   Communicate complex streaming concepts to technical and non-technical stakeholders   Operate with transparency and responsiveness to support high-performing teams   Skills and Experience   Required Experience   7+ years of experience in the data engineering field with significant streaming data specialization   Bachelor's degree in Computer Science, Engineering, or related STEM field   Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools   Proven experience with AWS MSK and Amazon Kinesis services in production environments   Strong background in real-time data processing and stream analytics   Technical Proficiencies   Streaming Technologies: Apache Kafka, Kafka Connect, Kafka Streams, Amazon MSK, Amazon Kinesis (Data Streams, Data Firehose, Analytics)   Programming Languages: Proficient in Python, Java, and Scala for streaming applications   Stream Processing Frameworks: Apache Spark Streaming, Apache Flink, AWS Lambda for stream processing   Data Serialization: Experience with Avro, Protocol Buffers, JSON, and schema registry management   Big Data Technologies: Hadoop ecosystem, Apache Spark, distributed computing concepts   Database Technologies: SQL and NoSQL databases, data warehousing solutions, time-series databases   Cloud and Infrastructure Skills   AWS Services: Deep knowledge of AWS streaming and analytics services (MSK, Kinesis, Lambda, EMR, Glue)   Containerization: Docker and Kubernetes for streaming application deployment   Infrastructure as Code: Terraform, CloudFormation for s