We are seeking a Senior Manager, Data Platform Software to lead our data platform team in designing, building, and maintaining a scalable and high-performance data infrastructure. This role involves managing teams working on data bus technologies like Kafka, real-time streaming services, data lakes, and analytics platforms to support business intelligence, machine learning, and AI-driven insights. The ideal candidate will have deep expertise in big data architecture, distributed systems, and cloud-native technologies, with strong leadership and strategic vision.
Key Responsibilities
- Define and drive the technical vision, architecture, and roadmap for the data platform, ensuring alignment with business goals.
- Lead cross-functional teams in the development and operation of a high-scale, low-latency data infrastructure.
- Drive best practices for data engineering, streaming, and analytics to ensure reliability, performance, and scalability.
- Collaborate with product, engineering, analytics, and business teams to define data requirements and solutions.
- Oversee the design and implementation of Kafka-based messaging systems for real-time data streaming.
- Manage the development and optimization of data lakes (e.g., AWS S3, Delta Lake, Apache Iceberg) to support large-scale analytics.
- Lead the adoption of real-time and batch processing technologies (e.g., Apache Flink, Spark, Snowflake, Databricks, BigQuery).
- Ensure efficient data governance, cataloging, lineage tracking, and access control across the data platform.
- Establish CI/CD pipelines, monitoring, and observability frameworks for data pipelines and streaming services.
- Optimize data ingestion, transformation, and storage for cost efficiency and performance.
- Lead incident management and troubleshooting for production data systems, ensuring high availability and reliability.
- Build and mentor a high-performing data engineering team, fostering a culture of ownership, learning, and innovation.
- Manage vendor relationships and oversee the integration of third-party data platform tools.
- Advocate for data-driven decision-making and align leadership on key data initiatives.
Required Qualifications
- 10+ years of experience in data engineering, platform architecture, or software development, with at least 6+ years in a leadership role.
- Hands-on experience with Kafka, Apache Flink, Apache Spark, and stream processing frameworks.
- Deep knowledge of cloud data architectures (Azure, AWS, or GCP), including serverless computing, Kubernetes, and containerized data services.
- Strong experience with data lake technologies (Delta Lake, Iceberg, Parquet, ORC) and warehouse solutions(Snowflake, BigQuery, Redshift).
- Proficiency in big data frameworks, distributed computing, and real-time analytics solutions.
- Expertise in SQL, Python, Scala, or Java for data processing and automation.
- Understanding of data security, compliance (GDPR, CCPA), and governance best practices.
- Proven experience in managing and scaling data platform teams in a fast-paced environment.
- Strong ability to influence and drive technical decisions across different levels of the organization.
- Excellent stakeholder management and ability to translate complex technical topics into business value.
Preferred Qualifications
- Experience with ML model deployment and AI-driven analytics on data platforms.
- Familiarity with real-time analytics tools (Druid, ClickHouse, Rockset, Pinot).
- Certifications in Azure/AWS/GCP data engineering.
Why Join Us?
- Work on cutting-edge real-time data streaming and analytics solutions at scale.
- Collaborate with world-class engineers to drive innovation.
- Lead a high-impact team in shaping the future of our data ecosystem.
