Job Duties and Responsibilities:

• Design, develop, and maintain scalable, high-performance data solutions to handle large volumes of data across multiple platforms.
• Architect and optimize advanced ETL/ELT pipelines using Hadoop Ecosystem, Python, Spark, Hive, Presto, SQL to process structured, semi-structured, and unstructured data at scale.
• Collaborate with engineers, product managers, data analysts and data scientists to understand data needs and represent key data insights visually.
• Build and maintain real-time data streaming applications using Kafka, Flink, or similar technologies.
• Create and contribute to frameworks that improve the efficacy of logging data, working with data infrastructure to triage issues and resolve them.
• Work with BI teams to design, build, and launch sophisticated data models and visualizations that support multiple use cases across different products or domains.
• Build and maintain tools for data automation, monitoring, and CI/CD workflows for data engineering processes.
• Provide strategic insights on leveraging data to drive user engagement, content optimization, and platform growth.

Technologies Involved / Skills required for the position:

• Hadoop Ecosystem (HDFS, Hive, Spark, Sqoop, Presto), Python, SQL, PySpark, Scala, Scuba, Daiquery, Impala, Kafka, Flink, MySQL, Teradata, Dataswarm (data pipeline framework in Python), Hue, Mercurial, Jenkins, GitLab CI/CD, Tableau, Power BI, M360, Unidash.