BIG DATA DEVELOPER in Pune at InfoCept

Primary responsibilities: Designing Hive/HCatalog data model includes creating table definitions, file formats, compression techniques for Structured & Semi-structured data processing Implementing Spark processing based ETL frameworks Implementing Big data pipeline for Data Ingestion, Storage, Processing & Consumption Modifying the Informatica-Teradata & Unix based data pipeline Enhancing the Talend-Hive/Spark & Unix based data pipelines Develop and Deploy Scala/Python based Spark Jobs for ETL processing Strong SQL & DWH concepts Preferred background:        Function as integrator between business needs and technology solutions, helping to create technology solutions to meet clients’ business needs         Lead project efforts in defining scope, planning, executing, and reporting to stakeholders on strategic initiatives         Understanding of EDW system of business and creating High level design document and low level implementation document         Understanding of Big Data Lake system of business  and creating High level design document and low level implementation document Designing Big data pipeline for Data Ingestion, Storage, Processing & Consumption Tools & Technologies Big Data (must have) Hadoop Hive HDFS Spark YARN DWH concepts SQL Linux/Unix (OS) Programming Language (must have). Should be proficient in at least one of the below language: Scala Python