Big Data Engineer

Company: PriceSenz
Location: Irving
Posted on: May 17, 2020

Job Description:

The Big Data Engineer I is responsible for designing, developing, testing, tuning and building a large-scale data processing system for data products that allow HMS to improve quality, velocity and monetization of our data assets for both Operational Applications and Analytical needs. Responsibilities bull Responsible for design, development and delivery of data from operational systems and files into ODS, downstream Data Marts and files.bull Works with BA's, end users and architects to define and process requirements, build code efficiently and work in collaboration with the rest of the team for effective solutions. bull Research, develop and modify ETL processes and job according to the requirements.bull Develop Spark Jobs, Custom UDF using Spark RDDs, Data frames, and Spark SQL. bull Strong analytical SQL experience working with dimensional modeling.bull Troubleshoot and develop on Hadoop technologies including HDFS, Hive, NiFi, Hue, Pig, Flume, HBase, Spark, Kafka, Impala and Hadoop ETL development via tools such as Informatica, Talend.bull Build a service structure on AWS or AZURE capable of being deployed and scaled to runa variety of platform components dynamicallybull Knowledge of and experience with any Azure Data Platform components ndash Azure Data Lake, Data Factory, Data Management Gateway, Azure Storage Options, DocumentDB, Data Lake Analytics, Stream Analytics, EventHubs, Azure SQLbull Translate, load and present disparate data sets in multiple formats and multiple sources including JSON, Avro, text files, Kafka queues, and log data.bull Implement quality logical and physical ETL designs that have been optimized to meet the operational performance requirements for our multiple solutions and products. bull Experience to design the optimal performance strategy, and manage the technical metadata across all ETL jobs.bull Responsible for building solutions involving large data sets using SQL methodologies, Data Integration Tools like Informatica in any database.bull Deliver projects on-time and to specification with quality. Qualifications bull 8+ years of experience in managing data lineage and performing impact analysis.bull 5+ years of experience with any ETL tool development and relational DB with strong SQL skills.bull 4+ years of experience with Hadoop Eco System.bull Experience working in cloud services on Azure or AWS.bull Experience working in Data Management projects.bull Experience working in Hive or related tools on Hadoop, Performance tuning, File Format, executing designing complex hive HQL's, data migration conversion.bull Experience working with programing language like JavaScala or python. Python preferred.bull Experience working in agile environment. bull Experience working with Spark for data manipulation, preparation, cleansing. bull Experience working with ETL Tools (InformaticaDSSSIS) for data Integration. bull Experience designing and developing automated analytic software, techniques, and algorithms.bull Ability to handle multiple tasks and adapt to a constantly changing environment.bull Self-starter with the ability to work independently and take initiatives.bull Ability to translate ideas and business requirements into fully functioning ETL workflows. bull Ability to apply mastery knowledge in one of the relational database (DB2, MSSQL, Teradata, Oracle 8i9i10g11i)bull Expert ability and hands-on experience in SQL is a must.bull Experience with UnixLinux and shell scripting.bull Strong analytical and problem-solving skills.bull Excellent written and oral communication skills, with the ability to articulate and document processes and workflows for use by various individuals of varying technical abilities.bull Knowledge of HealthCare is a plus. Minimum Education bull MSBS in Computer Science, Information Systems, or related field preferred andor equivalent experience. Additional Skills bull Ability to work both independently and in a collaborative environment.bull Excellent problem-solving skills, communication skills and interpersonal skills.bull Ability to analyze information and use logic to address work-related issues and problems.bull Ability to demonstrate proficiency in Microsoft Access, Excel, Word, PowerPoint and Visio.bull Experience working in DevOps environment is a plus.bull Experience or knowledge of web architecture, (Javascript, SOAPXML, Weblogic, Tomcat) is a plus.bull Experience with an ORM framework, SOA Architecture, Microservices is a plus.bull Experience with Middleware components (ESB, API Gateway) is a plus.

Keywords: PriceSenz, Irving , Big Data Engineer, Engineering , Irving, Texas