Senior Big Data Engineer

Company: TechStar Consulting Inc.
Location: Irving
Posted on: November 19, 2021

Job Description:

Hi,

Role: Senior Big Data Engineer
Location: Irving TX

Required :
--- - At least 4 years of experience on designing and developing production data pipelines for data ingestion or transformation using tools from Hadoop stack (HDFS, Hive, Spark (PySpark/Scala), HBase, Kafka, NiFi, Oozie, Apache Beam, Apache Airflow etc.).
--- - At least 4 years of experience in the following Big Data Platforms: Cloudera / Horton Works / Snowflake / AWS EMR / RedShift / AWS Glue.
--- - At least 4 years of developing applications with Monitoring, Build Tools, Version Control, Unit Test, TDD, Change Management to support DevOps
--- - At least 2 years of experience with SQL and Shell Scripting experience
--- - Experience troubleshooting JVM-related issues.
--- - Familiarity with machine learning implementation using Spark ML / Tensorflow.
--- - Experience in data visualization tools like Cognos, Arcadia, Tableau.
--- - Experience in data warehousing modeling techniques.
Preferred:
--- - Familiarity with JAVA development on platforms like PCF, Kubernetes will be a plus
--- - 2+ years' experience with Amazon Web Services (AWS), Google Compute or another public cloud service
--- - 2+ years of experience working with Streaming using Spark or Flink or Kafka
--- - 2+ years of experience working with Dimensional Data Model and pipelines in relation with the same
--- - Intermediate level experience/knowledge in at least one scripting language (Python, Perl, JavaScript)
--- - Hands on design experience with data pipelines, joining data between structured and unstructured data
--- - Familiarity of SAS programming will be a plus
--- - Experience implementing open source frameworks & exposure to various open source & package software architectures (AngularJS, ReactJS, Node, Elastic Search, Spark, Scala, Splunk, Apigee, and Jenkins etc.).
--- - Experience with various NOSQL databases (Hive, MongoDB, Couchbase, Cassandra, and Neo4j) will be a plus
--- - Experience in Ab Initio technologies including, but not limited to Ab Initio graph development, EME, Co-Op, BRE, Continuous flow)

Job Requirements:
Integrate new data and big data management technologies and software engineering tools
Utilize the new big data tools
Ensure the data architecture can be extendable for big data solutions
Implement data tools to support analytics and data scientist team
Breathe data systems architecture engineering
Design the data architecture and data integration layers
Ensure that data center networks
Develop business critical data solutions
Manage the migration of data from legacy systems to new data solutions
Troubleshoot data processing and regular data loads
Integrate new data technologies/tools across the enterprise
Evangelize data best practices and implement analytics solutions
Discover data across many different systems, data sources, and data types
Collect and store big data
Executing data center infrastructure engineering projects About AECOM
Executing data center infrastructure engineering projects
Implement data extraction tools with integration of a variety of data sources and data formats
Solve big data problems with smart algorithmic solutions
Assist with data-related technical issues
Build and integrate data from various resources and manage big data

Keywords: TechStar Consulting Inc., Irving , Senior Big Data Engineer, Engineering , Irving, Texas