Introduction
• On this function, you’ll work in our IBM Shopper Innovation Heart (CIC), the place we ship deep technical and trade experience to a variety of private and non-private sector shoppers around the globe. These facilities provide our shoppers domestically primarily based expertise and technical experience to drive innovation and adoption of latest expertise.
A profession in IBM Consulting is rooted by long-term relationships and shut collaboration with shoppers throughout the globe.
You’ll work with visionaries throughout a number of industries to enhance the hybrid cloud and AI journey for essentially the most modern and invaluable firms on the planet. Your skill to speed up affect and make significant change to your shoppers is enabled by our strategic companion ecosystem and our strong expertise platforms throughout the IBM portfolio; together with Software program and Purple Hat.
Curiosity and a continuing quest for information function the inspiration to success in IBM Consulting. In your function, you’ll be inspired to problem the norm, examine concepts outdoors of your function, and give you artistic options leading to floor breaking affect for a large community of shoppers. Our tradition of evolution and empathy facilities on long-term profession progress and improvement alternatives in an setting that embraces your distinctive expertise and expertise.
Your Function and Obligations
As a Huge Knowledge Engineer, you’ll develop, preserve, consider, and check huge information options. You’ll be concerned in information engineering actions like creating pipelines/workflows for Supply to Goal and implementing options that deal with the shoppers wants.
Your major obligations embrace:
- Design, construct, optimize and help new and present information fashions and ETL processes primarily based on our shoppers enterprise necessities.
- Construct, deploy and handle information infrastructure that may adequately deal with the wants of a quickly rising information pushed group.
- Coordinate information entry and safety to allow information scientists and analysts to simply entry to information at any time when they want too.
Required Technical and Skilled Experience
- Developed the Pysprk code for AWS Glue jobs and for EMR.. Labored on scalable distributed information system utilizing Hadoop ecosystem in AWS EMR, MapR distribution..
- Developed Python and pyspark packages for information evaluation.. Good working expertise with python to develop Customized Framework for producing of guidelines (identical to guidelines engine).
- Developed Hadoop streaming Jobs utilizing python for integrating python API supported purposes..
- Developed Python code to assemble the information from HBase and designs the answer to implement utilizing Pyspark. Apache Spark DataFrames/RDD’s had been used to use enterprise transformations and utilized Hive Context objects to carry out learn/write operations..
- Re – write some Hive queries to Spark SQL to scale back the general batch time
Most well-liked Technical and Skilled Experience
- Understanding of Devops.
- Expertise in constructing scalable end-to-end information ingestion and processing options
- Expertise with object-oriented and/or practical programming languages, resembling Python, Java and Scala”