Lead Data Engineer (hp)

Job Summary

Lead data visualization and data engineering components for Machine Learning and Advanced Analytics solutions. The professional should have deep expertise in data visualization and data engineering skills with good understanding of machine learning and advanced analytics projects.

Responsibilities

• Develop visualizations in Power BI which will be the user interface of the machine learning and advanced analytics solutions.
• Build DAX queries in Power BI to build custom components of Power BI dashboard.
• Integrate Power BI with cloud platforms like Databricks, Datalakes and databases.
• Extract relevant information from source systems, either from Databricks environment, Datalakes etc.
• Build optimal data models for Power BI visualizations and other purposes.
• Build E2E data pipelines for machine learning and advanced analytics projects.
• Understanding of machine learning and advanced analytics projects, which will help in collaborating with data scientists.
• Automate data engineering and data visualization processes, so that data refresh happens automatically.
• Directly work with business stakeholders, understand their requirements, explain technical components of the solution in non-technical language.
• Work on Databricks cloud platform with Microsoft Azure cloud.
• Data quality monitoring and automated testing to ensure quality data & identify issues.
• Collaborate with data owners for extracting data from multiple sources as per project requirement.
• Develop scalable data engineering pipeline which can scale across geographies and businesses.
• Design optimal architecture keeping project requirements in mind, scope of future expansion of project, company’s data landscape etc.
• Responsible for cyber security approval for data engineering pipeline.
• Technically guide and mentor junior data engineers and visualization professionals.
• Collaborate with technical teams within HP Finance to identify best practices & new data engineering technologies & then drive their usage across teams.
• Responsible for building reusable data engineering components.

Knowledge and Skills:

• 6+ years advanced knowledge on Power BI and DAX queries. Also understanding of other visualization solutions.
• Advanced SQL knowledge, experience with relational databases, query authoring (SQL)
• Knowledge to extract, process & store data in Federated Data Lake.
• Experience of automating data monitoring and testing.
• Experience in Spark or Python package of Spark. Will be used for distributed computing within platform like Databricks.
• Experience in building automated data engineering pipelines
• Strong communication, collaboration skills, teamwork & ability to influence
• Familiarity with Python preferred.