Lead Data Engineer

Design, build and maintain scalable data infrastructure, including data lakes, pipelines, and metadata repositories to ensure timely and accurate data delivery.

Collaborate with data scientists to develop and support data models, integrate data sources, and assist in machine learning workflows and experimentation environments.

Create and enhance large-scale batch and real-time data processing systems to improve operational efficiency and meet business objectives.

Utilize Python, Apache Airflow, and AWS services to automate data workflows and processes, ensuring efficient scheduling and monitoring.

Leverage AWS services to manage data storage and compute resources, aiming for high performance, scalability, and cost-efficiency.

Implement rigorous testing and validation procedures to guarantee the reliability, accuracy, and security of data processing workflows.

Keep abreast of industry best practices and emerging technologies in data engineering and data science to suggest optimizations and innovative solutions.

Proficient in Python for data processing (pandas, pyspark), workflow automation (Apache Airflow), and experience with AWS services (Glue, S3, EC2, Lambda).

Experience working with Kubernetes and Docker for managing containerized environments in the cloud.

Hands-on experience with columnar and big data databases (Athena, Redshift, Vertica, Hive/Hadoop), along with version control systems like Git.

Strong familiarity with AWS services for cloud-based data processing and management.

Experience with CI/CD tools such as Jenkins, CircleCI, or AWS CodePipeline for continuous integration and deployment.

Expertise in building and managing robust data architectures and pipelines for large-scale data operations.

Ability to support data science workflows, including collaboration on data preparation, feature engineering, and enabling experimentation environments.

Familiarity with Langchain for building data applications involving natural language processing or conversational AI frameworks.

Experience with AWS Sagemaker or Databricks for enabling machine learning environments.

Familiarity with both RDBMS (MySQL, PostgreSQL) and NoSQL (DynamoDB, Redis) databases.

Experience with enterprise BI tools like Tableau, Looker, or PowerBI.

Familiarity with distributed messaging systems like Kafka or RabbitMQ for event streaming.

Experience with monitoring and log management tools such as the ELK stack or Datadog.

Knowledge of best practices for ensuring data privacy and security, particularly in large data infrastructures.

Attractive salary

Bonus equivalent to a month's salary

Performance-based incentives

Access to professional English training

Comprehensive health coverage

Generous annual leave allowance

PEGASI – IT Recruitment Consultancy | Email: recruit@pegasi.com.vn | Tel: +84 28 3622 8666