INTL
WWR
전문가
외주
원격 가능
Toptal: Senior Data Engineer – AWS Data Lake & Pipeline Architecture
예상 기간
6개월 이상 (장기계약)
난이도
전문가
기술 스택
AWS
Spark
AWS Glue
EMR
Kafka
Python
Terraform
SQL
AI 분석 요약 기회점수 72/100
AWS 기반 Data Intelligence Platform 구축을 지원하는 시니어 데이터 엔지니어 프로젝트입니다. 데이터 모델링, ETL 파이프라인 설계·자동화, 검색/추천 엔진 통합, 데이터 레이크 운영을 담당하며 Spark, Glue, EMR, Kafka/Kinesis 등을 활용합니다. 구조화/비구조화 데이터를 모두 다루고 Terraform과 Jenkins로 IaC 및 CI/CD를 구축합니다. 7~9년차의 숙련된 클라우드 데이터 엔지니어에게 적합합니다.
핵심 요구사항
- 7~9년차 클라우드 데이터 모델링·ETL·인프라 설계 경험
- 대규모 복잡 SQL 작성 및 최적화, 관계형 DB 베스트 프랙티스
- Spark, Glue, EMR, Kafka 또는 Kinesis 실무 경험
- Java, Python, Scala 프로그래밍 능력
- Terraform IaC 및 Jenkins CI/CD 구축 경험
- NoSQL·graph·document·time-series 등 다양한 DB 지식
산출물
- 확장 가능한 고성능 데이터 솔루션 아키텍처 및 구현
- 자동화된 데이터 파이프라인 및 배포 체계
- 데이터 모델 및 기술 문서/명세서
- 데이터 거버넌스·접근제어·보안 체계
매력 포인트
- 명확하고 상세한 업무 스펙
- Toptal 기반 장기 원격 계약
- 최신 AWS 데이터 스택 경험 축적 가능
주의사항
- 예산·기간 정보 미기재
- Java/Python/Scala 동시 요구 등 요구 기술 범위가 매우 광범위함
프로젝트 유형장기계약
적합 개발자7~9년차 시니어 데이터 엔지니어 (AWS + Spark + SQL)
프로젝트 원문 설명
Headquarters:
Summary
We are seeking a Data Engineer to support the development of a Data Intelligence Platform. This role focuses on data modeling, data services, data pipelines, and cloud-based data infrastructure for reporting, analytics, and data science.
General information
This Data Engineer will support data-related operations across a broader Data Intelligence Platform team. The work includes building and consuming web services, integrating search technologies, supporting personalization and recommendation engines, and applying data engineering best practices across the platform.
The role spans cloud-based data architecture, database performance, pipeline automation, and support for data scientists, researchers, and internal business units. The environment includes AWS-based infrastructure and a mix of structured and unstructured data sources.
Tasks and Deliverables
- Participate in the architecture design and implementation of high-performance, scalable, and optimized data solutions.
- Create data models from scratch using strong SQL fundamentals.
- Write and optimize in-application SQL statements.
- Ensure the performance, security, and availability of databases.
- Prepare documentation and technical specifications.
- Handle database procedures such as upgrades, backups, recovery, and migration.
- Profile server resource usage and optimize configurations as necessary.
- Design, build, and automate the deployment of data pipelines and applications.
- Integrate data from on-premise databases and external data sources using REST APIs and harvesting tools.
- Collaborate with business units and data science teams on data access, transformation, processing, and reporting needs.
- Support implementation, technical issues, and training related to the data lake ecosystem.
- Work with the team to manage AWS resources, including EMR and ECS clusters.
- Support provisioning, monitoring, configuration, and maintenance of AWS tools.
- Evaluate and promote new cloud technologies that improve capabilities and lower operating costs.
- Support automation efforts using Infrastructure as Code with Terraform and CI/CD tools such as Jenkins.
- Work with the team to implement data governance, access control, and security risk reduction.
Required experience
- 7-9 years of experience designing and developing cloud-based data models, ETL pipelines, and infrastructure.
- Experience working with both structured and unstructured data.
- Strong proficiency with SQL across popular databases.
- Experience optimizing large, complex SQL statements.
- Knowledge of best practices for relational databases.
- Experience configuring database engines and orchestrating clusters.
- Ability to plan resource requirements from high-level specifications.
- Ability to troubleshoot common database issues.
- Experience with Spark, Glue, EMR, and Apache Kafka or AWS Kinesis.
- Experience with version control tools such as Git or Subversion.
- Experience using automated build systems and CI/CD workflows.
- Experience programming in Java, Python, and Scala.
- Knowledge of data structures and algorithms.
- Knowledge of relational, NoSQL, graph, document, key-value, and time-series databases.
- Knowledge of scalable data model design and management.
- Knowledge of ML model deployment.
- Knowledge of AWS cloud platforms.
- Knowledge of TDD and BDD.
- Strong interest in improving software development skills, frameworks, and technologies.
Engagement highlights
- Opportunity to work across data modeling, data services, and data science within a broader Data Intelligence Platform.
- Exposure to a varied technical environment spanning AWS, ETL pipelines, databases, search technologies, and recommendation systems.
- Direct collaboration with data science teams and internal stakeholders on reporting, transformation, and platform capabilities.
To apply: https://weworkremotely.com/remote-jobs/toptal-senior-data-engineer-aws-data-lake-pipeline-architecture
Summary
We are seeking a Data Engineer to support the development of a Data Intelligence Platform. This role focuses on data modeling, data services, data pipelines, and cloud-based data infrastructure for reporting, analytics, and data science.
General information
This Data Engineer will support data-related operations across a broader Data Intelligence Platform team. The work includes building and consuming web services, integrating search technologies, supporting personalization and recommendation engines, and applying data engineering best practices across the platform.
The role spans cloud-based data architecture, database performance, pipeline automation, and support for data scientists, researchers, and internal business units. The environment includes AWS-based infrastructure and a mix of structured and unstructured data sources.
Tasks and Deliverables
- Participate in the architecture design and implementation of high-performance, scalable, and optimized data solutions.
- Create data models from scratch using strong SQL fundamentals.
- Write and optimize in-application SQL statements.
- Ensure the performance, security, and availability of databases.
- Prepare documentation and technical specifications.
- Handle database procedures such as upgrades, backups, recovery, and migration.
- Profile server resource usage and optimize configurations as necessary.
- Design, build, and automate the deployment of data pipelines and applications.
- Integrate data from on-premise databases and external data sources using REST APIs and harvesting tools.
- Collaborate with business units and data science teams on data access, transformation, processing, and reporting needs.
- Support implementation, technical issues, and training related to the data lake ecosystem.
- Work with the team to manage AWS resources, including EMR and ECS clusters.
- Support provisioning, monitoring, configuration, and maintenance of AWS tools.
- Evaluate and promote new cloud technologies that improve capabilities and lower operating costs.
- Support automation efforts using Infrastructure as Code with Terraform and CI/CD tools such as Jenkins.
- Work with the team to implement data governance, access control, and security risk reduction.
Required experience
- 7-9 years of experience designing and developing cloud-based data models, ETL pipelines, and infrastructure.
- Experience working with both structured and unstructured data.
- Strong proficiency with SQL across popular databases.
- Experience optimizing large, complex SQL statements.
- Knowledge of best practices for relational databases.
- Experience configuring database engines and orchestrating clusters.
- Ability to plan resource requirements from high-level specifications.
- Ability to troubleshoot common database issues.
- Experience with Spark, Glue, EMR, and Apache Kafka or AWS Kinesis.
- Experience with version control tools such as Git or Subversion.
- Experience using automated build systems and CI/CD workflows.
- Experience programming in Java, Python, and Scala.
- Knowledge of data structures and algorithms.
- Knowledge of relational, NoSQL, graph, document, key-value, and time-series databases.
- Knowledge of scalable data model design and management.
- Knowledge of ML model deployment.
- Knowledge of AWS cloud platforms.
- Knowledge of TDD and BDD.
- Strong interest in improving software development skills, frameworks, and technologies.
Engagement highlights
- Opportunity to work across data modeling, data services, and data science within a broader Data Intelligence Platform.
- Exposure to a varied technical environment spanning AWS, ETL pipelines, databases, search technologies, and recommendation systems.
- Direct collaboration with data science teams and internal stakeholders on reporting, transformation, and platform capabilities.
To apply: https://weworkremotely.com/remote-jobs/toptal-senior-data-engineer-aws-data-lake-pipeline-architecture
WWR에서 원본 확인
원본 보기