Cloud Data Platform Engineer

Koud is looking for a Cloud Data Platform Engineer to work with one of our clients.

The position focuses on Databricks platform engineering, cloud infrastructure, automation, security, governance, and operational reliability across Azure and/or AWS environments. The professional will be responsible for designing, implementing, administering, and optimizing scalable and secure data platforms, supporting data engineering, analytics, machine learning, and artificial intelligence initiatives.

Key Responsibilities:

• Design, implement, and administer Databricks workspaces, clusters, and platform components across Azure and/or AWS environments;
• Build and maintain scalable, secure, highly available, and reliable Databricks environments;
• Define platform standards, architectures, governance frameworks, and operational best practices;
• Configure and administer Unity Catalog, Delta Lake, workspace governance, and identity and access controls;
• Implement platform lifecycle management, upgrades, capacity planning, and optimization;
• Provision and manage cloud infrastructure using Infrastructure as Code tools, particularly Terraform;
• Automate platform deployments, configuration management, and operational processes;
• Design secure networks, private connectivity, and integration patterns across cloud environments;
• Implement backup, disaster recovery, high availability, and business continuity solutions;
• Monitor platform health, availability, performance, and capacity;
• Develop observability solutions for infrastructure and data workloads, including monitoring, logging, and auditing;
• Identify and troubleshoot issues related to the platform, networking, infrastructure, and workloads;
• Conduct root cause analyses and implement corrective actions;
• Implement enterprise security controls, RBAC, least-privilege access models, secrets management, encryption, and key management;
• Support compliance requirements and security audits;
• Develop operational runbooks, support processes, and reusable automation standards;
• Work closely with Data Engineering, Data Science, AI Engineering, and Cloud Infrastructure teams;
• Provide technical guidance on platform architecture, operational excellence, and cloud engineering best practices;
• Support the technical development of other professionals and contribute to the continuous evolution of the platform.

Requirements:

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or related fields, or equivalent practical experience;
• Strong experience in Cloud Engineering, Platform Engineering, Infrastructure Engineering, or related areas;
• Strong hands-on experience implementing and administering Databricks environments;
• Strong experience with enterprise cloud platforms, particularly Azure and/or AWS;
• Experience designing secure, scalable, and highly available cloud solutions;
• Solid knowledge of Databricks administration, including workspace management, cluster configuration and optimization, workflows, and job scheduling;
• Strong hands-on experience with Unity Catalog and Delta Lake;
• Strong experience with Infrastructure as Code, particularly Terraform;
• Experience with CI/CD pipelines using Azure DevOps and/or GitHub Actions;
• Knowledge of Python, PowerShell, and/or Bash for automation;
• Knowledge of cloud networking, including VNET/VPC design and Private Endpoints;
• Experience with RBAC, identity federation, secrets management, encryption, and key management;
• Experience with monitoring and observability tools such as Azure Monitor, CloudWatch, and Log Analytics;
• Strong problem-solving skills and the ability to work independently in complex enterprise environments;
• Good communication skills and the ability to collaborate with different technical teams and stakeholders;
• Fluent English.

Preferred:

• Databricks Certified Platform Administrator certification;
• Databricks Certified Data Engineer Professional certification;
• Azure Administrator Associate or Azure Solutions Architect certification;
• AWS Solutions Architect certification;
• Experience implementing enterprise AI and Machine Learning platforms;
• Experience in regulated environments, particularly in the Insurance or Financial Services sectors;
• Experience with AI and Machine Learning capabilities within the Databricks ecosystem.

Additional Information:

• Hybrid position, with 3 days per week working on-site in Pinheiros, São Paulo.