Job Description

Dear Partners,

Resourcesys is a highly recognized provider of professional IT Consulting services in the US.

Here is our open requirement which can be filled immediately. Kindly respond to this requirement with your consultant resume, contact and current location info to speed up the interview process.

Job Title:- Databricks Architect

Location:- Atlanta, Georgia (Remote)

Duration:- Contract

Job Description

Key Responsibilities:

• Architect and Design Solutions:
  Lead the architecture and design of Databricks-based data solutions that support data engineering, machine learning, and real-time analytics.
• Data Pipeline Design:
  Design and implement ETL (Extract, Transform, Load) pipelines using Databricks, Apache Spark, and other big data tools to process and integrate large-scale data from multiple sources.
• Collaborate with Stakeholders:
  Work with business and data teams to understand requirements, identify opportunities for automation, and design solutions that improve data workflows.
• Optimize Data Architecture:
  Create highly optimized, scalable, and cost-effective architectures for processing large data sets and managing big data workloads using Databricks, Delta Lake, and Apache Spark.
• Implement Best Practices:
  Define and promote best practices for Databricks implementation, including data governance, security, performance optimization, and monitoring.
• Manage Databricks Clusters:
  Manage and optimize Databricks clusters for performance, cost, and reliability. Troubleshoot performance issues and optimize the use of cloud resources.
• Data Governance and Security:
  Implement best practices for data governance, security, and compliance on the Databricks platform to ensure that data processing and storage meet organizational and regulatory standards.
• Automation and Optimization:
  Automate repetitive tasks, streamline data processes, and optimize data workflows to improve efficiency and reduce operational costs.
• Mentorship and Training:
  Mentor and provide guidance to junior engineers, ensuring the team follows best practices in the development of data pipelines and analytics solutions.
• Keep Up-to-Date with Trends:
  Stay current with emerging technologies in the big data and cloud space, and recommend new solutions or improvements to existing processes.

Required Skills & Qualifications:

• Technical Expertise:
  • Extensive experience with Databricks, Apache Spark, and cloud platforms (AWS, Azure, or GCP).
  • Proficiency in programming languages such as Python, Scala, or SQL.
  • Strong understanding of distributed computing, data modeling, and data storage technologies.
  • Hands-on experience with Delta Lake, Spark SQL, and MLlib.
• Experience with Cloud Services:
  • Expertise in deploying and managing data platforms and workloads on cloud environments like AWS, Azure, or GCP.
  • Familiarity with cloud-native services like S3, Redshift, Azure Blob Storage, and BigQuery.
• Data Engineering Skills:
  • Experience designing, building, and optimizing ETL data pipelines.
  • Familiarity with data warehousing concepts, OLAP, and OLTP systems.
• Machine Learning (ML) Knowledge:
  • Experience in integrating machine learning workflows with Databricks, building models, and automating model deployment.
• Leadership and Collaboration:
  • Strong leadership and communication skills to interact with both technical and non-technical stakeholders.
  • Experience in leading cross-functional teams and mentoring junior team members.

Preferred Skills:

• Advanced Databricks Knowledge:
  In-depth experience with Databricks components, such as notebooks, jobs, and collaboration features.
• DevOps & CI/CD:
  Experience with DevOps practices, automation, and CI/CD pipelines in data engineering.
• Data Governance:
  Strong knowledge of data governance principles, such as metadata management, data lineage, and data quality.
• Certifications:
  • Databricks Certified Associate Developer for Apache Spark.
  • Cloud certifications (e.g., AWS Certified Solutions Architect, Azure Solutions Architect Expert).

Education & Experience:

• Education:
  Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field (or equivalent work experience).
• Experience:
  5+ years of experience in data architecture, engineering, and working with cloud platforms (preferably with Databricks and Apache Spark).

Thanks

Varun Anand

10294 A Baltimore National Park, Ellicott City, MD - 21042 | Ellicott City, MD

| Email: varun@resourcesys. com

Job Tags

Similar Jobs