Data & AI Engineering onboarding pack

This document IN PROGRESS and is TEMPORARILY PUBLIC [In Progress 11/07/2024]

Welcome to aiimi!

This is your onboarding pack. It is not intended to teach you to become a data engineer. Rather, it aims to quickly acclimate you to the aiimi landscape and help you to understand our commonly used approaches and technologies.

In addition, all aiimi employees are enrolled to the Aiimi Leadership Training Program (ALTP), which will help develop you further as a consultant at aiimi.

Note that '00 Resources' contains at a glance documents which should help you during client engagements. Use these when you need quick inspiration.

The rest of the curricula is in its suggested order.

---

Onboarding curriculum:

• 00 Resources

  • Useful links

• 01 Orientation: [WiP]

  • Data engineering specific intros

• 02 Consulting basics:
  • Customer focus
  • Professionalism
  • Presentation skills
  • Scrum & Agile working
  • What makes aiimi different
  • Understanding client needs
  • Technical expertise
  • Staying up to date with industry developments
  • Leadership
  • Role model

• 03 Our commonly used tech stack and recommended learning pathways:

  • Languages: PySpark, SQL,
  • Azure eco-system, Databricks, etc.
  • Microsoft learn, suggested certifications etc.
  • Version control

• 04 Our commonly used architectural patterns:

  • Data lake and Data Lakehouse Medallion architecture
  • Data modeling techniques (Inmon, Kimball, OBT, etc)
  • Data warehousing concepts (ETL, OLAP vs. OLTP), etc

• 05 Case studies: [WiP]

  • Successful projects
  • Lessons learned, mistakes we’ve made
  • Examples of how different practices operate together e.g. data engineering, software, data science etc.

• 06 Technical playbooks: [WiP]

  • Code examples

• 07 Advanced techniques: [WiP]

  • Pipeline best practices
  • Spark optimisation
  • AI Engineering
  • Software engineering best practices
  • CI/CD
  • Docker

• 08 aiimi style guide:

  • PySpark
  • SQL

Contributing

To contribute to this repo first clone it, make a new branch, make your changes/additions, commit your code, then open a pull request with your proposed changes.

Example steps...

Clone the repo:

git clone https://github.com/AiimiLtd/de-ai-onboarding.git

Change to the directory:

cd de-ai-onboarding

Create a new branch:

git checkout -b your-new-branch-name

Make your changes.

Commit your changes with a short message:

git add .
git commit -m "added new feature to enhance X"

Push your changes:

git push origin your-new-branch-name

On GitHub, open a pull request (PR) to the main branch of the original repo.

Assign reviewers. Address any feedback from reviewers. Merge Your Changes. Once approved, your pull request will be merged.