↓Skip to main content

Michael Ayoub

reachme@michael.ayoub.io - linkedin.com/in/michaelayoub - ayoub.io

Skills #

Cloud: Experience with both AWS: CloudFormation (CF), Cloud Development Kit (CDK), Managed Streaming for Kafka (MSK), Lambda, DynamoDB, Fargate with Elastic Container Service (ECS), Kinesis Streams and Data Firehose, Glue, Athena, S3; Azure: Event Hubs, Logic Apps, Stream Analytics, Azure Active Directory

Platform Administration: Elasticsearch with Logstash and Kibana (ELK), Kafka, Linux (Ubuntu and CentOS)

Data Engineering: Extracting data from APIs and databases; writing complex transformations in Python, Pandas, and SQL; building data pipelines with Logstash; executing multi-step data workflows with dependency resolution using Makefiles and Luigi; using dbt for analytics engineering in SQL Server

Development: Python (FastAPI), Java, JavaScript/TypeScript and Node.js (Hono), SQL (T-SQL on SQL Server, Athena/Trino)

Analysis: Experience in developing functional design documents; successful at bridging the gap between business and technical realms; preparing prototypes; user-centered design; lean product development.

Experience #

[March 2022 – Present] Engineer III for Mutual of Omaha (Omaha, Nebraska)

  • Presented new data lakehouse designs using Apache Iceberg for transactional semantics in S3
  • Used DuckDB for data migration and backfill
  • Successfully navigated cross-functional and interdisciplinary design sessions around complex topics like sensitive (confidential) event data and JSON Schema design
  • Design and proof of concept event catalog in Spotify Backstage to show event schemas, producer/consumer authorization, and producer/consumer relationships
  • Built distributed workflow system using functional event sourcing, TypeScript, AWS Lambda, DynamoDB, and DynamoDB Streams
  • Built an analytics platform for MuleSoft APIs using TypeScript, AWS Lambda, Glue, Kinesis Streams & Firehose, and Athena
  • Hosted team book clubs to mentor and grow other engineers (Designing Data Intensive Applications, The Rust Programming Language, and Algorithmic Thinking)

[July 2021 – March 2022] Senior Consultant for Object Partners (acquired by Improving) (Omaha, Nebraska)

  • Built Python APIs using FastAPI to expose data in Snowflake, deployed to Azure Kubernetes Service
  • Proof of concept using dbt for extract-load-transform (ELT) workloads in Snowflake, paving the way for data engineers
  • Built developer tools and wrote documentation for using Apache Beam with Apache Flink on Azure Kubernetes Service

[January 2021 – July 2021] Senior Technical Solution Architect (API / Event Driven Architecture) for Mutual of Omaha (Omaha, Nebraska)

  • Responsible for enabling developers to use services for implementing event driven architectures in public (AWS) and private clouds, like AWS Managed Streaming for Kafka (MSK), SNS/SQS, and EventBridge
  • Responsible for participating in third party SaaS and PaaS solution evaluation, feature comparisons, and requirements analysis
  • Automated generation and association of SASL/SCRAM principals in AWS Secrets Manager to AWS MSK clusters using Cloud Development Kit (CDK) and usage of those secrets in Fargate services
  • Built data analytics pipelines with Python, dbt, Ansible, and SQL Server for presenting MuleSoft Anypoint platform metrics, KPIs, and transaction logs for other developers
  • Proof of concept deployment of Spotify Backstage developer portal, including writing plugins in React and TypeScript pulling data from internal APIs to expose backend system information

[April 2020 – January 2021] Technical Solution Architect (API / Event Driven Architecture) for Mutual of Omaha (Omaha, Nebraska)

  • Responsible for enabling developers to implement event driven architecture in public (AWS) and private clouds
  • Enabled AWS Managed Streaming for Kafka (MSK) for developers using Cloud Development Kit (CDK) and TypeScript
  • Automated generation of key stores for client certificate authentication to Kafka using AWS Lambda and Java
  • Wrote documentation for onboarding development teams to Kafka and covering core concepts like topics, partitions, consumer groups, and producers/consumers

[October 2019 – April 2020] Senior Software Engineer (API / Event Driven Architecture) for Mutual of Omaha (Omaha, Nebraska)

  • Responsible for automating configuration of Apache Kafka clusters with Ansible, documenting best practices for integrating applications with Kafka, and being a North Star for internal teams on leveraging streaming data platforms.
  • Deployed Kafka to on-premise virtual machines using Ansible
  • Proof of concept evaluation of Apache Atlas as a catalog of events and streaming applications
  • Gained familiarity with MuleSoft Anypoint Platform

[September 2017 – October 2019] Information Security Analyst (Development and Automation) for Kiewit Technology Group (Omaha, Nebraska)

  • Responsible for architecting, building, and maintaining a SIEM (Security Information and Event Management) data platform from open source tools, including Elasticsearch, Logstash, and Kibana (ELK Stack), Kafka, TheHive, and ElastAlert
  • Driving implementation of Kappa Architecture streaming data solutions with Kafka Streams and KSQL across multiple teams, including a proof of concept building comprehensive joined views of objects in SAP based on change-data-capture (CDC) streams of SAP tables into Kafka
  • Deployed Logstash data pipelines to Docker Swarm, enabling scalable ingestion for peak activity (400+ GB/day, 370MM+ logs/day) and one-command packaging and deployment of new ETL containers, enabling our analysts to gain insights from data more quickly
  • Trained team members on JSON and using REST APIs from Python, cURL, and PowerShell
  • Deployed Traefik and OpenFaaS onto Docker Swarm, providing a flexible platform for service development
  • Developed Python services for data extraction from 3rd-party services including Office 365 Management API into Kafka, with scheduled execution on Docker Swarm
  • Developed complex ETL in Pandas to extract meaning from SAP Audit Logs, enabling real-time monitoring of SAP transaction code execution
  • Automated IR (Incident Response) case management in TheHive, including a custom ElastAlert connector that uses a version-controlled repository of Incident Response Plans
  • Developed data Enrichment suites, including front-end tools for analyst maintenance of contextual data, a PostgreSQL DB + PostgREST API utilizing triggers in PostgreSQL for auditing of changes, and Ruby scripts for Logstash to enrich security logs with up-to-the-second intelligence
  • Proof of concept using Spark SQL and the Elasticsearch-Hadoop bridge to develop rich interactive analytics system for TheHive data

[October 2016 – September 2017] Business Analyst for Kiewit Technology Group (Omaha, Nebraska)

  • As a business analyst, responsible for working with domain experts to gather requirements, and lead testing efforts on new product development
  • Developed complex ETL (extract, transform, load) of $1B+ of SAP transactional data into Ariba Spend Analysis, using Python and T-SQL
  • Developed proof of concept apps for Microsoft Hololens using C# and Unity, including the MR (Mixed Reality) Sharing libraries for Hololens-to-Hololens networking and voice recognition

[November 2015 – October 2016] Systems Engineer for Kiewit Energy (Lusby, Maryland).

  • Responsible for material master data strategy for all engineered equipment on a $4B LNG plant expansion project, including organization, nomenclature, and tagging
  • Realized $1.1MM in efficiency gains by designing, developing, and deploying a new material management system handling over $12MM/year in transactions with complex cost accounting across engineering disciplines.
  • Reduced time-to-replenish by hours every day with a reporting application in Java that combined transactional data, SAP warehouse information, and a map of the warehouse to produce optimized pick tickets
  • Improved engineer efficiency and decreased errors by automating material data upload generation

[August 2014 – November 2015] Business Analyst for Kiewit Technology Group (Omaha, Nebraska).

  • Started as an intern in August 2014. Hired full time in February 2015.
  • Responsible for writing functional specifications for SAP enhancements, writing and executing test plans, and facilitating UAT (user acceptance testing).
  • Reverse-engineered Windows Mobile 6.5 browser interface to SAP, and injected JavaScript into final deployments to adapt the shortcut keys to our use cases and particular equipment

Presentations #

MoO Conf 2023: internal conference talk on building streaming data platforms using Kafka, Debezium, and ksqlDB, and reactive live dashboards with Svelte and SvelteKit

Education #

[2009-2014] Bachelor of Science, Mathematics at Florida State University. Minor in Computer Science.

[2009-2014] Bachelor of Arts, Philosophy at Florida State University. Minor in Russian.