Address
Form of work
SalaryLocation: This is a hybrid role based in their Chicago office and will require you to be in office Tuesdays and Thursdays.
What’s so interesting about this role?
Our client is looking to bring on a Senior Data Engineer with chops in cutting edge real-time streaming technologies and ambitions to achieve high quality and reliability with TDD, automation, and continuous delivery. .
In this role, you will have the opportunity to work with our product teams, building models and APIs to drive new features, delivers analysis to further improve engagement in existing features, and empowers our business with real-time insights to drive growth in market share, engagement, and revenue. We see 1 trillion events per year and process 10TB of data daily.
What’s the job?
- Design, develop and deliver data products to production, complying with internal data governance, security and scalability of our system.
- Moving implementation to ownership of real-time and batch processing and data governance and policies.
- Maintain and enforce the business contracts on how data should be represented and stored.
- Stay on top of new technologies through R&D and prototyping to continuously improve our big data architectures and systems to streamline how we deliver value with high quality to our end users
- Implementing ETL processes, moving data between systems including S3, Snowflake, Kafka, and Spark.
- Work closely with our Data Scientists, SREs, and Product Managers to ensure software is high quality and meets user requirements.
What we’ll love about you
- 5+ years of experience working with data at scale, including Data Engineering, business intelligence, data science, or related field.
- 7+ years experience using Python,
- Experience using big data technologies (Snowflake, Airflow, Kubernetes, Docker, Helm, Spark, pySpark, )
- Significant experience with relational databases and query authoring (SQL) in Snowflake or other distributed Databases.
- Experience with agile engineering practices such as TDD, Pair Programming, Continuous Integration, automated testing, and deployment.
- Experience with building stream-processing systems, using solutions such as Kafka, Storm or Spark-Streaming
- Experience with dimensional data modeling and schema design in Data Warehouses
- Familiar with ETL (managing high-quality reliable ETL pipelines)
- Be familiar with legal compliance (with data management tools) data classification, and retention.
Location : This role is twice a week in a hybrid role minimum.
