People who write Python like dictionaries. People who write a lot of Python hate dictionaries. And it’s easy to see why! It’s tempting and easy to pass around random information between functions and modules as dictionaries, define your configs as dictionaries, process API responses as dictionaries…and it gets unwieldy fast.

In this article, we will be taking about:

  1. How we can tame dictionaries with dataclasses.
  2. Then we can also talk about how we can also use marshmallow to validate the data before it gets converted into a dataclass.
  3. Then we will talk about how we can use desert to keep all of the above DRY.
Image for post
The TLDR of this article

Note that while…


1. Extremely slow start times

https://medium.com/capital-one-tech/aws-glue-an-etl-solution-with-huge-potential-91a04a2a0712

The number one most annoying thing about Glue is that the startup times are extremely slow. It can take up to 20 minutes to start up a Glue job (but can take a little less time if you had run it recently) and that is not counting the time it takes to actually run the job. Compare that to the startup time of GCP’s Dataproc which typically takes around 60–90 seconds. This means that debugging a Glue job can often be a long, arduous process in which half of your time is just spent waiting for the job to…


I’m writing this because when I had this problem, there didn’t seem to be any resources online detailing how to solve this (not on Stack Overflow, AWS support page or etc). Hopefully even if your problem is not EXACTLY the same, this doc will still help. Also, if there is a bug with anything or if there’s anything I could have done easier, please let me know!

Also, I will be using the AWS CLI a lot and also the jq command a lot to parse the json I get from the CLI. So make sure that you have jq…

Jessica Chen Fan

Principal Software Engineer @Curology. Previously @ Etsy, TwoSigma, Amazon, Microsoft. Avid artist: instagram.com/jfan_art & singer: soundcloud.com/jfan6

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store