I used AI extensively during my internship to write code, yet surprisingly I didn’t feel like my workload was that light. Turns out the hard part about Data Science was never the code. It was:

• Figuring out the right questions to ask.
• Figuring out how to clean and stitch the damn data together.

Both of these require creativity, a quality that I don’t believe LLMs possess yet.

Figuring out the right questions to ask obviously requires creativity, but what about data cleaning and merging requires creativity? Consider this: at some point, in order to answer a very interesting question my manager came up with, I had to figure out a way to assign employee IDs to a certain transactional table. I cannot divulge too much since I signed an NDA, but let’s just say there was no convenient JOIN key in order to append that employee ID column to the table and the solution ended up involving matching different columns across 4 tables from 2 different cloud databases (there was a legacy system and a new system), a different matching rule for each campus, and a manual exploration loop to find the best pseudo-JOIN key for each campus. Not to mention long the way we found serious data quality/consistency issues, which every time we encountered we had to go on a sidequest rabbit hole figuring out why the data is like that so we can fix it or remove it from our end analysis. And after all of that we still ended up with only 80% of the rows matched.

Another time, I had to assign Fund Source Categories to each row in a payroll dataset. Someone had already done a mapping of fund codes to Fund Types which I could use. The problem: there were 4 different codes, and some of the codes weren’t in the mapping at all. 2 of them corresponded directly to the mapping, we’ll call them the golden codes. Problem is the golden codes were like 40% missing, and so I could only assign 60% of the rows in the payroll data. The other 2 codes looked like the golden codes, and at first I thought I could use them as a substitute, but I would later find they are actually ever so slightly different: a number gets replaced with a letter, completely different codes that could actually be mapped deterministically to the golden codes, etc. I had tried asking Codex for how we could match the rows better and it failed to notice those patterns. It was too fixated on using the golden codes, which were a lost cause given how much of it was missing. So these were all things I found out myself in a dataset of >1M rows! I also learned of some other patterns I could exploit: many of the fund types had a “range” of codes, using employee ID fund type sources from previous years to conditionally impute fund type sources in this year. The final product was a 15-step Python pipeline that tries every damn pattern I could find using the 4 codes given to me, achieving a 99.9% match rate and imputing the other 0.1%.

While we used AI to write the code that did all those steps, I doubt an LLM would have been able to conceive of such a scrappy, dirty, and roundabout way of stitching all this together. And yet it was the only way it could be done. If a Data Science AI startup can figure out a way to make the AI figure all that out from a single prompt “add Employee ID to this table”, then I’ll concede. Maybe if someone figured out how to build AI knowledge bases for all this…But for now, as long as data is messy and scattered everywhere, I don’t think Data Science is being replaced by AI until at least 2030.

Or really, maybe the bigger problem is my company needs to have better data hygiene…