I used AI extensively during my internship to write code, yet surprisingly I didn’t feel like my workload was that light. Turns out the hard part about Data Science was never the code. It was:

• Figuring out the right questions to ask.
• Figuring out how to clean and stitch the damn data together.
  Both of these require creativity, a quality that I don’t believe LLMs possess yet.

Figuring out the right questions to ask obviously requires creativity, but what about data cleaning and merging requires creativity? Consider this: at some point, in order to answer a very interesting question my manager came up with, I had to figure out a way to assign employee IDs to a certain transactional table. I cannot divulge too much since I signed an NDA, but let’s just say there was no convenient JOIN key in order to append that employee ID column to the table and the solution ended up involving matching different columns across 4 tables from 2 different cloud databases (there was a legacy system and a new system), a different matching rule for each campus, and a manual exploration loop to find the best pseudo-JOIN key for each campus. Not to mention long the way we found serious data quality/consistency issues, which every time we encountered we had to go on a sidequest rabbit hole figuring out why the data is like that so we can fix it or remove it from our end analysis. And after all of that we still ended up with only 80% of the rows matched.

While we used AI to write the code that did all those steps, I doubt an LLM would have been able to conceive of such a scrappy, dirty, and roundabout way of stitching all this together. And yet it was the only way it could be done. If a Data Science AI startup can figure out a way to make the AI figure all that out from a single prompt “add Employee ID to this table”, then I’ll concede. Maybe if someone figured out how to build AI knowledge bases for all this…But for now, as long as data is messy and scattered everywhere, I don’t think Data Science is being replaced by AI until at least 2035.