Summary

This article advocates data-heavy teams and workflows to move out of Excel and into modern tools like cloud databases for storage, ETL pipelines for data processing, and web dashboards for visualizations. Not only does Excel have a hard limit at 1M rows, but moving to a modern data stack would streamline many workflows and save a lot of manual time. Analysts’ time is best spent analyzing, interpreting, and asking good questions about the data – NOT copy and pasting data from one Excel sheet to another Excel sheet.

Hard Limits

First of all, Excel has a hard limit at 1M rows. Beyond 1M rows, moving away from Excel is no longer just “nice” or “more modern”, it is literally impossible to process all your data anymore in Excel. Maybe the data you want to analyze doesn’t have over 1M rows now, but what about in the future? What if you want to do some longitudinal/time series analysis incorporating decades of data? There are trade-offs and considerations for premature optimization, but in most cases, if you expect to have a lot of data work in the long term, you should be moving away from Excel.

Manual Effort Scales D x t x K x N

Say for example I have an established Excel-based workflow as follows. I need to pull data from sales, so I send an email to them detailing what data I need, they send me back an Excel file, and and then I process it myself taking t minutes of manual effort. Well, ok now I want to contextualize that data with revenue and expenses, so I contact accounting, they send me back an Excel file, and that’s another t minutes. And I do this for the next D departments I need data from. Total for one dashboard: D x t minutes. Ok the stakeholders really like the dashboard and they want it updated regularly, so now I have to refresh the data K times every year.

So for one dashboard, it takes D x t x K minutes. But does your organization only have one dashboard? Am I the only employee making dashboards? Let’s assume N dashboards across your entire organization.

Assuming 15 minutes of manual processing per data source, 2 data sources per dashboard, 10 dashboards, all refreshed 12 times a year. That’s 15x2x10x12 = 3600 minutes = 60 hours.

How do we optimize this? Well I would assume D (data sources), K (refresh regularity), N (num dashboards) are non-negotiable, AKA all the dashboards are important and worth maintaining. This leaves us with t. Well, I’ll tell you this right now: the modern data stack can make t = 0, because you only need to write the code once and a computer can execute it for you forever. The trade-off is an upfront cost in writing the code. But in the long term, once the code has been written, t = 0.

Well we can’t move everything out of Excel

And you’d be right! Accounting, small teams, one-time analyses from 1 small data source – if D, N, and/or T is small, Data Engineering would be overkill. But for any serious data-intensive work; one where you need to connect data between different departments, crunch massive amounts of numbers, or process >1M rows, Excel is not the tool. It was not designed for heavy data work.

Code is Self-Documenting

And the best part? Modern data stacks use coding languages, and code is self-documenting. When you need to do some sort of data transformation in Excel, you do it visually; drag and drop and click and whatever. But the steps aren’t documented, unless you get your employee to write an SOP.
Compare this to writing code. When you are writing code, you are by definition writing out the steps to do the data transformation so that a computer can execute it. Code by itself already documents a workflow. Can’t write code? No worries. AI is good enough that it can read the code, explain it to you, and write understandable documentation of it. But in Excel workflows, none of the steps are inherently written down. It’s all in the employee’s head.

“What if the Data Engineer leaves?”

- you might object. Well, What if the guy maintaining the Excel sheet leaves?
At least with code anyone can just run the script in the click of a button. I’ll do you one better, the Data Engineer can probably one-shot vibe-code a lil’ app for it. Upload your file here, press this button, and get your cleaned data or summary visualization, back in 10 seconds.
At least with code it’s self documenting. Don’t understand the code? No problem, pop it into your favorite LLM and ask it what the code is doing.
With code, knowledge of the workflow becomes institutionalized within the organization. With Excel, knowledge of the workflow stays in the employee’s head. Good luck if they suddenly leave or get sick.