Today, I want to shift gears a little bit and talk about technology. This podcast is focused on speaking to the people building a business, managing business processes. If you’re a technician, this is probably not going to be the best podcast for you, because we’re not going to be getting into the nitty-gritty of the technical details of what it takes to build business intelligence and data warehousing solutions, we’re going to do kind of the opposite.
I feel like technology is a wonderful thing. It’s an enabler. It’s what helps us get this job done. In the end, it’s a tool. It’s extremely important, but at the same time, I see, in many cases, that it gets in our way. We just have this tendency to want to jump to the latest and greatest technologies all the time, and today I want to talk about why we should just forget about technology.
As I was taking notes for this episode, I realized that this could come across as saying, “Don’t ask questions.” And that’s not it. You need to understand what type of technology is being used to implement your solution to some extent. That reminds me of the movie A Few Good Men where, I believe it was, the Colonel was on the stand in a courtroom being questioned and he had a famous response. I went and looked it up and I found this quote. He said, “I have neither the time nor the inclination to explain myself to a man who rises and sleeps under the blanket of the very freedom that I provide and then questions the manner in which I provide it.” That is a good quote, I thought, and it’s not what I’m saying. Here is the reason I’m bringing it up. We’re not saying, “Don’t question the people or the company that’s implementing your solution about the technology they’re using.” We’re just saying that should not be the focus here. The type of technology being used should be the least of your concerns.
What we’re talking about with building a Reportopia is solving business problems, helping you operate more efficiently, capturing revenue opportunities, and reducing expenses and risks. That’s what we’re talking about. How we do that is using technology, but I will make the case that you should use the most boring technologies out there, frankly. Not the new, shiny technology that every tech head wants to use. That’s not the approach we should take, and that’s what we’re going to talk about today.
Now I also preface this and say, the audience of this episode, and really the whole podcast, is the mid-market. We’re not talking about how Google should implement their business intelligence solutions, or Apple, or any mega-large company. It’s going to be a different world. We’re talking about companies that are all the way from the small startups up to, I’d say, the 500-1,000 employees or less, something in that ballpark is usually the audience that I’m trying to speak to. That’s important because the size of an organization has an impact on the type of technology that needs to be used to solve these problems.
The good thing about being in this segment that I’m talking about is your problems are quite easily solved with technology today. There’s no need to take excessive risk on new technologies that are unproven or difficult to find resources that know how to use them. They’re so new that there’s no way anybody could be a true expert in them. There’s no reason for that. We’ve got tried-and-true technologies that solve the problem well, and they do it efficiently, and those are the technologies we need to be using.
I was on a sales call recently that kind of perked my ears up, because I hear these things quite often. When I get on a call and we’re getting to know each other, it’s a new company, and I explain a little bit about LeapFrogBI. Then the client or the prospect explains a little bit about their needs. When the client is talking about their needs, they’re not talking about the business needs. They’re thinking about the implementation, the technology selection, the architecture, all these things. I’m, of course, thinking in the back of my head, “Where’s the value opportunity here? Are they just not communicating it to me, or is it that they’re focused on the technology? Am I talking to the wrong person? Am I talking to the implementer here and not the people that are needing to have a problem solved?”
And that’s where the idea of this episode came from. To me, building a Reportopia is about adding value.It’s about helping organizations meet their goals. If you’re in healthcare, it’s about serving patients better. If you’re in e-commerce, it’s about selling products that your customers want in value. If you’re in marketing, well, then it’s about getting data into a format that your subscribers can use to serve their clientele better. All these things are going to use technology to be successful, but technology is not the focus. The focus should be on solving the problem.
And that’s where we get into this kind of a slippery slope. When we are working with technicians, we spend our careers learning about technology. We know the nitty-gritty of every little gotcha within our area of expertise, and we’re excited when we see that something new has come out that may solve those problems. At the same time, those new things, while they may solve the problems that we’re aware of, they may also introduce other problems. I would say there’s no reason for companies in the mid-market to go out and try to spend time and money on these new solutions.
In our practice at LeapFrogBI, we are using relational databases to serve clients that have large solutions. I’m talking about terabyte-size solutions. Most organizations don’t need that size of a data warehouse, not in the mid-market. Certain industries have larger data quantities than others, but, whenever we’re building solutions to solve specific problems, we’re not talking about moving every piece of data we can ever find into some big data repository and then figure out how we’re going to use it. That’s not the approach we take. The approach we should be taking is, we find the value opportunity and then we build a solution to fulfill that particular opportunity. When we do that, we’re not bringing in all the data that we can ever find and put into some big repository. Instead, we’re bringing in the information we need and solving the problem that we need to solve.
So I’m going to talk a little bit about technology because I have a pretty good example. At LeapFrogBI, we use the relational database engine. If you’re not familiar, this is basically the type of technology that applications have been using to store the information that you’re inputting into a front end. They use these relational databases pretty much across the board. There are some exceptions, but relational databases have been around forever. They’re proven, they’re economic, they’re efficient. The relational database is a very good solution for the needs that we have in building these reporting solutions.
There are other options out there. A couple of options that are available are in the Azure set of services. One is, an Azure SQL Database. This is basically a relational database, same technology, except it’s serverless, meaning that you purchase just the database. You don’t have to host the instance of SQL Server; you don’t have to have a virtual machine or actual hardware. You don’t have to have an operating system. You just have a single database that you’re subscribing to.
Why not do that? Why not get rid of all this other stuff, such as the server and the operating system etc., and just subscribe to the database. I’ve looked at that. We’re always looking for better opportunities to improve our technology solution and we absolutely would recommend to our clients to move to those solutions once they’ve been proven. When I say “proven”, I’m not talking about six months in the market, and you got a couple people who know how to use it. No, that’s not a solution that our clients or that I would recommend that anyone in the mid-market go after. A proven solution is one that’s been around for years that has plenty of people that know how to use it, we know what the pros and cons are of it, we know how to get the job done. That’s what I mean by “proven”.
Let’s go back to a couple of options, if we were going to use Azure SQL DB, let’s first talk about cost. First, the cost of a single database would be around $500 per month for a single database, and that may not mean a lot to you because I know I’m not talking to a technical audience here, but I think you understand what $500 a month means.
The first problem we have is, whenever you’re building a solution, you’re not just building one database. You’ll need, usually, at least three, if not more. You must have a development environment, you’ll have a production environment, and usually you’ll have some type of a reference database which maintains things like the goals that you’re trying to meet or things that aren’t found in your data source. You have at least three of them here and oftentimes we’ll have as many as a dozen or more, let’s just say it’s three. That $500 a month just turned into $1,500 a month. Now we’re talking about, significant amount of money.
That’s not where the problems end, because when you subscribe to this Azure SQL Database, well, you don’t have a SQL Server Agent. The agent is what we need to orchestrate a lot of the processes that must happen. I’m not going to go too far in the details. If you just think about it as a set of jobs, and that’s exactly what the SQL Server Agent calls its processing tools, “jobs”, and these jobs need to do A, B, and C. We can automate the process of executing those things by using the agent’s jobs. We can schedule things, we can set up notifications and alerts, ect.
We don’t have that in Azure SQL DB, but there is another solution, it’s the automation resource in SQL. Now we need to go subscribe to another resource, we need to learn how to use that other resource, and guess what? That other resource is not just a generic SQL Server Agent resource or a specific SQL Server Agent resource, it is an automation resource that can do anything. We use the resource for other reasons, but it’s not specifically built to execute jobs within a SQL Server environment.
So the point here is you can use an automation resource to get the same work done, but it’s not going to be nearly as efficient or as purpose-built as a SQL Server Agent. What does that mean? Well, it means more cost. Means you need to build more. It means you have more difficulty in maintaining and adjusting to changing needs.
There’s a few issues. One more thing. In that situation where I was describing what SQL Azure DB has, you pay for it by the database and we’re saying we’re going to need at least three of them. Well, in SQL Azure DB, you can’t query across those databases by default, you must jump through some hoops. I’ve done the jumping through hoops, and I can tell you that that’s not an approach that you’re going to find very appealing when you’re having to build this reporting solution using this elastic DB. Basically, you’ve got to do some things to enable the databases to talk to one another. That is just a feature that’s natively inside of the SQL Server if you hosted yourself.
To recap, we have an option. We can go host SQL Server as our technology in our own operating system on our own virtual machine, or we can use SQL Azure DB. I gave you a few reasons why I don’t like the Azure SQL DB solution.
#1: You get one database; it is going to cost you around $500 a month for even a small database. That is measured in DTUs or data transfer units. #2: We don’t have a SQL Server Agent, so now we must go build something else using automation to solve that problem. #3: We can’t do cross-database queries without jumping through hoops. Those are three pretty good reasons, I think, to not go down that road. And another reason is, if you want to stop the billing on an Azure SQL Database, you’ve got to tear the whole thing down. In other words, you’ve got to decommission the resource, deallocate the whole thing, and when you do that, you’re going to lose the information. If you want to stop the billing, you need to back up the database, take the whole thing offline and then terminate the resource. Not just stop the resource, but terminate it, and then rebuild it when you’re ready to go again. It’s just not feasible to do that.
There is another option, subscribing to a managed instance in the cloud.
The cloud has a lot of resources that are available to us. It solves a couple of problems. #1: I can have as many databases as I want in this instance. Solve that problem. #2: It has a SQL Server Agent. That’s another problem solved. #3: You can do cross-database queries because you’re basically running an entire instance in the cloud, just not on a server that you’re hosting.
It is pretty good, but at a minimum you’re going to spend $1,000 a month for the SQL DB instance and there’s other problems. For example, when you run a SQL DB instance and you try to back up the databases that are in that instance, you’re not going to get a backup file that can be restored to anything other than a SQL Managed Instance. That’s important. I don’t think anybody out there likes to be stuck. If you have a SQL Managed Instance, you can’t even back up those files and restore them if you change your mind and decide you want to go on virtual machine running SQL Server instance, because it’s two different versions of SQL Server.
There are ways around that, you can try to create a backpack, but I would just say, “Go and try it.” See how long it takes to create a backpack file to overcome that problem. You’re going to end up jumping through a lot of hoops there as well. And the point of all this is not to say that any of these solutions are not appropriate for certain workloads. They certainly are. I’m just saying that these solutions are not appropriate for a reporting solution back end for the middle-market.
In that example I gave where we have a SQL Instance running, it’s just another resource you can provision in Azure. I was saying that’s going to cost you around $1,000 a month minimum. It’s very likely it’s going to be a heck of a lot more depending on the workload. Let’s compare that to the tried-and-true method of running SQL Server on a virtual machine, we use Azure Virtual Machines because we try to reduce the cost.
If you run SQL Server in a virtual machine and you set up your architecture so that you don’t have direct queries hitting that database, in our case, we use Power BI. We push datasets out to a Power BI repository and then we can remove the dependency on the data warehouse itself. That means that you can turn that virtual machine off, and with the virtual machine, you can deallocate that machine and stop the billing. If you’re paying for SQL Server by the minute, hypothetically, there’s other ways of doing it, but this is the way we like to get started. It’s the cheapest way of doing things and gives us the most flexibility, prevents us from being stuck with one certain solution.
We can run some of our small clients for as little as $100 a month, sometimes even less, if you can believe it. $100 a month, and that’s with the full, standard edition, current version of SQL Server with all the tools that we need. When we decide that we need to use a file system, we’ve got a file system. When we need an agent? We’ve got an agent. And for comparing apples to apples, I have to say that now we have a server to manage. Well, that might cost us about 50 cents a month. So, it’s not going to cost us much time to manage an instance of an operating system in a virtual machine, because we know how to do that. We can automate most of it and reduce all those costs.
So in this little long-winded scenario, at LeapFrogBI, we still choose to use virtual machines, single-tenant. We don’t co-mingle client information, when single-tenant virtual machine is running a full instance of SQL Server, we got all the tools that we need, and in some cases, it can cost as little as $100 a month. Eventually it’s going to cost more than that for most clients, frankly. Most clients are going to probably be in the $500 a month and some of our large clients might be over $1,000 a month. I’m just trying to compare apples to apples when we’re talking about using, SQL Server in a VM versus SQL Azure Database or SQL Managed Instance.
The numbers don’t lie, but the cost is not the only equation. It’s great that we can do this for $100 a month using a VM running SQL compared to $1,500 in a SQL DB scenario or $1,000 or more in a SQL Managed Instance scenario. Obviously, the lower cost is important, but probably more important, in my opinion, is this idea of not getting stuck. If I decide, one day, that I’m not using Azure anymore, I don’t want to have to recreate everything I just spent the last few years building. That would be a catastrophe. I just want to have an easy way to know that if I decide to move to AWS, or if I decide to take this stuff into hardware that I’m running in-house, I want to have that option. Otherwise, we’re stuck. We’re going to be stuck to some extent, because we got to pick the tools and the solutions that we need to get the job done, but there’s no reason to go overboard with that. We can maintain some of this flexibility.
There’s going to be some people out there that might think that I’m anti-technology and there’s nothing further from the truth. I love technology. I love new technology. I like going out and playing with the new stuff. I’m just saying, I don’t recommend using any of the new stuff until it’s proven long-term, and by long-term, I can’t imagine how a new technology can be proven in less than five years. There’s so much to know, there’s so much to learn, and there’s no reason for the clients that we work with in the mid-market-sized organizations to take on that cost. It’s just too high.
If you do want to do that, I wouldn’t consider that to be part of your reporting solution. I’d say that’s some type of other side project. It’s too important. Not only is it costly to fail because of the cost of the effort that you’re putting into this, but it’s also very costly to not capture the opportunity. Often, that’s the bigger cost. We’re trying to capture hundreds of thousands, if not millions of dollars’ worth of opportunity on an ongoing basis. If we don’t capture that opportunity, well, that’s bad. Do we really want to risk that on our desire to go out and play with the newest tools? And I think the answer is “No.”
Another thing to consider about new technologies, I think I mentioned it earlier, is it’s very difficult to be an expert and it’s also very difficult to find experts because it’s new. No one has had time to go out and really put it through its paces, and really figure out what is new technology good or not good at.
Now, of course, the marketing material’s going to say it’s good at everything, right? “It solves all the problems.” I remember when we first started getting these MapReduce-type programs like Hadoop and that sort of thing and the buzz was, “Oh, that’s the end of data warehousing”. I thought, “What does that even mean?” Because data warehousing is not even a technology. Data warehousing could be on any technology. Even if Hadoop replaced the relational database engine, then you would have a data warehouse in this other technology, but some of this stuff is just nonsense, I think. It’s just people being excited about the new technologies, which is great, but there’s no reason for the mid-market to take that risk. Let Google take that risk. They have a lot more resources to go invest in these things that may or may not be successful.
It’s difficult to find people that are qualified in these solutions, and it takes a long time to become an expert. You certainly need to be an expert in the technology you’re using, or else you’re going to lose focus on what we’re trying to do. Capture value. Instead, you’re going to be thinking about how to overcome technical problems that no one else has solved before. You’re on the cutting edge, so now you’ve changed from being an insurance company to being a company that’s going to try to figure out the latest and greatest data technology. That’s not where you want to be.
Another analogy I want bring up is kind of about how I was raised. I grew up in a small town in Southeast Texas called Cheek, and my family always had a farm. It was a soybean farm, and my dad worked in a refinery. It was a small-town type of upbringing. We always had equipment to do the job that we needed to get done. I’d say it was kind of old equipment, for that matter. It’s not like we would go out and buy a brand-new John Deere and get new implements for that equipment. Then go out there and double the amount of acreage that we could work on overnight, that wasn’t the way we did business. Not to say that it’s not nice to have new equipment, believe me, there was many a times when I thought, “Man, this stuff is junk. We’re working on it all the time.”
So having new equipment can solve some of those problems. Although, buying new equipment isn’t exactly the cat’s meow when it comes to having to work on your equipment because new stuff breaks down as well. Another side note, my family took our first venture in a RV. I bought a new travel trailer, and if you want to talk about new stuff breaking down, I’ve never seen something break down so often as a travel trailer. I’ve learned, that in the travel trailer market, it’s better to buy used. The new stuff just breaks down all the time. You must get the kinks worked out until you get to the point where you have something that’s reliable.
Anyway, so back to my farm analogy. We had old equipment, and we had to work on it a lot. But at the end of the day, when you go out into the field and you’re plowing. You’re on a tractor pulling a plow. Even though I never had the opportunity to pull a new plow, I would bet that the dirt had no idea if it was an old plow or a new plow that was being pulled across the ground. The dirt just doesn’t care.
The same thing is true when we’re talking about building these Reportopia solutions. It doesn’t really matter at the end of the day if you use note cards or if you use the shiniest new technology. What really matters here is that we’re able to capture opportunity. I know that I probably sound like a broken record, but I see this so often. People lose sight of what we’re trying to achieve and instead, they’re so focused on technology that they just become blinded to the overall goal.
It’s not about the technology. I would say, for the business and the business-focused people. You’ve got to go and find someone that you can trust. You need to find someone that has the skills or company that has the skills to do what you need done. Hopefully they have a proven track record of doing something like what you need. So you have a level of confidence that this company knows how to do it, and you can trust them.
That’s kind of the leap of faith, but what you don’t want to be doing is reading about some new-fangled technology and going back to those people and saying, “Hey, what about this? Maybe you should try this out.” That’s not the right approach. If you’re in an industry, focus on the problem at hand. Don’t focus on all the technologies that might be better. There’s a time for that, maybe you want to dedicate a meeting every year to go out and just evaluate all the new technologies, but don’t let that disrupt what you’re trying to accomplish.
At the end of the day what I’m suggesting here is, with the focus being on businesses in the mid-small market, forget about technology. Use the boring stuff. It’s cheaper. It’s tried-and-true. It is going to work. There’s no reason to take risk on using any new technology. You’re going to pay a premium for it, you’re going to have difficulty in finding and retaining resources that are going to work on it. I’ve listed several different reasons why I think that’s the wrong approach, to be chasing the new technology.
One other thing I want to say is, “guess what?” As soon as you chase that new technology, there’s going to be another new one. It’s going to come quickly, and that means that you’re no longer in the new technology. That’s going to become a very, very expensive endeavor if the mode of operation is that we’re going to stay on the latest and greatest technology, so to speak.
I hope that the message that I’m trying to relay here is coming across the way that I intended. Again, I’m not saying, “Don’t ask questions.” You need to understand everything about your solution. You need to be able to trust the people that are implementing your solution, but don’t chase technology. Let’s use the tried-and-true, proven approach and stick with it. Let’s focus on what matters, adding business value.
