Name: The Hidden Bill of Bad Data: How Inaccurate Infrastructure Drives Overspend and Risk
Uploaded: 2026-03-04T16:01:52.020Z
Duration: 34 min 24 s
Description: The Hidden Bill of Bad Data: How Inaccurate Infrastructure Drives Overspend and Risk

Transcript for "The Hidden Bill of Bad Data: How Inaccurate Infrastructure Drives Overspend and Risk": Hi, everyone. Thanks for joining us for our webinar, the the hidden bill of bad data. We're just gonna give everyone a chance to settle in, get ready for for this Wednesday morning or afternoon webinar, depending where you are. Thanks for joining. Let's just give it twenty or thirty more seconds before we get started. Okay. I think we'll we'll get started now. We're we're a couple minutes after. Hi, everyone. I'm Lauren Malhoite, the director of solution architecture at IP Fabric. And today, I would like to talk about the hidden bill of bad data. So I'm gonna show some slides, but I'll also be giving you some little snippets of demos, live demos of what's possible today with IP Fabric, both alone and also how we can help you use the tech stack that you're already working with, so that you could use it to its full potential. And I wanna start with a a very simple idea, which is that every error in your cloud and infrastructure data has a price tag. So some of those price tags are are small and annoying like you might expect, but some can actually change the trajectory of an entire network. So across industries, we're spending more on technology than ever before. I know this is probably not shocking to hear, but, we're spending over $5,000,000,000,000 globally each year according to Gartner. When I've spoken privately with CIOs, CTOs, even CFOs, I hear the same frustrations, which is we're spending more than ever. We still don't know what's really running on our network, what we really own, and and most importantly, where our risk really sits. And that's because the single biggest blind spot in most enterprises isn't observability. It isn't even security. Right? It's bad infrastructure data. It's incomplete, inaccurate, out of date, and sometimes even contradictory. And when the data about your network is wrong, everything that depends on it. Right? Maybe you have a new AI AI ops practice that you're trying to implement, your cost models, your risk models, security posture, transformation strategies. These all inherit that wrongness. And what I wanna show you today is that the problem is actually probably far bigger, more expensive, but also more fixable than most organizations realize. So today, I'm gonna walk you through some customer stories based on real life conversations that I've had, and then talk about how we can improve this wrongness, and and work towards more efficient spending and more mature security and compliance postures all at the same time. I wanna encourage anyone to, ask questions anytime throughout. Love to make it interactive. But I'm gonna start with a real story from an energy company that we're working with, and they've gone through multiple, mergers over the years. And like most organizations in that situation, their network became a a patchwork of inherited designs and duplicate tooling, inconsistent standards, and, of course, stale documentation. So they had suspected some, CMDB drift, some change management database drift, but they didn't know the extent. And by the way, when I say CMDB, this this applies to anything that lists your inventory, right, or acts as a source of truth for the com for this company, it was their CMDB, but I hear it all the time about IPAMs or asset managers. Right? Anything that that is a list of your your inventory. And these solutions, by the way, are very necessary. But if they're not supported by something automated and up to date, the source of truth quickly becomes stale. So based on our customer suspicions, we ran a factual comparison between what their CMDB said they had and what was actually operating on their production network. And this was the basis for a proof of concept that we ran on a on just a sample part of their production network. And by the way, when this is a use case for us, we we always run this comparison in our POCs. So their CMDB listed 425 devices, but the actual network only had 383 devices, meaning that 42 devices were what I refer to as phantom or devices that were probably once in production but were were pulled for some reason and and not documented or or updated. Now 42 devices isn't a rounding error. It's it's 10% of that sample alone. So across the full estate, for for this customer, about 1,100 network devices, we could estimate that this phantom inventory problem was costing them anywhere from 50,000 to $300,000 a year in unnecessary maintenance renewals and possible refresh costs. And this was only the beginning. So they also found a 102 incorrect host names, 96 incorrect serial numbers, and almost every Cisco model name was wrong. Now this isn't on Cisco. It was maybe a user input or or some other thing happening in automation that made the model names wrong, but this also isn't cosmetic. Right? Host names determine monitoring scopes, automation workflows being properly assigned, and even security and compliance policy adherence, where serial numbers determine RMA support eligibility. And serial numbers are often that key to understand this device is the same as this device even though they have different names. Right? Model names drive, life cycle automation, EOL, those kinds of things, CVE workflows, and replacement planning. So in other words, all of the places you really don't want to be wrong. And keep in mind, this was still just a a sample part of their network. And I'm gonna get to the real kicker in just a minute, but I I wanna show you an example. I told you there would be some live demos today, so I'm gonna run you through a a quick demo of how we show, you know, the the attachment, the the alignment between Phantom devices, for example, and and how this actually is a a cost lever. So right now, I'm using, an internal ServiceNow integration that we've built that actually shows, you know, what what this could be costing according to industry standards. So just to prove that it's live, I'm actually gonna log in, in real time and connect to my ServiceNow lab. So I have a ServiceNow lab. I have a network gear table because we're we're talking about the network infrastructure in this case. And so now I've I've connected IP fabric to ServiceNow. I'm just gonna do a dry run, although I could write directly to my network gear table. And when I do that dry run, I could include, you know, switch stacks and fabric extenders. I can even sync relationships, literally, just the config, not just the state, but how do these devices connect to each other. And then I run that sync. And, you know, bear with me again. I'm doing it live, but it should be fairly quick since we're only doing a dry run. You can see that it was fairly quick. And what I've done here is I've I've applied industry standard costs of things. So I put the cost impact as a thousand dollars. This is changeable in our environment. But for an enterprise access switch, it's about $5,000, so the the maintenance fees could be anywhere from 500 to $1,000 per year. Right? That's where I got that number. For compliance risk, right, I'm using HIPAA and PCI standards here where it could be $50,000 per incident. And then for for security breaches, I'm using the data from IBM. Right? $4,800,000 on average for a data breach. How does that break down? So you can you can change these numbers to apply to what what is actually going on in your environment, but then I'm using the real numbers, the real differences between my IP fabric very accurate discovery and what's in the CMDB. And so I can tell there are 127 phantom devices. Those devices that are in the CMDB but not actually in production anymore, we might be paying maintenance fees on those. You know, often the vendor expects us to kind of true up at the end of the year. This is how I can know I'm truing up, in the right way. 30 missing devices. So the opposite problem where I have devices that are in production that aren't in my CMDB. So they're ghost devices or documentation wasn't updated. We multiply that number for a quick answer. Right? What is that times 50,000? And that's where we get that. And then this last part is, we have quality change devices, meaning they're in both the CMDB and in the IP fabric discovery, but the configurations are different. Right? We've matched the serial numbers, but maybe one is matching my my security policy and one is not. Hopefully, you know, the the production is matching our security policy so that we're not, you know, nervous about the data breach. But we should know about these things. Right? And so that's just a a quick, demo to show you how this this happens in real time and how real numbers get attached to this type of inventory information. But with that, I'm going to kinda give you the the last part, of this company's journey with us. So they used different levels of security to determine network segmentation, which is pretty common. Right? Strict rules about which levels can communicate with each other. Right? Maybe I have a PCI network that can't communicate with a non PCI network or ITOT, those kinds of things. So using end to end path validation, they found several places where, the segmentation didn't match policy. You may be thinking that that this is rare, but I actually make a point to ask every single person that I meet with, how are you currently checking to make sure your segmentation policies are being followed? And what's rare is that they're able to tell me that they do anything but manually spot check this. So you're not allowed if if you're if that's what you're doing, but there is a better way. Right? When you look at my slide here in my end to end path analysis, your eye goes straight to the firewall. I've got a a clean little arrow taking you there. And green is good usually. Right? But in this case, the green is probably going to cost you another kind of green. I'm talking about money here. This firewall should actually be red, denoting that it's blocking traffic, that that traffic is segmented. And if you wanna understand why this really matters, just look at the headlines. Right? Duke Energy was fined 10,000,000 for NERC CIP violations tied to cybersecurity posture. The Colonial pipe pipeline ransomware event cost around 20,000,000 between ransom and operational disruption. And in the energy sector, more broadly, a data recovery and rebuild effort after a malware incident can hit $50,000,000 and has. Now this is the exact same story, like I said, for other industries. Right? PCI, HIPAA, even the war, voluntary security frameworks require that you have not only segmentation configured, but they require that you have evidence of that segmentation. And sometimes it's actually the harder part to produce. Now luckily, our customer wasn't breached, but they had no idea they were walking around with the same blind spots that lead organizations to those types of situations. And the irony is that they they didn't need to purchase a a new SIEM or millions of dollars in in new firewalls or even a new segmentation project or product, but they just needed accurate data. So I wanna give you another live show of, what this looks like in, IP Fabric. And what I'm gonna do is I'm gonna go to my diagrams and just cut right to the end to end path analysis feature. So you can see I have a source and destination put in here. I could do CIDR ranges too. It doesn't have to be host to host. It could be network to network, host to network, all of those things. You can pick, you know, what what protocols you're using and then actually simulate what that path is gonna look like. And I'm gonna pick a different snapshot here so that it's a little bit more interesting. We are a snapshot based tool, a point in time tool that that that gives you a simulation of your entire network. So this is just one end to end path. But this is how that path was behaving at that moment in time that we did the collection. Maybe we do two, maybe we do four collections a day. You know, that's that's kind of up to our customers. But you can see here that the firewall is red, and maybe that's a good thing. Right? We want that traffic to block. But the nice thing we can do here then is say what our intended status is. Maybe we want it to pass. Maybe we want it to fail. We can specify that and save it. And then you can save dozens or hundreds of these paths, and now every single time we run a new discovery, you know, at least daily, probably two times, maybe more, every single day, we can automate the check that those, paths are behaving as intended. So I can quickly show you the end to end path verification, just one of our built in intent checks. I can click on that that two number, the red two, you know, denoting that that these things are not behaving as intended. And now it's something to show the auditor and something to check every day, make sure my segmentation is behaving as it should be. Okay. So back to back to my slides here. Again, you know, you might be thinking that these these are outliers, but they're not. Right? In fact, across all of our new customers, the source of truth inaccuracy show up 100% of the time, and I and I'm not being hyperbolic here. The reasons are structural. They are, there are devices in the CMDB that that aren't really in the network anymore. We talked about that when I showed you the ServiceNow plugin. This is due to things like leftovers from projects and migrations. And then, of course, the other thing, the the devices that are in the network that aren't in the CMDB due to things like shadow IT or or or bad documentation. And then, like I said, lastly, the the devices that show up in both places but with different configuration details. And this is probably the most dangerous because you think you're actually covered, right, but it's a a false sense of security. These are the the wrong serial numbers, the host names, things like that. So, you know, most organizations, even the largest of enterprises, are still using manual documentation for for a lot of what I've just shown shown you and and what I'm about to show you. It takes six months to complete. It's obsolete by the time it's ended, and oftentimes, you're paying, you know, exorbitant consultancy fees to get this information. They're using monitoring tools that tell you when something broke. Right? Monitoring is absolutely necessary, but it doesn't usually tell you what, how, or where something broke. Right? And this problem's universal because the network is changing daily, but the systems used to describe it aren't being updated. Okay. So I wanna give you another example. Different company, different problem, but the same fee, which is bad data quietly costing real money. And this time, it's a global manufacturing company, and they were wildly overbuilt. And it it's not that nobody suspected this, but they didn't understand just how much and what the business impact was. And and by the way, that's also a common theme that we see a lot. So over the last decade, every time they, took on a new project, they did what most teams do under pressure. What we've actually been trained to do as network engineers, which is copy the last design for every pod. Same topology, same port counts, and it works. But it also means that the network is growing in a way that has nothing to do with actual traffic patterns, device utilization, or business requirements. So when we took a snapshot of just one of their regional networks, here's what we found. Dozens of devices with utilization under 10%, redundant switches that weren't providing any meaningful redundancy, and then ports powered up, licensed, cooled, but basically doing nothing. And when we looked at the data from one region, we found about a 100 devices, and I'm just gonna kinda use round numbers here, that were powered, supported, and maintained, but contributing little or no real utilization. And these were, you know, enterprise access switch that I that I mentioned earlier, the kind that typically cost between 4 and $6,000 each to purchase. And like I said, support contracts for hardware in that range are usually run 10 to 20% of the hardware value per year. So using the the conservative end of that range, about $500 per device, those 100 low value devices represented roughly $50,000 a year in maintenance and support, again, in that one region. So once we validated redundancy and capacity requirements, we confirmed that about half of those devices were truly redundant and that, therefore, safe to remove from the refresh plan. And with hardware in that class, again, typically costing about $5,000 per unit, avoiding the replacement of 50 of them prevented a quarter of $1,000,000 CapEx event. But this is the part I want to emphasize. Nothing was done wrong. Right? Teams were making decisions based on the information that they had. It's just that the information wasn't accurate enough to reveal the waste. So once they had accurate data, the savings were actually immediate and obvious. Now I wanna give you the opposite example here, because not every organization is bloated or overbuilt. Some are trying to do things in in a much leaner way. So one one of our customers and by the way, just heard this example again come up this week. In Europe, it comes up a lot, but they they had adopted a sparing strategy. So instead of over provisioning the network everywhere, they ran production lean and kept a a small pool of spare switches usually on the shelf. So the idea is that you buy fewer devices overall because you're not deploying extra equipment that you don't actually need. You actually pay lower maintenance and licensing because spares don't require the full support contracts that production devices do. And then when something fails, you just swap it out from the spare pool instead of paying for downtime costs and an RMA waiting for an RMA and all of that stuff. So when done well, sparing reduces both CapEx and OpEx. And ultimately, leadership believed that they were saving hundreds of thousands a year by running lean. But when we ran a discovery, we found that a surprising number of those spares were no longer spares at all. So over the last year or two, different teams had quietly pulled these spares off the shelf, put them into production, and none of it was updated in their their CMDD or their asset system. So the spare pool they thought had eight devices only had two. And instead of running, you know, what they thought was a well calibrated sparing strategy, they were running a a right sized production network but with no real backup plan. And that's the real message here. Right? You can have the the right architecture, the right processes implemented, even the right financial model, but if your data's wrong, the strategy collapses underneath you. And you're not just at risk of outages, You're at risk of unknowingly running your business without the resilience that you promised your customers. Okay. So let's shift a little bit from cost optimization to something maybe a little bit more painful, which is why downtime lasts as long as it does. When an outage first hits, of course, the first question we all ask is what changed? And if your network data is stale or incomplete or or contradictory, teams waste time trying to rebuild reality under pressure. Right? Device by device, hop by hop, that's what drags out mean time to repair, MTTR. Right? In manufacturing, Siemens puts unplanned downtime anywhere from 39,000 to 2,000,000 per hour. One customer of ours actually saw their MTTR drop from hours to just about thirty minutes once they had accurate end to end visibility into paths, dependencies, and config and state. Right? Saving millions in avoided downtime. In finance, of course, trading and payments, time is literally money. McKinsey actually estimates settlement processing delays cost $1,000,000,000 per year. One of our customers actually cut their MTTR by 98%, taking a six hour interruption down to just minutes and therefore reducing both their operational and financial risk. And, of course, in health care, downtime isn't just about lost revenue. It impacts, you know, delivery of care. So Ponemon estimates that downtime cost at a clinic is about $7,500 per minute. Outages can, of course, though, disrupt EHR, imaging, labs, monitoring, right, where where reliability and and delivery of care matter just as much as cost. So the pattern is always the same. If your data is wrong, your MTTR is long. And when teams can see the real state and configuration and dependencies in the network, those outages actually shrink dramatically. So I wanna close I'll do one more quick demo, but I wanna close with something very practical, what good actually looks like. Because the organizations who get this right aren't doing anything exotic. They're they're doing three things consistently from what I've seen. First, they maintain a continuously updated model of the entire network. It's not just a Visio diagram. It's a living digital twin that reflects every device and relationship across everywhere, right, across your campus, your branch, your data center, your cloud, all of those things in the same place. And it's a digital twin of your entire multi vendor network and security infrastructure, anything that moves frames or packets. Second, they use snapshots to understand not just the network as it exists today, but how it changes over time. Right? And I I told you how how this is a point in time based tool. They can answer questions that most teams can't answer without days of investigation. Right? What changed this morning? What will break if we move this workload or update this policy? And this is how MTTR collapses from hours or days down to minutes, not because engineers are suddenly working faster, but because that guesswork disappears. And third, they assure that the network behaves the way it was designed. Right? They check that routing sessions are stable. They verify segmentation boundaries are actually enforced end to end. This is how organizations, avoid those multimillion dollar penalties and outages, not just by responding to them after the fact. So I promised one more quick demo, and what I'm gonna do is just kind of show you a little bit. Right? We we talked about that network digital twin building these diagrams, building them every day, every you know, twice, three times a day. And here you can see, you know, we've got our sites. We've got our devices in the site. We can see how they're connected, the protocols used. We can click in, get more information about the devices or the connections. This is a digital twin. This is a simulation of your entire end to end network. I want to show you snapshot comparison as well. Because we are a point in time based tool, we can compare things historically to yesterday, maybe the time between last night and this morning when something broke, maybe to six months ago if we're checking compliance posture maturity. I'm gonna run this same end to end path analysis that I ran before, but this time, I'm gonna compare snapshots. So I'm gonna select the the snapshot maybe taken after a change. And now I can see, right, in red, these are the things that have been removed. In green, these are the things that have been added since my last snapshot or since the the snapshot that I I chose. Alright. This is the evidence that your auditors are looking for. This is how we reduce MTTR. And lastly, I just wanna show our dashboard. I showed you a little bit about finding the the end to end path analysis verification, but there's so much more. A 160 built in intent checks, things like end of sale, end of support, management consistency, right, AAA, NTP, SNMP, routing stability. And, of course, you could create your own. Anything that we collect data on, you can build an intent check for. And then all you have to do is actually click in, get that information. I'll choose a a different snapshot, for example, for end of support, and we can see that information, avoid those security problems, and plan our refreshes accordingly. And now just quickly back to my I know I'm I'm running out of time here, so I just have this last slide. I wanna leave you with four questions you can take back to your teams because the answers will tell you exactly where your exposure is. One, what percentage of your network inventory has been checked against the the last thirty days of reality? Right? Has it been updated today? Why not today? Two, how much of your MTTR is spent finding the data, not fixing the issue? Three, how many devices are you paying to maintain that no longer exist on your network? And vice versa, how many exist that you're not protecting? And then four, if an auditor walked in tomorrow, could you prove that your infrastructure data is accurate? Because the truth is the hidden bill of bad data is always going to get paid. Right? You pay it in unnecessary renewals, in outages, in compliance exposure, in delayed transformation because your your data is wrong. So fixing your infrastructure data isn't an engineering project. Right? It's one of the most powerful financial controls available to you. And the organizations that take this seriously reduce cost, reduce risk, and move faster than their competitors. That's all I have for today. I would love if we have any questions. I know we only have a few minutes left. Hey, everyone. I'm I'm joining the stage, the woman behind the curtain to moderate the q and a. We do already have a couple of of questions. But if anyone has any other questions, you still have time to add them in the q and a feature in the upper right hand, corner of this screen. But for the time being, I can kick off with, probably the easiest of the questions we've gotten so far. Lauren, you've already talked a little bit about how IP fabric fits in with monitoring tools with, of course, like, ServiceNow CMDBs. But is IP fabric just a discovery tool, or is it something more? Like, how else does it feed into the network tool ecosystem? Would. insight there. Yeah. Yeah. Great question. So, I mean, I do want to accentuate our our discovery capabilities. We do it like no one else does, which is through using SSH into the CLI, or we do it to APIs when necessary. So we're not using SNMP. So a, we're we're less of a burden on your network, and we're actually getting accurate information from things like neighbor relationships, CDP, LLDP, you know, our tables, all all of the things. But then we we go on to the assurance part. Right? What I showed in those dashboards and and the network diagramming. So all of this, like I said, is available out of the box. These 160 intent checks, the ability to create your own custom dashboards, the ability to automatically have these these diagrams created for you according to your sites. It it's essentially plug and play. And then, of course, we're all open APIs. So anything I can do in the the UI, I can do via the API programmatically, which, of course, means, yeah, we're gonna work not only with your CMTB, but with your SIM, with your your your other security tools, other inventory tools, other monitoring tools, other you know, pretty much anything that requires you to have accurate inventory information, and then everything that we offer out of the box within IPFabric as well. Awesome. Thanks for for the quick summary. And, I guess, now on to kind of the more in-depth or, like, high level questions. So you've already talked about, like, the the price tags associated with downtime, with noncompliance, with underutilized devices, all that. And it's very clear that that data affects so many different parts of the network and of IT budgets. But I'm curious because there's so many different things you can kind of choose from. What is the fastest way to prove to leadership, especially that your CMDB data has a a financial impact? Yeah. So I showed a little bit of that example in the ServiceNow where we're literally attaching the price tag based on industry standards to some of that information or the gaps in information, I should say. But what I would also say is anything in our our dashboards here, could be considered a price tag. So, you know, end of end of sale, end of support, that means that those devices aren't being updated anymore with maintenance and security patches. And, therefore, I can attach a, you know, a security and compliance price tag to that. Other places where, let's say, you know, Telnet isn't built in, Devices with Telnet access. If Telnet's turned on, right, that's another security problem. Any one of these 160, right, fan module state, I can attach to outages. Routing stability. Again, you know, I can attach to outage times and therefore risk as well as, you know, loss of revenue. So I would I would challenge anyone to find just even one intent check-in here that doesn't have a price tag, allocated to it. That is pretty compelling. Not gonna lie. But yeah. Okay. We've got the last of our questions so far. But if anyone else wants to add anything, now is your chance. Basically, this last one, though, is the buzziest of the three, dare I say, which is, of course, about AI and automation. Is there an AI based approach to improving this automation for for managing your CMDB? Is there a quicker, easier, better, faster, cheaper way to do it? Sure. You know, I I I wanna be careful not not to get too buzzy and say that AI is, the way to do everything. But I I think it is a way to unlock some of this information in an easier way. And so we are, right now, we will be releasing an MCP server for IP Fabric. We we're already using it internally. We're demoing it for our prospects and customers, and so we'll we'll be releasing that in the next, I would say, month or two. And then what that will do is open up the ability to kind of do whatever you like. You know, if we think of the MCP as the wrapper for all of our APIs and our APIs are are completely open, then you can do things like, you know, give me a compliance report based on my you know, the the current NERC SIP standards, and and how you would apply that for the network piece of NERC SIP or for the network piece of PCI. So compliance reports is just one example, but there are a lot of other examples out there kind of doing the the snapshot comparison, right, where we give you the snapshot comparison like I showed you. We also do a side by side of the configs right in the the product. But then you could take your your chat with with the MCP server and say, you know, provide context around this. When did that change? Can you tell me, you know, exactly what changed based on, you know, the information from IP Fabric? So we a lot of people are already using us as the glue in their AIOps practices. So Red Hat, I know, did a webinar with us last year about how they are building their AIOps practice and using IP Fabric as the glue, as they said, for AIOps. So, yeah, I think absolutely that there's a real possibility here for and and, in fact, more and more importance on the accuracy of the data because AI is going to exacerbate timelines. That was great. And, yeah, for anyone who's interested in watching that webinar, it's available on our YouTube channel. I think the title is unlocking AI ops, so easy to find. But it looks like that's all we had for questions for now. And, of course, if anyone has any other questions, you can always reach out to us using the contact link. It's listed under the docs just as contact us. Send us an email. We're happy to answer your questions that way as well. But I think that's all for me. Lauren, I guess, take it away until the if you have anything else to say. No. Thanks, for watching. Feel free to reach out with questions. And, yeah, we'd love to show you some more. Thanks, everyone. Awesome. Thanks all.