A Cautionary Tale of AI and VLANs

-

 

This turned out to be a cautionary tale. At the beginning I was hoping to find an AI success story that demonstrated the power of how it lets you wade into the deep end of a domain and find success. It turned into a grueling week of troubleshooting rabbit holes that the AI insisted I get to the bottom of. It's a powerful thing, but it takes a mindset that I'm still trying to figure out.

I have what you might call a home lab and was thinking about adding some smart home devices. After a bit of yak shaving it became clear that I needed to setup a VLAN in order to partition the traffic. While I was at it I'd setup networks for IP cameras and my work computer. Last year I switched over to a proper firewall running OPNsense, so I figured all this would be a nice exercise to get practical experience with things I was generally already familiar with.

Shortly after firing up ChatGPT I had a high level plan and a better understanding of how to design a network. With a couple managed switches ordered, I got right to work. The AI advice sounded correct and was matched general understanding of Networking and how VLANs should be configured. You probably know this already, but the AI is very reassuring. Every step of the way it let me know how clever I was for asking the right questions. I especially appreciated the way it insisted on designing the architecture ahead of time. It gave me a lot of confidence that I was going to have a solid foundation to my network that would evolve elegantly over time. Unfortunately, the hallucinations started almost immediately.

The first problems were fairly innocuous. The advice regularly referred to OPNsense settings that don't exist. These were easy to ignore and I explained them away by saying the information could based on a different version. My human intelligence along with the actual documentation easily filled in the gaps and before I knew it, DHCP was configured and seemed to work. I plugged in a device and it got the correct IP!

Things went downhill from here as the first major issue presented itself. For some reason my device couldn't ping the firewall and the firewall couldn't ping my device. This was a bit puzzling because the link was up and the IP address was assigned in the correct range. Clearly they could communicate at some level. The AI was happy to guide me through the issue and was quite confident that this was definitely a Layer 2 problem in the managed switch. After running through a conveniently generated troubleshooting checklist, the results remained conclusive: what I was looking at was clearly a Layer 3 issue in my firewall.

At this point I was starting to realize that the usefulness of the AI was beginning to approach zero. The prescribed root cause continued to waffle between the Network and Data Link layers. Ultimately, I went off script. The AI had previously recommended I put my switch into "Basic VLAN" mode. With nothing to lose I ventured into "Advanced VLAN" mode. It presented me with a new setting called the PVID table. With some trial and error, I was able to get ICMP and DHCP to work. The AI had previously told me that "Advanced VLAN" mode was notoriously buggy... when pressed on this issue it wasn't able to back up the claim. Go figure. 

With that I had my first "access" port configured, which is just the beginning of the story. I had multiple trunk ports to configure, access points and IP cameras to migrate, and a house full of people asking if I was doing something to the internet. To make a long story short the process continued for several days. By the end of it I had a pretty good idea about what I was doing and how things needed to work.

Overall, the experience left me extremely nervous about how easy it is for the AI to write code for us. None of the issues I hit were particularly sophisticated, but networks are fairly binary things. They work or they don't, and that property makes small bugs pretty catastrophic and easy to spot.

So what are the takeaways, I certainly learned a few things:

  • AI probably allowed me to configure this network faster than I could have done it on my own. The high level advice was sound and it helped me wire my brain to think about the problem in the right way.
  • The hallucinations were present at every step of the way, regardless of whether or not I was able to identify them.
  • AI is a powerful and dangerous tool that should be used cautiously.
  • You can't trust anything the AI tells you.
  • OPNsense is a nice piece of software.
Location: Boston, MA, USA

Comments

Post a Comment

Popular posts from this blog

Terminal UI the Easy Way: BubbleTEA

-
Image
For the past 18 months I've been tinkering with a terminal application that serves as an Algorand node frontend. I had a number of requirements in mind when choosing the technology: a full screen terminal application, works over ssh, extensible to allow it to evolve with different utilities and views of the system over time, be programmed with the Go programming language.  In the past I've created these sorts of programs with bash, but it's not fun to manage terminal control characters and deal with re-drawing with bash. Using bash has always led my applications to be simplistic, typically never getting beyond simple spinners and progress bars. I knew about ncurses, but didn't want to lose the portability of bash -- some of these scripts were even POSIX compliant, and honestly I just didn't want to deal with low level interfaces and C bindings. Enter bubbletea . The team at Charm has built the tools I didn't even know I needed. There is an impressive array of d...
Read more

XCode4: UITabBarController with UINavigationController using Interface Builder

-
Image
If this helped you, please consider downloading my FREE  iPhone app " Hydrate Yourself "and leaving a review. There are a  whole   bunch  of  tutorials  to create a Navigation Controller inside a TabBarController with XCode3.  But there are a number of small changes in XCode4 which can trip up someone not familiar with them.  So this guide is essentially an updated version of those other guides, designed to create the same simple demo using the new tool. 1. Create a new Project We are going to start with a Tab Bar Application, so go ahead and create a new project as usual. I'm calling mine Navigation Tab Bar. 2. Select the Tab Bar Click MainWindow.xib and you should see the integrated Interface Builder. On the left is a section called Objects, select the Tab Bar Controller. 3. Add the Navigation Controller On the right side of your screen click the Utilities box. This is new in XCode4 and should look familiar if you've been...
Read more

Easy and accurate SVG to DXF conversion using Inkscape and pstoedit

-
If you've ever worked with a vector tool to design something to be carved with a CNC routing machine, you may have needed to convert the resulting SVG file into a DXF. In my workflow I often use Inkscape to generate my SVG graphic, but my CAD software is CamBam and it requires a DXF file. Many people have tried to make the perfect Inkscape plugin for exporting DXF, there is  Better DXF Export ,  Better Better DXF Export  and  Big Blue Saw's DXF Export . I've had the most luck with the last one, but it was still finicky and often broke when updating Inkscape or OSX. Not only that, but at the end of the Big Blue Saw page, Simon mentions that he doesn't even use the plugin most of the time, instead relying on a command line tool called pstoedit. With that in mind I made a small shell script which will convert an SVG file to DXF, you can save the script and run it directly or add the functions to your startup script. The script is available on github under the proj...
Read more