Infraspec Garage
Infraspec Garage is where a lot of our engineering side quests happen. And where the odd little things we build get a proper home.
A lot of these projects start pretty casually. Someone wants to try something, so they build it.
We love making a space where people can tinker around. An experiment could end up being our own little inside joke. But a lot of it blossoms into something genuinely useful that runs most of our infrastructure and informs the quality of work we bring to the table for our clients.
This place, right here, is a glimpse into our creative bent of mind and our constant need to tinker and experiment.
Welcome to Infraspec Garage.
Why there’s a gaming PC running AI in our office
There’s a large RGB-lit gaming PC sitting in the Infraspec office.
It doesn’t really get much gaming done, to be honest.
We put it together because we wanted the whole team to have easy access to GenAI tools day to day. Separate ChatGPT subscriptions for everyone would’ve just added up. Plus, keeping cloud infrastructure running all day for occasional use didn’t really make a lot of sense.
So we went with the giant glowing machine. Obviously.
We ran self-hosted models on it with Ollama and used Open WebUI for the chat interface. We also connected a single ChatGPT subscription through the API, instead of everyone needing a separate one.
Having the machine around made it easy to mess around with new models too. People could try an idea while it was still half-baked and see what came of it.
The first time Chetan wrote about it, the machine had already been running for almost 90 days.
Now, more than a year later, it’s still running AI for the team.
Leaving the giant glowing thing out of the Infraspec Garage would just be rude at this point.
(And yes, it still looks like it should be running Cyberpunk 2077.)
The lore behind 5,300 images in 24 hours
PicFusion started its life as a fun internal project at Infraspec for generating personalised AI images.
We started dropping generated photos right next to real ones and asking people to pick the real image. People had a hard time telling them apart, which, of course, we absolutely loved. Mwahahaha!
We opened PicFusion up to a small group outside Infraspec too. The cap on numbers was a very literal one sadly. There was only so much hardware to go around. But even with limited access, people kept PicFusion extremely busy.
There was a 24-hour stretch where people generated 5,300 images! How cool is that?
Of course, that kind of volume requires a lot of fine-tuning and inferencing, which can get pretty expensive, especially at that kind of volume. We were already working with a very hard hardware limit, and there was only so much of PicFusion’s workload it could handle. So, to tackle this problem, we split PicFusion across both cloud and on-prem infrastructure to keep that cost under control.
People were squinting at the photos trying to work out which face was real.
We just had to make sure to keep all the generations running without letting the cost run with them. Sheesh.
5,300 images in a day was a lovely number to see.
Why one Raspberry Pi now helps set up the others
We had a bunch of Raspberry Pis that needed the same setup every time a new one joined the network. After doing that by hand oh-so-many times, we thought the Pis could probably take some of that work off our plate.
We already had one Pi as our main server. It kept the boot files and filesystem in one place. The rest could boot over the network and grab what they needed from there.
We still had to set up each new Pi. It needed its own files and config, so we wrote a Bash service to take care of that too.
Now, when a new Pi starts the netboot process, the service checks whether it has seen that device before. It keeps track of the Pis it has already seen, and prepares the files and config that the new ones need to boot.
The service keeps running in the background, so the next Pi just needs to show up and look cute. Everything else is already taken care of.
At this point, one Raspberry Pi is handling onboarding for every new Pi that joins the network.
Quite a job for a tiny little computer that fits in your hand. :)
Good ideas rarely arrive fully formed.
We make room to tinker, test and turn useful experiments into dependable systems.