Archive for the 'Uncategorized' Category
Hitek Homeless May 11th, 2015
I’ve spent the last couple of weeks neck deep in new technology. That’s always awesome. The end result is that a couple of our humble projects are going to be running on a considerably more robust platform than the one we administered professionally for a regional telco a decade ago. Also, I spent about $10 learning. Ouch!
As a telco, we ran a very large file server that handled all of our users web sites in one central location. It was nice because we could scale out to multiple web servers for redundancy and scalability. However, when the NFS server went down, the ENTIRE cluster had to be massaged in the right order to bring it back online. Talk about your nightmare scenario: send a new tech over to trace cables and just wait for the network wide outage to occur.
Our new infrastructure is a much smaller filesystem – we only have a couple of websites on it. However, we have three (3!!!) servers acting as redundant file servers. As they are all mirrors and geographically separated from each other, we can lose two entire regions before we have a real problem.
MySQL has been the defacto standard for sql servers on a budget for years. We ran this in an ISP environment and had the forethought to have a master-slave setup. In the budget telco world of a decade past, when we lost a SQL server, everything crapped out until someone manually logged into the slave server and told it to become master. Yay… we didn’t lose everything. Boo, it was still an outage.
Our new infrastructure has clustered master-master sql servers. I can bring a new one online without even telling the existing servers about it! We also have three of these running in different geographical areas. We can lose an entire region and keep chugging along. If we lose two regions, someone has to login and tell the remaining server that it’s ok to think for yourself. It’s quite an upgrade from the old days, but not quite as nice as the fileserver.
Then there’s the whole web server thing. It needs to talk to the sql server and file server. Thanks to modern magic, it can seamlessly talk to either service even if it’s own closest server crashes. It would be hard to mention all the differences in our new implementation versus the old way of doing things, but our new web server software is a) f*cking fast as sh!t. b) has built in caching to reduce the overall work load c) is really good on memory usage.
Back in the telco days, budget was always a thing. We simply couldn’t get the money for load balancing hardware and didn’t have the people/time to do it properly in software. Today, that’s not even a concern. I can get multi-region load balancing that let’s me put all of the above infrastructure in 4 different data centers for less than the cost of a 12 pack of microbrew once a month. Compare that to round robin DNS – where an outage means that users can keep hitting ‘reload’ and get a working server if they are persistent. I’ll take dirt cheap load balancing over DNS hackery!
Want to know the best part? All of this costs about half as much as we’ve been paying every month to run a single server at our current hosting provider. That’s right.. one big server, with no redundancy, a broken serial console and backups that don’t even boot if we have an outage. I’m TERRIFIED of rebooting our server because the console doesn’t work, our backups don’t boot and tech support says ‘pay for OS support and buy more capacity’ any time I mention that their services don’t actually work.
Meanwhile, I’ve intentionally rebooted 1-2 servers on the new system in the middle of SQL imports and filesystem modifications without a hiccup. Everything just works. Better yet.. I can add systems to the cluster if we need the capacity or upgrade the existing servers to larger systems anytime I want. We can literally get 3x the redundancy and twice the capacity for the same price. Yet, I only need half the capacity we currently pay for.
We also have to ability to add auto-scaling. That means we’d boot servers that contact our sql/file server and just serve web pages. And we could do it magically as needed and only pay for it while they were needed. Fancy.
Currently, I think or backend cluster has the extra capacity to handle our web services as well. But, if we need it in the future, we can boot thin web servers on-demand and kill them overnight to avoid unnecessary charges, Talk about the icing on your chocolate covered doughnut,
FYI, google is offering a $300/2 month credit on compute engine. Don’t be like me and sign up before heading to Baja – losing your credit!
UPDATE (8:30am, 5.5 hours after cutover completion):
Cutover went well this morning… there’s nothing like a 1am maintenance window to remind me why I don’t want to work for the telcos again.
However, I had to scale up our servers in the middle of watching a movie as they were starting to lag a bit. 20 minutes of rebooting servers, without even a blip of an outage, and we’ve scaled our capacity vertically 4-fold.
Sadly, this raises costs. Now, our hosting costs are closer to 3/5 of what they were prior to cutover. Still, a nice savings. Did you see the part where I took the fileservers and sql servers offline and replaced them each in under 5 minutes with bigger servers and couldn’t even tell anything was happening from the client side? That’s pretty fancy for a couple of home-brewed websites about camping in the woods and where to dump poop.
After looking at the costs of inter-zone bandwidth and just how much of it we were using, I got to thinking about how to lower that cost. A little filesystem hackery was enough to drop our inter-zone bandwidth to 1/3 or less and reduce the CPU overhead enough that we could scale servers back DOWN! Who’d think the lowly symlink would be capable of saving 2/3 of our server costs as well as 2/3 of our bandwidth costs. Crazy!
I’m learning that the real power in cloud computing is in getting the smallest building blocks you can and using them as efficiently as possible.
Hitek Homeless February 26th, 2014
Here’s a little bit of last winter that I just ran across. Sorry about any bitrot. I’m too lazy to look for pictures at the moment. I recommend setting your imagination to 8 or higher.
After Rain and I ran away from home (well, from Jenn and the idea of working for a living), we made our way through Arizona, New Mexico and most of Texas before finally arriving at Corpus Christi and finally got to see something besides desert and high desert for the first time in months.
Padre Island National Seashore
I made it to North Padre Island the last night of February – a couple weeks before Texas spring break started. North Padre Island is part of the national seashore and allows beach camping on 60 miles of unimproved beach. The first few miles are accessible to passenger vehicles, after which things get a bit hairy. My advice to anyone wanting to go far down the beach is to pay attention to the tides and drive in and out during low tide. It’s best to match up low tide and daylight hours so you can drive as fast as possible and watch out for humps and debris in the damp sand. This is a really nice couple of hours driving at the right time. During high tides, you’ll need four wheel drive and be going very slowly.
The rules for length of stay at Padre are really friendly for snowbirding. 14 days in the park, 48 hours out. Rinse and repeat until you’ve stayed 56 days in a year. Expect rust. Expect salt spray to foul up electronics. Expect unicorns to attack (true story).
I drove about 15 miles out upon arrival, cooked up some sausages and made dinner. It was a beautiful clear night; just a mild chill in the air considering it was the end of February. The only lights I could see were a few oil rigs out on the Gulf and a couple of boats. We slept with one wing open to lessen the wind blowing through the rig all night. I do like having a toasty sleeping bag and a little dog with sharp ears.
The next morning, I got up and took my time making breakfast. It was Saturday, so there were a few people driving out past us. The van with the wings open and off in the high sand drew a LOT of stares. It took a while to get used to this, but the Texas beaches are kind of a cruising scene.
The day was gorgeous and the weather report from NOAA (I was picking it up via a handheld radio my buddy Phil gave me this winter) gave me a pretty good idea of when to expect low tide. About an hour prior to low tide, I broke camp and we continued on down the beach on the hardpack. I drove a bit too fast and the occasional hump snuck up on me, but we made it down to Big Shell. In fact, we made it there before anyone else that day.
Rain and I goofed around. It was her first time on the beach and she absolutely loved the soft sand after a season in the rocky desert. The ocean became something that we HAD TO DO at least twice a day. As usual, there was no running into the ocean on her own, it was all about chasing sticks and saving them from being swept out to sea. She did, however, run around like a crazed lunatic in the sand while I was talking with one of the locals that showed up a bit later. Fun to watch.
A few more people showed up and I ended up sharing a camp with a New Englander on his way home (the long way, obviously) from an aborted trip to Panama. It turned out that he uses freecampsites.net. Small world. I was disappointed that he had to take off the following day, but I really couldn’t complain about the sheer serenity of my camp. The only people I could see were the fishermen across the channel at South Padre Island. Come Monday, there was very little of even that until the following weekend.
I fell into a simple rhythm of waking up sometime after the sun was up and having breakfast before taking Rain down to the beach for a swim. The rest of the day was spent reading or just staring out at the pelicans feeding or dolphins swimming up and down the channel. I might have a snack at mid day and supper around dark.
I still don’t have any refrigeration in the van, so I developed the habit of buying a pack of sausages or some other meat and cooking it all right away. It goes from the grill/frying pan directly into a ziplock. I would typically make a meal with one sausage for dinner the night I cooked a pack of five. The next two days I’d have a sausage with breakfast and one with dinner. After that meat was scarce until the next trip to town.
Meals were very simple affairs. I alternated grits and pancakes for breakfast. I’m from the South, so grits actually means grits, eggs, cheese and sausage if there was any. Lunch was usually just a peanut butter and honey sandwich, ramen noodles or a couple of quesadillas. Evenings, I alternated beans and rice with pasta. Boring fare I suppose, but it was all simple to prepare, no refrigeration required and really cheap.
All told, I ended up staying 10 days out at Big Shell before I ran out of fresh water and had to head back to town. About a week into my stay, I got an email from a friend and former coworker. It seems his current employer has an opening in one of those ridiculously high-paying jobs doing some bleeding edge work that I would probably enjoy quite a bit. Try as I might, I couldn’t think of it as anything but a temporary position that would give me a financial cushion so I could continue doing exactly what I was already doing.
The more I examined the situation, the less it seemed that taking a six figure job working for someone else would actually benefit me long term as much as continuing to sit on a beach and stare at the pelicans. Sure, I have to work a bit to get by, but I don’t actually need a hundred grand a year to live, eat and enjoy life. I can live on such a small fraction of that corporate income. Mostly, I work when I actually feel like working and on whatever pleases me.
There I was, sleeping on a beach with the windows open curled up next to an awesome dog and spending my days doing exactly what I wanted to do. Meanwhile, my buddy was up in Colorado slogging to work in the snow and sitting in a cubicle a minimum of 40 hours a week. More likely, he was salaried and pushing 50+ hours. How much money would it take to actually make that change in one’s lifestyle? And what the hell would you do with the money when you had it?
After thinking about it for a couple of days, all I could really come up with was to buy a newer version of the same van and have some customization done. So I could drive to exactly where I was and watch it rust. So here I am. Still poor. Still happy.
57 miles down the beach I ran into the first of many people I’d run into stuck on the beach. This guy was out camping over the weekend with his girlfriend and her kids only to find he had a dead battery Sunday morning. They had two vehicles but no jumper cables. I can’t really say much, I didn’t have any cables either as I’d left them with Jenn. I did have a battery charger and a solar system though, so we sat in the van BS’ing while charging his battery. He offered Rain and I a gallon of water which we accepted happily as we’d run out of water that morning.
The last few miles off the beach I noticed there were a lot more people around than on the way in. Spring break had begun while I was sitting at the end of the world navel gazing. Nothing like a scruffy hippie roaming the beaches in a creepy white van while kids from every school in the state are out partying.
I filled up on water, bought some more meat and headed back to Padre, but I didn’t feel like going terribly far out. I found a spot about 8 miles out where I could just barely get a trickle of internet access. There was a group of guys a little ways down the beach that Rain fell in love with. I think it was the boomerang they were throwing really. I’d look up from a book and notice that she was off visiting the neighbors again. They were nice kids and seemed to like her, so I let her go visit periodically.
After a few days in this spot, we had a relatively calm evening that seemed perfect for a little fire juggling. Before I tossed the first torch in the air I had a couple of Federal agents out there enforcing the county’s fire ban (the county beaches had bonfires left and right). As the torches were freshly lit and there were two park rangers standing watch, the ‘senior’ one tells the trainee to let me do it so he can video tape it with his new sunglasses. (WTF?) The neighbors cheered from the safety of their own camp; the hall monitors had come by to run all their ID’s the day before.
Five minutes later, Federal agents start asking for ID, about weapons, illegal aliens and heroin – I really can’t make this crap up. We established that I am not required to carry ID to juggle, that there are certainly weapons in the van, that I’m not going to talk about them any further. No, I didn’t see any illegals, but I’ll keep extra food and water on hand since you seem so concerned with their welfare. Did you really just accuse me of smuggling heroin as an excuse to search my home? Also, it is illegal to juggle fire during a fire ban even if you are standing in knee deep water.
After determining that they really didn’t want to get a warrant, they left with a large bag of trash I’d picked up from around my camp just in case a ranger stopped by. They seemed a bit upset that I asked them to perform such a menial task. I actually had to argue with these ‘public servants’ to convince them to put a bag of trash in the back of their truck and haul it to the dumpster that they drove past ten times a day.
The next morning, I found my neighbors had gone and left a giant box of ‘groceries’ outside my open window. I think they were living on chips and trail mix all week. Yummy snacks.
I was getting close to my fourteen days, so I decided to explore a bit. I didn’t make it all that far after breaking camp. I went as far as North Beach, which is still part of the park, but doesn’t require a permit to get in. The weather was still amazing and the wind was pretty light, so I spent most of they day just juggling on the beach with the occasional splash in the ocean to cool off.
TL;DR: I won a four month, no-expenses-paid, vacation on the beaches of Texas. Park Rangers hate fire and don’t like to touch icky trash bags.
jenn September 3rd, 2013
We just ate our first country ham. Well, Johnny grew up eating it, but it’s OUR first country ham. We have been buying different kinds of shelf stable meats. This week, we bought 3lbs of sliced country ham. It was sitting on a shelf, unrefrigerated, in the grocery store. Now it’s sitting on a shelf, unrefrigerated, in the camper. We cooked up a slice this morning for egg, ham and cheese sandwiches. Hope we live through the week.
jenn August 27th, 2013
Rain and I were out on a hike/walk. I saw an unmarked, overgrown trail going down into the deep gorge that we camped on the rim of. I like to search the unknown and walking along the road is boring. Anyway, we were both tired of walking on the road. Off it we went.
The trail went a few hundred feet and then seemed to end at an overlook. I wondered what was below me. I was definitely standing on a rocky outcropping. I know there are know caves within 10s of miles, so I thought that I might be on top of one.
As much as I wanted to go down, It was just Rain and I. Johnny had no idea where we were. The ridge was very steep and covered with dead leaves. If I slipped and hit my head, rescue wouldn’t show up for hours or even days. It would be foolish of me to go by myself. So, after taking a minute to weigh my choices, I decided to head down.
I plotted my decent. Most of the trees were too far from the path to offer much support. The support they did offer came in the form of dead limbs. I was stuck using thorny vines for balance. No matter. I made it to the bottom where I wanted to be. Rain had little to no trouble.
Once there, I found what was going to be, or what used to be (before a collapse) a cave. Actually, thinking back, it had the precision of something man made. It was a roughly cut rock shelter 3 feet deep and 8 feet wide with a 6 foot ceiling. Limestone walls shot out along the ridge on either side of it. The sight of which pushed all recognition of danger out of my head, even if Rain acted like she wanted to go back up the trail. It was time to go ridge walking and look for caves!
We scrambled over rocks and climbed under downed trees. I poked my head in holes. Rain sniffed leaves. While fun, it was pretty uneventful. I saw this spot on the ridge below us that looked very promising. We started heading down. Then, I saw it. I picked it up and examined it. I looked at the seemingly concave, rocky wall that was our destination. I looked at the object again. As bad as I wanted to go further, I couldn’t. In my hand, I held a sun bleached femur with a bear’s teeth marks etched into it.
I turned back. Kicking myself the whole way, I retreated. We both made it back up the gorge much more quickly than we had descended it. No more poking my head in holes. No more exploring. We now had a destination.
We weren’t in any real danger. Only perceived. While it crossed my mind, the bone surely was too thick to be human. I assume a camper had brought a cow bone for their pup and a visiting bear lifted it from the campsite. It was old. Just because there WAS a bear there doesn’t mean that there IS a bear there. Hiking around here, you pass all kinds of bear scat, marking and dinning locations.
I chickened out. But on the bright side, I lived another day. I am planning on returning in the next two days. I moved the bone, so hopefully I wont get spooked this time 🙂 Haha I just know there is a cave a little ways down that ridge around those rocks. Cavers last words? Or, perhaps their first.