"but a steak, like, I could use that as a washcloth"

News and Mailing lists almost back

2009-03-08T06:26:25Z

Angelo and Alex managed to get the Alex's news spool to finally import into INN, so we just need to configure the news server and plug in the spools and we should be all set.

I'm working on getting mailman back up; we don't have the lists anymore, so people are going to have to subscribe themselves when it does come back up.

Progress report

2009-03-06T14:25:28Z

Mail delivery is online and the spools have emptied down. Last I checked, there were about 40 messages in the queue (out of a total of ~18k that we started with).

Mail submission is working, you should be able to send mail both local and via authenticated ssl if you're remote. There does seem to be a configuration problem causing fury and maybe others to reject, looks like IPv6 config. We're looking into it.

IMAP and POP are online. SASL was crashing because it was linked against the wrong version of kerberos. Dan W. got that fixed, thankfully.

None of the fancy filtering stuff is online - OpenSPF, SpamAssassin-via-start-settings, Greylisting, Amavis/ClamAV are all out right now. We'll be working on this tonight.

Mailing lists are gone, newsmail/mailnews are gone. We'll be rebuilding those soonish.

News is progressing. Angelo managed to get the old DNEWS spools running on a test machine. We'll start to get INN installed on whitefox and migrate the spools to that. More progress on that sometime between tonight and Sunday.

Alex Grant is fighting with getting his news cache into a test server to be able to import them into the new news server, but we have the raw data so it's just a matter of getting it imported. Worst comes to worse, there are several hard ways to do it.

Delivery onlinish

2009-03-06T03:02:55Z

The delivery half of mail is running, draining out the relay. Whitefox's 60.19 ip is still firewalled, letting the relay finish first. Once that happens, we'll unfirewall it.

Imap and pop should be kinda up. Mail submission is still down right now.

Mail delivery almost ready

2009-03-06T01:31:43Z

We're nearing the final stages of having mail /delivery/ back online, with several caveats.

We will have procmail processing; spamassassin, nmh, razor-agents are installed and look like they're working.

All of the other crazy configuration stuff we had - openspf, greylisting, amavis/clamav and such will not be in. We're going for simple for now.

Actually getting to your mail will come later. Imap and pop are having a hard time (saslauthd and friends are throwing core files). But with mail delivery online, forwards and procmail processing will happen, and mail reading via pine and mutt and such /might/ work.

News recovered

2009-03-05T12:53:05Z

------- Forwarded message -------
From: "Alex Grant"
Subject: News recovered
Date: Thu, 05 Mar 2009 06:20:50 -0500

Oh hai guyz. Just letting everyone know that I have a nearly-complete
backup of news, in INN's tradspool format, up to the day before the
server went down. Since I previously set my news client to cache the
contents of every message ever posted, I wrote a script to convert
from one format to the other and there you go. (It's "nearly-complete"
because it doesn't include groups I wasn't subscribed to; however, the
only groups I wasn't subscribed to are the ones nobody's posted
anything in for years, so we should be able to recover those
completely from the Really Old Backup)

Rebuilding

2009-03-04T21:20:35Z

Rebuilding efforts continue. Got the base OS installed on a FreeBSD geom mirror after much work. Started working on installing packages when i got up around 11 am.

Disk recovery efforts are looking poorer and poorer. It's looking like out of 4 disks, we only have a good image of one, which means we only have half the stripe set. We're still looking into this, but it's looking like our best bet for recovering the news spool is going to be to use the old dnews spool + what people find in their news client caches.

Getting mail back up is #1 priority, but it's going to be a while because of all the crazy crap we had configured. We did find some of the configs on a test machine, so that'll certainly help tremendously, but it's going to be a lot of time spent tweaking and testing. We'll keep posting updates.

News is back-burner right now. What we have is the old spool from when we were using dnews. We're going to need to set up a separate machine, get dnews working on it, and then export news from that. It probably won't be easy. I'm guessing that news will continue to be down for several days after we get mail working. There's still hope in recovering most of the spool, but that will be a slow effort even after we get news working again. BTW, all you newsmail users, if you've got archives of your email, please get in contact with us.

Update 5:57 PM: The FUDForum installation that John Musbach setup seems to have a cache of news posts. We haven't looked into it much yet, but it's likely that it'll have a lot of the more recent messages, if not every message from the server. This is some very needed good news.

CSH Mail Server - Long live news

2009-03-04T03:43:50Z

This just keeps getting worse. Please read this entire message, of if you don't care to, read the bottom: we might need your help.

Turns out the news spool was probably stored on /usr or /var, which are medium-term dead right now.

If i recall correctly, when we migrated to innd, innd wanted the news spool somewhere under /usr/local/. It was so small and we had the space for it, so we let it continue to use that directory.

We haven't lost all of news - we still have what was left in /news_domain, which is what the old dnews server was using, which is current as of Aug 2008.

At worst, we lose all news since Aug 2008.

We have two long-shots to get the news since then
1) Try to reinterleave the disks by hand from disk images
We're getting the images of the old data onto hactar right now. That effort can start as soon as the images are available.

2) This sounds bad, but it's not entirely impossible: recover the news posts from caches from clients.

Thus, my request to anybody reading this: Zip up your client's cache of news and upload it to your user directory on house systems, then make a comment here that you've done so. If you need quota, let us know and we'll give it to you. If you decide to post it somewhere else (please dont!) make sure the rest of the world can't read it!

Update 11:50pm: Rusty has brought up the extremely good idea of sending the drives off to a data recovery outfit. Given that the drives aren't damaged, there's a lot of hope in recovering the data.

Setting up the mail relay server

2009-03-04T03:01:13Z

"Well, the firewall hasn't updated yet, so mailrelay won't be visible."
"Well, two non-responsive mail servers can't be worse than one" =)

CSH Mail Problems Day 3

2009-03-03T19:28:33Z

The restore from ITS is going tragically slow (Angelo told me about around 1:20 pm that we only had about 2.5 GB recovered out of ~23).

Actually, looking at the console just now, the recovery aborted, apparently a network glitch. FML.

We're now going to have ITS restore the data to some disks they have over there, just so that we can get the data off the tapes as fast as possible, once that is done, we can swing by with some removable disks or something to get the data back to CSH.

Watching the restore right now, it looks like it's moving at around 5-15 MB/second, so it should be done in a reasonable amount of time.

Update 4:30pm: Backup server index isn't showing anything in /usr, but reports indicate that /usr was backed up; guessing index is AFU. /mail_domain and such are inbound to hactar. Probably going to need to build base OS from scratch.. not looking forward to this. gotta run!

Update 6:51pm: Good news: mail and news spools are intact. Bad news: /usr and /var are gone. All this time, networker was backing up /usr/compat/linux/[usr,var]. I feel like an ass for never seeing this.

CSH Mail Server Problems

2009-03-02T20:23:44Z

Whitefox got rebooted due to a power problem, then the raid card refused to boot up because its hold-up battery is dead. We've been thus-far unsuccessful at convincing it to not give a damn. Seriously shitty hardware.

Our options:

1) Get the card to not care about the failed battery and boot anyway
Can't get the card to load its bios to even play around with settings/etc.

2) Get a new card
Can't find one.

3) Trick the card into thinking there's a battery there
Tried that, still complains.

4) Reinterleave the blocks by hand on another machine
Question marks.

5) Restore from backups onto a new array definition.
Possible loss of data.

It's looking like we're going to be forced to restore from backups, which I'm not looking forward to because we'll lose a little bit of mail and whatnot since the last backup.

Chris Lockfort has been working on it since it happened last night, but hasn't been able to recover it yet. I'll be going in today after work to help.

Update 6:10 pm:

We're going ahead with bare-metal restore from backups. The raid card is starting to respond, after some magic, so while we acquire backups from tape onto spare disk, we're going to continue to try to magic up the raid card.
Opcomm also has a twitter account now:
http://twitter.com/cshrtp

Back of the envelop (literally)

2008-12-05T06:08:50Z

Gasoline: 34.8 MJ/L
3.78 L/Gal
--> 131.544 MJ/gallon

My car has a 17.5 gallon tank.
Averaging 60 miles/hour, 350 miles trip, in 5.88 hours is almost exactly one tank.

131.544 MJ/gallon * 17.5 gallons == 2302.02 MJ of raw chemical energy in my car

2302.02 MJ in 5.88 hours is 391.5 MJ/hour is 108.750 kilowatts

My car consumes ~108 kilowatts. At 25% efficiency, it's generating ~27 kilowatts power.

Sanity: 27 kilowatts is 38.57 horsepower. That about makes sense.

Noise

2008-11-10T19:34:09Z

I turn on my MP3 player, forgetting what song was on last, and all that comes out is random electronic noise. Err, mp3 player is broken?

No, no... just aphex twin.

OMFG

2008-10-29T09:23:31Z

For the last half hour someones goddamn car broke, causing the horn to sound like someone was just leaning on the steering wheel. For a goddamn half hour. The guy who owned it and the landlord couldn't figure out how to kill it. With half the apartment building, the landlord, the owner and several police standing there, they still couldn't figure out how to shut it off. Seriously. I see them standing there without a clue from my window, run out there with my tool kit and kill the guys battery in less than 30 seconds. Was that so hard?

antiduh @ 2008-10-05T10:12:00

2008-10-05T14:13:14Z

The sorted and sundry

2008-09-22T07:33:28Z

Car is still running great, gave it a wash the other day, the trees outside of my apartment have been covering it with sticky sap, guh.

The guitar is going, but I'm a little stalled because I can't find songs that I can play and that I like, stuff that has a focus more on single string notes and picking instead of chords. So far I've got 'All Apologies' and parts of 'Wish You Were Here' down. I started just jamming the other day and came up with a simple song, which is still stuck in my head.

Started dating a girl by the name Julie. Taking it nice and slow this time. She's very different than me, which is part of why I like her. And she's patient =)

Speaking of relationships, my sister got married on the sixth, which was a great experience. Getting to see my family and extended family was a lot of fun. The part that sticks out the most was seeing some of my cousins and their kids, who I hadn't seen in nearly 6-7 years, since before college. What a shock to see these people all grown up after knowing them only as kids.

Work tomorrow. This week we'll probably be prepping for a migration of a 250kloc codebase to .Net 3.5. It all compiles and works and has been moderately tested by now, we just need to get everybody else up to speed on tools and such, then convert the projects and commit.

Time for sleep..

Rockin' out.

2008-08-28T19:25:14Z

So, I bought a guitar the other day after finally deciding that I really want to learn to play. So far it's been pretty awesome - I've only played for a total of about 5 hours and I can kinda make it through the intro to All Apologies. I'd be practicing more, but my fingers are killing me. Yay for learning new things

Happy little accidents

2008-08-24T18:48:20Z

I just found out that I had my automatic payments for my car insurance set a little to high (I hadn't noticed that my premium had gone down), and with an extra payment of $56 bucks today, my insurance is payed off for the rest of the year. It's like finding a couple $20s in the laundry, except in this case I found 40 of them.

Newly discovered music

2008-08-20T02:48:31Z

The Wombats (Check out Let's Dance to Joy Division, Kill The Director)
Silverson Pickups (Check out: Well thought out twinkles, Lazy Eye, Melatonin)
Rilo Kiley - Under The Blacklight
Wolf Parade - At Mount Zoomer
Bjork - Live At The Royal Opera House
Ladytron - Velocifero
Chemical Brothers - We are the night. (Salmon Dance!)

"Sun Microsystems is about to horsefuck the database world, and nobody sees it coming"

2008-07-28T21:48:17Z

But will it actually work? Sun published a few papers about HyTM last year, and they were a shot across the bow to Oracle. ... Rewritten using Sun’s scheme, BDB’s locking subsystem achieves several orders of magnitude more throughput as the number of threads grows linearly. Oracle owns Sleepycat Software, the developers of BerkeleyDB. The dicks have been thrown down, and Sun’s is dragging in the dirt.

http://www.theregister.co.uk/2008/07/28/sun_dziuba_tm/

antiduh @ 2008-07-19T02:32:00

2008-07-19T06:45:08Z

Superconducting magnet quench is when magnets that are made out of superconductors begin to stop superconducting because they were allowed to heat up too much. When the magnet begins to stop superconducting, it begins to heat up more, causing the quench to become a runaway process until the magnet is destroyed.

The Large Hadron Collider has a total stored energy of 10.6 gigajoules in all of its guide magnets, all of which must be dissipated in less than 200 milliseconds in the case of a magnet quench.

To dissipate such a quench in 200 ms, at a minimum power must flow at a rate of 53 gigawatts.

Kevin 1, Canandaigua 2

2008-06-23T06:32:56Z

http://www.gmap-pedometer.com/?r=2012569

I finally made it to Canandaigua and back on the bike. Twice before, I had tried to ride from my apartment to Canandaigua. The first time I tried (last year), I did made it there, but I didn't make it back - I had to call it quits because I was just too tired and the conditions too dangerous. The second time, earlier this year, I didn't make it there - I was trying a different route, and got lost somewhere in Farmington. By the time I had realized I was lost, I was already too far off path to have enough energy to backtrack, make it there, and make it back, nevermind having enough time left in the day. Thank you to the folks who got me out of that one. I was in bad shape.

This time, I made it. I beat the Canandaigua beast. I left around 1:00pm on Saturday, made it there at about 4:00pm, then after 66 miles and 5:41 of riding time, I made it back around 7:30 (just in time for Dinosaur, to boot).

It was a rough ride too. In Fairport, my tire slipped while I was trying to bike up some painted incline under a bridge, and I ended up banging my knee and ankles pretty bad. On the way there, somewhere in Farmington and Canandaigua, it started to rain, turing into a downpour with thunder and lightning, and that lasted for the better portion of an hour and a half. Route 332 gets pretty high up there, it's pretty much the highest point of the ride, and I happend to be on top of it on a metal bike covered in salty water just when the light show started. And of course I blew a tire (what ride would be complete without one?) somewhere in Farmington on the way back.

All in all, I think was a great experience. The countryside is wide and beautiful, and the view of the lake is wonderful.

As for stats, I'm up to 685 on the odometer, ~400 since this april. I'm hoping to do 1000 miles this year, but I don't know if it's going to happen. In the three months of April, May and June I've done 400 miles, 133 miles a month. If I'm to hit 1000 by the end of September, early October, I've got to push that up to ~200 miles a month. We'll see.

And I'm still planning on doing my run to Buffalo (and back?). Anybody willing is welcome to come, but it won't be an easy ride.

2008 Corporate Challange

2008-05-30T01:59:22Z

Ran it in 50:10, only walked for about 8 minutes of it. For 2009, I really want to hit 35, 40 minutes tops.

Mountain biking totally did not prepare me in the least for this. My feet and calves are killing me.

Emulin helps fight the diabetehs

2008-05-18T00:23:35Z

From http://www.local6.com/health/16314509/detail.html?rss=orlpn&psp=news:

The report featured Central Floridian Linda Hambelton, who takes 69 pills a day to treat her diabetes.

Hambelton has had three strokes, three heart attacks and 22 eye operations after suffering temporary blindness. She had a pancreas transplant and a kidney transplant.

.. or, "holy crap why am I still alive".

Huh, and I just saw that video..

2008-04-18T06:30:06Z

David Byrne and Brian Eno reuniting for new album

"I don't care when people die!"

2008-03-25T13:50:59Z