the thank you that keeps on thanking

The best way to thank your friends and family for helping you move?

Food poisoning.

Including myself and the lovely wife, nine of the dozen or so people who helped us move were struck down by the Subway sandwich platter. I don't think I've ever been so sick. About 24 hours of sick, sick, sickness followed by a week of not eating solid food.

MarkMail, sweet!

Last May, at the Mark Logic User Conference, Jason Hunter showed off MarkMail. It's finally available to the public at http://markmail.org/

Various links about it:

• An overview at Mark Logic's site
  
• A message from Mark Logic's CEO
  
• An article at O'Reilly OnJava
  

One part that wowed me is the ability to view attachments within the interface. Maybe I'm easily impressed. For an example, go here for email with a PDF attached. Click on the link for the attachment. Search for other attachments with a search string like 'extension:ppt' or 'extension:doc'

who knew?

Finally added labels to the blog posts. Who'd have guessed? Tied for the most frequent label:

• computers
• i am lame

so close, yet so far

The appraisal is complete and our buyers haven't complained so it must have covered the sale price. Our buyers decided to do the inspection themselves, which we found odd. It's a fairly new home, and we are paying for a home warranty, so maybe they don't think much could be wrong. But, we're baffled that they wouldn't spend a couple hundred to get a professional to look over the house to verify there aren't any non-obvious problems.

The sellers on our new home were happy to extend the closing date on our contract for another couple weeks, so everything is looking good for the closing and move in the next 2 weeks.

Our buyers came by on Sunday to do the inspection, said they needed 3 hours. We decided to head up to the new house, visit our friends across the street, and rake leaves. I'd assumed it would be about an hour's job.

15 orange bags, 6 blisters, and 4 hours later, we were done.

Did I mention we've got 7 big trees on the new lot? 2 ash trees, 3 maples, and another 2... something else ... and a couple assorted smaller trees -- peach, crab apple, birch. The ashes and had dropped all their leaves, the maples and other big trees still had the majority of their leaves. Guess what I get to look forward to doing again in a few days?

We roll back home around 8pm, and realize our buyers had left the back door out of the garage wide open. Awesome!

Soon all the contract drama will be over, and all that will be left is moving 9 years of accumulated crap. And then we can kick back on the huge back deck at the new house.

where's the suggestion box?

The employer frequently elicits employee ideas for additional revenue streams. I have decided on one that beats my previous "Nominal fees for helping minors purchase alcohol" stream.

1. Form Hannah Montana tribute band, 'Amanda Bandana'
2. Prey on gullible parents thinking they're purchasing Hannah Montana tickets.
3. Profit

However, brief googling suggests 'Amanda Bandana' is already taken for a teen's myspace/livejournal/etc.

This is as disappointing a googling as when I found that 'Hell Toupee' was already taken. Back to the lame-cover-band-name drawing board.

under contract!

Lots of interest on the house -- typically 5-10 viewings a week, for the 5-6 weeks we'd been listed. No offers till about 1.5 weeks ago when we got two in one day: one good offer, one low offer.

The 'good offer' went south fast. They had acted very flaky, so we were disappointed but not surprised.

• Our agent is told (on Wednesday): "We'll send you an offer today!"
• No offer. Our agent calls them and is told: "Oh, they're putting an offer in on a different house"
• 24 hours later, we get their offer. They've only offered $500 earnest money, ask for %3 in closing costs, and don't have show any money down for the loan.
  
• We decide to counter-offer, asking for $1000 in earnest money, and say we'll cover $5000 in closing costs rather than %3. Send the counter-offer Friday morning.
  
• Saturday: We hear from their agent, "I'll be talking to my clients this afternoon"
• Sunday: No word
• Monday: No word... Our agent calls, and is told "I'll be talking to my clients this afternoon"
• Monday evening: Their agent says the deal is off. Leaves some nonsensical message for our agent saying we should have just accepted the offer and we could have closed in 3 days. I don't know why he thought he could get the inspection, appraisal, and loan stuff all worked out in that time.

In the meantime, we'd also counter-offered the other offer. And did the counter-offer thing every day last week. The buyers appeared solid financially, and were motivated to buy in our neighborhood -- their cousins had just bought up the street from some desperate sellers (they had been on the market 6 months). That house also probably screwed everyone in the subdivision on appraisals (they'd listed in May for $280 and gradually dropped to $250, accepting a $237 offer). It was nearly identical to our house -- 100 sq feet more, but hideous inside. Despite the hideousness, it was clear from the start that our buyers really, really didn't want to spend more than their cousins.

After talking them up $7k from their initial offer, we ended up accepting quite a bit less than the asking price. However, with the current market, and the fact that we'd already found our new house and were under contract on it, we decided that we couldn't wait for the perfect offer. At the price we accepted we wouldn't have any worries with appraisal and it gives us enough profit to put 20% down on the new house with a bit left over.

Our agent's co-workers are amazed she got us under contract in ~5 weeks. Some of it was us being priced right, in the right location, and our hard work keeping the house clean. But a huge part of it was how hard she worked. Lots of marketing, including at least one open house a week. She was awesome, and very funny. Best of all, unlike a lot of horror stories we'd heard about realtors, she was always available. Any time we had questions, she was very quick to respond to phone calls or e-mails.

If you're selling (or buying) or planning to, I have no reservations recommending her. Shauna Thomas at the Sugarhouse Keller Williams (599-9126). Shauna also hooked us up with a mortgage broker: Rain Wallace at Citywide Home Loans. We didn't have as much one-on-one time with Rain, but she was great -- quick to answer phone/email questions, patiently showed us our options, and helped us lock in a loan with a great rate.

nuggets from the dumb monkey sack

We've got a new developer in the group. As he's installing all the supporting tools he'll need for the projects he'll be working on, he came to me when Ant stopped working. Odd messages about Ant core/tools version incompatibilities.

We fiddle and fiddle with JAVA_HOME, with ANT_HOME, with PATH... No luck. Same messages, despite the Ant we want being right there at the start of PATH and in ANT_HOME.

Like a revelation from heaven, I realize... wait... what's in CLASSPATH? Well, there's a Documentum Jar in there... but that wouldn't have stuff to do with Ant, right?

The documentum Jar in question turned out to not have any classes in it, but it did have a giant list of other Jars in the manifest's classpath. Including Ant. More specifically, a local install of Ant 1.2 beneath the Documentum installation.

As much as I'd like to blame Sun for not providing a non-insane standard for application startup, I think its more appropriate to vent at EMC/Documentum for creating an installer that modifies the system's classpath environment variable to add their kitchen-sink Jar. For as much money as they charge, you'd think they could write an installer that doesn't screw up your system.

I'm not sure why I was surprised. Nearly every other interaction I've had with Documentum has been an exercise in agony. They must have a burlap sack full of exceptionally dumb monkeys deciding how their software gets installed and configured. Other nuggets from the monkey sack:

• No desktop application needs to talk to more than content server, right? Well, if they do, they can manually edit this INI file in c:\windows.
  
• No desktop application needs to talk to more than one version of a content server right? Well... if they do, they can uninstall/re-install to do so.

I can excuse their crappy, obfuscated APIs. Still hate them, but I can understand inflicting them on software developers. But to screw things up for all the end-users, QA, and support that need to use their crappy desktop apps, seems mind-numbingly stupid.

blog comment spam

I had to enable moderation on the comments due to the spam getting added. Moderating seemed like a better solution than going in and deleting the comments after-the-fact.

The first thing that clued me in to their spaminess:

These articles are fantastic; the information you show us is interesting for everybody and is really good written. It’s just great!! [more sales pitch and spammy URLs here...]

Anyone who actually read my blog would know that it has never been, and never will be any of those things.

hardware problems are teh suck

Why does my home PC pick the worst time to decide to stop booting? I really didn't want to spend 4 hours tonight fiddling in the XP recovery console, the BIOS settings, and running disk tests.

At first, I thought it was the height of inconvenience that I'd packed up my Windows XP install disk, had to search the basement, unpack the box, re-pack the box. Then, unpack the box again after realizing I forgot to grab the floppy disk with the SATA drivers.

However, that turned out to be minor compared to having to reboot twice every time I changed something in the XP recovery console. Once to get out of the console, and again so that the XP install disk would release itself from the CD-ROM drive. Gah!

System Rescue CD and the XP recovery console both seemed happy to read the disks, no problem getting into the partitions and looking at files. Tried fixboot, tried fixmbr, ran DFT's tests on both hard disks. No love from the boot gods.

Finally, decided to start fiddling with the BIOS. Swapped the order in which the hard disks are booted (SATA then IDE), and it came up. Not sure why I didn't notice it before. Also not sure why the BIOS implies it'll check disk #1 then disk #2 when booting, but only actually try #1 and give up.

Also not sure what would have screwed it up. I re-installed Ad-Aware, and Spybot S&D over the weekend. We also had a couple power-bumps a couple days ago.

In less annoying news, System Rescue CD is 4-6x as awesome as it was the last time I tried it a few months ago.

found it!

The pictures of our house are now on the online listing... all the paint colors and floor photographed well. And, our cat is now famous.



So far, things are looking pretty good for selling our house. Lots of interest. Open house this Sunday. Our realtor is so optimistic, she's going to have a 2 co-workers there to help her out.

Hopefully it'll all work out because we found the house to buy. The location we wanted, near my parents, near friends. Great neighborhood. We'll be going under contract on it tonight. I'm not sure if I feel relieved, or more panicky.

Best of all... the new house optionally includes 6 chickens. I know it's a bizarre and sad thing to have always wanted... but I really have liked the idea of having chickens. We're already speculating that our cat is finally going to be even more fascinated and intrigued by the outside. Maybe enough to get over her fear of the unknown.

sweet spot

After many self-imposed deadlines came and went, finally listed our house for sale this weekend.
We felt bad about letting the deadlines pass, but we wanted finish all the painting and staging of the house before we listed. Turns out that was a very good idea.

Listed it about 3pm Saturday. Around 5pm Saturday we got a from our realtor saying there was someone who wanted to look. Aaaiiiiieeee! We weren't ready. The house was a mess. Paint cans, cleaning supplies, unpacked rooms. A couple hours of last-minute cleanup got 90% of the rooms ready for viewing. They seemed to like the house, and also said that even with our mess, it was one of the top two homes they'd seen in the area.

Got a wake-up call Sunday from our realtor. Another group wanted a viewing. We didn't have to do much because of the previous day's frantic cleanup. We run a few errands while they look at the house with a realtor. Afterwards, we get back home to be accosted by a family in a mini-van asking about the house.

Late Sunday afternoon, I finally get a breather and mow the lawn. Interrupted once by a couple while mowing the front lawn. Interrupted a second time, and nearly had a heart attack, while mowing the backyard when a realtor waltzes through the gate with another couple in tow asking if they can view the house.

I guess we hit the sweet spot w/ regard to location, price, and size of home. We don't even have the pictures online yet (that should happen today). It's a big relief, seeing the amount of interest in the home. And, seeing that our work over the past 2 months was worth it.

If only our own house hunt was as successful. Still having found the right home after viewing probably 70+ homes. At least we're done with our own home now and we can focus on finding our new home.

update: Got a call from the lovely wife. Another realtor and their clients just dropped by "We were in the neighborhood...". 4 viewings in the 2.5 days it's been listed. Aiiiiieeeee. We're beginning to wonder when we'll have time to do the final cleaning for the open-house next weekend.

stressful vacation, house hunting, awesome basement

Just finished a week of vacation. 'Vacation' being code for fixing up the house -- finished laying the laminate floors, installing new baseboards, some new bathroom cabinets, and lots of painting.

Still not terribly near the end point. Plenty more painting and packing to do before we'd feel comfortable having potential buyers look at the house.

The lovely wife and I have been house hunting with our realtor. So many open houses, they're all a blur now.

The price on one with a super-awesome basement has finally dropped enough to be worth it. Those are indeed built-in sofas. The tales those cushions could tell.

But seriously, that house has a great location, great square footage, and the extra fantastic basement. And, the price has dropped enough that we should be able to update the kitchen and a couple other out-of-date rooms prior to moving in.

Of course, the basement would have to stay as-is. It's just too awesome.

crappy neighbor water hazard

Three times over the last couple weeks, my and one of my neighbor's backyards have flooded. The picture is the view from my yard into the neighbor's. Our house isn't that effected... but the water is right against the neighbors foundation. Not good.

Seems another neighbor can't figure out their inflatable above-ground pool.

• The first time, I think they forgot to turn on the filter and decided to the drain the funky water
• The second time, they started refilling the pool and left for the day.
  
• The third time (two days after flood #2), they hadn't tightened a coupling and didn't notice the leak.

On the plus side... I've talked to my fellow-floodee neighbor more in the last week than I have in the ~2 years they've been in their house.

On the minus side. The flooder neighbor seems more annoyed that we're upset, rather than apologetic. I understand that once the water is out there isn't much they can do about it. But, it'd be nice if they seemed slightly concerned about things they do that effect their neighbors.

mmm... pie!

A friend helped us move our furniture around for the work we're doing on the house. Last time, he and his girlfriend raved about Left Fork Grill (68 W. 3900 South, 266-4322). We had him over again this weekend to move furniture back into the finished room, and as a reward for helping us we stopped there for breakfast.

I had a breakfast sandwich (sausage and egg), the lovely wife french toast, and our friend had a fried egg sandwich. Everything was delicious. And, they made hash browns correctly -- the right ratio of crisp to tender, and not greasy.

To top it all off, they have the best pie ever. The lovely wife had banana cream, I had raspberry-rhubarb. Excellent crust, freshly made fillings. Both were great. Many others think so too.

It's shot right up to top of our list of breakfast spots -- great food, reasonable prices.

A shame that it's only open for breakfast and lunch. Pie for breakfast may become a family tradition.

finally, a chance to wear a toolbelt

We've been thinking of moving for a couple years now. But, we'd wanted to get a few things fixed up with the house. I needed to get the yard and flower beds into better shape, and we wanted to replace some of our carpet.

A few months ago, we'd bought some glueless laminate flooring at Costco while it was on sale (ended up saving around $150 because of the sale). Finally got around to starting the project last weekend. Finally realized the extent to which one of our cats ruined the carpet. Ugh.

The flooring went down quick. Not as easy as the instructions make it seem. But, it looks nice, and gave me my once-every-two-years excuse to wear a toolbelt.









And, in odd things I've stuck in my pie-hole news:

• The 'Hog Burger' at Britton's (964 East Union Square in Sandy). A burger with grilled onions nestled between a grilled-cheese sandwich with bacon, and a second grilled-cheese sandwich with tomato. The first half was good. The second half was 'what have I gotten myself into?'
  

impatient book review

I couldn't be bothered to wait the year(s?) for the U.S. release of Steven Erikson's newest Malazan book, Reaper's Gale.

After comparing the Canada/UK prices on the respective Amazon sites, and plugging in the conversion rate, it was cheaper to order from the UK and have it shipped across the Atlantic. Odd.

It was very good. Heartbreaking (and other not-so-heartbreaking) deaths of main characters. Resolution of plot lines. Very enjoyable.

I also picked up Night of Knives, by Ian Cameron Esslemont. Also set in the Malazan universe. It was pretty good, but not great. It could have used another round of editing -- typos and a couple confusing sections. But, it was fast-paced and fun.

roller derby -- violence, short skirts, beer -- what's not to like?

The lovely wife and I went to the roller derby last night. Lots of fun. Lots of violence. Lots of hot chicks in short skirts zooming around the track and knocking each other down. And beer! Mmm... Beer.

It was an All Star match between the local league and their opponents from Idaho.

Salt Lake Software Symposium, day #2

Last day. Still pretty good, but didn't rock as much as the first.

Attended a couple more of Jared Richardson's sessions: "Agile Testing Strategies", and "Software Development Techniques". Lots of good real-world examples, and lots of inspiration for changing our processes. Definitely thinking of picking up Ship It!

Neal Ford's "Pragmatic Extreme Programming" was good. But, I think it'll be easier to work in lessons from Jared's sessions (on days 1 and 2) into our existing process.

I wasn't going to attend Brian Sam-Bodden's "Complex Builds with Ant" session. From the slides, it appeared like I'd already knew the major tips on my own. But, I was wiped out by the last session. Decided it was worth going to a topic I was familiar with in order to pick an expert's brain. Only three attendees, so the session went fast and I was able to ask a lot of questions.

More sour-grapes today over all the Ruby rah-rah-rah. Again, Neal Ford declared Ruby the winner in the dynamic language race on the JVM. Neal is a very smart guy, a great presenter, and also great to talk to one-on-one. I don't have any issue with him advocating Ruby. Any language that makes developers more productive is a good thing. But, it seems disingenuous to declare a victor in a wide open race to a room full of people who have little to no experience with dynamic languages.

Personally, I prefer Python over Ruby. Others can and should disagree. Here's a fair, but slightly biased (since the author is familiar with Python), comparison of Python vs. Ruby.

I'm not sure who to blame for lack of Python excitement in the Java world (or at least at this conference). Maybe with the recent revitalization of the Jython project, it will get more Java developer's attention.

Microsoft's CLR (and the new DLR and Silverlight) seems very exciting. See here and here and here. The second link is a screencast showing interoperation between Ruby, JavaScript, Python, and VB.

All in all, the conference was very good. I'll definitely plan on going next year.

Salt Lake Software Symposium, day #1

Since this blog is all about (pick one):

• lame book reviews
• lame software conference anecdotes
• odd things I've put in my mouth
  

Here's more software conference anecdotes.

Spent today at the Salt Lake Software Symposium (aka No Fluff Just Stuff). So far, the conference is very good. Attended presentations by Neal Ford, Jared Richardson, and Brian Sletten. All great presenters, showing off great technology.

A friend has tried 2-3 times to explain Aspect Oriented Programming to me. It sounded... like more trouble than it was worth. Brian Sletten's AOP presentation was very useful in cluing me in. Seeing the AspectJ/Eclipse tools in action got me all excited to try it out. The immediate use-case I have would to remove a bunch of cut-n-paste concurrent locking boilerplate in my pipeline application. If I understood it correctly (always a questionable assumption), I can replace all that code scattered throughout my class with an aspect that'll do the

  writeLock();
  try {
       // .... do stuff....
  } finally {
       releaseWriteLock();
  }

logic all from a single place for all the setters methods in my class. And a similar readLock() aspect for my getters.

Brian also presented NetKernel. Also awesome. Still trying to get a handle on how exactly we can leverage it for our needs. His Mashup/SemanticWeb/RDF presentation was also good.

Jared Richardson's presentation Shippers Unite was very good. The best part, calling sales people 'Sales Critters'. He had lots of great advice, and great best-practice type information.

Neal Ford's SOA presentation, and his keynote after dinner were also very good. His keynote, "Polyglot Programming" was very interesting. Contrasted the increasing cruft, and complexity (and irrelevancy) of the Java language against the awesomeness of the Java platform. Described the increasing place Dynamic Languages will have within the software industry. In particular their place inside managed runtime environments (either JVM or CLR).

The only major disappointment I experienced was the (apparent) disdain the conference presenters seemed to show for Python/Jython. Lots of Groovy/Ruby/JRuby/Grails rah-rah-rah. I know Rails is the new poster child, and Ruby folks should be proud. But, to give Python/Jython/IronPython hardly a mention. And, the one time the word Python appeared on a slide, to explicitly declare Ruby the winner over Python... seems premature.

Yeah, yeah. Sour grapes. Maybe I should try Ruby before deciding I don't like it... but the Perl-like $ and @ syntax rubs me the wrong way. And, maybe I should just be happy about the rising tide of dynamically typed languages that'll lift all boats.

discrimination!

Someone brought a delicious looking treat in to work today.

The sign next to it:

APRICOT CAKE
____________

(no nuts)


And, in other lame joke news, my new favorite joke (the bottom panels) . I'm just that classy.

mmm... pie.

It's turns of phrase like this that keep me going back to Worse Than Failure :

Of course, upon his return to work, fate crapped on his hope and served him up a big, steaming slice of shattered dream pie. The familiar stench of hot computers and death descended on him while he tried to shut out the familiar fear of dying in the room and having the network absorb nutrients from his decaying body.

san francisco -- day #4 -- "You like extreme?"

Final conference sessions today. OK, but nothing fantastic.

Great lunch at House of Nanking. The waiter arrived, said "Your first time here?" When we said yes, he took our menus. No idea what the dishes were, but they were all super-excellent.

Then got a taxi to the airport. After a few minutes of silence, the driver asks, "You like extreme?"

I'm not sure I hear him right. He asks again, "You like extreme?"

Yep. I had heard him right.

Pause....... I'm not sure if I'm going to get invited to a cock fight, or what. I guess that maybe he's asking if it's OK to play his favorite thrash metal CD. "Uhhh.... extreme music?" I ask.

He explains, "No! Extreme! Like X! .... Games! Extreme. Sports."

I respond, "umm. A little?"

The rest of the ride was enclosed in blissful silence.

san francisco -- day #3

Noisey street. Some sleepy sessions. A couple awesome ones -- displaying debugging and profiling support for XQuery.

Then the final session of the day. At first, I'm thinking... blah, a mailing list archive viewer? Oh, how wrong I was. Sweet zombie Jesus, it was cool. Or maybe Jason Hunter is an great speaker. Or maybe both?

It was also very applicable to the work we're doing -- transforming and enhancing XML and non-XML assets into an normalized XML form and storing the results in MarkLogic.

san francisco -- day #2

Noisey street + hotel room close to street = no sleep.

As the opening session begins, I realize I can't focus. Then I notice the strobing migraine precursor.... Today is going to be awesome!

Interesting sessions. One of the sessions on the technical track of the conference seemed like a waste of time, but most of the others were interesting. And an added plus, presentations by the vendor and one of our competitors seem to validate our current architectural approach.

san francisco -- day #1

Flew to San Francisco for a technical conference. Flight uneventful, and in a teeny CRJ200. Only 12 and a half rows of seats.

Googled up restaurants near the hotel. I've been craving Vietnamese all the time lately, so I decided on a hole-in-the-wall place with good reviews -- Golden Flower Vietnamese. Just a couple minutes walk from the hotel.

Pretty good, but not the best I've had. The imperial rolls were greasy, and more mystery-meaty than I'd have preferred. The bun with grilled pork and shrimp was tasty.

In order of worst to first:
Pho Cali -> Golden Flower Vietnamese -> Pho Hoa -> Shanghai Cafe -> The deeee-licious versions Dien made for us over at Mark's house. Mmmmm.

Also, it's less fun to travel without coworkers. Wandering aimlessly is less fun to do on your own... it feels more like getting lost than having an adventure.

additional lame book reviews

Yes, more reviews. Why the wait? Because I finally read some books again during a few days of vacation.

Day Watch by Sergei Lukyaneko -- pretty similar to the first book, Night Watch. As good as the first, and sometimes as infuriating (wah? where'd that plot twist come from?). But pretty fun. Hopefully the new movie will be good as well.

Storm Front by Jim Butcher -- the lovely wife and I are enjoying the Dresden Files on SciFi, so I thought I'd pick up the first book in the series it was based on. Pretty good. On the fence on whether I'll pick up more.

Fiends of the Eastern Front by David Bishop -- What'll that crazy Hitler do next? Hire vampires to help win WWII, that's what. I liked the premise. I liked some scenes. But, ultimately very lame.

Did I mention it was lame? And full of typos? I usually blissfully skip right past most typos, not this time. That must mean they were 2-3x more that I missed.

Sample paraphrased ending scene of each book (in the 3-book omnibus):

books #1 and #2:
Lord Constanta (main baddie) : "Yes. You tried to kill me. Did I mention I'm completely (and lamely) invulnerable? It's because my Sire is so super bad-ass. I'll let you live to the next book, but let me tell you about my ludicrous world domination plans first...."

book #3: Yes. I'm spoiling the ending.

(further expository villains explaining their ludicrous world domination plan ... : "Yes, our plan is to darken the skies over Europe so we don't have to worry about our sunlight imposed curfew. Then we can drink blooooooooood all the time! Muauahahahhaa. No, I don't see the complication that that'll kill all life other than us... C'mon, man. We're gonna teach the stupid sun a lesson.")

And, the horrible, horrible, horrible ending:

The Sire (basically a big vampire bat you might expect to see in a Godzilla flick) : Bwaaaaah! Watch me fly outta this lake of blood. Oh nos! The day-walker hero is doing his homage to Slim Pickens in Dr. Strangelove by riding a silver-tipped V1 rocket into me. I esplode. When I gave Lord Constanta his resurrection powers, I forgot to give them to my self. Curse my strict FIFO ordering of completing my TODO list.

I was willing to give the books a little credit until the ultra-lame ending. Christ on a pogo stick, that was awful.

change the world?

I've been getting more and more excited by the OLPC project. Not only is it exciting technically, it seems like it has huge potential to change the world.

It'll take a few years before the educational goals bear fruit, but it'll have a huge technical impact much sooner. A lot of the cool technology could make its way out to non-OLPC hardware -- displays, power management, networking, security, exciting Python changes... lots of great stuff.

The Daily Python-URL pointed me here. A great introduction to the project, its goals, and the technology.

finally learned my lesson

I clearly remember thinking a year ago: "I've had incredible luck with my computer hardware. No disk, motherboard, power-supply, or other problems with any of the computers I've owned."

Since then, I've replaced the power-supply on my newest computer. TWICE. Both times, the hardware was still under warranty, so financially it hasn't been an issue. But the inconvenience of losing access to the files has been annoying.

After hearing good things about Mozy, I decided to give it a shot. So far, so good... Of course, it's only been an hour. Fingers-crossed for more hardware problems! Woo-hoo!

And, in somewhat related news, I'm finally taking the time to encrypt my sensitive files -- tax returns, contact info, schematics for man-eating robots, muffin recipes, etc. I'm giving TrueCrypt a shot. Because I'm cheap. Also, having a cross-platform encryption solution was a big plus. So far, easy to setup and use.

lesson of the day: just assume Co-Worker #1 is correct

I've been reading Java 2 Performance and Idiom Guide, it's quite good. But, I think I'm too interested in experimenting with the tweaks it suggests.

As I mentioned in my last post I'm working on a data-processing pipeline, and have spent the last couple days investigating why the performance (records/second) of a particular pipeline stage drops off like it does.

The software we're working on analyzes bibliographic records and creates a representative sample. Our users create mappings between one XML schema and another. They have to go through much fewer iterations if they've got a good sample to work with. And iterations are much shorter if they can test their mapping against a sample that's <500 records, rather than 16+ million records.

The oddly performing pipeline stage normalizes XML entities by replacing them with a numeric reference to their Unicode equivalent. Yeah, the DTD should do that... but it doesn't help when the data provider escapes the entities. Or when their DTD replaces the entity... with the entity... argh.

The graph below shows the records/second performance of three stages of a pipeline (y-axis). The x-axis is time. The yellow nodes are each file's entity-replacement performance. The blue nodes are the files getting split into 5000-record chunks and inserted into a Marklogic database. The pink nodes are each 5000-record chunk being analyzed. The first 500 or so files all have 30,000 records; the rest of the files are daily updates that vary in size -- that's why the tail end of each gets a little kooky.The initial steep drop-off for the first few files was understandable. It's one of the first stages of the pipeline, and the threads doing entity-replacement don't have to compete for I/O with other pipeline stages until the following stages have entity-replaced-output to work on. But the continued decline made me worry that there was a problem in the code.

Reviewed the code. Each file is getting processed by its own thread. Each thread should create SAX handling objects that aren't shared between threads. No static fields or other stuff to interact between threads...

But! I notice that the SAX handler uses a StringBuffer to accumulate a records-worth of text as it replaces entities. When endElement is reached it calls sbuf.setLength(0), which my fancy new book learnin' suggested could cause memory/performance problems.

And I set off on my non-adventure....
There are four series on the graph above.

1. Dark blue is the original run.
   
2. Pink is the result of replacing sbuf.setLength(0) with sbuf = new StringBuffer(). No fabulous improvement. Stupid book learnin'.
   
3. I decide that 'new StringBuffer()' meant that memory is getting re-allocated from initial the default StringBuffer size, which isn't as efficient as it could be. Yellow is the result replacing sbuf = new StringBuffer(INIT_LENGTH). INIT_LENGTH is a couple thousand bytes. Still no major improvement.
4. Re-review the code. Realize that the FileInputStream/FileOutputStream objects aren't wrapped with BufferedInputStream/BufferedOutputStream. The light blue nodes are the result of changing that... Still no major improvement. I assume this was because these input files are in Zip files, and output to GZIP files -- so were already buffered (I did increase their buffer sizes along with adding the buffering to the uncompressed-file handling).

Then I remember the comment from Co-Worker #1 when I showed the graph to her before I began my journey to nowhere: "Maybe its just file-size"

I fiddle with my statistics exporting code to include the file-size in the exported spreadsheet.
*@&#$*&!

And the KB/second graph looks like:Moral of the story: Get the most complete picture of a performance problem you can, and don't be swayed by the kooky little performance trivia you read.

a programming post, odd.

At work, I've been working on a data processing pipeline. The meta-data for the pipeline (e.g. status of pipeline phases, data locations, etc) at runtime is held in a LRU cache and flushed to persistent storage. Prior to a couple weeks ago, the storage was just a bunch of files on the filesystem. It worked, but it was pretty slow to collect aggregate information out of it -- e.g. if you wanted a global view of all status within the pipeline, you had to traverse the meta-data graph loading each individual file as the traversal progressed.

I'm allergic to relational databases, and we're already using Marklogic/XQuery to store and transform our data, so we decided to try a new implemenation of the LRU cache with the storage in Marklogic. Performance wasn't impacted much (maybe an extra couple minutes for a 1500+ file pipeline that takes 10+ hours), and now status reports take a couple seconds rather than a few minutes. Hooray!

Since that was less painful, I decided I'd also try getting some performance statistics out of the meta-data in the database. There are a few helper functions to do sum/count/average, but I wanted to avoid iterating over the list multiple times. I found some psuedocode here, and implemented an XQuery version:

(:
This is an XQuery implementation of the variance algorithm on this page:
   http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

   credited to Donald Knuth, The Art of Computer Programming, vol 2: Seminumerical Algorithms, 3rd edn., p. 232 Boston: Addison-Wesley.

It relies on Marklogic's extensions to XQuery, so it probably won't
work on other XQuery implementations.
:)
define function putil:getSumMeanVariance($srcvals as xs:double*) as node()* {
   let $n := 0
   let $mean := 0
   let $S := 0
   let $sum := 0

   let $throwAway :=
       for $x in $srcvals
       let $delta := $x - $mean
       return (
           xdmp:set($n,$n + 1),
           xdmp:set($mean,$mean + ( $delta div $n ) ),
           xdmp:set($S, $S + $delta * ( $x - $mean )),
           xdmp:set($sum, $sum + $x )
       )
   let $variance :=
           if ($n gt 1) then
               $S div ($n - 1)
           else 0
   return (
       <count>{$n}</count>,
       <sum>{$sum}</sum>,
       <mean>{$mean}</mean>,       
       <variance>{$variance} </variance>,
       <stdev>{math:sqrt($variance)}</stdev>
   )
}


let $elapsedTimes :=
 for $pnode in collection($collName)[ .... uninteresting XPath predicate stuff here .... ]
 return $pnode/stats/time/end - $pnode/stats/time/start
return <stats>{getSumMeanVariance($elapsedTimes)}</stats>

lame book reviews: the revenge

While the lovely wife recovers from her surgery[1], I've been reading Carl Zimmer's fascinating blog: The Loom I've always been interested in evolution and biology. Once I started, I can't stop digging through his archive of posts. Great Stuff! I read a couple of his books a year or two ago. "Parasite Rex"-- creepy, disgusting, fascinating. and "Fish with Fingers, Whales with Legs", also very good.

Fiction-wise:

• Scott Smith, "The Ruins" -- meh. It had good bits, but I couldn't suspend my disbelief w/ regard to the villain. Definitely not a book to read while vacationing in Central America.
  
• Cormac McCarthy, "The Road" -- A man and his son wander across a post-apocalyptic wasteland. Heartbreaking. Good.
• Dan Simmons, "The Terror" -- 19th century Arctic explorers stuck for years in the ice while a monster slowly picks them off. Awesome.

[1] Since my brother-in-law reads this, I'd better explain. She had a lipoma removed. It was an outpatient procedure, everything went well, and she's recovering at home. Today is probably her worst birthday ever, but she's doing OK.

end of job hunt

After 2 interviews and one offer the job hunt is over. I decided to stay at the current employer.

On the same day I received the offer our CTO came to talk to us. He said pretty much everything I wanted to hear: my project isn't dead, he's signed the purchase order for the new hardware we've been waiting for, and our office will have an active role in the next major upgrade to our customer-facing system. That, combined with the fact that I really like the people I'm working with, made me decide to stick it out.

The offer was tempting. It was from the company that bought my ex-employer. I know the work is exciting, and the people are great. But, in the end I decided that I'd have much more control over my future at my current employer. I have more say in which projects I'll work on, as well as having a major hand in all the parts of their development cycles. And, the much shorter product lifetimes will keep my technical skills fresher.

Hopefully it'll turn out for the best.

favorite post ever

While reading the rec.arts.sf.written thread spawned by the news that HBO has bought the rights to produce a TV series based on George R. R. Martin's "Song of Ice and Fire", came across perhaps the most beautiful thing ever uttered on USENET. The thread is mostly a discussion, comparing Good vs. Evil as characterized in Martin versus JRR Tolkien.

> If killer ice zombies aren't evil, then the word has no meaning.

return of the lame book reviews

Enough job whining. Time for more lame book reviews. Hooray!

• A Dirty Job, by Christopher Moore -- Man discovers he is Death. Or, at least, one of Death's little helpers. Very funny. Very sad. Very good.
• Old Man's War, by John Scalzi -- Earth's Colonial Defense Force only accepts recruits that are over 75 years old. Very good
• The Android's Dream, by John Scalzi -- Best. First. Paragraph. Ever. "Dirk Moeller didn't know if he could fart his way into a major diplomatic incident. But he was ready to find out." Very funny. Very good.
  
• The Algebraist, by Iain M. Banks -- I didn't like it as much as his Culture novels, but still good.
  

job hunting

Well, I've decided to jump ship... Or, at the very least, test the waters very seriously. I've even updated my resume.

The job market is very good at the moment, with a number of cool sounding jobs. The ex-employer even has good jobs -- and even appears to be making money. Even better, I still have friends there. But... haven't decided if I want to get back into the C++, real-time, simulation world. But, the hardware/software at ex-employer is so, so cool.

worst new years gift, ever

Found out on Saturday the employer sent a wonderfully vague re-organization announcement on early-AM Friday.

I could have a job. I could be laid off. I could be asked to move to Manhatten to join that team.

Or, it might just reflect the management structure of the software development work at the company. Which would also suck, because my (immediate) boss is great.

Thank goodness we exercised some restraint this Christmas in terms of gifts for ourselves and family. We also transferred our credit card debt onto a 0% APR card a couple months ago... we were planning on paying it off over the next year, but now it might just get minimum payments until the job thing settles.

I'm not too worried. There are a ton of jobs in the Salt Lake area (according to monster and dice), and the company that bought my old employer's simulation business is doing very well and has positions. There are also a ton of cool sounding start-ups down in Utah county... the wife has put the kibosh on moving down there, but Draper/Bluffdale might be doable.

I'm mostly frustrated. I hadn't planned on staying at [employer's name withheld] for more than another year... but it would have been nice to leave on my terms rather than theirs. It would have also been nice if they'd done the layoff at a time when more than 5 people were actually in the office, and sending such an vague-but-ominous e-mail announcement.

Update: The job is safe. The vague e-mail implied the development positions would be cut/moved, but in fact none of the development positions at our location was involved. Our software architect was laid off, however... probably due to butting heads with the new CTO. The rest of the layoffs were spread across the groups.