which is better?

1. Laughing/crying at the right-wing, loony spam mails I get from my kook of an Aunt.
   
2. Laughing at the "in no way related" e-mail containing the facts behind the rumors that inevitably follows a few days later

The latest example, a forward from the doublespeak-tastic 'American Family Council', with the subject line 'Fwd: If you think things are bad now, just wait....'

{snip}

In case you may think I’m a “the sky is falling” type of person, you should know: When it comes to predictions, I am a very reserved person. But not on this one. I cannot overstate the damage a liberal victory will do to our country. The upcoming election is the most important in the history of our nation. Yes, if the liberals win you will lose some of your religious freedoms and free speech rights. Churches and pastors will not be exempt. You will not be allowed to say certain things about a particular group. Every item of the homosexual agenda will be approved. All the laws protecting the unborn will be wiped away.

{snip}

And 5 days later, the next e-mail from her with the subject line, 'In Fairness to B. Obama. AND J. McCain.....Snopes.com' :

There are so many derogatory Political Emails floating thru space out there, that I am personally tired of having to wade thru them all - and acknowledge that it's hard to get the straight-forward Truth....

I think it might do us all well to consult those who have done the research on these (either bogus or real-truth) emails. (See links below)

Click here: snopes.com: Politics (Barack Obama)

Click here: snopes.com: Politics (John McCain)

My favorite part of the inevitable follow-up e-mail is the lack of self-awareness or any hint of an apology for adding to the onslaught of misinformation herself. Despite the fact that no one has asked her to, I'm glad she's warned us that she's "personally tired of having to wade thru" all those spam messages she passes along.

I've decided whenever I get baited into a political or religious discussion with my family, I'll make the saving throw by chiming in with, "While I appreciate that these Gay Agendas (*music sting*) make our meetings well organized, they're really making me re-think my conversion to Secret Islamism."

P.S. Also plus too as well : http://www.factcheck.org/

worst. impression. ever.

Dear Autoflush Toilet,

You ain't foolin' me! You're not a bidet!

Why does your sensor think my back is transparent?

You're horrible.

Love,
Steve



P.S. Now that you're thinking about me pooping, I'd like to point out that I deftly avoided use of any onomatopoeia.

P.P.S. Mostly because I couldn't connect the dots between this post and the punchline using the consignment store named "Grunts & Postures" near the first apartments of my wife and I.

P.P.P.S. First one to comment with a "full of shit" joke gets 15 demerits.

P.P.P.P.S. Yes, despite the demerits, it is still worth it.

hudson == teh awesome

Hudson is great. Lightyears beyond CruiseControl. Everything configurable through the (awesome) UI, a ecosystem of plugins, and incredibly simple installation.

We don't have that complex a build, but our SCM team looks at me like I'm a crazy person when I point out yet another issue with the build. After the umpteenth time I've found yet another build problem, I decided to give Hudson a try. We need to checkout some modules from the trunk, and others from a branch. It was bad enough when their chosen solution to that problem meant we couldn't automatically trigger the builds with a check-in -- builds had to be forced from the UI, then a pre-build step did a CVS checkout of the modules from the appropriate branches. The last straw was realizing that the pre-build script also contained some logic that overwrote any changes we made to our build.xml file with some out-of-synch version of build.xml that one of them had created at some point in the past. The sort of thing you'd like a heads up about.

With Hudson, builds have been a snap to set up -- monitor these CVS modules, run this script from the repository if any changes are checked-in, the build artifacts will match this regex pattern allow download of them from the UI, keep only X number of builds on the CI server.

Hudson doesn't let you checkout from a mixture of HEAD/branch either -- but it's simple enough to set up multiple build commands, the first of which is doing the branch checkout. We won't get a complete changelog or CI automation, but it's good enough.

disturbing. hilarious.

lessons in conflation : compare != sort

For months, the business people have complained that the rules engine sporadically produces different results on Windows than HP-UX. They'd complained previous developers said, "sorting is different on Windows and Unix". I did my best to restrain any reply more caustic than, "That's silly."

Until this month they couldn't point me to bills that had the problem at the same time as we had log files from our regression tests (or find me any archived e-mails from the original developers). Given that we're probably the only people running the application on both HP-UX and Windows -- during our regression testing on the two platforms -- the bug hasn't been that high a priority.

Finally tracked down the cause of the issue to be differences in the qsort() algorithm of the C runtime lib -- specifically how qsort() handles two elements that compare as equal (!). For some reason, the Windows qsort() performs some extra comparisons and 1 extra swap.

The list we're sorting looks something like :

1. $23.96
2. $23.96
3. $23.96
4. $23.96
5. $40.95

On Unix, qsort() and this particular compare (descending order by amount) results in the following :

5. $40.95
1. $23.96
2. $23.96
3. $23.96
4. $23.96

On Windows, its qsort() and the same compare function resulted in:

5. $40.95
3. $23.96
1. $23.96
2. $23.96
4. $23.96

The values we're comparing are equal... but what we're actually sorting is the value and a reference to where it started in the list. Because the sorting produces different results we end up annotating the bill lines differently on Windows than Unix.

Changing how we do the comparison fixed the problem. Now we compare left/right values, if they are equal we compare the originating line number of the values.

And now we get consistent results no matter the platform:

5. $40.95
1. $23.96
2. $23.96
3. $23.96
4. $23.96


Next time, in Steve's job rants:

• Typedefs on top of typedefs on top of typedefs. Oh yeah, and because of typedefs A, B, C, and D it turns out 95% of calls within the system look like pass-by-value but are in fact pass-by-reference. It took me ~2 months to realize that was going on. 4 months later and it still gives me the heebie jeebies.
  

loving/hating Fit

We waste a huge amount of time on our legacy code base because we have no Unit tests, and no standard way the project's developers test their changes.

It's typically some error-prone manual process -- run some SQL in Toad, copy results to Excel, run the bill, re-run SQL, compare to the old results with eyeballs, pray.

I read about Fit a few months ago, and it seemed like it'd solve a lot of our problems. Blackbox testing would help build tests up around our legacy codebase. It'd also help bridge the communication gaps between onshore/offshore development, and development/business.

Finally took the time to play with it this week -- specifically, DbFit. Without any customization, I was able to write some simple SQL in a wiki page to generate a new test case for the issue I was working on this week.

I'm in love. It seems like with some minor customization we can make some huge leaps in our productivity. But I really hate trying to navigate the Fit/Fitnesse web sites for reference information. Ugh.

additional offshoring adventures

My project at work is a set of Java applications for which communicate over UDP / RMI. The port numbers they talk over is configurable. In our hosted environment our servers may be running the apps for multiple clients. So, the users (or our tech support people) have to configure unique port #'s in each set of config files if there are multiple s running against different databases. It's a annoying, but the least of our worries at the moment.

Monday this week, I realize our configuration editor doesn't give the user the ability to edit a certain set of port #s. I ask the offshore team to add that ability to the config UI.

When I come in the next day the offshore lead has decided the existing way we do things is too error prone. He's made a great change so that the configurable settings are just a range of port #'s and the application(s) will pick unused ports within the range.

After reviewing the changes, the only issue I found was in some output files created by the apps once they pick a port number. Once the port # is selected, an output file will be written. It was implemented with a DataOutputStream to write an int primitive to the file, and read back in with DataInputStream. I'd prefer it to be human readable, so I respond with some notes saying to make the file a text file rather than a data file.

I come in the following day, and find that the only thing he's changed is to give the file a ".txt" file extension.

Wah?

It's the 'little' things like this that drive me crazy. Things I assume don't need much specification. It's particularly annoying because this engineer is so awesome -- he's always quick to turn around any solution, and always asks great questions on our calls. And, he did a great job of taking the initiative to fix a major flaw just a day earlier.

further adventures in offshoring

Have not had time to do a code review for the project I've inherited. Decided to use FindBugs to pick off low hanging fruit.

aiiiiiiiieeeee.

• bitwise '&' operator used instead of boolean '&&' operator
• '==' operator used for Long/Integer/many-other-reference-types
• String.equals() comparing a String to a DO object
• Enum classes with their own extra-special equals() methods. Ick.
• Classes with equals() but no hashCode()
  
• Who implements finalize()?
• many, many more
  

what can make me watch an infomercial?

MR T

an afternoon I'll never get back, thanks Eclipse!

Spent Friday afternoon fighting with Eclipse. Around lunchtime, my development environment stopped working... When launched from Eclipse the app couldn't find the log4j, Spring, or Hibernate config files. After many false starts and much gnashing of teeth, finally noticed it wasn't all config files that weren't getting copied into the projects' output folders but only .xml files.

Finally managed the right phrase in Google that found me the answer: http://dev.eclipse.org/newslists/news.eclipse.platform/msg74562.html

Apparently, Eclipse plugins may update one of the preferences that tells the builders which file extensions it shouldn't copy to the build output path. To fix it, you have to go to 'Preferences... -> Java -> Compiler -> Building' and then check all the extensions entered into the Filtered Resources text field.

I'm not sure if it was a plugin change, or (probably more likely) changes I was making to my launch configurations to not run out of the 'dist' directory into which the inter-project Ant build packages everything.

I'm not sure if I should be happy that the offshore team I've now inherited at least got the Ant build right. Or, if I should be concerned about whether they've actually been running the code / configuration they think they are when running under Eclipse.

One more in a long line of complaints I've had regarding how this set of applications is configured. Normally, I'm the one on whatever-team-I'm-on pushing for extensive runtime configurability. I don't know why this project's configuration setup -- not the Spring/Hibernate, the home-grown .properties file reading and command-line handling stuff -- rubs me the wrong way. The nonsensical or nonexistent documentation, the obfuscated search paths, the unhelpful error messages, or the fragility of the whole system when you're missing one piece... all of the above?


Even with that lost afternoon, I still love Eclipse. After all, it's free. Oooh, free. One plugin that now I can't live without: Remote System Explorer.

I've got to run Windows at work, and the corporation-approved SSH client sucks balls. When you resize a terminal window... what would you expect to happen? More columns/rows? Bah! That's too obvious and old-school for Attachmate Reflection's SSH client-- instead the mother f*&#$*#&er resizes the font. Badly.

PuTTY is mediocre. The corporation's security agent can't be disabled, and it prevents the cygwin install from running completely.

Enter Remote System Explorer. Multiple terminal windows. Does the 'right' thing when resizing or scrolling. Gives me a file-tree view of the remote system via SFTP. I can edit remote files within Eclipse and it automagically saves them to the remote system.

Eclipse is a beast, and a bit of overkill for a shell window. But, I've already got it open. And, unlike my Ubuntu system running under VMWare... I don't feel like the corporation's jackbooted thugs will re-educate me if/when they find I'm using it.

conclusions

• If you write an application where the order of command-line options matter, you're a bad person.
• I need to use the adjective 'Diabolical' more often.
  

so many resignations, none of them mine.

Two friends at current employer resigned. Unfortunately, they were both awesome engineers, so the future looks bleak. The only silver lining is there will be less competition for the 'good' projects. I was looking forward to working with them and learning from them.

One friend at old employer resigned. Finally. Sucks for those that are left behind, but it's a great move for him.

As for me, I'm still trying to decide how long to stick it out at the current employer. Finally feel like I'm getting the hang of my current project. We're continuing to refine our processes, and we've got two new engineers that seem to be very good. Now that I've settled into the rut, I've been knocked out of it to take over as tech lead for one of the people that's leaving. More responsibility, higher profile, more interesting work... I should want that right? And not the cozy rut?

In less wishy-washy-I-am-lame news, things I'm enjoying:

• Dexter
• Dr. Horrible
  
• 'A Fire Upon the Deep' by Vernor Vinge. I couldn't put it down. Read until 4am night before last, and 2am last night. Very good.

I bought Toll of the Hounds a month ago (delivered right after the return from vacation). Decided I'd re-read the Malazan series from the beginning before starting on the new book. Again. I'm amazed that it's holding together on the third read through. Picking up nuances and hints I'd completely missed before. And this is the third read through. I've made it through book four, and decided to read 'A Fire Upon the Deep' as a palate cleanser.

another sign of the apocalypse

I was struck speechless while perusing the beer at the supermarket yesterday.

Bud & Clamato == Chelada?

It's not quite intriguing enough to want to try myself. But, inflicted upon someone else... Perhaps a six pack as a house warming gift for my enemies

don't eat the chicken pizza

Back from a week in Cancun! Luckily, we only got horribly sick on the last day. Just in time to fly home. Awesome.

We stayed at the Dreams Tulum Resort. For an all-inclusive, it was reasonably priced, and very nice. The staff was awesome. The food at the resort's restaurants was all very good.

The majority of the week was spent listening to downloads of the Geek Show Podcast, sipping mojitos, and staring at the ocean.

Thursday, we took a tour to Chichen Itza. Oh lord... it was a long drive from Tulum. But, the ruins were worth it. Very interesting, it was great to see the scale of things. Unfortunately, climbing the steps is now verboten.

Included in the tour package was a lunch at the buffet at the Mayaland hotel right next to the ruins. It looked risky, but we were lulled into complacency by the luck we had with our resort's food. All four of us got sick that night and the next day.

On the way back from Chichen Itza, we stopped at an amazing cenote. We only had a half hour, so we decided not to take a dip.

further lessons in patience

I've been tracking down why a run of the rules engine on the Windows platform produced different results for a particular bill was different from Unix.

After debugging the problem, I tell the business peeps the same thing the other 2 developers have been saying for the past month:

• I can't reproduce that unexpected behavior on Windows or Unix, using my builds or the official release builds.
  
• Also like the other 2 devs, I tell them there's a bug in the code we need to fix. The unexpected behavior is actually the desired behavior and fixing the bug will result in the desired results all the time.

Through pure chance... I realize the order in which bills work their way through the system will affect the outcome of the regression test. If bill A is part of a claim X, and claim X has one or more bills that haven't yet been reprocessed at the time bill A is pulled out of the FIFO bill processing queue... when the rules engine queries the database for the bills that are part of claim X, the engine won't retrieve the yet-to-be-processed bills.

This fits a big symptom of our inconsistent results for this and other bills. We see an unexpected result, and we try to reproduce it after-the-fact... no luck.

I can finally reproduce the original problem, but only if I run the regression test prep scripts, and then manually process a handful of bills from the claim and leave some stuck in limbo.

I naively ask the person that runs the test suite, "I think I found the problem, but to confirm I need to look at the log files from the regression test run on Windows. Do we still have the logs?"

I'm told, "I don't know, I don't think there is one. We were told that on Windows the log files aren't produced consistently."

I look into it for 5 minutes. It turns out no one bothered to configure a log file in the INI file. So, no logs were produced.

I'm dumbstruck. It's the latest in a long string of stupendous tales the previous developers have told the business people and test team. Lots of mythology around why the system acts oddly ... memory leaks, bad casts and pointer references, 'Windows can't do the log files', 'Windows sorts differently', 'You need to enter the password in all caps... No, not with the shift key, you have to enter it with CAPS LOCK'

WTF?

To be fair... I am hearing all these anecdotes second-hand and filtered through the non-technical team. But, taken as a whole I can't decide whether it lessens my confidence in the BAs and dev management, in that they couldn't successfully call the engineers on their bullshit... or if it just confirms my opinion that the engineers were lazy and outright lied about how things worked because they didn't want to take the time to correct the problems.

triumph! ... oh, dammit

Friday was a bittersweet day.

On our horrible project-released-every-month, they are frequently late starting their regression test, and very often late with the release. In 2007, it was only released on-time twice.

As bad as that is, the good(?) thing is they don't release until their regression test says the only differences are the ones they expect. Unfortunately, that typically means a team of 12+ BA's, developers and testers work the weekend.

But not this month. We started the regression test when we were scheduled to on Monday. And, rather than posting the release on the drop-dead Monday date, we passed the release off to customer-facing site on Friday. This is the first time in years that that has happened.

The process-improvements Jeff and I have been insisting on deserve some of the credit.
Odd things. You know, completely out-of-left-field things like code-reviews and communicating. It's unclear how much of this month's success was the amount of work included in this release, and how much of it was our process changes.

Unfortunately, the day of triumph was made bittersweet by Jeff's resignation. Probably not accurate to call it a resignation, since he was a contractor and decided not to pursue extending his contract. Or, to be precise... not wait until the final day of his contract to find out if they were going to extend his contract.

I don't blame him for finding something else, and I may follow him soon enough. The monthly release meat-grinder can be made less painful... but it'll never be fun.

ready. set. go?

I'm coming to the realization that I was so focused on leaving the previous employer that I didn't think critically enough before accepting the offer at current employer. Beyond the many dead horses I've already beaten, there were plenty of hints to which I should have paid attention prior to accepting the offer.

More mixed feelings today after a meeting discussing our project's migration from usage under the current application framework to its replacement over the next 2-3 years. Felt good that our hard work to stabilize the crumbling infrastructure is recognized by the business people. Felt worse when I realized that 2+ years into a multi-million dollar project, they're finally discussing the actual mechanics and workflow of how both systems are going to be developed and maintained together. "Oh yeah... we want to release in October. We need to figure that out."

Nearly all the positives about the new job have been snuffed out. There is the potential for it to get better, but it's hard to hold on to the glimmer of hope. Other than feeling like I'm giving up, it's getting harder to find reasons to stay.

Friends at 4 different employers are looking for people... positions with varying levels of awesomeness. Or, I could strike out on my own and find something different.

But, what do I want? That's the million dollar question.

Someplace more engineer-y than my most recent work in content publishing or healthcare. Some of the work was great and challenging... other parts have been mind-numbing or downright eeeeeeeeviiiiiil. Not that the previous work in simulation was fluffy kittens.

Someplace that views R&D as a vital part of their business plan, rather than whining by engineers for 'fun' work. Maintenance can be fun -- debugging an issue, finding the cause of the crazy and esoteric problems is like unlocking a puzzle and can be very rewarding. But, it's not something that you look forward to doing as your sole activity for the next 3+ years.

Something more back-end rather than UI or web-applications. I'm pretty sure that new fangled interwebernet thing will never work. It's all TUBES! TUBES I TELLS YA!

It'd also be nice to land someplace where I could use Python for more than just throw-away utilities, and not be viewed as a rabble-rouser.

pink eye!

Liz's brother's family visited us for a couple days while they were in town for a funeral. Bart turned us on to one of our new favorite shows, Corner Gas.

Good silly fun. Needs more explosions.

waaaaaah?

I was feeling good this morning. Ran the regression suite overnight through Purify, and it had found some problems. Uninitialized memory references. Who would have thought that hundreds of 5-50 field structs in a 300 kloc code base with no standard initialization routines would have this problem. Bizarre.

So far I've resisted the urge to go fix all the potential instances of the problems, and only fixed the problems exposed by the regression suite. Maybe for next month's release.

Then I spent about 4 hours in conference calls. At about the third hour in I realize the crick in my neck that I'd blamed on sleeping oddly was more in fact from the previous day's conference calls. Jeff helpfully points out over IM, "Yes, this meeting is a pain in the neck."

Then I finally get a response from technical support on 2 of my tickets, after they'd spent ~3 weeks in a black hole.
"Due to ... blahblahlbha... we cannot install open source applications on any servers."

The offending applications? Zip. Unzip. GNU Make.

I can't decide to laugh or cry. The HP-UX server in question already has the GCC toolchain, GNU tar, ant, CVS, CruiseControl and the list goes on. Not to mention widespread use of Hibernate, Spring, ehcache, and tons of Apache and Jakarta projects throughout our Java and .Net products. WTF?

I've learned my lesson. Attempts at following procedure will only be made as a last resort. I need anti-action-item-Wonder-Woman-bracelets to deflect IT-related action items onto my coworkers.

Speaking of playing dress up...

I have to keep reminding myself I have better things to do than spend $250 on t-shirts.

further lessons in not believing anyone

More whining about work.

It's becoming increasingly apparent that the previous development staff of Project A were lazy. Definitely not stupid, only people confident in their brilliance could produce such a poorly documented and nightmarish code base.

Lazy in terms of, "Hey, that's acting weird... it seems hard to figure out too. I'll blame X and the business people will live with it."

The latest example is the supposedly 'inconsistent' regression test results. When Jeff and I joined the project, we were repeatedly warned in reverent tones of the odd non-deterministic regression test. The first time the 10k+ bills in the regression suite are run, some bills fail. If you run those bills through again, they work.

On the face of it, it seems very odd. The way it was presented to us it seemed that the failing bills were random, and on re-run the identical bill would produce different results. Various things were blamed: memory leaks, buffer overrun, bad casts.

As we dug into it, it became clear that wasn't the case:

• the bill processing engine wasn't re-started between the failure and success.
• for at least one large set of the failing bills, the same bills fail every time the regression test is run.
• The regression test's preparatory script actually modifies the bill in such a way that they become illegal bills
• the engine sees the illegal bill, and tweaks it such that works again
• but, many of the failing bills are only partially tweaked because they have manual override codes. The partial tweaking sets the bill up to fail the first time, but corrects things enough that it will work the second time through the system.

Definitely a bug in the engine. The data that's reset by the prep script shouldn't make it behave that badly.