an afternoon I'll never get back, thanks Eclipse!

Spent Friday afternoon fighting with Eclipse. Around lunchtime, my development environment stopped working... When launched from Eclipse the app couldn't find the log4j, Spring, or Hibernate config files. After many false starts and much gnashing of teeth, finally noticed it wasn't all config files that weren't getting copied into the projects' output folders but only .xml files.

Finally managed the right phrase in Google that found me the answer: http://dev.eclipse.org/newslists/news.eclipse.platform/msg74562.html

Apparently, Eclipse plugins may update one of the preferences that tells the builders which file extensions it shouldn't copy to the build output path. To fix it, you have to go to 'Preferences... -> Java -> Compiler -> Building' and then check all the extensions entered into the Filtered Resources text field.

I'm not sure if it was a plugin change, or (probably more likely) changes I was making to my launch configurations to not run out of the 'dist' directory into which the inter-project Ant build packages everything.

I'm not sure if I should be happy that the offshore team I've now inherited at least got the Ant build right. Or, if I should be concerned about whether they've actually been running the code / configuration they think they are when running under Eclipse.

One more in a long line of complaints I've had regarding how this set of applications is configured. Normally, I'm the one on whatever-team-I'm-on pushing for extensive runtime configurability. I don't know why this project's configuration setup -- not the Spring/Hibernate, the home-grown .properties file reading and command-line handling stuff -- rubs me the wrong way. The nonsensical or nonexistent documentation, the obfuscated search paths, the unhelpful error messages, or the fragility of the whole system when you're missing one piece... all of the above?


Even with that lost afternoon, I still love Eclipse. After all, it's free. Oooh, free. One plugin that now I can't live without: Remote System Explorer.

I've got to run Windows at work, and the corporation-approved SSH client sucks balls. When you resize a terminal window... what would you expect to happen? More columns/rows? Bah! That's too obvious and old-school for Attachmate Reflection's SSH client-- instead the mother f*&#$*#&er resizes the font. Badly.

PuTTY is mediocre. The corporation's security agent can't be disabled, and it prevents the cygwin install from running completely.

Enter Remote System Explorer. Multiple terminal windows. Does the 'right' thing when resizing or scrolling. Gives me a file-tree view of the remote system via SFTP. I can edit remote files within Eclipse and it automagically saves them to the remote system.

Eclipse is a beast, and a bit of overkill for a shell window. But, I've already got it open. And, unlike my Ubuntu system running under VMWare... I don't feel like the corporation's jackbooted thugs will re-educate me if/when they find I'm using it.

conclusions

• If you write an application where the order of command-line options matter, you're a bad person.
• I need to use the adjective 'Diabolical' more often.
  

so many resignations, none of them mine.

Two friends at current employer resigned. Unfortunately, they were both awesome engineers, so the future looks bleak. The only silver lining is there will be less competition for the 'good' projects. I was looking forward to working with them and learning from them.

One friend at old employer resigned. Finally. Sucks for those that are left behind, but it's a great move for him.

As for me, I'm still trying to decide how long to stick it out at the current employer. Finally feel like I'm getting the hang of my current project. We're continuing to refine our processes, and we've got two new engineers that seem to be very good. Now that I've settled into the rut, I've been knocked out of it to take over as tech lead for one of the people that's leaving. More responsibility, higher profile, more interesting work... I should want that right? And not the cozy rut?

In less wishy-washy-I-am-lame news, things I'm enjoying:

• Dexter
• Dr. Horrible
  
• 'A Fire Upon the Deep' by Vernor Vinge. I couldn't put it down. Read until 4am night before last, and 2am last night. Very good.

I bought Toll of the Hounds a month ago (delivered right after the return from vacation). Decided I'd re-read the Malazan series from the beginning before starting on the new book. Again. I'm amazed that it's holding together on the third read through. Picking up nuances and hints I'd completely missed before. And this is the third read through. I've made it through book four, and decided to read 'A Fire Upon the Deep' as a palate cleanser.

another sign of the apocalypse

I was struck speechless while perusing the beer at the supermarket yesterday.

Bud & Clamato == Chelada?

It's not quite intriguing enough to want to try myself. But, inflicted upon someone else... Perhaps a six pack as a house warming gift for my enemies

don't eat the chicken pizza

Back from a week in Cancun! Luckily, we only got horribly sick on the last day. Just in time to fly home. Awesome.

We stayed at the Dreams Tulum Resort. For an all-inclusive, it was reasonably priced, and very nice. The staff was awesome. The food at the resort's restaurants was all very good.

The majority of the week was spent listening to downloads of the Geek Show Podcast, sipping mojitos, and staring at the ocean.

Thursday, we took a tour to Chichen Itza. Oh lord... it was a long drive from Tulum. But, the ruins were worth it. Very interesting, it was great to see the scale of things. Unfortunately, climbing the steps is now verboten.

Included in the tour package was a lunch at the buffet at the Mayaland hotel right next to the ruins. It looked risky, but we were lulled into complacency by the luck we had with our resort's food. All four of us got sick that night and the next day.

On the way back from Chichen Itza, we stopped at an amazing cenote. We only had a half hour, so we decided not to take a dip.

further lessons in patience

I've been tracking down why a run of the rules engine on the Windows platform produced different results for a particular bill was different from Unix.

After debugging the problem, I tell the business peeps the same thing the other 2 developers have been saying for the past month:

• I can't reproduce that unexpected behavior on Windows or Unix, using my builds or the official release builds.
  
• Also like the other 2 devs, I tell them there's a bug in the code we need to fix. The unexpected behavior is actually the desired behavior and fixing the bug will result in the desired results all the time.

Through pure chance... I realize the order in which bills work their way through the system will affect the outcome of the regression test. If bill A is part of a claim X, and claim X has one or more bills that haven't yet been reprocessed at the time bill A is pulled out of the FIFO bill processing queue... when the rules engine queries the database for the bills that are part of claim X, the engine won't retrieve the yet-to-be-processed bills.

This fits a big symptom of our inconsistent results for this and other bills. We see an unexpected result, and we try to reproduce it after-the-fact... no luck.

I can finally reproduce the original problem, but only if I run the regression test prep scripts, and then manually process a handful of bills from the claim and leave some stuck in limbo.

I naively ask the person that runs the test suite, "I think I found the problem, but to confirm I need to look at the log files from the regression test run on Windows. Do we still have the logs?"

I'm told, "I don't know, I don't think there is one. We were told that on Windows the log files aren't produced consistently."

I look into it for 5 minutes. It turns out no one bothered to configure a log file in the INI file. So, no logs were produced.

I'm dumbstruck. It's the latest in a long string of stupendous tales the previous developers have told the business people and test team. Lots of mythology around why the system acts oddly ... memory leaks, bad casts and pointer references, 'Windows can't do the log files', 'Windows sorts differently', 'You need to enter the password in all caps... No, not with the shift key, you have to enter it with CAPS LOCK'

WTF?

To be fair... I am hearing all these anecdotes second-hand and filtered through the non-technical team. But, taken as a whole I can't decide whether it lessens my confidence in the BAs and dev management, in that they couldn't successfully call the engineers on their bullshit... or if it just confirms my opinion that the engineers were lazy and outright lied about how things worked because they didn't want to take the time to correct the problems.

triumph! ... oh, dammit

Friday was a bittersweet day.

On our horrible project-released-every-month, they are frequently late starting their regression test, and very often late with the release. In 2007, it was only released on-time twice.

As bad as that is, the good(?) thing is they don't release until their regression test says the only differences are the ones they expect. Unfortunately, that typically means a team of 12+ BA's, developers and testers work the weekend.

But not this month. We started the regression test when we were scheduled to on Monday. And, rather than posting the release on the drop-dead Monday date, we passed the release off to customer-facing site on Friday. This is the first time in years that that has happened.

The process-improvements Jeff and I have been insisting on deserve some of the credit.
Odd things. You know, completely out-of-left-field things like code-reviews and communicating. It's unclear how much of this month's success was the amount of work included in this release, and how much of it was our process changes.

Unfortunately, the day of triumph was made bittersweet by Jeff's resignation. Probably not accurate to call it a resignation, since he was a contractor and decided not to pursue extending his contract. Or, to be precise... not wait until the final day of his contract to find out if they were going to extend his contract.

I don't blame him for finding something else, and I may follow him soon enough. The monthly release meat-grinder can be made less painful... but it'll never be fun.

ready. set. go?

I'm coming to the realization that I was so focused on leaving the previous employer that I didn't think critically enough before accepting the offer at current employer. Beyond the many dead horses I've already beaten, there were plenty of hints to which I should have paid attention prior to accepting the offer.

More mixed feelings today after a meeting discussing our project's migration from usage under the current application framework to its replacement over the next 2-3 years. Felt good that our hard work to stabilize the crumbling infrastructure is recognized by the business people. Felt worse when I realized that 2+ years into a multi-million dollar project, they're finally discussing the actual mechanics and workflow of how both systems are going to be developed and maintained together. "Oh yeah... we want to release in October. We need to figure that out."

Nearly all the positives about the new job have been snuffed out. There is the potential for it to get better, but it's hard to hold on to the glimmer of hope. Other than feeling like I'm giving up, it's getting harder to find reasons to stay.

Friends at 4 different employers are looking for people... positions with varying levels of awesomeness. Or, I could strike out on my own and find something different.

But, what do I want? That's the million dollar question.

Someplace more engineer-y than my most recent work in content publishing or healthcare. Some of the work was great and challenging... other parts have been mind-numbing or downright eeeeeeeeviiiiiil. Not that the previous work in simulation was fluffy kittens.

Someplace that views R&D as a vital part of their business plan, rather than whining by engineers for 'fun' work. Maintenance can be fun -- debugging an issue, finding the cause of the crazy and esoteric problems is like unlocking a puzzle and can be very rewarding. But, it's not something that you look forward to doing as your sole activity for the next 3+ years.

Something more back-end rather than UI or web-applications. I'm pretty sure that new fangled interwebernet thing will never work. It's all TUBES! TUBES I TELLS YA!

It'd also be nice to land someplace where I could use Python for more than just throw-away utilities, and not be viewed as a rabble-rouser.

pink eye!

Liz's brother's family visited us for a couple days while they were in town for a funeral. Bart turned us on to one of our new favorite shows, Corner Gas.

Good silly fun. Needs more explosions.

waaaaaah?

I was feeling good this morning. Ran the regression suite overnight through Purify, and it had found some problems. Uninitialized memory references. Who would have thought that hundreds of 5-50 field structs in a 300 kloc code base with no standard initialization routines would have this problem. Bizarre.

So far I've resisted the urge to go fix all the potential instances of the problems, and only fixed the problems exposed by the regression suite. Maybe for next month's release.

Then I spent about 4 hours in conference calls. At about the third hour in I realize the crick in my neck that I'd blamed on sleeping oddly was more in fact from the previous day's conference calls. Jeff helpfully points out over IM, "Yes, this meeting is a pain in the neck."

Then I finally get a response from technical support on 2 of my tickets, after they'd spent ~3 weeks in a black hole.
"Due to ... blahblahlbha... we cannot install open source applications on any servers."

The offending applications? Zip. Unzip. GNU Make.

I can't decide to laugh or cry. The HP-UX server in question already has the GCC toolchain, GNU tar, ant, CVS, CruiseControl and the list goes on. Not to mention widespread use of Hibernate, Spring, ehcache, and tons of Apache and Jakarta projects throughout our Java and .Net products. WTF?

I've learned my lesson. Attempts at following procedure will only be made as a last resort. I need anti-action-item-Wonder-Woman-bracelets to deflect IT-related action items onto my coworkers.

Speaking of playing dress up...

I have to keep reminding myself I have better things to do than spend $250 on t-shirts.

further lessons in not believing anyone

More whining about work.

It's becoming increasingly apparent that the previous development staff of Project A were lazy. Definitely not stupid, only people confident in their brilliance could produce such a poorly documented and nightmarish code base.

Lazy in terms of, "Hey, that's acting weird... it seems hard to figure out too. I'll blame X and the business people will live with it."

The latest example is the supposedly 'inconsistent' regression test results. When Jeff and I joined the project, we were repeatedly warned in reverent tones of the odd non-deterministic regression test. The first time the 10k+ bills in the regression suite are run, some bills fail. If you run those bills through again, they work.

On the face of it, it seems very odd. The way it was presented to us it seemed that the failing bills were random, and on re-run the identical bill would produce different results. Various things were blamed: memory leaks, buffer overrun, bad casts.

As we dug into it, it became clear that wasn't the case:

• the bill processing engine wasn't re-started between the failure and success.
• for at least one large set of the failing bills, the same bills fail every time the regression test is run.
• The regression test's preparatory script actually modifies the bill in such a way that they become illegal bills
• the engine sees the illegal bill, and tweaks it such that works again
• but, many of the failing bills are only partially tweaked because they have manual override codes. The partial tweaking sets the bill up to fail the first time, but corrects things enough that it will work the second time through the system.

Definitely a bug in the engine. The data that's reset by the prep script shouldn't make it behave that badly.

python + elixir + pyyaml == yay!

And so... I'm playing with a little toy script to ease some pain at work. We've got a bunch of bills in a very large database, the bill data our application reads/modifies is split across a number of tables.

At the moment, the developers manually run SQL scripts to peek at various bits of data. Definitely a huge waste of time.

So, I'm playing with Python + Elixir + PyYAML to get a script that I can just pass in the bill ID and it'll query the 5-6 tables and serialize the bill to something more human readable. If I play my cards right, I'll never have to deal with Toad again.

Elixir works its magic and I can query my database. The autoload didn't work, and there are a couple tables with 50-100+ columns. Serializing it to YAML without repeating myself is now the trick.

I'm sure there's a better way to do this, but here's what I came up with... applied to the Elixir tutorial. Not the most exciting thing ever. But, it's the start of building a better testing framework. Doing the same for the real tables will make it much easier to compare a bill before and after running through our application.

#!/usr/bin/python

# -*- coding: latin-1 -*-



from elixir import *



from yaml import load, dump

try:

    from yaml import CLoader as Loader

    from yaml import CDumper as Dumper

except ImportError:

    from yaml import Loader, Dumper





def _toYamlRep(ent):

    """

    Given an elixir entity, query the entity's members via its __dict__ 

    and return a dict

    """

    ret = {}

    for (k,v) in ent.__dict__.items():

        if k.startswith('_') or k == 'row_type':

            # don't print out the 'hidden' keys

            continue

        if v:

            ret[k] = str(v)

    return ret



class YamlEntity(Entity):

    def toYamlRep(self):

        """

        Wrap the _toYamlRep() dict in another dict, use the class' name as the header.

        """

        return {self.__class__.__name__ : _toYamlRep(self) }



class Movie(YamlEntity):

    title = Field(String(30))

    year = Field(Integer())

    description = Field(Text())

    director = ManyToOne('Director')



    def __repr__(self):

        return '<Movie: "%s" (%d)>'%(self.title,self.year)





class Director(YamlEntity):

    name = Field(String(60))

    movies = OneToMany('Movie')



    def __repr__(self):

        return '<Director: "%s">'%(self.name)





def main():

    metadata.bind = "sqlite://"

    setup_all()

    create_all()



    rscott = Director(name="Ridley Scott")

    glucas = Director(name="George Lucas")

    alien = Movie(title="Alien", year=1979, director=rscott)

    swars = Movie(title="Star Wars", year=1977, director=glucas)

    brunner = Movie(title="Blade Runner", year=1982, director=rscott)

    

    session.flush()



    for m in Movie.query().all():

        print dump(m.toYamlRep(),Dumper=Dumper,default_flow_style=False)



    cleanup_all()



if __name__ == '__main__':

    main()

And, here's the output:

Movie:
  director: '<Director: "George Lucas">'
  director_id: '2'
  id: '3'
  title: Star Wars
  year: '1977'

Movie:
  director: '<Director: "Ridley Scott">'
  director_id: '1'
  id: '4'
  title: Alien
  year: '1979'

Movie:
  director: '<Director: "Ridley Scott">'
  director_id: '1'
  id: '5'
  title: Blade Runner
  year: '1982'

cruisecontrol == teh suck

Every time I try to dive into configuring CruiseControl, I hate it a little bit more.

can it be? finally done? ... WTF?

And now, the servers have been installed (all except for the HP-UX I may give up on).

after the following loop through the tech support desk:

1. go to system A
2. go to system B
3. go to system C
4. go to system D
5. go to system F
6. no, go to system C. enter text 'XYZ A'
7. FAIL
8. no, go to system C. enter text 'XYZ B'
9. SUCCESS

The admin rights have been granted.

It seems obvious now, but if you're not given a definitive answer when you ask "Doesn't Umbrella Corp already have a pool of licenses we can pull from?", keep asking.

Turns out, we had the licenses all along. SWEET ZOMBIE JESUS, that was good times.

progress?

When last we saw "Steve's lame-assed adventures in IT", I was relating my struggles with our IT department. This morning's update:

• I reply, "I don't know what group name you're asking for? How do I find that information?"
• Tech-support person #6 replies, "Global group used by your team to access servers."
• I reply, "I have a 'global group to access servers'? I log in using my XYZ domain account. I don't know what group you're asking for. Is the global group one of the access groups listed in service request system F?"
• Tech-support person #6 replies, "Yes, global group is a access group."

I don't know if this is progress or not.

Further complicating matters... the process improvements Jeff and I have suggested appear to have helped us meet this month's release. The group's communication is improving, the code reviews caught some subtle problems, and it seems like meeting the deadline may not only be because of the handful of items they dropped off the release plan. And, Jeff and I are spitballing some ideas to for a unit testing framework so the developers can test their modifications without manually running SQL statements to verify the result of their changes.

It seems like we could be successful. If only we're not ground up by the monthly release chaos.

But, is it worth it? I'd hate to run away after only 2 months, but it gets very discouraging. There are at least two friends who have jobs to fill, one unknown but potentially awesome and another known and good-to-awesome.

I don't feel like I'm growing technically. But, I'm definitely growing in terms of dealing with projects, customers, and other developers onshore-and-offshore. Maybe the pain is worth it. On the plus side, Jeff and I have the complete support of the business-side to make the changes we're planning. They're suprisingly enthusiastic.

are you there Mordac? it's me steve

Because the purpose of my life is to serve as a warning for others, please note the following:

1. Don't spend the first 3 weeks after your initial "I need software X" request, lackadaisically monitoring the request's progress every 2-3 days. Expect to spend 1-3 hours per day following up on e-mails and on the phone.
2. Don't bother trying to navigate the 'helpful' self-service website, or the dozen+ service request web apps. Go directly to the Help Line. While one-on-one interaction with actual people is scary, at least there will be someone to hear your plaintive cries.
3. If tech-support person #1 helpfully forwards your e-mail to person #2 for clarification, don't assume that person #1 would take note of person #2's out-of-office reply that says they'll be out for the rest of the month.
4. Don't assume #3 wouldn't happen twice. The same day.
   
5. Assume every person 'helping' you is actually actively preventing you from reaching your
   goal through a clever misinformation campaign meant to drive you slowly insane.

As the owner of the action item, it does make me feel slightly better to be an obstruction to the people waiting on me to get this done. Their pitiful cries bring delight to my cruel, cruel heart.

Last week's status meeting:

Person Z asks, "Steve, any progress on ?"

I reply, "I've got it installed on the two wintel servers I had admin rights to. I can't find anyone to give me a straight answer on who owns the other two wintel servers, or the 'right' process for requesting admin access. I've entered a Unix service request on the fifth server, I assume someone may see it eventually."

The assorted people in the room nod sympathetically, apparently my struggles are par for the course.

Project Manager F asks, "Is there anything we can do to help?"

"Pray."


Mordac's latest trick, masquerading as other IT service members:

• As persona #1, point me to intranet URL #1 telling me to use service request system A to apply for admin rights.
• I submit request, push the approval through my management chain.
  
• As persona #2, reject request. "System A is only to request admin access to your desktops. Not servers. You need to follow the set of instructions at this URL."
• URL #2 describes in the vaguest way possible how to gain access to various intranet web applications through service request system B
  
• I reply, "WAH? Person #1 told me this, and its clearly documented at URL #1. If that's wrong, it needs to be updated. I follow your instructions, but I don't see which access group I need to belong to gain access rights, can you clarify?"
• Persona #2 forwards it along to persona #3.
  
• Persona #3 says "You need to follow the instructions at URL #3, and use service request system C."
• System C looks like its asking for the right information. I submit the request.
  
• The request is rejected by Persona #4, "You can't do that for your specific user id. You have to enter your team's group name."
• This is the first I've heard that my team has a group name.
  
• No one on my team is aware we have a group name, or what it may be. Not even those who already have admin on those servers.
  
• No reply from Persona #4 yet.
  

project A makes baby jesus cry

And so... for some reason I'd gotten it into my head to volunteer for Project A at work. Project A is a rules engine for processing ... things... Myself and another fellow newbie on the project (Hi Jeff!) are slowly coming up to speed on the crazy.

How crazy? Let me count the ways...

• Release every month!
• Without a project manager, development manager, or release manager. Who needs 'em? Crybabies.
  
• We can't control the # or scope of change requests coming in each month. State regulations drive all the changes to the system. Some nebulous decision process allows us to occasionally drop some changes from the queue... but we don't know which until the deadline has been blown.
  
• Crumbling 12+ year old code base. Not given love or documentation over the years. Occasionally tossed a bucket of fish heads. You can tell which dark corners survived from the halcyon days -- they use the hacked-in exception handling in what appears to be the correct way. While I appreciate the cleverness of adding exception handling to C (preprocessing macros around some setjump/longjump magic), I'd appreciate it more if anyone had bothered to write a brief document describing said 'right' way. An example in the .h file? Pshaw! That's for suckers! In the 3 weeks I've spent diving through the code to correct memory leaks, there's at least 4-5 different exception handling idioms. Many possibly broken sections of code, swallowing exceptions with no comment to explain whether the author did accidentally or on purpose. If there aren't comments, it's usually right... for certain values of right. If there are comments, the author clearly thinks exceptions either work like Java's or has no clue.
• Functions are named as ambiguously as possible with regard to whether it returns a reference to a list that shouldn't be modified, or a copy of the list that's safe to modify and needs to be cleaned up by the caller.
  
• It doesn't matter that developers create the builds pushed to customers, right?
  
• Oh, and developers have been doing the source control labeling too.
• Oh, and they're doing both of those tasks half-assed
  
• The developers have been too lazy or too scared of the HP-UX makefile to allow it to work without manually copying all source files from their CVS checkout directory to the application directory.
  
• You know what'd be a great idea? Basing the fancy new .Net-based Product B's rules engine on Project A. Wait... let's add a Java JNI wrapper around its re-packaged Project A. Awww, yeah... now that's good and f*^@#$ed up.
  
• Cherry on top: some genius decides to branch to support Project B. "Branches are neat! Wait... branches are hard... oh well, we'll just branch this one directory that has the DAO stuff, that code definitely has to be different between Project A and B. That's what branching is for, right?" The branched directory also contains nearly identical files that will be slowly, and not-so-slowly , diverging to meet each project's mostly identical monthly release requirements. What? Labels on the branch/trunk to indicate merged code? Nah, that'd be too helpful. I should consider myself lucky that the branch has only existed for a year.
• Oh yeah, we're releasing Project B mid-Summer. Sweet!

Somehow, I'm the calming influence on Jeff. Probably up to the point where he reads this rant. C'mon, we're in it together! You know, like Musketeers! But less flouncy.

I'm pretty sure I'm living in a sitcom written by The Daily WTF. The alternative is unthinkable.

This is Mark's cue to waltz through with a comment where he doesn't say a thing. Which I appreciate.

more ponytails than should be allowed by law

Decided to go to the April UJUG meeting a couple nights ago. The presenters varied a little in quality, but were average to great. The topic was interesting, I would have enjoyed more "What went horribly wrong" war stories.

I thought it was telling that even though Maven was held up as a savior for many of the projects, everyone who spoke up mentioned their love/hate relationship with it. It sounded like the LDS church's team had embraced it the most, and a big part of their success with it appears to be because they've dedicated 2-3 engineers to supporting it full time within their organization.

Turnout was huge, 100+. I think I prefer the smaller, and less formal, UPyUG meetings. And it's not just because I think software developers with long hair look silly. So many ponytails at the UJUG meeting. And at least one beret.

Spring, best thing since sliced bread?

I've been reading a lot about the Spring Framework lately. It's been aggravating me... realizing how much easier it could have made the last 2 years of my life, if I'd dug into it before starting on my last big project.

I'm enjoying Spring in Action. Only a few chapters into it. So far a good introductory book. I like the light tone of the examples, but the jokes are becoming grating.

I also picked up Expert One-on-One J2EE Development without EJB. I'm pretty sure it wins for the creepiest cover on a technical book ever. Kids, if he pulls up to your playground in a ice cream truck, RUN AWAY!

what? just because we don't have a customer we have to stop working?

The job switch has been... odd. Spent the first week and a half scrambling, reading non-stop to catch up with the rest of the team on the Java frameworks we'll be using.

Spent the second week and a half wondering what I'll be working on. The original project was put 'on hold' due to lack of business funding. We'll be shuffled off onto different projects for the time being, likely never to return to the original project.

It's good times.

chastened by mother nature for act of hubris

After enjoying spring-ish weather, walking around the new neighborhood, and doing some cleanup in the yard... we decided to purchase some patio furniture for the back deck.

I feel I must apologize to everyone in the valley for resulting snow fall. We had at least 2-3 inches at our house.

adrift

Not sure why I feel more adrift now that I've not only made the decision to leave my current employer, I've done a bunch of job hunting, interviewed and accepted an offer.

I thought once I'd made my decision, things would have solidified in my mind. Maybe I'm realizing how much I'll miss my co-workers. Nothing like them being laid off, and conspiring with them against 'the man,' to bring you closer together. Their last day is Friday the 29th. My last day is the following Friday, and I'll start the new job the next Monday.

It was difficult to choose where to go. I could go someplace unknown, or I could go back to the ex-employer, or I could follow some friends that left current-employer last year. Doors #2 and #3 seem fairly recession proof; military work for the former, health for the latter.

All the reasons I had for picking door #2 were comfort-based -- very close to the new house, great friends, and interesting work. Door #3 is more of an unknown and a ton more responsibility, but with a safety net of other good friends. Ultimately picked door #3, the riskier job that could be huge career-wise.

eclipse + cdt + cygwin = hours of misery! hooray!

Holy dammit Christmas, every step along the way to debug a simple hello world app is an exercise in misery.

• it appears to have found make/gcc/etc... but then won't actually run anything. Launching eclipse via a batch script that adds c:\cygwin\bin to the path before starting eclipse fixes this. It would have been nice if eclipse threw up more red flags about not finding the things it needed.
  
• I get everything building, launch the debugger and it complains that it can't find the source file. This was answered by the FAQ... once I found the FAQ.
  

I hate that I didn't think to look for the FAQ as the first step. I hate that the top google search results were all 3 or more years out of date. I hate Eclipse's web site -- it's as bad as my employer's intranet site.

demotivation

After a Holiday season filled with indecision over staying at my employer, I got motivated again last week. I decided that there was interesting work to be done, and that I really like everyone I'm working with.

Then, the bombshell Wednesday. Lay offs. Because while we're profitable, we're not growing fast enough for our parent company. Why not outsource all our content production? All our competitors are doing it, so it must work! Quick, outsource everything in a month and a half!

Out of my group of 10 (including my manager), I'm one of 4 software developers left. The content production team was hit much worse, only 2 left out of over 20.

Now we're looking at an avalanche of legacy code maintenance work for at least the next few months as we train outsourced software developers to maintain our convoluted production system that has grown into a horrible mess over the last 10-15 years. There are a ton of services and applications that interact in odd and poorly documented ways. As if that wasn't bad enough, anything over 5 years old frequently isn't in source control.

We have an experienced set of users who know how to push the data through the tangle of applications. We'll lose that in a little over a month.

Fun times ahead.

house chaos, kitchen chaos, more lame book reviews

We've slowly been settling into the house. Hampered in large part by our decision to remodel the kitchen. So far, so good. It has stayed on schedule, and is turning out well.

A little over one more week of no kitchen sink or appliances to go, then we'll be able to stop living off of the kindness of friends and family, the mini-fridge and microwave, and waaaaaaay too many take-away and fast food dinners.

Further lame book reviews accumulated during the move and holidays. To further distill my always lame, and always blessedly brief reviews, they will be presented in list form.

First the Good:

• Camouflage, by Joe Haldeman. Interesting alien(s).
  
• They Call me Naughty Lola, edited by David Rose. A collection of personals from the London Review of Books. Bizarre and very funny.
• Do Androids Dream of Electric Sheep (aka Bladerunner), Philip K. Dick.
  
• Also decided to pick up some science fiction classics I had never read. All pretty fun. A Deepness in the Sky, by Vernor Vinge. Also, The Mote in God's Eye, and The Gripping Hand, by Larry Niven and Jerry Pournelle.
  

Next, the mediocre:

• Storm Front, by Jim Butcher
• The Summer Tree, The Wandering Fire, and The Darkest Road -- by Guy Gavriel Kay. Re-read these. I remember picking them up years ago because I'd heard such good reviews. I remembered not liking them much. I thought I'd give them another go since my tastes have changed quite a bit since then. Still found them very annoying. Stilted dialogue, and the whole college-kids transported into alternate medieval reality thing kept making me think Ren Faire.

the thank you that keeps on thanking

The best way to thank your friends and family for helping you move?

Food poisoning.

Including myself and the lovely wife, nine of the dozen or so people who helped us move were struck down by the Subway sandwich platter. I don't think I've ever been so sick. About 24 hours of sick, sick, sickness followed by a week of not eating solid food.

MarkMail, sweet!

Last May, at the Mark Logic User Conference, Jason Hunter showed off MarkMail. It's finally available to the public at http://markmail.org/

Various links about it:

• An overview at Mark Logic's site
  
• A message from Mark Logic's CEO
  
• An article at O'Reilly OnJava
  

One part that wowed me is the ability to view attachments within the interface. Maybe I'm easily impressed. For an example, go here for email with a PDF attached. Click on the link for the attachment. Search for other attachments with a search string like 'extension:ppt' or 'extension:doc'

who knew?

Finally added labels to the blog posts. Who'd have guessed? Tied for the most frequent label:

• computers
• i am lame

so close, yet so far

The appraisal is complete and our buyers haven't complained so it must have covered the sale price. Our buyers decided to do the inspection themselves, which we found odd. It's a fairly new home, and we are paying for a home warranty, so maybe they don't think much could be wrong. But, we're baffled that they wouldn't spend a couple hundred to get a professional to look over the house to verify there aren't any non-obvious problems.

The sellers on our new home were happy to extend the closing date on our contract for another couple weeks, so everything is looking good for the closing and move in the next 2 weeks.

Our buyers came by on Sunday to do the inspection, said they needed 3 hours. We decided to head up to the new house, visit our friends across the street, and rake leaves. I'd assumed it would be about an hour's job.

15 orange bags, 6 blisters, and 4 hours later, we were done.

Did I mention we've got 7 big trees on the new lot? 2 ash trees, 3 maples, and another 2... something else ... and a couple assorted smaller trees -- peach, crab apple, birch. The ashes and had dropped all their leaves, the maples and other big trees still had the majority of their leaves. Guess what I get to look forward to doing again in a few days?

We roll back home around 8pm, and realize our buyers had left the back door out of the garage wide open. Awesome!

Soon all the contract drama will be over, and all that will be left is moving 9 years of accumulated crap. And then we can kick back on the huge back deck at the new house.

where's the suggestion box?

The employer frequently elicits employee ideas for additional revenue streams. I have decided on one that beats my previous "Nominal fees for helping minors purchase alcohol" stream.

1. Form Hannah Montana tribute band, 'Amanda Bandana'
2. Prey on gullible parents thinking they're purchasing Hannah Montana tickets.
3. Profit

However, brief googling suggests 'Amanda Bandana' is already taken for a teen's myspace/livejournal/etc.

This is as disappointing a googling as when I found that 'Hell Toupee' was already taken. Back to the lame-cover-band-name drawing board.

under contract!

Lots of interest on the house -- typically 5-10 viewings a week, for the 5-6 weeks we'd been listed. No offers till about 1.5 weeks ago when we got two in one day: one good offer, one low offer.

The 'good offer' went south fast. They had acted very flaky, so we were disappointed but not surprised.

• Our agent is told (on Wednesday): "We'll send you an offer today!"
• No offer. Our agent calls them and is told: "Oh, they're putting an offer in on a different house"
• 24 hours later, we get their offer. They've only offered $500 earnest money, ask for %3 in closing costs, and don't have show any money down for the loan.
  
• We decide to counter-offer, asking for $1000 in earnest money, and say we'll cover $5000 in closing costs rather than %3. Send the counter-offer Friday morning.
  
• Saturday: We hear from their agent, "I'll be talking to my clients this afternoon"
• Sunday: No word
• Monday: No word... Our agent calls, and is told "I'll be talking to my clients this afternoon"
• Monday evening: Their agent says the deal is off. Leaves some nonsensical message for our agent saying we should have just accepted the offer and we could have closed in 3 days. I don't know why he thought he could get the inspection, appraisal, and loan stuff all worked out in that time.

In the meantime, we'd also counter-offered the other offer. And did the counter-offer thing every day last week. The buyers appeared solid financially, and were motivated to buy in our neighborhood -- their cousins had just bought up the street from some desperate sellers (they had been on the market 6 months). That house also probably screwed everyone in the subdivision on appraisals (they'd listed in May for $280 and gradually dropped to $250, accepting a $237 offer). It was nearly identical to our house -- 100 sq feet more, but hideous inside. Despite the hideousness, it was clear from the start that our buyers really, really didn't want to spend more than their cousins.

After talking them up $7k from their initial offer, we ended up accepting quite a bit less than the asking price. However, with the current market, and the fact that we'd already found our new house and were under contract on it, we decided that we couldn't wait for the perfect offer. At the price we accepted we wouldn't have any worries with appraisal and it gives us enough profit to put 20% down on the new house with a bit left over.

Our agent's co-workers are amazed she got us under contract in ~5 weeks. Some of it was us being priced right, in the right location, and our hard work keeping the house clean. But a huge part of it was how hard she worked. Lots of marketing, including at least one open house a week. She was awesome, and very funny. Best of all, unlike a lot of horror stories we'd heard about realtors, she was always available. Any time we had questions, she was very quick to respond to phone calls or e-mails.

If you're selling (or buying) or planning to, I have no reservations recommending her. Shauna Thomas at the Sugarhouse Keller Williams (599-9126). Shauna also hooked us up with a mortgage broker: Rain Wallace at Citywide Home Loans. We didn't have as much one-on-one time with Rain, but she was great -- quick to answer phone/email questions, patiently showed us our options, and helped us lock in a loan with a great rate.

nuggets from the dumb monkey sack

We've got a new developer in the group. As he's installing all the supporting tools he'll need for the projects he'll be working on, he came to me when Ant stopped working. Odd messages about Ant core/tools version incompatibilities.

We fiddle and fiddle with JAVA_HOME, with ANT_HOME, with PATH... No luck. Same messages, despite the Ant we want being right there at the start of PATH and in ANT_HOME.

Like a revelation from heaven, I realize... wait... what's in CLASSPATH? Well, there's a Documentum Jar in there... but that wouldn't have stuff to do with Ant, right?

The documentum Jar in question turned out to not have any classes in it, but it did have a giant list of other Jars in the manifest's classpath. Including Ant. More specifically, a local install of Ant 1.2 beneath the Documentum installation.

As much as I'd like to blame Sun for not providing a non-insane standard for application startup, I think its more appropriate to vent at EMC/Documentum for creating an installer that modifies the system's classpath environment variable to add their kitchen-sink Jar. For as much money as they charge, you'd think they could write an installer that doesn't screw up your system.

I'm not sure why I was surprised. Nearly every other interaction I've had with Documentum has been an exercise in agony. They must have a burlap sack full of exceptionally dumb monkeys deciding how their software gets installed and configured. Other nuggets from the monkey sack:

• No desktop application needs to talk to more than content server, right? Well, if they do, they can manually edit this INI file in c:\windows.
  
• No desktop application needs to talk to more than one version of a content server right? Well... if they do, they can uninstall/re-install to do so.

I can excuse their crappy, obfuscated APIs. Still hate them, but I can understand inflicting them on software developers. But to screw things up for all the end-users, QA, and support that need to use their crappy desktop apps, seems mind-numbingly stupid.

blog comment spam

I had to enable moderation on the comments due to the spam getting added. Moderating seemed like a better solution than going in and deleting the comments after-the-fact.

The first thing that clued me in to their spaminess:

These articles are fantastic; the information you show us is interesting for everybody and is really good written. It’s just great!! [more sales pitch and spammy URLs here...]

Anyone who actually read my blog would know that it has never been, and never will be any of those things.

hardware problems are teh suck

Why does my home PC pick the worst time to decide to stop booting? I really didn't want to spend 4 hours tonight fiddling in the XP recovery console, the BIOS settings, and running disk tests.

At first, I thought it was the height of inconvenience that I'd packed up my Windows XP install disk, had to search the basement, unpack the box, re-pack the box. Then, unpack the box again after realizing I forgot to grab the floppy disk with the SATA drivers.

However, that turned out to be minor compared to having to reboot twice every time I changed something in the XP recovery console. Once to get out of the console, and again so that the XP install disk would release itself from the CD-ROM drive. Gah!

System Rescue CD and the XP recovery console both seemed happy to read the disks, no problem getting into the partitions and looking at files. Tried fixboot, tried fixmbr, ran DFT's tests on both hard disks. No love from the boot gods.

Finally, decided to start fiddling with the BIOS. Swapped the order in which the hard disks are booted (SATA then IDE), and it came up. Not sure why I didn't notice it before. Also not sure why the BIOS implies it'll check disk #1 then disk #2 when booting, but only actually try #1 and give up.

Also not sure what would have screwed it up. I re-installed Ad-Aware, and Spybot S&D over the weekend. We also had a couple power-bumps a couple days ago.

In less annoying news, System Rescue CD is 4-6x as awesome as it was the last time I tried it a few months ago.

found it!

The pictures of our house are now on the online listing... all the paint colors and floor photographed well. And, our cat is now famous.



So far, things are looking pretty good for selling our house. Lots of interest. Open house this Sunday. Our realtor is so optimistic, she's going to have a 2 co-workers there to help her out.

Hopefully it'll all work out because we found the house to buy. The location we wanted, near my parents, near friends. Great neighborhood. We'll be going under contract on it tonight. I'm not sure if I feel relieved, or more panicky.

Best of all... the new house optionally includes 6 chickens. I know it's a bizarre and sad thing to have always wanted... but I really have liked the idea of having chickens. We're already speculating that our cat is finally going to be even more fascinated and intrigued by the outside. Maybe enough to get over her fear of the unknown.

sweet spot

After many self-imposed deadlines came and went, finally listed our house for sale this weekend.
We felt bad about letting the deadlines pass, but we wanted finish all the painting and staging of the house before we listed. Turns out that was a very good idea.

Listed it about 3pm Saturday. Around 5pm Saturday we got a from our realtor saying there was someone who wanted to look. Aaaiiiiieeee! We weren't ready. The house was a mess. Paint cans, cleaning supplies, unpacked rooms. A couple hours of last-minute cleanup got 90% of the rooms ready for viewing. They seemed to like the house, and also said that even with our mess, it was one of the top two homes they'd seen in the area.

Got a wake-up call Sunday from our realtor. Another group wanted a viewing. We didn't have to do much because of the previous day's frantic cleanup. We run a few errands while they look at the house with a realtor. Afterwards, we get back home to be accosted by a family in a mini-van asking about the house.

Late Sunday afternoon, I finally get a breather and mow the lawn. Interrupted once by a couple while mowing the front lawn. Interrupted a second time, and nearly had a heart attack, while mowing the backyard when a realtor waltzes through the gate with another couple in tow asking if they can view the house.

I guess we hit the sweet spot w/ regard to location, price, and size of home. We don't even have the pictures online yet (that should happen today). It's a big relief, seeing the amount of interest in the home. And, seeing that our work over the past 2 months was worth it.

If only our own house hunt was as successful. Still having found the right home after viewing probably 70+ homes. At least we're done with our own home now and we can focus on finding our new home.

update: Got a call from the lovely wife. Another realtor and their clients just dropped by "We were in the neighborhood...". 4 viewings in the 2.5 days it's been listed. Aiiiiieeeee. We're beginning to wonder when we'll have time to do the final cleaning for the open-house next weekend.

stressful vacation, house hunting, awesome basement

Just finished a week of vacation. 'Vacation' being code for fixing up the house -- finished laying the laminate floors, installing new baseboards, some new bathroom cabinets, and lots of painting.

Still not terribly near the end point. Plenty more painting and packing to do before we'd feel comfortable having potential buyers look at the house.

The lovely wife and I have been house hunting with our realtor. So many open houses, they're all a blur now.

The price on one with a super-awesome basement has finally dropped enough to be worth it. Those are indeed built-in sofas. The tales those cushions could tell.

But seriously, that house has a great location, great square footage, and the extra fantastic basement. And, the price has dropped enough that we should be able to update the kitchen and a couple other out-of-date rooms prior to moving in.

Of course, the basement would have to stay as-is. It's just too awesome.

crappy neighbor water hazard

Three times over the last couple weeks, my and one of my neighbor's backyards have flooded. The picture is the view from my yard into the neighbor's. Our house isn't that effected... but the water is right against the neighbors foundation. Not good.

Seems another neighbor can't figure out their inflatable above-ground pool.

• The first time, I think they forgot to turn on the filter and decided to the drain the funky water
• The second time, they started refilling the pool and left for the day.
  
• The third time (two days after flood #2), they hadn't tightened a coupling and didn't notice the leak.

On the plus side... I've talked to my fellow-floodee neighbor more in the last week than I have in the ~2 years they've been in their house.

On the minus side. The flooder neighbor seems more annoyed that we're upset, rather than apologetic. I understand that once the water is out there isn't much they can do about it. But, it'd be nice if they seemed slightly concerned about things they do that effect their neighbors.

mmm... pie!

A friend helped us move our furniture around for the work we're doing on the house. Last time, he and his girlfriend raved about Left Fork Grill (68 W. 3900 South, 266-4322). We had him over again this weekend to move furniture back into the finished room, and as a reward for helping us we stopped there for breakfast.

I had a breakfast sandwich (sausage and egg), the lovely wife french toast, and our friend had a fried egg sandwich. Everything was delicious. And, they made hash browns correctly -- the right ratio of crisp to tender, and not greasy.

To top it all off, they have the best pie ever. The lovely wife had banana cream, I had raspberry-rhubarb. Excellent crust, freshly made fillings. Both were great. Many others think so too.

It's shot right up to top of our list of breakfast spots -- great food, reasonable prices.

A shame that it's only open for breakfast and lunch. Pie for breakfast may become a family tradition.

finally, a chance to wear a toolbelt

We've been thinking of moving for a couple years now. But, we'd wanted to get a few things fixed up with the house. I needed to get the yard and flower beds into better shape, and we wanted to replace some of our carpet.

A few months ago, we'd bought some glueless laminate flooring at Costco while it was on sale (ended up saving around $150 because of the sale). Finally got around to starting the project last weekend. Finally realized the extent to which one of our cats ruined the carpet. Ugh.

The flooring went down quick. Not as easy as the instructions make it seem. But, it looks nice, and gave me my once-every-two-years excuse to wear a toolbelt.









And, in odd things I've stuck in my pie-hole news:

• The 'Hog Burger' at Britton's (964 East Union Square in Sandy). A burger with grilled onions nestled between a grilled-cheese sandwich with bacon, and a second grilled-cheese sandwich with tomato. The first half was good. The second half was 'what have I gotten myself into?'
  

impatient book review

I couldn't be bothered to wait the year(s?) for the U.S. release of Steven Erikson's newest Malazan book, Reaper's Gale.

After comparing the Canada/UK prices on the respective Amazon sites, and plugging in the conversion rate, it was cheaper to order from the UK and have it shipped across the Atlantic. Odd.

It was very good. Heartbreaking (and other not-so-heartbreaking) deaths of main characters. Resolution of plot lines. Very enjoyable.

I also picked up Night of Knives, by Ian Cameron Esslemont. Also set in the Malazan universe. It was pretty good, but not great. It could have used another round of editing -- typos and a couple confusing sections. But, it was fast-paced and fun.

roller derby -- violence, short skirts, beer -- what's not to like?

The lovely wife and I went to the roller derby last night. Lots of fun. Lots of violence. Lots of hot chicks in short skirts zooming around the track and knocking each other down. And beer! Mmm... Beer.

It was an All Star match between the local league and their opponents from Idaho.

Salt Lake Software Symposium, day #2

Last day. Still pretty good, but didn't rock as much as the first.

Attended a couple more of Jared Richardson's sessions: "Agile Testing Strategies", and "Software Development Techniques". Lots of good real-world examples, and lots of inspiration for changing our processes. Definitely thinking of picking up Ship It!

Neal Ford's "Pragmatic Extreme Programming" was good. But, I think it'll be easier to work in lessons from Jared's sessions (on days 1 and 2) into our existing process.

I wasn't going to attend Brian Sam-Bodden's "Complex Builds with Ant" session. From the slides, it appeared like I'd already knew the major tips on my own. But, I was wiped out by the last session. Decided it was worth going to a topic I was familiar with in order to pick an expert's brain. Only three attendees, so the session went fast and I was able to ask a lot of questions.

More sour-grapes today over all the Ruby rah-rah-rah. Again, Neal Ford declared Ruby the winner in the dynamic language race on the JVM. Neal is a very smart guy, a great presenter, and also great to talk to one-on-one. I don't have any issue with him advocating Ruby. Any language that makes developers more productive is a good thing. But, it seems disingenuous to declare a victor in a wide open race to a room full of people who have little to no experience with dynamic languages.

Personally, I prefer Python over Ruby. Others can and should disagree. Here's a fair, but slightly biased (since the author is familiar with Python), comparison of Python vs. Ruby.

I'm not sure who to blame for lack of Python excitement in the Java world (or at least at this conference). Maybe with the recent revitalization of the Jython project, it will get more Java developer's attention.

Microsoft's CLR (and the new DLR and Silverlight) seems very exciting. See here and here and here. The second link is a screencast showing interoperation between Ruby, JavaScript, Python, and VB.

All in all, the conference was very good. I'll definitely plan on going next year.

Salt Lake Software Symposium, day #1

Since this blog is all about (pick one):

• lame book reviews
• lame software conference anecdotes
• odd things I've put in my mouth
  

Here's more software conference anecdotes.

Spent today at the Salt Lake Software Symposium (aka No Fluff Just Stuff). So far, the conference is very good. Attended presentations by Neal Ford, Jared Richardson, and Brian Sletten. All great presenters, showing off great technology.

A friend has tried 2-3 times to explain Aspect Oriented Programming to me. It sounded... like more trouble than it was worth. Brian Sletten's AOP presentation was very useful in cluing me in. Seeing the AspectJ/Eclipse tools in action got me all excited to try it out. The immediate use-case I have would to remove a bunch of cut-n-paste concurrent locking boilerplate in my pipeline application. If I understood it correctly (always a questionable assumption), I can replace all that code scattered throughout my class with an aspect that'll do the

  writeLock();
  try {
       // .... do stuff....
  } finally {
       releaseWriteLock();
  }

logic all from a single place for all the setters methods in my class. And a similar readLock() aspect for my getters.

Brian also presented NetKernel. Also awesome. Still trying to get a handle on how exactly we can leverage it for our needs. His Mashup/SemanticWeb/RDF presentation was also good.

Jared Richardson's presentation Shippers Unite was very good. The best part, calling sales people 'Sales Critters'. He had lots of great advice, and great best-practice type information.

Neal Ford's SOA presentation, and his keynote after dinner were also very good. His keynote, "Polyglot Programming" was very interesting. Contrasted the increasing cruft, and complexity (and irrelevancy) of the Java language against the awesomeness of the Java platform. Described the increasing place Dynamic Languages will have within the software industry. In particular their place inside managed runtime environments (either JVM or CLR).

The only major disappointment I experienced was the (apparent) disdain the conference presenters seemed to show for Python/Jython. Lots of Groovy/Ruby/JRuby/Grails rah-rah-rah. I know Rails is the new poster child, and Ruby folks should be proud. But, to give Python/Jython/IronPython hardly a mention. And, the one time the word Python appeared on a slide, to explicitly declare Ruby the winner over Python... seems premature.

Yeah, yeah. Sour grapes. Maybe I should try Ruby before deciding I don't like it... but the Perl-like $ and @ syntax rubs me the wrong way. And, maybe I should just be happy about the rising tide of dynamically typed languages that'll lift all boats.

discrimination!

Someone brought a delicious looking treat in to work today.

The sign next to it:

APRICOT CAKE
____________

(no nuts)


And, in other lame joke news, my new favorite joke (the bottom panels) . I'm just that classy.

mmm... pie.

It's turns of phrase like this that keep me going back to Worse Than Failure :

Of course, upon his return to work, fate crapped on his hope and served him up a big, steaming slice of shattered dream pie. The familiar stench of hot computers and death descended on him while he tried to shut out the familiar fear of dying in the room and having the network absorb nutrients from his decaying body.

san francisco -- day #4 -- "You like extreme?"

Final conference sessions today. OK, but nothing fantastic.

Great lunch at House of Nanking. The waiter arrived, said "Your first time here?" When we said yes, he took our menus. No idea what the dishes were, but they were all super-excellent.

Then got a taxi to the airport. After a few minutes of silence, the driver asks, "You like extreme?"

I'm not sure I hear him right. He asks again, "You like extreme?"

Yep. I had heard him right.

Pause....... I'm not sure if I'm going to get invited to a cock fight, or what. I guess that maybe he's asking if it's OK to play his favorite thrash metal CD. "Uhhh.... extreme music?" I ask.

He explains, "No! Extreme! Like X! .... Games! Extreme. Sports."

I respond, "umm. A little?"

The rest of the ride was enclosed in blissful silence.

san francisco -- day #3

Noisey street. Some sleepy sessions. A couple awesome ones -- displaying debugging and profiling support for XQuery.

Then the final session of the day. At first, I'm thinking... blah, a mailing list archive viewer? Oh, how wrong I was. Sweet zombie Jesus, it was cool. Or maybe Jason Hunter is an great speaker. Or maybe both?

It was also very applicable to the work we're doing -- transforming and enhancing XML and non-XML assets into an normalized XML form and storing the results in MarkLogic.

san francisco -- day #2

Noisey street + hotel room close to street = no sleep.

As the opening session begins, I realize I can't focus. Then I notice the strobing migraine precursor.... Today is going to be awesome!

Interesting sessions. One of the sessions on the technical track of the conference seemed like a waste of time, but most of the others were interesting. And an added plus, presentations by the vendor and one of our competitors seem to validate our current architectural approach.

san francisco -- day #1

Flew to San Francisco for a technical conference. Flight uneventful, and in a teeny CRJ200. Only 12 and a half rows of seats.

Googled up restaurants near the hotel. I've been craving Vietnamese all the time lately, so I decided on a hole-in-the-wall place with good reviews -- Golden Flower Vietnamese. Just a couple minutes walk from the hotel.

Pretty good, but not the best I've had. The imperial rolls were greasy, and more mystery-meaty than I'd have preferred. The bun with grilled pork and shrimp was tasty.

In order of worst to first:
Pho Cali -> Golden Flower Vietnamese -> Pho Hoa -> Shanghai Cafe -> The deeee-licious versions Dien made for us over at Mark's house. Mmmmm.

Also, it's less fun to travel without coworkers. Wandering aimlessly is less fun to do on your own... it feels more like getting lost than having an adventure.

additional lame book reviews

Yes, more reviews. Why the wait? Because I finally read some books again during a few days of vacation.

Day Watch by Sergei Lukyaneko -- pretty similar to the first book, Night Watch. As good as the first, and sometimes as infuriating (wah? where'd that plot twist come from?). But pretty fun. Hopefully the new movie will be good as well.

Storm Front by Jim Butcher -- the lovely wife and I are enjoying the Dresden Files on SciFi, so I thought I'd pick up the first book in the series it was based on. Pretty good. On the fence on whether I'll pick up more.

Fiends of the Eastern Front by David Bishop -- What'll that crazy Hitler do next? Hire vampires to help win WWII, that's what. I liked the premise. I liked some scenes. But, ultimately very lame.

Did I mention it was lame? And full of typos? I usually blissfully skip right past most typos, not this time. That must mean they were 2-3x more that I missed.

Sample paraphrased ending scene of each book (in the 3-book omnibus):

books #1 and #2:
Lord Constanta (main baddie) : "Yes. You tried to kill me. Did I mention I'm completely (and lamely) invulnerable? It's because my Sire is so super bad-ass. I'll let you live to the next book, but let me tell you about my ludicrous world domination plans first...."

book #3: Yes. I'm spoiling the ending.

(further expository villains explaining their ludicrous world domination plan ... : "Yes, our plan is to darken the skies over Europe so we don't have to worry about our sunlight imposed curfew. Then we can drink blooooooooood all the time! Muauahahahhaa. No, I don't see the complication that that'll kill all life other than us... C'mon, man. We're gonna teach the stupid sun a lesson.")

And, the horrible, horrible, horrible ending:

The Sire (basically a big vampire bat you might expect to see in a Godzilla flick) : Bwaaaaah! Watch me fly outta this lake of blood. Oh nos! The day-walker hero is doing his homage to Slim Pickens in Dr. Strangelove by riding a silver-tipped V1 rocket into me. I esplode. When I gave Lord Constanta his resurrection powers, I forgot to give them to my self. Curse my strict FIFO ordering of completing my TODO list.

I was willing to give the books a little credit until the ultra-lame ending. Christ on a pogo stick, that was awful.

change the world?

I've been getting more and more excited by the OLPC project. Not only is it exciting technically, it seems like it has huge potential to change the world.

It'll take a few years before the educational goals bear fruit, but it'll have a huge technical impact much sooner. A lot of the cool technology could make its way out to non-OLPC hardware -- displays, power management, networking, security, exciting Python changes... lots of great stuff.

The Daily Python-URL pointed me here. A great introduction to the project, its goals, and the technology.

finally learned my lesson

I clearly remember thinking a year ago: "I've had incredible luck with my computer hardware. No disk, motherboard, power-supply, or other problems with any of the computers I've owned."

Since then, I've replaced the power-supply on my newest computer. TWICE. Both times, the hardware was still under warranty, so financially it hasn't been an issue. But the inconvenience of losing access to the files has been annoying.

After hearing good things about Mozy, I decided to give it a shot. So far, so good... Of course, it's only been an hour. Fingers-crossed for more hardware problems! Woo-hoo!

And, in somewhat related news, I'm finally taking the time to encrypt my sensitive files -- tax returns, contact info, schematics for man-eating robots, muffin recipes, etc. I'm giving TrueCrypt a shot. Because I'm cheap. Also, having a cross-platform encryption solution was a big plus. So far, easy to setup and use.

lesson of the day: just assume Co-Worker #1 is correct

I've been reading Java 2 Performance and Idiom Guide, it's quite good. But, I think I'm too interested in experimenting with the tweaks it suggests.

As I mentioned in my last post I'm working on a data-processing pipeline, and have spent the last couple days investigating why the performance (records/second) of a particular pipeline stage drops off like it does.

The software we're working on analyzes bibliographic records and creates a representative sample. Our users create mappings between one XML schema and another. They have to go through much fewer iterations if they've got a good sample to work with. And iterations are much shorter if they can test their mapping against a sample that's <500 records, rather than 16+ million records.

The oddly performing pipeline stage normalizes XML entities by replacing them with a numeric reference to their Unicode equivalent. Yeah, the DTD should do that... but it doesn't help when the data provider escapes the entities. Or when their DTD replaces the entity... with the entity... argh.

The graph below shows the records/second performance of three stages of a pipeline (y-axis). The x-axis is time. The yellow nodes are each file's entity-replacement performance. The blue nodes are the files getting split into 5000-record chunks and inserted into a Marklogic database. The pink nodes are each 5000-record chunk being analyzed. The first 500 or so files all have 30,000 records; the rest of the files are daily updates that vary in size -- that's why the tail end of each gets a little kooky.The initial steep drop-off for the first few files was understandable. It's one of the first stages of the pipeline, and the threads doing entity-replacement don't have to compete for I/O with other pipeline stages until the following stages have entity-replaced-output to work on. But the continued decline made me worry that there was a problem in the code.

Reviewed the code. Each file is getting processed by its own thread. Each thread should create SAX handling objects that aren't shared between threads. No static fields or other stuff to interact between threads...

But! I notice that the SAX handler uses a StringBuffer to accumulate a records-worth of text as it replaces entities. When endElement is reached it calls sbuf.setLength(0), which my fancy new book learnin' suggested could cause memory/performance problems.

And I set off on my non-adventure....
There are four series on the graph above.

1. Dark blue is the original run.
   
2. Pink is the result of replacing sbuf.setLength(0) with sbuf = new StringBuffer(). No fabulous improvement. Stupid book learnin'.
   
3. I decide that 'new StringBuffer()' meant that memory is getting re-allocated from initial the default StringBuffer size, which isn't as efficient as it could be. Yellow is the result replacing sbuf = new StringBuffer(INIT_LENGTH). INIT_LENGTH is a couple thousand bytes. Still no major improvement.
4. Re-review the code. Realize that the FileInputStream/FileOutputStream objects aren't wrapped with BufferedInputStream/BufferedOutputStream. The light blue nodes are the result of changing that... Still no major improvement. I assume this was because these input files are in Zip files, and output to GZIP files -- so were already buffered (I did increase their buffer sizes along with adding the buffering to the uncompressed-file handling).

Then I remember the comment from Co-Worker #1 when I showed the graph to her before I began my journey to nowhere: "Maybe its just file-size"

I fiddle with my statistics exporting code to include the file-size in the exported spreadsheet.
*@&#$*&!

And the KB/second graph looks like:Moral of the story: Get the most complete picture of a performance problem you can, and don't be swayed by the kooky little performance trivia you read.

a programming post, odd.

At work, I've been working on a data processing pipeline. The meta-data for the pipeline (e.g. status of pipeline phases, data locations, etc) at runtime is held in a LRU cache and flushed to persistent storage. Prior to a couple weeks ago, the storage was just a bunch of files on the filesystem. It worked, but it was pretty slow to collect aggregate information out of it -- e.g. if you wanted a global view of all status within the pipeline, you had to traverse the meta-data graph loading each individual file as the traversal progressed.

I'm allergic to relational databases, and we're already using Marklogic/XQuery to store and transform our data, so we decided to try a new implemenation of the LRU cache with the storage in Marklogic. Performance wasn't impacted much (maybe an extra couple minutes for a 1500+ file pipeline that takes 10+ hours), and now status reports take a couple seconds rather than a few minutes. Hooray!

Since that was less painful, I decided I'd also try getting some performance statistics out of the meta-data in the database. There are a few helper functions to do sum/count/average, but I wanted to avoid iterating over the list multiple times. I found some psuedocode here, and implemented an XQuery version:

(:
This is an XQuery implementation of the variance algorithm on this page:
   http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

   credited to Donald Knuth, The Art of Computer Programming, vol 2: Seminumerical Algorithms, 3rd edn., p. 232 Boston: Addison-Wesley.

It relies on Marklogic's extensions to XQuery, so it probably won't
work on other XQuery implementations.
:)
define function putil:getSumMeanVariance($srcvals as xs:double*) as node()* {
   let $n := 0
   let $mean := 0
   let $S := 0
   let $sum := 0

   let $throwAway :=
       for $x in $srcvals
       let $delta := $x - $mean
       return (
           xdmp:set($n,$n + 1),
           xdmp:set($mean,$mean + ( $delta div $n ) ),
           xdmp:set($S, $S + $delta * ( $x - $mean )),
           xdmp:set($sum, $sum + $x )
       )
   let $variance :=
           if ($n gt 1) then
               $S div ($n - 1)
           else 0
   return (
       <count>{$n}</count>,
       <sum>{$sum}</sum>,
       <mean>{$mean}</mean>,       
       <variance>{$variance} </variance>,
       <stdev>{math:sqrt($variance)}</stdev>
   )
}


let $elapsedTimes :=
 for $pnode in collection($collName)[ .... uninteresting XPath predicate stuff here .... ]
 return $pnode/stats/time/end - $pnode/stats/time/start
return <stats>{getSumMeanVariance($elapsedTimes)}</stats>

lame book reviews: the revenge

While the lovely wife recovers from her surgery[1], I've been reading Carl Zimmer's fascinating blog: The Loom I've always been interested in evolution and biology. Once I started, I can't stop digging through his archive of posts. Great Stuff! I read a couple of his books a year or two ago. "Parasite Rex"-- creepy, disgusting, fascinating. and "Fish with Fingers, Whales with Legs", also very good.

Fiction-wise:

• Scott Smith, "The Ruins" -- meh. It had good bits, but I couldn't suspend my disbelief w/ regard to the villain. Definitely not a book to read while vacationing in Central America.
  
• Cormac McCarthy, "The Road" -- A man and his son wander across a post-apocalyptic wasteland. Heartbreaking. Good.
• Dan Simmons, "The Terror" -- 19th century Arctic explorers stuck for years in the ice while a monster slowly picks them off. Awesome.

[1] Since my brother-in-law reads this, I'd better explain. She had a lipoma removed. It was an outpatient procedure, everything went well, and she's recovering at home. Today is probably her worst birthday ever, but she's doing OK.

end of job hunt

After 2 interviews and one offer the job hunt is over. I decided to stay at the current employer.

On the same day I received the offer our CTO came to talk to us. He said pretty much everything I wanted to hear: my project isn't dead, he's signed the purchase order for the new hardware we've been waiting for, and our office will have an active role in the next major upgrade to our customer-facing system. That, combined with the fact that I really like the people I'm working with, made me decide to stick it out.

The offer was tempting. It was from the company that bought my ex-employer. I know the work is exciting, and the people are great. But, in the end I decided that I'd have much more control over my future at my current employer. I have more say in which projects I'll work on, as well as having a major hand in all the parts of their development cycles. And, the much shorter product lifetimes will keep my technical skills fresher.

Hopefully it'll turn out for the best.

favorite post ever

While reading the rec.arts.sf.written thread spawned by the news that HBO has bought the rights to produce a TV series based on George R. R. Martin's "Song of Ice and Fire", came across perhaps the most beautiful thing ever uttered on USENET. The thread is mostly a discussion, comparing Good vs. Evil as characterized in Martin versus JRR Tolkien.

> If killer ice zombies aren't evil, then the word has no meaning.

return of the lame book reviews

Enough job whining. Time for more lame book reviews. Hooray!

• A Dirty Job, by Christopher Moore -- Man discovers he is Death. Or, at least, one of Death's little helpers. Very funny. Very sad. Very good.
• Old Man's War, by John Scalzi -- Earth's Colonial Defense Force only accepts recruits that are over 75 years old. Very good
• The Android's Dream, by John Scalzi -- Best. First. Paragraph. Ever. "Dirk Moeller didn't know if he could fart his way into a major diplomatic incident. But he was ready to find out." Very funny. Very good.
  
• The Algebraist, by Iain M. Banks -- I didn't like it as much as his Culture novels, but still good.