Salt Lake Software Symposium, day #1

Since this blog is all about (pick one):

• lame book reviews
• lame software conference anecdotes
• odd things I've put in my mouth
  

Here's more software conference anecdotes.

Spent today at the Salt Lake Software Symposium (aka No Fluff Just Stuff). So far, the conference is very good. Attended presentations by Neal Ford, Jared Richardson, and Brian Sletten. All great presenters, showing off great technology.

A friend has tried 2-3 times to explain Aspect Oriented Programming to me. It sounded... like more trouble than it was worth. Brian Sletten's AOP presentation was very useful in cluing me in. Seeing the AspectJ/Eclipse tools in action got me all excited to try it out. The immediate use-case I have would to remove a bunch of cut-n-paste concurrent locking boilerplate in my pipeline application. If I understood it correctly (always a questionable assumption), I can replace all that code scattered throughout my class with an aspect that'll do the

  writeLock();
  try {
       // .... do stuff....
  } finally {
       releaseWriteLock();
  }

logic all from a single place for all the setters methods in my class. And a similar readLock() aspect for my getters.

Brian also presented NetKernel. Also awesome. Still trying to get a handle on how exactly we can leverage it for our needs. His Mashup/SemanticWeb/RDF presentation was also good.

Jared Richardson's presentation Shippers Unite was very good. The best part, calling sales people 'Sales Critters'. He had lots of great advice, and great best-practice type information.

Neal Ford's SOA presentation, and his keynote after dinner were also very good. His keynote, "Polyglot Programming" was very interesting. Contrasted the increasing cruft, and complexity (and irrelevancy) of the Java language against the awesomeness of the Java platform. Described the increasing place Dynamic Languages will have within the software industry. In particular their place inside managed runtime environments (either JVM or CLR).

The only major disappointment I experienced was the (apparent) disdain the conference presenters seemed to show for Python/Jython. Lots of Groovy/Ruby/JRuby/Grails rah-rah-rah. I know Rails is the new poster child, and Ruby folks should be proud. But, to give Python/Jython/IronPython hardly a mention. And, the one time the word Python appeared on a slide, to explicitly declare Ruby the winner over Python... seems premature.

Yeah, yeah. Sour grapes. Maybe I should try Ruby before deciding I don't like it... but the Perl-like $ and @ syntax rubs me the wrong way. And, maybe I should just be happy about the rising tide of dynamically typed languages that'll lift all boats.

discrimination!

Someone brought a delicious looking treat in to work today.

The sign next to it:

APRICOT CAKE
____________

(no nuts)


And, in other lame joke news, my new favorite joke (the bottom panels) . I'm just that classy.

mmm... pie.

It's turns of phrase like this that keep me going back to Worse Than Failure :

Of course, upon his return to work, fate crapped on his hope and served him up a big, steaming slice of shattered dream pie. The familiar stench of hot computers and death descended on him while he tried to shut out the familiar fear of dying in the room and having the network absorb nutrients from his decaying body.

san francisco -- day #4 -- "You like extreme?"

Final conference sessions today. OK, but nothing fantastic.

Great lunch at House of Nanking. The waiter arrived, said "Your first time here?" When we said yes, he took our menus. No idea what the dishes were, but they were all super-excellent.

Then got a taxi to the airport. After a few minutes of silence, the driver asks, "You like extreme?"

I'm not sure I hear him right. He asks again, "You like extreme?"

Yep. I had heard him right.

Pause....... I'm not sure if I'm going to get invited to a cock fight, or what. I guess that maybe he's asking if it's OK to play his favorite thrash metal CD. "Uhhh.... extreme music?" I ask.

He explains, "No! Extreme! Like X! .... Games! Extreme. Sports."

I respond, "umm. A little?"

The rest of the ride was enclosed in blissful silence.

san francisco -- day #3

Noisey street. Some sleepy sessions. A couple awesome ones -- displaying debugging and profiling support for XQuery.

Then the final session of the day. At first, I'm thinking... blah, a mailing list archive viewer? Oh, how wrong I was. Sweet zombie Jesus, it was cool. Or maybe Jason Hunter is an great speaker. Or maybe both?

It was also very applicable to the work we're doing -- transforming and enhancing XML and non-XML assets into an normalized XML form and storing the results in MarkLogic.

san francisco -- day #2

Noisey street + hotel room close to street = no sleep.

As the opening session begins, I realize I can't focus. Then I notice the strobing migraine precursor.... Today is going to be awesome!

Interesting sessions. One of the sessions on the technical track of the conference seemed like a waste of time, but most of the others were interesting. And an added plus, presentations by the vendor and one of our competitors seem to validate our current architectural approach.

san francisco -- day #1

Flew to San Francisco for a technical conference. Flight uneventful, and in a teeny CRJ200. Only 12 and a half rows of seats.

Googled up restaurants near the hotel. I've been craving Vietnamese all the time lately, so I decided on a hole-in-the-wall place with good reviews -- Golden Flower Vietnamese. Just a couple minutes walk from the hotel.

Pretty good, but not the best I've had. The imperial rolls were greasy, and more mystery-meaty than I'd have preferred. The bun with grilled pork and shrimp was tasty.

In order of worst to first:
Pho Cali -> Golden Flower Vietnamese -> Pho Hoa -> Shanghai Cafe -> The deeee-licious versions Dien made for us over at Mark's house. Mmmmm.

Also, it's less fun to travel without coworkers. Wandering aimlessly is less fun to do on your own... it feels more like getting lost than having an adventure.

additional lame book reviews

Yes, more reviews. Why the wait? Because I finally read some books again during a few days of vacation.

Day Watch by Sergei Lukyaneko -- pretty similar to the first book, Night Watch. As good as the first, and sometimes as infuriating (wah? where'd that plot twist come from?). But pretty fun. Hopefully the new movie will be good as well.

Storm Front by Jim Butcher -- the lovely wife and I are enjoying the Dresden Files on SciFi, so I thought I'd pick up the first book in the series it was based on. Pretty good. On the fence on whether I'll pick up more.

Fiends of the Eastern Front by David Bishop -- What'll that crazy Hitler do next? Hire vampires to help win WWII, that's what. I liked the premise. I liked some scenes. But, ultimately very lame.

Did I mention it was lame? And full of typos? I usually blissfully skip right past most typos, not this time. That must mean they were 2-3x more that I missed.

Sample paraphrased ending scene of each book (in the 3-book omnibus):

books #1 and #2:
Lord Constanta (main baddie) : "Yes. You tried to kill me. Did I mention I'm completely (and lamely) invulnerable? It's because my Sire is so super bad-ass. I'll let you live to the next book, but let me tell you about my ludicrous world domination plans first...."

book #3: Yes. I'm spoiling the ending.

(further expository villains explaining their ludicrous world domination plan ... : "Yes, our plan is to darken the skies over Europe so we don't have to worry about our sunlight imposed curfew. Then we can drink blooooooooood all the time! Muauahahahhaa. No, I don't see the complication that that'll kill all life other than us... C'mon, man. We're gonna teach the stupid sun a lesson.")

And, the horrible, horrible, horrible ending:

The Sire (basically a big vampire bat you might expect to see in a Godzilla flick) : Bwaaaaah! Watch me fly outta this lake of blood. Oh nos! The day-walker hero is doing his homage to Slim Pickens in Dr. Strangelove by riding a silver-tipped V1 rocket into me. I esplode. When I gave Lord Constanta his resurrection powers, I forgot to give them to my self. Curse my strict FIFO ordering of completing my TODO list.

I was willing to give the books a little credit until the ultra-lame ending. Christ on a pogo stick, that was awful.

change the world?

I've been getting more and more excited by the OLPC project. Not only is it exciting technically, it seems like it has huge potential to change the world.

It'll take a few years before the educational goals bear fruit, but it'll have a huge technical impact much sooner. A lot of the cool technology could make its way out to non-OLPC hardware -- displays, power management, networking, security, exciting Python changes... lots of great stuff.

The Daily Python-URL pointed me here. A great introduction to the project, its goals, and the technology.

finally learned my lesson

I clearly remember thinking a year ago: "I've had incredible luck with my computer hardware. No disk, motherboard, power-supply, or other problems with any of the computers I've owned."

Since then, I've replaced the power-supply on my newest computer. TWICE. Both times, the hardware was still under warranty, so financially it hasn't been an issue. But the inconvenience of losing access to the files has been annoying.

After hearing good things about Mozy, I decided to give it a shot. So far, so good... Of course, it's only been an hour. Fingers-crossed for more hardware problems! Woo-hoo!

And, in somewhat related news, I'm finally taking the time to encrypt my sensitive files -- tax returns, contact info, schematics for man-eating robots, muffin recipes, etc. I'm giving TrueCrypt a shot. Because I'm cheap. Also, having a cross-platform encryption solution was a big plus. So far, easy to setup and use.

lesson of the day: just assume Co-Worker #1 is correct

I've been reading Java 2 Performance and Idiom Guide, it's quite good. But, I think I'm too interested in experimenting with the tweaks it suggests.

As I mentioned in my last post I'm working on a data-processing pipeline, and have spent the last couple days investigating why the performance (records/second) of a particular pipeline stage drops off like it does.

The software we're working on analyzes bibliographic records and creates a representative sample. Our users create mappings between one XML schema and another. They have to go through much fewer iterations if they've got a good sample to work with. And iterations are much shorter if they can test their mapping against a sample that's <500 records, rather than 16+ million records.

The oddly performing pipeline stage normalizes XML entities by replacing them with a numeric reference to their Unicode equivalent. Yeah, the DTD should do that... but it doesn't help when the data provider escapes the entities. Or when their DTD replaces the entity... with the entity... argh.

The graph below shows the records/second performance of three stages of a pipeline (y-axis). The x-axis is time. The yellow nodes are each file's entity-replacement performance. The blue nodes are the files getting split into 5000-record chunks and inserted into a Marklogic database. The pink nodes are each 5000-record chunk being analyzed. The first 500 or so files all have 30,000 records; the rest of the files are daily updates that vary in size -- that's why the tail end of each gets a little kooky.The initial steep drop-off for the first few files was understandable. It's one of the first stages of the pipeline, and the threads doing entity-replacement don't have to compete for I/O with other pipeline stages until the following stages have entity-replaced-output to work on. But the continued decline made me worry that there was a problem in the code.

Reviewed the code. Each file is getting processed by its own thread. Each thread should create SAX handling objects that aren't shared between threads. No static fields or other stuff to interact between threads...

But! I notice that the SAX handler uses a StringBuffer to accumulate a records-worth of text as it replaces entities. When endElement is reached it calls sbuf.setLength(0), which my fancy new book learnin' suggested could cause memory/performance problems.

And I set off on my non-adventure....
There are four series on the graph above.

1. Dark blue is the original run.
   
2. Pink is the result of replacing sbuf.setLength(0) with sbuf = new StringBuffer(). No fabulous improvement. Stupid book learnin'.
   
3. I decide that 'new StringBuffer()' meant that memory is getting re-allocated from initial the default StringBuffer size, which isn't as efficient as it could be. Yellow is the result replacing sbuf = new StringBuffer(INIT_LENGTH). INIT_LENGTH is a couple thousand bytes. Still no major improvement.
4. Re-review the code. Realize that the FileInputStream/FileOutputStream objects aren't wrapped with BufferedInputStream/BufferedOutputStream. The light blue nodes are the result of changing that... Still no major improvement. I assume this was because these input files are in Zip files, and output to GZIP files -- so were already buffered (I did increase their buffer sizes along with adding the buffering to the uncompressed-file handling).

Then I remember the comment from Co-Worker #1 when I showed the graph to her before I began my journey to nowhere: "Maybe its just file-size"

I fiddle with my statistics exporting code to include the file-size in the exported spreadsheet.
*@&#$*&!

And the KB/second graph looks like:Moral of the story: Get the most complete picture of a performance problem you can, and don't be swayed by the kooky little performance trivia you read.

a programming post, odd.

At work, I've been working on a data processing pipeline. The meta-data for the pipeline (e.g. status of pipeline phases, data locations, etc) at runtime is held in a LRU cache and flushed to persistent storage. Prior to a couple weeks ago, the storage was just a bunch of files on the filesystem. It worked, but it was pretty slow to collect aggregate information out of it -- e.g. if you wanted a global view of all status within the pipeline, you had to traverse the meta-data graph loading each individual file as the traversal progressed.

I'm allergic to relational databases, and we're already using Marklogic/XQuery to store and transform our data, so we decided to try a new implemenation of the LRU cache with the storage in Marklogic. Performance wasn't impacted much (maybe an extra couple minutes for a 1500+ file pipeline that takes 10+ hours), and now status reports take a couple seconds rather than a few minutes. Hooray!

Since that was less painful, I decided I'd also try getting some performance statistics out of the meta-data in the database. There are a few helper functions to do sum/count/average, but I wanted to avoid iterating over the list multiple times. I found some psuedocode here, and implemented an XQuery version:

(:
This is an XQuery implementation of the variance algorithm on this page:
   http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

   credited to Donald Knuth, The Art of Computer Programming, vol 2: Seminumerical Algorithms, 3rd edn., p. 232 Boston: Addison-Wesley.

It relies on Marklogic's extensions to XQuery, so it probably won't
work on other XQuery implementations.
:)
define function putil:getSumMeanVariance($srcvals as xs:double*) as node()* {
   let $n := 0
   let $mean := 0
   let $S := 0
   let $sum := 0

   let $throwAway :=
       for $x in $srcvals
       let $delta := $x - $mean
       return (
           xdmp:set($n,$n + 1),
           xdmp:set($mean,$mean + ( $delta div $n ) ),
           xdmp:set($S, $S + $delta * ( $x - $mean )),
           xdmp:set($sum, $sum + $x )
       )
   let $variance :=
           if ($n gt 1) then
               $S div ($n - 1)
           else 0
   return (
       <count>{$n}</count>,
       <sum>{$sum}</sum>,
       <mean>{$mean}</mean>,       
       <variance>{$variance} </variance>,
       <stdev>{math:sqrt($variance)}</stdev>
   )
}


let $elapsedTimes :=
 for $pnode in collection($collName)[ .... uninteresting XPath predicate stuff here .... ]
 return $pnode/stats/time/end - $pnode/stats/time/start
return <stats>{getSumMeanVariance($elapsedTimes)}</stats>

lame book reviews: the revenge

While the lovely wife recovers from her surgery[1], I've been reading Carl Zimmer's fascinating blog: The Loom I've always been interested in evolution and biology. Once I started, I can't stop digging through his archive of posts. Great Stuff! I read a couple of his books a year or two ago. "Parasite Rex"-- creepy, disgusting, fascinating. and "Fish with Fingers, Whales with Legs", also very good.

Fiction-wise:

• Scott Smith, "The Ruins" -- meh. It had good bits, but I couldn't suspend my disbelief w/ regard to the villain. Definitely not a book to read while vacationing in Central America.
  
• Cormac McCarthy, "The Road" -- A man and his son wander across a post-apocalyptic wasteland. Heartbreaking. Good.
• Dan Simmons, "The Terror" -- 19th century Arctic explorers stuck for years in the ice while a monster slowly picks them off. Awesome.

[1] Since my brother-in-law reads this, I'd better explain. She had a lipoma removed. It was an outpatient procedure, everything went well, and she's recovering at home. Today is probably her worst birthday ever, but she's doing OK.

end of job hunt

After 2 interviews and one offer the job hunt is over. I decided to stay at the current employer.

On the same day I received the offer our CTO came to talk to us. He said pretty much everything I wanted to hear: my project isn't dead, he's signed the purchase order for the new hardware we've been waiting for, and our office will have an active role in the next major upgrade to our customer-facing system. That, combined with the fact that I really like the people I'm working with, made me decide to stick it out.

The offer was tempting. It was from the company that bought my ex-employer. I know the work is exciting, and the people are great. But, in the end I decided that I'd have much more control over my future at my current employer. I have more say in which projects I'll work on, as well as having a major hand in all the parts of their development cycles. And, the much shorter product lifetimes will keep my technical skills fresher.

Hopefully it'll turn out for the best.

favorite post ever

While reading the rec.arts.sf.written thread spawned by the news that HBO has bought the rights to produce a TV series based on George R. R. Martin's "Song of Ice and Fire", came across perhaps the most beautiful thing ever uttered on USENET. The thread is mostly a discussion, comparing Good vs. Evil as characterized in Martin versus JRR Tolkien.

> If killer ice zombies aren't evil, then the word has no meaning.

return of the lame book reviews

Enough job whining. Time for more lame book reviews. Hooray!

• A Dirty Job, by Christopher Moore -- Man discovers he is Death. Or, at least, one of Death's little helpers. Very funny. Very sad. Very good.
• Old Man's War, by John Scalzi -- Earth's Colonial Defense Force only accepts recruits that are over 75 years old. Very good
• The Android's Dream, by John Scalzi -- Best. First. Paragraph. Ever. "Dirk Moeller didn't know if he could fart his way into a major diplomatic incident. But he was ready to find out." Very funny. Very good.
  
• The Algebraist, by Iain M. Banks -- I didn't like it as much as his Culture novels, but still good.
  

job hunting

Well, I've decided to jump ship... Or, at the very least, test the waters very seriously. I've even updated my resume.

The job market is very good at the moment, with a number of cool sounding jobs. The ex-employer even has good jobs -- and even appears to be making money. Even better, I still have friends there. But... haven't decided if I want to get back into the C++, real-time, simulation world. But, the hardware/software at ex-employer is so, so cool.

worst new years gift, ever

Found out on Saturday the employer sent a wonderfully vague re-organization announcement on early-AM Friday.

I could have a job. I could be laid off. I could be asked to move to Manhatten to join that team.

Or, it might just reflect the management structure of the software development work at the company. Which would also suck, because my (immediate) boss is great.

Thank goodness we exercised some restraint this Christmas in terms of gifts for ourselves and family. We also transferred our credit card debt onto a 0% APR card a couple months ago... we were planning on paying it off over the next year, but now it might just get minimum payments until the job thing settles.

I'm not too worried. There are a ton of jobs in the Salt Lake area (according to monster and dice), and the company that bought my old employer's simulation business is doing very well and has positions. There are also a ton of cool sounding start-ups down in Utah county... the wife has put the kibosh on moving down there, but Draper/Bluffdale might be doable.

I'm mostly frustrated. I hadn't planned on staying at [employer's name withheld] for more than another year... but it would have been nice to leave on my terms rather than theirs. It would have also been nice if they'd done the layoff at a time when more than 5 people were actually in the office, and sending such an vague-but-ominous e-mail announcement.

Update: The job is safe. The vague e-mail implied the development positions would be cut/moved, but in fact none of the development positions at our location was involved. Our software architect was laid off, however... probably due to butting heads with the new CTO. The rest of the layoffs were spread across the groups.

worst christmas gift, ever

Pink Eye.

more lame books reviews!

I made a second pass through Steven Erikson's 6 Malazan books (~4000 pages). Definitely held up on the re-read. Found lots of foreshadowing I'd missed the first time. Reading them one after the other also allowed me to catch some references I'd missed the first time. Lots of fun. Sometimes gory. Sometimes very funny. Highly recommended.

Just finished S.M. Stirling's Dies the Fire. I love good post-apocalyptic fiction. It was pretty good, but the premise made me want to throw the book across the room a few times. I can see all electronic devices being knocked out at once... But gunpowder? And steam engines? It seemed like the author just wanted an excuse to get people into chain mail whacking each other with swords as soon as possible. The way the skilled characters found each other seemed way too easy (blacksmiths, horse trainers, ex-soldiers).

It was a good read. But I found myself bored or frustrated every-other-chapter when the author would focus on the Wiccan/ex-SCA characters. I don't think its just that they're Wiccan -- I'd be just as annoyed at a book with Christian characters that constantly hit me over the head with their faith.